Pretraining Large Language Models with NVFP4

ArXiv: 2509.25149

🎯 Pitch

A 12B-parameter language model trained entirely in 4-bit precision for 10 trillion tokens nearly matches the accuracy of FP8 baselines, shattering the assumption that such extreme quantization inevitably sacrifices quality. The key is a specific recipe combining random Hadamard transforms, two-dimensional block scaling, stochastic gradient rounding, and selective high-precision layers—remove any one, and training fails.

---

1. Executive Summary

This technical report introduces a NVFP4-based pretraining technique for stable and accurate 4-bit floating point training of large language models, validating the approach by training a 12-billion-parameter hybrid Mamba-Transformer model on 10 trillion tokens—the longest publicly documented training run in 4-bit precision to date. The methodology integrates four key mechanisms: Random Hadamard transforms to bound block-level outliers (redistributing outlier magnitudes across 16×16 matrix tiles to approach a Gaussian distribution), two-dimensional block scaling for consistent representations across forward and backward passes (grouping weight elements into 16×16 blocks that share scale factors, preventing chain rule violations when tensors are transposed during backpropagation), stochastic rounding on gradients for unbiased estimation (probabilistically rounding gradient values to avoid systematic bias from consistent quantization direction), and selective retention of high-precision layers (keeping roughly 15% of linear layers—primarily the final blocks—in BF16). The NVFP4-trained model achieves an MMLU-pro accuracy of 62.58%, nearly matching the 62.62% of an FP8 baseline, while the format itself delivers a 2–3× arithmetic throughput boost over FP8 on NVIDIA Blackwell hardware—establishing that billion-parameter models can be pretrained at 4-bit precision over multi-trillion-token horizons with negligible accuracy loss only when these four stabilizing techniques are applied together, since ablations removing any single component cause measurable convergence degradation.

2. Context and Motivation

The Core Problem: FP8 Is the Floor, Not the Ceiling

The paper addresses a deceptively straightforward problem: can we train large language models using 4-bit floating point arithmetic instead of 8-bit, without sacrificing accuracy? This matters because, while 8-bit floating point (FP8 and MXFP8) training has become widely adopted across the industry (Micikevicius et al., 2022; DeepSeek-AI et al., 2024; Mishra et al., 2025), there is enormous pressure to push narrower still. The economics of frontier model training make this urgent: training a state-of-the-art LLM today requires "on the order of tens to hundreds of yottaflops" (Section 1), representing massive investments of time, compute, and energy. Each halving of numerical precision theoretically unlocks a doubling of arithmetic throughput and halves memory consumption—gains that compound directly into faster training, lower cost, and reduced energy footprint.

The jump from 32-bit to 16-bit, and then from 16-bit to 8-bit, each unlocked substantial efficiency improvements that enabled the current generation of frontier models. FP4 represents the next logical step in this progression, but it crosses a threshold where the dynamic range of the format—only 2 exponent bits and 1 mantissa bit (the E2M1 format)—becomes severely constrained. An FP4 number can only represent eight distinct positive values: 0.5, 1, 1.5, 2, 3, 4, 6, along with their negatives and zero. Training requires representing weight updates, activation values, and gradient signals that can span many orders of magnitude, and the question is whether this tiny representational palette can faithfully capture the computations needed for stochastic gradient descent over trillions of tokens.

The paper does not frame this as merely an engineering optimization. It is a numerical feasibility question: is there a fundamental lower bound on training precision below which the optimization dynamics of large-scale neural network training break down? The 10-trillion-token 12B-parameter experiment described in Section 3 is designed to answer exactly this—it is "the longest publicly documented training run in 4-bit precision to date" (Abstract), deliberately chosen to probe whether 4-bit training is stable over realistic training horizons, not just toy-scale demonstrations.

Why This Problem Matters: The Hardware Has Already Arrived

The motivation is not speculative—it is driven by hardware that already exists. NVIDIA Blackwell GPUs provide native Tensor Core support for 4-bit microscaling formats, including both MXFP4 (the Open Compute Project standard) and NVFP4 (NVIDIA's enhanced variant). Table 1 (Section 2) shows that on Blackwell hardware, FP4 GEMM operations deliver 4× arithmetic throughput on GB200 chips and 6× on GB300 chips compared to BF16, and approximately 2× and 3× compared to FP8, respectively. Memory usage is also roughly halved when using FP4 operands.

This means the hardware capability already exceeds the current software frontier. The industry has GPUs that can compute FP4 matrix multiplications at dramatically higher throughput than FP8, but lacks a demonstrated training methodology that can use this capability without degrading model quality. The gap between hardware capability and algorithmic readiness is the specific problem this paper aims to close. The authors state this directly: "This technical report presents an in-depth analysis of large language model pretraining using NVFP4" and positions the work as validating that the hardware's raw throughput can be translated into real training efficiency for production-scale models.

The practical implications are substantial. If FP4 training becomes standard, organizations training frontier models could potentially:

• Reduce training time by roughly 2–3× for the same model architecture
• Train larger models within the same compute budget and wall-clock time
• Cut energy consumption proportionally, which matters both for cost and environmental impact

Prior Approaches and Where They Fall Short

The paper situates itself within a rapidly evolving landscape of narrow-precision training research, but each prior approach has specific limitations that create the gap this work fills.

FP8 training (the current state-of-the-art). FP8 formats (E4M3 and E5M2) and their microscaling variant MXFP8 are now mainstream, with demonstrated success in training models at the scale of DeepSeek-V3 (DeepSeek-AI et al., 2024) and Nemotron-H (NVIDIA, 2025b). The methodology for FP8 training is relatively mature: keep some sensitive layers in higher precision, use block-based scaling factors computed online, and apply round-to-nearest-even quantization. However, FP8 cannot access the 4–6× throughput boost that FP4 hardware provides. The gap is not methodological but fundamental to the format width: FP8 simply cannot deliver the same arithmetic density as FP4 on Blackwell hardware.

MXFP4 (the Open Compute Project standard). Prior work has explored training with MXFP4, the 4-bit microscaling format defined by the Open Compute Project (Rouhani et al., 2023; Open-Compute-Project, 2023). Several research groups have attempted FP4 training at moderate scales. Tseng et al. (2025b) trained language models with MXFP4, demonstrating feasibility at smaller scales. Chmiel et al. (2025) proposed "FP4 All the Way," attempting fully quantized 4-bit training. Chen et al. (2025) applied MXFP4 training to vision transformers, addressing oscillation issues. Castro et al. (2025) with "Quartet" and Wang et al. (2025) with FP4 optimization explored variations on 4-bit training methodology. Zhou et al. (2025) investigated FP4 pretraining specifically.

However, these prior efforts have specific limitations that this paper's NVFP4 work addresses:

• Scale of validation is modest. Many prior studies demonstrate FP4 training on models with hundreds of millions of parameters trained on hundreds of billions of tokens—meaningful, but not sufficient to prove stability at the scale where these techniques matter most. The 12B/10T-token experiment in this paper is roughly an order of magnitude larger in both model size and token horizon than most prior FP4 demonstrations.

• MXFP4 format limitations. The MXFP4 format itself has inherent accuracy constraints that limit training quality. As the paper explains in Section 2 and Appendix B.4, MXFP4 uses a single-level scaling scheme where block scale factors are restricted to powers-of-two (stored in UE8M0 format). This means values cannot be scaled to fit perfectly into the FP4 representable range. After scaling, the block's absolute maximum value will either overflow and saturate (losing information) or round down to a smaller FP4 sample (wasting dynamic range). In the worst case, MXFP4 can lose both the ±4 and ±6 representable values and nearly one binade of dynamic range—a significant fraction of FP4's already limited palette. This power-of-two scaling constraint is the direct motivation for NVFP4's two-level scaling approach with E4M3 block scales (which have mantissa bits for fractional precision) plus an FP32 per-tensor scale to restore the range that the E4M3 format sacrifices. The paper quantifies this precisely in Section 5 (Figure 6): MXFP4 training requires 36% more tokens to match NVFP4's loss on an 8B model, directly demonstrating that the format design matters for training efficiency, not just inference.

• Methodology is not comprehensively validated. Individual prior works have proposed some of the components used in this paper—stochastic rounding for gradient bias reduction, Hadamard transforms for outlier management, selective high-precision layers—but no prior work has combined all of them, validated their interaction at billion-parameter scale, and shown through ablation studies that each component is individually necessary for convergence. The paper's ablation in Figure 4 demonstrates that removing stochastic rounding, Hadamard transforms, two-dimensional weight scaling, or sufficient high-precision layers each independently degrades training for the 12B model, establishing that FP4 training at this scale requires a system of interacting techniques, not any single algorithmic innovation.

The outlier problem in quantization. A well-documented challenge across all narrow-precision training research is that activation and gradient tensors in LLMs contain "outliers"—values with magnitudes far exceeding the typical distribution (Dettmers et al., 2022; Xiao et al., 2024; An et al., 2025). These outliers arise systematically in specific feature dimensions and are a consequence of how attention mechanisms concentrate information. In FP4's tiny dynamic range, outliers are catastrophic: a single large value in a block forces the block scale factor to expand, causing all smaller values in that block to quantize to zero. The microscaling approach (Rouhani et al., 2023) mitigates this by computing per-block scale factors—so an outlier only affects its local 16-or-32-element neighborhood, not the entire tensor—but outliers still degrade the accuracy of their containing block. Prior work addresses this through various transformations: SmoothQuant (Xiao et al., 2024) shifts the quantization difficulty from activations to weights by mathematically equivalent scaling, QuaRot (Ashkboos et al., 2024) applies Hadamard transforms for inference quantization, and HALO (Ashkboos et al., 2025) extended this to optimizer states. However, these prior approaches were developed primarily for inference-time quantization or for specific components (weights, KV-caches), not for the full end-to-end training pipeline where all three tensor types—weights, activations, and gradients—must be quantized in both forward and backward passes.

The paper's application of Random Hadamard transforms specifically to weight gradient (Wgrad) inputs builds on this prior work but makes a crucial empirical observation: the transforms help for Wgrad but hurt when applied to forward pass or activation gradient GEMMs (Appendix E.4.1, Figure 11). This is non-obvious—intuitively, transforming any tensor should help by spreading outliers—but the paper hypothesizes that the transform itself introduces additional quantization error from the matrix multiplication, and this added error outweighs the outlier-mitigation benefit for Fprop and Dgrad. This specificity (transform only Wgrad inputs, not Fprop or Dgrad) is a practical finding that prior work had not established.

Chain rule violations from quantization. A subtler problem that prior narrow-precision training work often ignored is that quantization can break the chain rule of backpropagation. In standard training, the backward pass computes gradients through the same mathematical function used in the forward pass—this is what makes backpropagation a correct application of the chain rule. However, when tensors are quantized, the forward and backward passes can apply different quantization operations because the tensor dimensions are transposed: in the forward pass, scaling and transforms apply along input channels; in the backward pass, the tensor is transposed so operations apply along output channels. If the quantization depends on which dimension is being scaled (as 1D block scaling does), the weight tensor has one quantized representation $w_{\text{fprop}}$ in the forward pass and a different representation $w_{\text{bprop}}$ in the backward pass. The backward pass then computes $\partial x = w_{\text{bprop}}^T \partial y$, differentiating a different function than was used in the forward pass $y = w_{\text{fprop}} x$.

The paper identifies this as a specific mechanism for accuracy degradation (Section 4.3): "This occurs because the backward pass transposes the tensors, which changes the dot-product dimension. As a result, the same tensor can have two distinct quantized representations, effectively breaking the chain rule since backpropagation no longer differentiates the same function used in the forward pass." Prior FP8 training work (DeepSeek-AI et al., 2024) mitigated this with 2D block scaling for weights—using 128×128 blocks that are invariant to transposition. This paper adopts a similar approach (16×16 blocks for NVFP4) and validates that it is essential: the ablation in Figure 14 (Appendix E.5) shows that using 1D scaling along different dimensions (creating different forward and backward representations) degrades loss compared to either 1D scaling along the same dimension or 2D block scaling. This is a systematic treatment of a problem that many prior FP4 training papers either did not address or did not validate at scale.

Gradient bias from deterministic rounding. When quantizing gradient tensors—which represent the direction and magnitude of parameter updates—deterministic rounding (such as round-to-nearest-even) can introduce systematic bias. If the true gradient value tends to fall between two representable FP4 numbers, deterministic rounding consistently pushes it in the same direction, creating a cumulative error over many training steps that can steer optimization away from true minima. This is a known problem from the low-precision training literature (Alistarh et al., 2017; Chmiel et al., 2023; Castro et al., 2025), and prior work has proposed stochastic rounding as a solution: probabilistically round to one of the two nearest representable values with probability inversely proportional to distance, making the expected rounding error zero.

The paper adopts this approach but contributes specificity about where stochastic rounding matters. The ablation in Appendix E.3 (Figure 10) shows that stochastic rounding on gradient tensors is essential for convergence, but applying it to activation or weight tensors causes divergence. This aligns with the hypothesis that gradient bias is the primary threat—gradients accumulate over many steps, so bias compounds, while activation and weight quantization errors are "one-shot" per forward/backward pass and can be partially compensated by the optimizer. Additionally, stochastic rounding of forward-pass tensors is detrimental because it introduces more variance than nearest rounding without the bias-reduction benefit, since forward-pass errors don't accumulate in the same way.

How This Paper Positions Itself

The paper positions itself as a methodology validation paper, not a novel algorithm paper. Even the NVFP4 format itself is not new—it was introduced in a prior NVIDIA blog post (Alvarez et al., 2025) for inference. The contribution is demonstrating that when the format's improved numerical properties are combined with a specific, carefully ablated set of training techniques, LLM pretraining at production scale becomes feasible in 4-bit precision for the first time.

This self-positioning is evident in the paper's structure and rhetoric:

• The title emphasizes the format, not a novel algorithm. "Pretraining Large Language Models with NVFP4" frames the work as applying and validating an existing format, not inventing one.
• The abstract highlights "the first successful demonstration" and "the longest publicly documented training run in 4-bit precision to date." The contribution is the demonstration, not a single technical novelty.
• Section 4 ("Training Methodology") is framed as a set of recommendations rather than claimed inventions: stochastic rounding, Hadamard transforms, 2D block scaling, and mixed precision are all individually known techniques. The contribution is combining them into a coherent, validated recipe and showing that all are necessary.
• Section 5 compares NVFP4 against MXFP4 explicitly to make the case that the format choice matters—the 36% token penalty for MXFP4 (Figure 6b) is a quantitative argument that NVFP4's design improvements translate to training efficiency, not just theoretical numerical precision.

The paper also distinguishes itself from prior work by what it does not claim. It explicitly states that "this report is primarily concerned with the underlying algorithms and methodology rather than with runtime efficiency or system-level optimizations" (Section 1). This narrows the scope: the paper establishes numerical feasibility, leaving the engineering work of translating the 4–6× theoretical throughput into wall-clock speedups to future system-level optimization work. It is a necessary precursor—you cannot optimize what you cannot make converge—but it is deliberately not claiming to have solved the full deployment pipeline.

The connection to the Nemotron-H model family (NVIDIA, 2025b,a) is strategic. Rather than training a generic GPT-style model, the paper chooses the same hybrid Mamba-Transformer architecture used in NVIDIA's publicly benchmarked models, making the results directly comparable to known FP8 baselines. This strengthens the validation: the 62.58% vs. 62.62% MMLU-pro comparison (Table 2) is not an abstract loss curve comparison but a direct demonstration that downstream capability is preserved, which matters for practitioners deciding whether to adopt FP4 training. The choice of a hybrid architecture—with Mamba-2 blocks alongside self-attention—also demonstrates that the methodology is not specific to pure transformer architectures, though the paper does not claim universality across all architectures.

Finally, the paper's ablation structure (Figure 4, Appendix E) positions the work as a system contribution where the interaction between techniques matters as much as any individual technique. The base method—naively quantizing all layers to NVFP4 with standard 1D scaling and nearest rounding—diverges immediately (Appendix E.1). Each technique added incrementally improves convergence, but no single technique is sufficient alone. This systems-level understanding—that FP4 training is achievable only through the combination of format improvements (NVFP4 over MXFP4), outlier management (Hadamard transforms), chain rule integrity (2D scaling), unbiased gradients (stochastic rounding), and selective precision retention (mixed precision)—is the paper's core intellectual contribution, more than any individual component.

3. Technical Approach

3.1 Reader Orientation

This paper presents a training methodology—a set of coordinated techniques for modifying how a large language model performs arithmetic during pretraining—that enables the model to compute using 4-bit floating point numbers (NVFP4 format) instead of the standard 8-bit or 16-bit formats, without sacrificing convergence stability or final model quality. The core problem is that 4-bit numbers have such limited dynamic range (only 8 distinct positive values) that standard training procedures immediately diverge when all computations are naively quantized; the solution is a system of four interacting mechanisms—selective high-precision layers, Random Hadamard transforms on gradient inputs, two-dimensional block scaling for weight consistency across forward and backward passes, and stochastic rounding on gradients—that collectively maintain numerical fidelity where it matters while accepting quantization error where the model tolerates it.

3.2 Big-Picture Architecture (Diagram in Words)

The system modifies the standard neural network training loop at each linear (fully-connected) layer. A linear layer performs three matrix multiplications (GEMMs) per training step: one in the forward pass (Fprop) computing activations, and two in the backward pass computing activation gradients (Dgrad) and weight gradients (Wgrad). The architecture has five major components that transform how these GEMMs execute:

1. Mixed-precision layer assignment — a static configuration that designates approximately 85% of linear layers to run all three GEMMs in NVFP4, while keeping roughly 15% (primarily the final blocks) in BF16 or FP8. This is a compile-time decision based on empirical sensitivity analysis.

2. NVFP4 quantization engine — the per-GEMM procedure that converts BF16 or FP32 input tensors (weights, activations, gradients) into the NVFP4 format before the matrix multiplication executes on Tensor Cores, using a two-level scaling scheme (per-tensor FP32 scale × per-block FP8 scale) and a configurable rounding mode (nearest or stochastic). The quantized GEMM produces a higher-precision output (BF16 or FP32).

3. Random Hadamard transform module — applied selectively to the inputs of Wgrad GEMMs only. This multiplies blocks of 16 elements by a randomized orthogonal matrix to redistribute outlier magnitudes more uniformly, reducing the chance that any single block's scale factor is dominated by an extreme value. The transform is fused into the quantization step.

4. Two-dimensional weight scaling module — for weight tensors only, scale factors are computed over 16×16 blocks (spanning both input and output channel dimensions) rather than the standard 1×16 row-wise blocks used for activations and gradients. This ensures the quantized representation of a weight matrix is identical whether the matrix is used in the forward pass (multiplying by activations) or transposed for the backward pass (multiplying by output gradients).

5. Rounding mode selector — a per-tensor-type policy: gradient tensors use stochastic rounding (probabilistic rounding that preserves expected value), while weight and activation tensors use round-to-nearest-even. This policy is fixed during training; it is not learned or adapted.

Information flows through these components in a fixed sequence per training step. For each linear layer designated as NVFP4: (a) incoming BF16 activations and FP32 weights enter the forward GEMM; (b) the quantization engine converts both to NVFP4 using nearest rounding and the appropriate scaling scheme (2D for weights, 1D for activations), with Hadamard transforms NOT applied to forward-pass tensors; (c) the Tensor Core computes the matrix product in NVFP4 and accumulates the result in FP32, which is then converted to BF16 and passed to the next layer; (d) during backpropagation, the incoming BF16 output gradient and the transposed weight matrix enter the Dgrad GEMM—activations gradients are quantized with 1D scaling and nearest rounding, while weights use their pre-computed forward-pass quantization; (e) the Wgrad GEMM receives the transposed activation tensor (with Hadamard transform applied before quantization, using stochastic rounding) and the transposed output gradient (stochastic rounding, 1D scaling), producing a weight gradient in FP32 that flows to the optimizer; (f) the optimizer maintains weights in FP32, so weight updates are applied in full precision before the next iteration's quantization.

3.3 Roadmap for the Deep Dive

• First, the NVFP4 quantization procedure — the format itself and the mathematical steps for converting a high-precision tensor into NVFP4 — because every other component operates on or modifies this quantization process. Understanding the two-level scaling, the block structure, and the role of the global scale is prerequisite to understanding why Hadamard transforms help and why 2D scaling matters.

• Second, mixed-precision layer selection — which layers stay in high precision and why — because this is the first decision made (at configuration time) and the ablation studies show it is the single most critical factor for preventing training divergence. The empirical finding that final layers are disproportionately sensitive determines the structure of all subsequent experiments.

• Third, Random Hadamard transforms — the mathematical mechanism, why they are applied only to Wgrad inputs, and the trade-offs in matrix size and randomization — because these transforms modify the data that enters quantization, and their interaction with 2D scaling (they must NOT be applied to weights) creates a coupling between components that the paper navigates through specific design choices.

• Fourth, two-dimensional weight scaling and the chain rule problem — the precise mechanism by which inconsistent forward/backward quantization breaks backpropagation, and why 2D block scaling for weights (but not activations) resolves it — because this connects the abstract numerical format design to the concrete correctness of the optimization algorithm.

• Fifth, stochastic rounding — the bias-variance trade-off, why gradients need it but activations and weights are harmed by it, and which specific gradient tensors (both Dgrad and Wgrad inputs for the 12B model) require stochastic treatment — because this is the component that most directly addresses the accumulation of quantization error over many training steps.

• Sixth, the full integration and ablations — how these components interact, the order in which they are stacked, and the evidence (Figure 4) that all four are individually necessary at the 12B/10T-token scale — because the paper's core claim is not about any individual technique but about the necessity of their combination.

3.4 Detailed, Sentence-based Technical Breakdown

This is primarily a methodology and validation paper whose core contribution is demonstrating that a specific combination of four training techniques enables stable FP4 pretraining of billion-parameter models over multi-trillion-token horizons, and that removing any single technique measurably degrades convergence. The techniques themselves are individually known; the contribution is their integration, their scale of validation, and the ablation evidence that they form a necessary system.

---

NVFP4 Format and Quantization Procedure

The NVFP4 format is a 4-bit floating-point representation defined by the E2M1 type: one sign bit, two exponent bits, and one mantissa bit. This yields exactly 16 distinct representable values: zero (with two sign representations, +0 and -0), ±0.5, ±1, ±1.5, ±2, ±3, ±4, and ±6. The dynamic range spans from 0.5 to 6 (a factor of 12), which is approximately 3.58 binades (powers of two: log₂(6/0.5) ≈ 3.58). This tiny representable set is the fundamental constraint that all subsequent techniques must work around.

The quantization procedure converts a tensor from higher precision (BF16 or FP32) to NVFP4 through a two-level scaling strategy, applied block-by-block. The process is specified in Appendix B and works as follows:

Step 1: Global tensor-level scaling. Before any block-level processing, all values in the tensor are rescaled by a single FP32 scale factor so that the block-level scales (which will be stored in FP8) stay within representable range. The global encode scale is computed as:

$s_{\text{enc}} = \frac{6 \times 448}{amax_x}$

where $amax_x = \max_i(|x_i|)$ is the absolute maximum value across the entire tensor $x$, $6$ is the maximum representable magnitude in the E2M1 (FP4) format, and $448$ is the maximum representable magnitude in the E4M3 (FP8) format used for block-level scale factors.

What it computes: the single scalar multiplier that maps the tensor's largest-magnitude value to the product of the FP4 maximum and the FP8 maximum, effectively reserving headroom in the block-level scales. After this scaling, the largest local decode scale (which will be $amax_x / 6$) maps to $448$ in E4M3, ensuring it fits without saturation.

Why this form: the product $6 \times 448$ represents the total dynamic range that can be expressed by the two-level scheme—the FP8 block scale can represent factors up to 448, and the FP4 values within a block can represent up to 6× the block's base scale. By normalizing to this product, the global scale ensures that the block-level FP8 scale factors never need to exceed their representable range regardless of the tensor's original magnitude. Without this global scale, tensors with very large or very small magnitudes would require block-level scales that saturate or underflow in E4M3, losing information. The corresponding decode scale $s_{\text{dec}} = 1/s_{\text{enc}}$ is stored in FP32 and applied after the GEMM to restore the original magnitude scale.

An important implementation detail: computing the global $amax_x$ requires scanning the entire tensor, which induces an extra pass through device memory—once to find the maximum, and once to apply the scaling before conversion to FP4. The paper notes that the global scale could potentially span a smaller granularity (e.g., a row or block of elements) to avoid additional round-trips through device memory, suggesting this as an optimization opportunity rather than a fundamental constraint.

Step 2: Local block-level scaling. The tensor is partitioned into blocks of 16 contiguous elements (along the dot-product dimension for GEMM inputs). For each block $b$, a local decode scale is computed so that the block's own absolute maximum value maps to the FP4 maximum representable:

$s_{\text{dec},b} = \frac{amax_b}{6}$

where $amax_b = \max_{i \in b}(|x_i|)$ is the absolute maximum within that block.

What it computes: the per-block multiplier that normalizes the block's largest element to 6 (the maximum FP4 value). For a block where the largest element is 3, the scale is 0.5; for a block where the largest element is 0.75, the scale is 0.125. All values in the block are then divided by this scale (multiplied by its reciprocal during encoding), bringing them into the [−6, 6] range where FP4 quantization can operate.

Why this form: this is the standard "absolute maximum" quantization strategy used across microscaling formats. It maximizes the utilization of the FP4 representable range—the largest element uses the full precision available—while preventing saturation (values exceeding 6 and being clipped). The trade-off is that blocks with a single outlier will have all other values compressed toward zero, potentially losing small-magnitude information. This is the precise motivation for Hadamard transforms: by reducing the outlier-to-typical ratio within blocks, more of FP4's limited precision budget can be allocated to representing meaningful variation rather than accommodating a single extreme value.

Step 3: Block scale quantization to FP8. The local decode scale must be stored in FP8 (E4M3 format) because Tensor Cores read block scales from this format. Before quantization, the scale is multiplied by the global encode scale to keep it in range:

$s_{\text{dec},b,\text{e4m3}} = \text{e4m3}(s_{\text{dec},b} \cdot s_{\text{enc}})$

where $\text{e4m3}(\cdot)$ denotes rounding to the nearest representable E4M3 value. The purpose of multiplying by $s_{\text{enc}}$ here is to remap the largest local decode scale to the FP8 maximum representable (448), ensuring no saturation when stored in E4M3. Round-to-nearest-even is used for this quantization step.

The corresponding local encode scale is then recovered by inverting the quantized decode scale:

$s_{\text{enc},b} = \frac{1}{\text{fp32}(s_{\text{dec},b,\text{e4m3}}) \cdot s_{\text{dec}}}$

where $\text{fp32}(\cdot)$ converts the E4M3 value back to FP32. This inversion ensures that $s_{\text{enc},b} \cdot s_{\text{dec}} \cdot s_{\text{dec},b,\text{e4m3}} \approx 1$, meaning the original data magnitude can be approximately recovered after the GEMM. If this product deviates significantly from 1—which can happen if the E4M3 quantization step introduces substantial rounding error—the decoded values will be systematically scaled away from their intended magnitudes, introducing bias into the computation.

Step 4: Element-wise quantization to FP4. Each element $x_i$ in block $b$ is scaled by the local encode scale and quantized:

$\hat{x}_i = q(x_i \cdot s_{\text{enc},b})$

where $q(\cdot)$ is the FP4 quantization function—either round-to-nearest-even (for weights and activations) or stochastic rounding (for gradients). The resulting $\hat{x}_i$ is a 4-bit value in the E2M1 format, representing one of the 16 FP4 values.

Step 5: Tensor Core computation and decoding. The Tensor Core reads the quantized values $\hat{x}_i$ and the local decode scales $s_{\text{dec},b,\text{e4m3}}$ for both input operands (call them $x$ and $y$). It computes partial dot-products over each block of 16 elements, multiplies each partial product by the corresponding scale factors from both operands, and accumulates the results:

$\text{partial\_sum}_b = s_{\text{dec},b,\text{e4m3}}^x \cdot s_{\text{dec},b,\text{e4m3}}^y \cdot \sum_{k \in b} (\hat{x}_k \cdot \hat{y}_k)$

After the GEMM completes, the global decode scales $s_{\text{dec}}^x$ and $s_{\text{dec}}^y$ are applied to the full output matrix, restoring it to the original magnitude scale. The output is produced in FP32 or BF16 precision, maintaining high accuracy for the accumulation even though the multiplication operands were in FP4.

Key design decisions in the NVFP4 format compared to MXFP4: NVFP4 differs from the MXFP4 standard in three ways that together provide meaningful training advantages:

1. Block size 16 vs. 32: NVFP4 uses blocks of 16 elements rather than 32. This reduces the dynamic range within each block because outliers affect only half as many neighboring values. The consequence is that more blocks have their scale factor determined by typical values rather than extremes, preserving more information in the presence of outliers.

2. E4M3 block scales vs. UE8M0: NVFP4 stores block scales as E4M3 (3 exponent bits, 4 mantissa bits including the implicit bit) rather than UE8M0 (unsigned 8-bit exponent, power-of-two only). This provides fractional precision in the scale factor, allowing the block's absolute maximum to map much closer to 6 in FP4 rather than being forced to a power-of-two alignment. Appendix B.4 illustrates the concrete advantage: for a block with $amax = 3 + \delta$, MXFP4's power-of-two scale forces the amax to quantize to 3 in FP4 (wasting the ±4 and ±6 bins) and loses nearly one binade of dynamic range. NVFP4's fractional E4M3 scale can map the amax to a value near 6, utilizing all FP4 bins and the full dynamic range.

3. Two-level scaling (FP32 per-tensor + FP8 per-block) vs. single-level: Because E4M3 has reduced exponent range compared to UE8M0 (it trades dynamic range for mantissa precision), NVFP4 requires the additional FP32 per-tensor scale to handle values that would otherwise overflow the E4M3 format. This two-level approach effectively decouples the problem: the per-tensor scale handles global magnitude, while the per-block scale handles local variation with high precision. MXFP4's single-level UE8M0 scales handle both in one step, but at the cost of power-of-two coarseness.

The practical impact of these differences is quantified in Section 5 (Figure 6): training a 8B model with MXFP4 requires 36% more tokens (1.36T vs. 1T) to match NVFP4's validation loss, directly translating format design choices into training efficiency.

---

Mixed-Precision Layer Assignment

The most critical finding from the paper's ablation studies is that naively quantizing every linear layer to FP4 causes immediate training divergence. This is not a slow degradation—the model fails to converge at all. The mixed-precision strategy is therefore not an optimization to close a small loss gap; it is a prerequisite for training to proceed.

Which layers stay in high precision. For the 12B model, the authors choose a conservative configuration: the first two blocks and the final eight blocks of the network are kept in BF16. The model has 62 total blocks, each containing 2 linear layers (for FFN or Mamba-2 blocks) or multiple linear layers (for attention blocks). The high-precision layers represent 16% of the total linear layers. For the 8B model (Section 5), only the final eight blocks are kept in BF16 (approximately 15% of layers). The 1.2B ablation model (Appendix E.2) shows that keeping only the last four blocks in BF16 is sufficient for convergence, while keeping the first four blocks without the last ones is insufficient—training still diverges.

Why final layers are disproportionately sensitive. The paper observes through tensor analysis that "the last layers tend to have larger quantization errors in the weight gradients." This is an empirical finding rather than a theoretically predicted result, but it has an intuitive interpretation: the final layers of a language model produce logits over the vocabulary, where the output distribution must be precisely calibrated to minimize cross-entropy loss. The gradients flowing back from this loss function carry fine-grained information about which token probabilities need adjustment. When these gradients are quantized to FP4—where the minimum relative step between adjacent values can be as large as 50% (from 0.5 to 1)—the precision is insufficient to encode the subtle probability adjustments needed. Additionally, the weight gradients in these layers see the accumulated error from all preceding layers, making them the convergence bottleneck.

The paper explicitly notes that "quantization error metrics could potentially serve as a mechanism to determine which linear layers should remain in higher precision during training." This suggests an automated approach for future work: during an initial training phase, measure the quantization error per layer and use a threshold to decide which layers need higher precision, rather than relying on a fixed heuristic (e.g., "last N blocks").

Other operations kept in higher precision. Beyond the linear layer GEMMs designated for NVFP4, several operations are always kept in BF16 or FP32 regardless of the mixed-precision configuration (Section 4.1). Embedding layers and the output projection head remain in BF16 because they interact directly with discrete token indices where quantization error would manifest as incorrect token probabilities. Normalization layers (RMSNorm) and non-linearities (Squared ReLU) use their original precision because these operations have implementations that are not amenable to FP4 Tensor Core acceleration—the cost of converting to FP4 and back would exceed any throughput benefit. Attention components, including softmax and the query-key and attention score-value batched GEMMs, remain in BF16 because softmax is numerically sensitive (it involves exponentials and normalizations) and the batched GEMM operations in attention have different shapes than the large matrix multiplications where FP4 throughput benefits are largest.

Optimizer and gradient accumulation precision. The master weights (the FP32 copy maintained by the optimizer), the weight gradients used for gradient accumulation across microbatches and data-parallel replicas, and the optimizer states (Adam's first and second moment estimates) are all kept in FP32. This is standard practice in mixed-precision training: the optimizer performs small additive updates to weights that would be lost in FP4 precision, and the moment estimates require accumulation over thousands of steps where quantization bias would compound. Tensor parallel reductions—where partial results from different GPUs are summed—are performed in BF16, which provides sufficient precision for across-device communication without the bandwidth cost of FP32.

The ablation evidence for mixed precision necessity. Figure 9 (Appendix E.2) demonstrates the sensitivity experimentally on the 1.2B model trained for 1T tokens. When all linear layers are quantized to NVFP4 (the base method), training diverges. Keeping only the last layer in BF16 does not prevent divergence. Keeping the last two layers in BF16 also fails. Keeping the last four layers in BF16 converges, with validation loss approaching the BF16 baseline (within ~0.05 in absolute loss). Keeping the first four layers in BF16 (without the last layers) diverges—confirming that sensitivity is concentrated at the end of the network, not the beginning. Keeping the first two and last two layers in BF16 converges, but this uses more high-precision layers than keeping just the last four. The finding that the final layers are the bottleneck, not the initial layers, is non-obvious because initial layers process the raw input where small numerical errors might be expected to compound through the entire network. The paper's empirical result contradicts this intuition: error introduced early in the network can potentially be compensated by later layers, but error in the final layers directly affects the loss computation and has no subsequent layers to correct it.

---

Random Hadamard Transforms

Random Hadamard transforms address the outlier problem in FP4 quantization by mathematically rotating the data before quantization so that large-magnitude values are redistributed across multiple elements. The core idea is that outlier values are "spiky"—concentrated in a few feature dimensions—and a random orthogonal transform spreads this energy across all dimensions, making the distribution more uniform and easier to quantize.

Mathematical mechanism. A Hadamard matrix $H_d$ of size $d \times d$ is a normalized orthogonal matrix with entries constrained to $\pm 1/\sqrt{d}$. It can be constructed recursively: $H_2 = \frac{1}{\sqrt{2}} \begin{bmatrix} 1 & 1 \\ 1 & -1 \end{bmatrix}$, and $H_d = \frac{1}{\sqrt{2}} H_2 \otimes H_{d/2}$, where $\otimes$ denotes the Kronecker product. The key property is $H_d H_d^T = I$ (orthogonality), which means applying the transform to both operands of a matrix multiplication is lossless—the transforms cancel:

$C = (A H)(H^T B) = A (H H^T) B = A B$

where $A$ and $B$ are the two input matrices and $C$ is their product. This identity is what makes Hadamard transforms viable for training: they can be applied to the quantized domain without changing the mathematical result (in exact arithmetic).

Randomization via sign flips. A plain Hadamard transform can fail if outliers happen to align with the Hadamard basis—for instance, if a tensor has structured patterns that the fixed $\pm 1$ entries of $H_d$ do not effectively disperse. To prevent this, the paper uses a Random Hadamard transform, which multiplies the Hadamard matrix by a random diagonal sign matrix $S_d$:

$H = S_d H_d$

where $S_d$ is a $d \times d$ diagonal matrix whose entries are randomly chosen from $\{-1, 1\}$. This flips the signs of entire rows of $H_d$, creating a random orthogonal matrix. The randomization ensures that any structured outliers are unlikely to survive the transform regardless of their pattern.

Tiled application. The transform is applied in a tiled fashion: for a tensor of size $m \times k$, it is partitioned into $\frac{m}{d} \times \frac{k}{d}$ tiles of size $d \times d$, and each tile is multiplied by $H$. Specifically, for the tensor $X$, the transformed tensor is $X' = X H$ (right-multiplying by $H$, which transforms along the dot-product dimension—the dimension that will be contracted in the GEMM). The computational cost is $mkd$ multiply-adds and $d^2$ reads for the Hadamard matrix, which is small when $d$ is much smaller than $m$ or $k$. At the paper's chosen size of $d = 16$, and typical layer dimensions of 4096 or 5120, the transform cost is a small fraction of the GEMM itself. The paper notes that these transforms can be implemented as batched matrix multiplications and fused with other layers to reduce round-trips to device memory.

Where transforms are applied—and where they are not. This is the most empirically specific finding about Hadamard transforms in the paper. The authors test applying transforms to the inputs of all three GEMM types (Fprop, Dgrad, Wgrad) on the 1.2B model (Appendix E.4.1, Figure 11). The results are:

• Wgrad inputs (activation tensor): Applying the transform improves validation loss compared to no transform. The relative difference from the BF16 baseline reduces from approximately −3% (no transform) to approximately −1.5% (with transform on Wgrad).

• Fprop inputs (activation tensor): Applying the transform degrades validation loss compared to no transform. The relative difference worsens.

• Dgrad inputs (output gradient tensor): Applying the transform degrades validation loss compared to no transform.

The paper hypothesizes that the transform itself introduces additional quantization error (the matrix multiplication with $H$ must itself be quantized, and the transformed values may land between FP4 representable numbers in ways that nearest-rounding handles poorly), and for Fprop and Dgrad, this added error outweighs the outlier-mitigation benefit. Only for Wgrad—where the transform is applied to the activation tensor before computing weight gradients—does the outlier reduction dominate. This is a crucial practical finding: naive application of Hadamard transforms everywhere, which might seem like the conservative approach, actually hurts training.

Why weights cannot be transformed (interaction with 2D scaling). The paper explicitly states that Hadamard transforms are NOT applied to weight tensors, even in Wgrad GEMMs where activations are transformed. This is because applying transforms along the dot-product dimension would break the 2D scaling consistency described in Section 4.3: if weights were transformed differently in the forward and backward passes, the chain rule violation would introduce errors. Since the transform on operands of a GEMM inverts via $H H^T = I$ only when both operands are transformed, the decision not to transform weights means the inversion property is not used. Instead, the Wgrad transform is applied unilaterally—the activation tensor is transformed before quantization, and the weight gradient is computed directly from the transformed-quantized activations. This means the weight gradient is computed with respect to a different (rotated) version of the activations, which is mathematically not equivalent to the original gradient, but empirically the outlier mitigation benefit outweighs this distortion for Wgrad specifically.

Hadamard matrix size trade-off. The matrix size $d$ controls how many elements participate in each transform. Larger $d$ distributes outliers over more values, approaching a Gaussian distribution more closely (by the central limit theorem, since the Hadamard transform is essentially a sum of $\pm 1$ combinations). However, larger $d$ increases the compute cost of the transform and, critically, requires the tensor dimension to be divisible by $d$. The paper tests sizes of 2×2, 4×4, 16×16, and 128×128.

On the 1.2B model, all sizes from 2×2 to 128×128 show virtually no difference in validation loss—the model is too small for the choice to matter. On the 12B model (Appendix E.4.2, Figure 12), the difference becomes visible: switching from 16×16 to 4×4 after 3.4T tokens causes the relative loss error to worsen (the gap from FP8 increases), while switching to 128×128 provides a minor improvement (relative error becomes slightly less negative). The 4×4 degradation is attributable to insufficient outlier dispersal—with only 4 elements per block, the probability that an outlier is effectively averaged out is lower. The 128×128 marginal improvement confirms that larger matrices help, but the paper chooses $d = 16$ as a balance between accuracy and cost, noting that "conclusions from smaller scales may not always hold for larger models."

Randomization strategy. The paper tests three levels of randomization (Appendix E.4.3, Figure 13): (1) no random sign vector—using the standard Hadamard transform without sign flips; (2) a single fixed random sign vector used for all transforms across all layers throughout training; (3) a new random sign vector for every transform instance. On the 1.2B model, all three strategies perform identically. On the 12B model, the absence of random sign vectors causes a measurable degradation (relative loss error becomes more negative), while using a unique seed per transform shows no improvement over a single fixed seed.

The interpretation: at larger scales, structured outliers that align with the Hadamard basis become more likely because the model has more parameters and more opportunity to learn weight patterns that interact poorly with the fixed $\pm 1$ structure. A single random sign flip is sufficient to break this alignment; re-randomizing every transform adds no further benefit. The paper therefore uses a single fixed random seed across all layers for simplicity.

---

Two-Dimensional Weight Scaling and the Chain Rule Problem

This section addresses what is perhaps the most subtle numerical issue in FP4 training: quantization can break the chain rule of backpropagation by making the forward and backward passes compute different functions.

The problem. Consider a linear layer with weight matrix $W$. In the forward pass, the layer computes $y = Wx$ (plus any bias, though the paper's models do not use biases). During quantization, $W$ is converted to a quantized representation $W_{\text{fprop}}$ by applying scaling and rounding along the input-channel dimension (the dimension that contracts with $x$). In the backward pass, the activation gradient is computed as $\partial x = W^T \partial y$. However, the backward pass quantizes $W$ along a different dimension: because the matrix is transposed, what were rows become columns, and the scale factors that were computed along input channels in the forward pass would need to be computed along output channels in the backward pass. If the scaling is one-dimensional (1×16 blocks along the dot-product dimension), the forward pass groups 16 elements along input channels, while the backward pass groups 16 elements along output channels—producing different blocks, different scale factors, and therefore different quantized values. The result is $W_{\text{fprop}} \neq W_{\text{bprop}}$, and the backward pass computes $\partial x = W_{\text{bprop}}^T \partial y$, which is the gradient of a different function $y_{\text{bprop}} = W_{\text{bprop}} x$ than the one used in the forward pass $y_{\text{fprop}} = W_{\text{fprop}} x$.

Why this matters. Backpropagation's correctness relies on the chain rule: if $y = f(x)$, then $\partial L / \partial x = (\partial L / \partial y) \cdot (\partial y / \partial x)$. The gradient of $f$ is computed by evaluating the derivative of $f$ at the same point where $f$ was evaluated in the forward pass. If the backward pass uses a different quantized $W$, it is evaluating the derivative of a different function—the chain rule no longer holds, and the gradient estimate can point in an arbitrary direction relative to the true loss landscape. Over many training steps, this manifests as degraded convergence or instability.

The 2D block scaling solution. To ensure $W_{\text{fprop}} = W_{\text{bprop}}$, the paper proposes quantizing weights using two-dimensional blocks of size 16×16. In this scheme, each block spans 16 input channels and 16 output channels—a square region of the weight matrix rather than a 1D strip. When the matrix is transposed, a 16×16 block remains a 16×16 block (just transposed), so the same elements are grouped together and the same scale factor applies. The scale factor is computed once and used in both passes.

Specifically, for the forward pass, the weight matrix is partitioned into 16×16 tiles. For each tile, one scale factor is computed based on the maximum absolute value across all 256 elements in that tile. These scale factors are stored and used identically when the weight matrix is accessed in transposed form during the backward pass. The per-tensor FP32 scale is also shared.

Critically, 2D scaling is applied only to weights, not to activations or gradients. Activations and gradients use the standard NVFP4 1D scaling (1×16 blocks along the dot-product dimension). The paper's rationale is that activation quantization also has a chain rule concern—the forward pass quantizes activations along one dimension, and the backward pass would theoretically quantize them along another dimension when used in Dgrad—but empirically, training is less sensitive to inconsistencies in activation tensors than weight tensors. Additionally, weights are more tolerant of the coarser granularity of 2D scaling (256 elements sharing one scale versus 16) because "they can adapt to the FP4 values"—during training, the optimizer can adjust weight values to compensate for the reduced scaling granularity, whereas activation values are fixed by the forward computation and cannot adapt.

Empirical validation on 1.2B model (Appendix E.5, Figure 14). The paper tests three scaling configurations for the 1.2B model:

• 1×16 scales along different dimensions in forward and backward (the broken-chain-rule case): For weights, this produces the worst validation loss, with a relative difference from BF16 of approximately −3.5% to −4% throughout training. The gap widens during the learning rate decay phase, where fine-grained optimization becomes more important.

• 1×16 scales along the same dimension in forward and backward: For weights, this is the ideal case—same quantized representation in both passes—and achieves much better loss (relative difference approximately −1.5%). However, as the paper notes, this configuration cannot be implemented in practice because Tensor Cores require scaling factors along the dot-product dimension, which changes between forward and backward. This experiment is a "what if" ablation to isolate the effect of consistency.

• 16×16 block scales: For weights, this is the practical configuration that achieves consistency without requiring the same dimension. It improves loss over the inconsistent 1D case (relative difference approximately −2.5% to −3%), though not quite matching the ideal same-dimension 1D case. This gap is the cost of the coarser scaling granularity (256 elements per block vs. 16).

For activations (Figure 14, right panel), the inconsistent 1D scaling along different dimensions shows a mild degradation, primarily during the learning rate decay phase. The consistent 1D scaling along the same dimension performs similarly to the inconsistent case for most of training, diverging only near the end. This confirms the paper's claim that activations are less sensitive to chain rule violations than weights.

Interaction with Hadamard transforms. The paper notes that "applying Hadamard transforms exacerbates the inconsistency and further impacts model accuracy." This is why Hadamard transforms are explicitly NOT applied to weight tensors—they would introduce an additional dimension of forward/backward inconsistency beyond the scaling issue. This creates a design constraint: the outlier-mitigation benefit of Hadamard transforms can only be realized for GEMMs where weights are not transformed, which restricts transforms to Wgrad (where only activations are transformed) and makes Fprop and Dgrad ineligible (since they would require transforming weights or output gradients, respectively). The paper accepts this limitation, finding that Wgrad transforms are sufficient for the 12B model.

---

Stochastic Rounding for Unbiased Gradients

Stochastic rounding is a probabilistic quantization scheme designed to eliminate the systematic bias that accumulates when deterministic rounding consistently pushes values in the same direction.

The bias problem with deterministic rounding. In FP4 quantization with round-to-nearest-even, a high-precision value $x$ is mapped to the nearest representable FP4 value. However, because FP4 has only 8 positive values with irregular spacing—0.5, 1, 1.5, 2, 3, 4, 6—the "nearest" decision is asymmetric for many intervals. For example, values between 1.25 and 1.75 round to 1.5 (a width of 0.5), but values between 1.75 and 2.5 round to 2 (a width of 0.75). A value uniformly distributed in [1, 3] will, on average, be rounded downward more often than upward because the representable points are not evenly spaced. This creates a systematic bias: the expected value of the quantized number differs from the original value, $\mathbb{E}[q(x)] \neq x$.

For gradients, this bias is particularly harmful. Gradients represent the direction and magnitude of parameter updates. If thousands of gradient elements are each biased slightly in a consistent direction, the cumulative effect steers the optimizer toward a different point in parameter space than the true gradient would indicate. Over many training steps—tens of thousands for the 12B model—this bias can prevent convergence or cause convergence to a worse local minimum.

How stochastic rounding works. Instead of always rounding to the nearest representable value, stochastic rounding randomly chooses between the two nearest representable values with probability inversely proportional to their distances. For a value $x$ that lies between representable values $a$ and $b$ (with $a < x < b$):

$q_{\text{SR}}(x) = \begin{cases} a & \text{with probability } \frac{b - x}{b - a} \\ b & \text{with probability } \frac{x - a}{b - a} \end{cases}$

What it computes: a random choice between the lower and upper FP4 values, where the probability of choosing the upper value is exactly proportional to how close $x$ is to $b$. If $x$ is 70% of the way from $a$ to $b$, it rounds up with probability 0.7 and down with probability 0.3.

Why this form: the expected value of the stochastically rounded number is exactly the original value: $\mathbb{E}[q_{\text{SR}}(x)] = a \cdot \frac{b-x}{b-a} + b \cdot \frac{x-a}{b-a} = x$. This is an unbiased estimator—over many realizations, the rounding introduces zero expected error. The variance (the spread of individual rounding decisions around the true value) is non-zero, but bias—the systematic shift that accumulates over optimization steps—is eliminated.

This bias-variance trade-off is the key insight: deterministic rounding has zero variance but non-zero bias; stochastic rounding has non-zero variance but zero bias. For gradients, which are accumulated over many steps, bias is the dominant source of error because it compounds, while variance averages out (the optimizer effectively smooths over individual noisy gradient estimates). For weights and activations, which are computed fresh each step, bias does not accumulate in the same way, and the additional variance of stochastic rounding is purely harmful.

Where stochastic rounding is applied. The paper's ablation (Appendix E.3, Figure 10) on the 1.2B model shows the specificity of this choice:

• Gradient tensors with stochastic rounding: Training converges with loss approaching the BF16 baseline. The relative difference stabilizes around −1.5% to −2%.

• Gradient tensors without stochastic rounding: Training diverges. The validation loss grows rapidly after a few hundred billion tokens, indicating that the gradient bias accumulates to the point where the optimizer can no longer make progress.

• Stochastic rounding on activations or weights: Training diverges, and does so faster than with no stochastic rounding at all. The additional variance from stochastic rounding on forward-pass tensors apparently introduces more noise than the model can tolerate, even though the bias is reduced.

• Stochastic rounding on all backward-pass tensors (not just gradients): This shows little improvement over stochastic rounding on gradients only, suggesting that the divergence from stochastically rounding forward-pass tensors is the primary issue.

For the 12B model, the requirement is more specific: stochastic rounding must be applied to the gradients going into both Dgrad and Wgrad GEMMs. That is, both the activation gradient computation and the weight gradient computation require unbiased gradient inputs to achieve proper convergence. This is more demanding than the 1.2B model, which was stable with only Wgrad gradient rounding—a scale-dependent finding that reinforces the paper's theme that techniques become more critical at larger model sizes and longer token horizons.

Hardware support. The paper notes that NVIDIA Blackwell GPUs provide native support for stochastic rounding in FP4 conversion instructions, alongside the standard round-to-nearest-even mode. This means stochastic rounding can be performed in hardware at the same throughput as deterministic rounding, without software emulation overhead.

Comparison with prior work. The finding that stochastic rounding is beneficial only for gradients, and harmful for activations and weights, aligns with prior work (Chmiel et al., 2025; Castro et al., 2025). The paper's contribution is validating this at the 12B/10T-token scale and showing that the requirement extends to both Dgrad and Wgrad gradients at this scale, not just Wgrad as in smaller-scale experiments.

---

Integration and Interaction of Components

The four techniques are not independent; they interact in ways that constrain the design space.

Stacking order and necessity. Figure 8 (Appendix E.1) on the 1.2B model shows the cumulative effect of adding techniques to a base configuration (all layers NVFP4, standard 1D scaling, nearest rounding, last 4 blocks BF16):

• Starting point: BF16 baseline.
• "Last 4 in BF16" alone: Validation loss is already close to BF16 (relative error approximately −1%), showing that the 1.2B model is relatively robust.
• Adding 2D weight scaling: Small additional improvement.
• Adding Hadamard transforms on Wgrad: Further improvement.
• Adding stochastic rounding: Small additional improvement.

The benefits are incremental and order-dependent—if 2D scaling and Hadamard transforms are already in place, the marginal benefit of stochastic rounding is smaller than if it were added first. This is because the techniques address overlapping problems: 2D scaling prevents chain rule violations that would create spurious gradients, Hadamard transforms reduce outlier-induced quantization error that would bias gradients, and stochastic rounding removes residual bias. With the first two in place, there is less bias for stochastic rounding to correct.

For the 12B model (Figure 4), the ablation is more dramatic. Starting from the full methodology (all four techniques active) after 3.43T tokens of training, the paper removes one component at a time and continues training:

• Full methodology: Relative loss error from FP8 is approximately −0.5% to −1%, stable.
• Remove stochastic rounding: Error worsens to approximately −1.5% to −2%. The gap grows gradually, consistent with bias accumulation.
• Remove Hadamard transforms: Error worsens to approximately −1.5% and is volatile, indicating instability from outlier-dominated quantization.
• Remove 2D weight scaling: Error immediately jumps to approximately −1.5% and continues worsening, suggesting chain rule violations cause sustained degradation.
• Reduce high-precision layers to only last 4 blocks (from 8): Error is approximately −0.5%, essentially unchanged from the full methodology. This actually suggests the 12B model could have been trained with fewer high-precision layers—only the final 4 blocks rather than the final 8—which would increase the fraction of the network in FP4 from 84% to roughly 92%. This is a practical finding that future deployments could use to squeeze more throughput from the same methodology.

Why all four are necessary only at scale. A key insight from comparing the 12B and 1.2B ablations is that the smaller model is much more forgiving. The 1.2B model converges reasonably with only high-precision layers; adding 2D scaling, Hadamard transforms, and stochastic rounding provides incremental improvements. The 12B model, trained on 10× more tokens, requires all four for stability. The paper's interpretation is that larger models have more outliers (more parameters → more opportunity for extreme values to emerge), longer training horizons give bias more time to accumulate, and the hybrid Mamba-Transformer architecture may have specific sensitivity patterns that the smaller Transformer-only model does not exhibit. This scale-dependence is the paper's central methodological argument: FP4 training techniques validated only on small models and short training runs cannot be assumed to transfer to production-scale training.

Switching precision late in training (Appendix D). The paper reports that most of the FP4 loss gap comes from quantizing the forward pass, and that switching from NVFP4 to BF16 during the learning rate decay phase can close the remaining gap. Specifically, for the 12B model with NVFP4 throughout training, the relative loss error is approximately −1.5% at the end of training. If the precision is switched to BF16 for both forward and backward passes starting at 8.2T tokens (roughly 82% through training, just before learning rate decay begins at 80%), the loss gap closes to near zero—the model matches the FP8 baseline. If only the forward pass is switched to BF16 (keeping backward in NVFP4), the gap reduces to approximately −0.5%. If only the backward pass is switched, there is "no such improvement."

This finding has practical implications for deployment: the majority of training (the stable-phase 80%) can run in full NVFP4, and only a small fraction (the final 20% with learning rate decay) needs higher precision to match FP8 quality. Since forward-pass computations are roughly one-third of total GEMM operations (one Fprop vs. two backward GEMMs), switching only the forward pass means approximately 6% of total training compute runs in higher precision, while the remaining 94% benefits from FP4 throughput. The paper recommends switching precision "shortly before the onset of learning rate decay for full loss recovery, or at the very end for notable loss improvements with minimal effect on training runtime."

This result—that forward-pass quantization, not backward-pass, is the primary source of the loss gap—is interesting because it contradicts some prior findings (Chmiel et al., 2025 reported loss recovery from switching backward-pass precision). The difference may be due to the specific model architecture or the NVFP4 format's properties.

Summary of the recommended configuration. The paper's final recommendation for NVFP4 training (Section 4) is:

• Keep approximately 15% of linear layers in BF16 or MXFP8, concentrated in the final blocks of the network.
• Apply Random Hadamard transforms of size 16×16 to the activation inputs of Wgrad GEMMs only. Use a single fixed random sign vector shared across all layers.
• Use 2D block scaling (16×16) for weights to maintain consistent representations between forward and backward passes.
• Use 1D block scaling (1×16) for activations and gradients, since they are less sensitive to chain rule violations.
• Apply stochastic rounding to gradient tensors (inputs to both Dgrad and Wgrad). Use nearest rounding for weights and activations.
• For maximum accuracy, switch forward-pass computations to BF16 during the learning rate decay phase.

This configuration is the result of extensive ablation, and the paper presents it as a validated starting point for practitioners rather than a theoretically optimal solution. The authors note that future work should explore automating the mixed-precision layer selection via quantization error metrics and reducing the fraction of high-precision layers further.

4. Key Insights and Innovations

Innovation 1: The "Necessary System" Framing — FP4 Training Is a Compositional Stability Problem, Not a Single-Algorithm Problem

The paper's deepest conceptual contribution is not any individual technique but the empirical demonstration that FP4 training at production scale requires a system of interacting techniques, all of which are individually necessary. This is a fundamentally different framing from how narrow-precision training has been approached previously. Prior work on FP8 training (Micikevicius et al., 2022; DeepSeek-AI et al., 2024) and early FP4 explorations (Tseng et al., 2025b; Chmiel et al., 2025; Castro et al., 2025) typically advanced one or two techniques — a new rounding scheme, a better scaling strategy, a transformation for outliers — and demonstrated incremental improvements over baselines. The implicit model was: find the single biggest source of numerical error, fix it, and training will work.

This paper's ablation structure (Figure 4) refutes that model. Starting from the full methodology on the 12B model after 3.43T tokens, removing stochastic rounding degrades convergence. Removing Random Hadamard transforms degrades convergence. Removing 2D weight scaling degrades convergence. And crucially, the base method — naively quantizing all layers with standard 1D scaling and nearest rounding — diverges immediately, regardless of which single technique is added (Appendix E.1). No individual technique is sufficient; the combination is what creates stability. This is intellectually significant because it recasts FP4 training as a compositional stability problem: there are multiple independent failure modes (outlier-dominated quantization, chain rule violations, gradient bias accumulation, layer-specific dynamic range insufficiency), and each must be addressed simultaneously because any unmitigated failure mode will eventually cause divergence at scale.

The field has seen this pattern before in other domains — adversarial robustness requires both architectural changes and training modifications; large-batch training requires both learning rate scaling and normalization adjustments — but the paper is the first to articulate it clearly for narrow-precision training. The implication is that future progress on even narrower formats (FP3? INT4?) will likely require not a single breakthrough but a coordinated set of techniques, each targeting a different numerical failure mode. This is a research-agenda-shaping insight that goes beyond the specific NVFP4 results.

Evidence for this framing comes from Figure 4, where every ablation line diverges from the full-methodology baseline, and from Appendix E.1 (Figure 8), which shows the cumulative effect of stacking techniques on the 1.2B model. The paper's explicit recommendation structure in Section 4 — "keep a few sensitive linear layers in higher precision... apply Random Hadamard transforms... use 2D scaling... use stochastic rounding" — encodes this systems-level understanding as a recipe rather than a single claimed invention.

Significance beyond performance: This is a diagnostic reframing of the problem. The field's question shifts from "what's the one trick to make FP4 work?" to "what are the independent failure modes of FP4 training, and have I addressed all of them?" This is a more productive framing that guides systematic investigation rather than ad-hoc experimentation.

Comparison to prior work: Prior FP4 training papers (Tseng et al., 2025b; Chmiel et al., 2025; Castro et al., 2025) each contributed techniques (stochastic rounding, Hadamard transforms, mixed precision), but none combined all of them, validated their interaction at billion-parameter scale and multi-trillion-token horizons, and showed through removal ablations that each component is individually necessary. The conceptual leap is from "here are techniques that help" to "these techniques form a necessary set."

Fundamental vs. incremental: This is a fundamental shift in how the problem is conceptualized, not an incremental refinement of an existing technique. It changes the unit of contribution from "algorithm" to "system."

---

Innovation 2: Empirically Mapping the Sensitivity Landscape — Why the Last Layers, Why Wgrad, Why Not Fprop

The paper makes a series of specific, non-obvious empirical findings about where numerical sensitivity is concentrated in large-model training, and these findings collectively form a sensitivity map that is the paper's second major conceptual contribution. Before this work, the default assumption in mixed-precision training was that sensitive layers might exist, but there was no systematic understanding of which layers or why. Practitioners typically kept the first and last few layers in higher precision as a heuristic, without evidence about whether this was optimal or whether the middle layers could also be sensitive.

The paper's sensitivity findings are:

1. Final layers are disproportionately sensitive, not initial layers. Figure 9 (Appendix E.2) shows that keeping only the last 4 blocks in BF16 enables convergence for the 1.2B model, while keeping the first 4 blocks without the last ones causes divergence. This contradicts the intuitive assumption that errors introduced early in the network would compound and be most damaging. The paper's empirical result reverses this: early quantization error can potentially be compensated by later layers, but error in the final layers directly corrupts the loss computation with no subsequent correction possible. The paper speculates that weight gradients in final layers carry accumulated error from all preceding layers, making them the convergence bottleneck.

2. Random Hadamard transforms help Wgrad but hurt Fprop and Dgrad. Figure 11 (Appendix E.4.1) shows that applying transforms to Wgrad inputs improves loss, while applying them to Fprop or Dgrad inputs degrades it. This is non-obvious because the transform is mathematically lossless when applied to both GEMM operands (via the $H H^T = I$ cancellation), yet applying it unilaterally to Wgrad activations helps while unilateral application to Fprop activations hurts. The paper hypothesizes that the transform itself introduces quantization error, and the trade-off between outlier mitigation and added quantization noise tips differently for different GEMMs. For Wgrad, outlier reduction dominates; for Fprop and Dgrad, added noise dominates.

3. Stochastic rounding is beneficial for gradients but harmful for activations and weights. Figure 10 (Appendix E.3) shows that applying stochastic rounding to activations or weights causes faster divergence than using no stochastic rounding at all. This is a precise characterization of the bias-variance trade-off: gradients accumulate bias over thousands of steps, so unbiased stochastic rounding is worth the variance cost; activations and weights are computed fresh each step, so added variance is purely harmful. For the 12B model, the requirement is even more specific: stochastic rounding must be applied to gradients entering both Dgrad and Wgrad, not just Wgrad as was sufficient for the 1.2B model.

4. Weight tensors are more sensitive to chain rule violations than activation tensors. Figure 14 (Appendix E.5) shows that inconsistent 1D scaling for weights (different quantized representations in forward and backward) degrades loss substantially, while the same inconsistency for activations causes only a mild degradation during learning rate decay. The paper hypothesizes that weight errors materialize in activation gradients and propagate through the network during backpropagation, while activation errors are more localized.

These findings are not presented as theoretical predictions but as empirical discoveries validated at two scales (1.2B and 12B). Their collective significance is that they form a practical sensitivity map that guides where to invest numerical precision. The map says: protect weight representations (2D scaling), protect gradients from bias (stochastic rounding), protect Wgrad from outliers (Hadamard transforms), protect final layers from quantization (mixed precision), but don't bother with Hadamard transforms on Fprop/Dgrad, don't bother with stochastic rounding on activations, and don't bother keeping early layers in high precision. This is actionable engineering knowledge that prior work did not provide.

Significance beyond performance: This is a diagnostic advance. The paper isn't just saying "our technique works"; it's characterizing where the numerical fragility in large-model training is located, which enables future work to target interventions more precisely and to predict which techniques will transfer to new architectures.

Comparison to prior work: Prior mixed-precision training work (Micikevicius et al., 2022; DeepSeek-AI et al., 2024) typically treated layer sensitivity as a binary property (some layers need higher precision) without characterizing which layers or why. Prior Hadamard transform work (Ashkboos et al., 2024, 2025) applied transforms broadly without characterizing where they help versus hurt. Prior stochastic rounding work (Castro et al., 2025; Chmiel et al., 2025) advocated for gradient rounding but didn't demonstrate the harm of activation rounding at scale. The paper's contribution is the systematic mapping across techniques, layer positions, and GEMM types.

Fundamental vs. incremental: This is a fundamental empirical contribution that changes what we know about the sensitivity structure of neural network training, though the individual findings are empirical observations rather than theoretical advances.

---

Innovation 3: NVFP4 vs. MXFP4 as a Format-Design Validation Experiment — Format Details Matter for Training, Not Just Inference

The paper's Section 5 comparison between NVFP4 and MXFP4 training on an 8B model is, at first glance, a straightforward head-to-head evaluation. But the deeper conceptual contribution is that it validates a specific hypothesis about what matters in narrow-precision format design for training, and the answer is not what the field might have assumed.

The hypothesis is this: for training (as opposed to inference), the critical format properties are fractional-precision block scaling (E4M3 with mantissa bits rather than UE8M0 power-of-two) and smaller block size (16 vs. 32), because these properties improve the accuracy of representing outliers and reduce information loss from scale factor coarseness. The counter-hypothesis — implicit in the MXFP4 standard's design — is that power-of-two scaling is sufficient and the hardware simplicity of UE8M0 outweighs any precision loss.

The paper's key result (Figure 6b) quantifies the answer: MXFP4 requires 36% more training tokens (1.36T vs. 1T) to match NVFP4's validation loss on the 8B model. This is a direct translation of format design choices into training efficiency — MXFP4 wastes over a third of its compute budget compensating for its numerical inferiority. The mechanism for this loss is explained in Appendix B.4: MXFP4's power-of-two scaling constraint means some FP4 bins (±4, ±6) can be effectively wasted, and up to one binade of dynamic range can be lost, reducing FP4's already-tiny palette from 8 values to effectively 6 in the worst case. NVFP4's E4M3 block scales with fractional precision preserve all FP4 bins and the full dynamic range.

This finding is significant because the field might reasonably have assumed that at training time — where weights are repeatedly updated and can potentially adapt to quantization constraints — format differences would matter less than at inference time. The paper's result contradicts this: format precision matters at least as much for training, because the accumulation of quantization error over billions of training steps amplifies small per-step differences. A 36% token penalty is enormous at frontier-model scale, representing months of additional training time and millions of dollars in compute.

The comparison also serves a broader function in the paper's argument: it justifies why NVFP4 specifically (rather than any 4-bit format) is the right substrate for the training methodology. If MXFP4 could be made to work with the same techniques, the paper's contribution would be purely methodological. But Figure 6a shows that even with identical training methodology (same mixed precision, same Hadamard transforms, same stochastic rounding), NVFP4 converges to a 1.5% relative loss error while MXFP4 reaches 2.5% — the format itself imposes a floor on achievable accuracy. The methodology is necessary but not sufficient; the format must also be numerically adequate.

Significance beyond performance: This is a validation of a design philosophy for narrow-precision formats. It argues that future formats (FP3? FP2?) should prioritize precise scaling with fractional range over simplified hardware scaling, and that block size matters independently of total bit width. It also provides a quantitative methodology for comparing format designs — measure the token overhead required to match a baseline — that future format proposals could adopt.

Comparison to prior work: Prior microscaling format work (Rouhani et al., 2023) established the concept but did not provide training-scale comparisons between format variants at equivalent precision. Prior NVFP4 work (Alvarez et al., 2025) introduced the format for inference but did not validate it for training. The paper provides the first direct, controlled comparison of two 4-bit formats for large-scale pretraining, quantifying the training efficiency gap.

Fundamental vs. incremental: This is an incremental but practically important validation. The concept that more precise scaling helps is not surprising, but the magnitude of the effect (36% token overhead) and its demonstration at production scale make a compelling case that format design details are not micro-optimizations — they are first-order determinants of training cost.

---

Innovation 4: The Forward-Pass Quantization Bottleneck — Identifying Where the Remaining Loss Gap Lives

The paper makes a diagnostic finding with practical implications that goes beyond the "how to train in FP4" core: most of the residual loss gap between FP4 and FP8 training comes from quantizing the forward pass, not the backward pass. This is demonstrated in Appendix D (Figure 7): switching only the forward pass from NVFP4 to BF16 during the learning rate decay phase reduces the relative loss error from approximately −1.5% to approximately −0.5%, while switching only the backward pass provides "no such improvement." The full loss gap can be closed by switching both forward and backward passes to BF16, or nearly closed by switching just the forward pass.

This finding is conceptually interesting because it is counter to some prior results. Chmiel et al. (2025) reported that switching the backward pass to higher precision recovered loss in their FP4 training setup. The difference may stem from the specific format (NVFP4 vs. their setting), model architecture (hybrid Mamba-Transformer vs. pure Transformer), or scale (12B vs. smaller models). Regardless, the paper's finding that forward-pass quantization is the primary bottleneck is a specific empirical claim that changes where practitioners should focus their precision budgets if they need to close the last bit of loss gap.

The practical upshot is significant: since forward-pass GEMMs are roughly one-third of total GEMM operations per training step (one Fprop vs. two backward GEMMs — Dgrad and Wgrad), switching only the forward pass to higher precision during the final 20% of training means approximately 6% of total training compute runs in higher precision, while 94% benefits from FP4 throughput. This is a much more favorable trade-off than switching the backward pass, which would require roughly 12% of compute in higher precision. The paper recommends switching "shortly before the onset of learning rate decay for full loss recovery, or at the very end for notable loss improvements with minimal effect on training runtime."

Significance beyond performance: This is a diagnostic finding that decomposes the loss gap by source (forward vs. backward quantization) and provides actionable guidance for precision budgeting. It also raises a theoretical question that the paper does not answer: why is forward-pass quantization more harmful than backward-pass? The paper's hypothesis (Section 3.4 in the prior analysis) is that weight gradients can adapt to quantization during optimization — weights can be adjusted to values that quantize favorably — while forward-pass activations are fixed by the input and the current weight state, offering no adaptation mechanism.

Comparison to prior work: This finding is presented in contrast to Chmiel et al. (2025), and the discrepancy suggests that the location of quantization sensitivity may be format-dependent or architecture-dependent, which is itself a useful caution against over-generalizing from any single study.

Fundamental vs. incremental: This is an incremental empirical finding — it doesn't change the conceptual understanding of FP4 training, but it provides a practically useful decomposition that can guide precision allocation in deployment.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The primary validation experiments use a 10-trillion-token corpus based on NVIDIA (2025b), consisting of high-quality curated and synthetic data including general web crawl, Wikipedia, math, code, academic data, crawl++, multilingual, and synthetic SFT-style data. Training uses a phased data-blending approach (Feng et al., 2024): Phase 1 covers 70% of training with a diversity-promoting data mixture, Phase 2 spans 20% with primarily high-quality datasets, and Phase 3 covers the final 10%. For the 8B model comparison, the same dataset is used with only two phases split between the first 60% and last 40% of training. Downstream evaluation is performed on a comprehensive benchmark suite spanning knowledge-intensive reasoning (MMLU, MMLU-Pro 5-shot, AGIEval English CoT), mathematics (GSM8k CoT, MATH), coding (HumanEval+, MBPP+), multilingual tasks (Global MMLU, MGSM), and commonsense understanding (ARC Challenge, HellaSwag, OpenBookQA, PIQA, Winogrande).

• Base model(s). Three model architectures are used: (1) A 12B-parameter hybrid Mamba-Transformer model following the Nemotron-Nano-12B-v2-Base architecture (NVIDIA, 2025b) with 62 blocks (6 Self-Attention, 28 FFNs, 28 Mamba-2), model dimension 5120, FFN dimension 20480, 40 query heads, 8 key-value heads, state dimension 128, and 8 Mamba groups. This serves as the primary validation architecture because it has been shown to achieve competitive accuracies across multiple benchmarks and represents a production-scale model. (2) An 8B-parameter hybrid Mamba-Transformer with 52 blocks (4 Self-Attention, 24 FFNs, 24 Mamba-2), model dimension 4096, 32 query heads, and 4 key-value heads, used for the NVFP4 vs. MXFP4 comparison. (3) A 1.2B-parameter standard Transformer architecture with 20 blocks, model dimension 2048, FFN dimension 6144, 16 query heads, and 8 key-value heads, used for ablation studies. The 12B model is chosen to demonstrate feasibility at a scale where practitioners would consider adopting FP4 training; the 1.2B model enables rapid iteration on methodology variations.

• Metrics. The primary training metric is validation loss, tracked continuously throughout the 10T-token training run on a held-out validation set (Section 3, Figure 2). The paper also reports the relative difference from the FP8 baseline, defined as (FP8 loss − experiment loss) / FP8 loss, expressed as a percentage—negative values indicate the experiment is worse than FP8 (Section 4, Figure 4). For downstream evaluation, accuracy is reported on each benchmark task using standard evaluation protocols (5-shot for MMLU and MMLU-Pro, chain-of-thought prompting for GSM8k and AGIEval English, pass@1 for coding tasks). The aggregate metric "General" (Table 2) appears to be an average across general-domain tasks. All evaluations are performed in BF16 precision regardless of training precision, ensuring fair comparison of the learned model quality rather than inference-time quantization effects.

• Baselines. Two reference training configurations are used: (1) FP8 baseline — the 12B model trained following the methodology in NVIDIA (2025b) and DeepSeek-AI et al. (2024), where all linear layers are computed in E4M3 except the first block and last two blocks which remain in BF16, with scale factors applied on 128×128 blocks for weights and 1×128 blocks for activations and gradients, computed online and stored in FP32. This represents the current production-standard narrow-precision training approach. (2) BF16 baseline — used for the 8B model comparison and 1.2B ablation studies, representing full-precision training without quantization. No comparison is made against 4-bit training baselines from prior work (e.g., MXFP4 with Tseng et al., 2025b methodology, or Chmiel et al., 2025 "FP4 All the Way"), which is a notable omission—the paper compares NVFP4 against FP8 and BF16, but not against prior FP4 training methods at equivalent scale.

• Generation budget / compute accounting. The paper does NOT measure compute in FLOPs or wall-clock time. Instead, the unit of comparison is training tokens processed—the primary experiments train on 10 trillion tokens for the 12B model and 1 trillion tokens for the 8B and 1.2B models, with validation loss tracked as a function of tokens seen. This makes the comparison fair across precision formats, since every configuration processes exactly the same amount of data. The paper explicitly states that "this report is primarily concerned with the underlying algorithms and methodology rather than with runtime efficiency or system-level optimizations" (Section 1), meaning that the claimed 2–3× throughput boost from FP4 over FP8 (Table 1) is a hardware capability that is NOT measured or validated in training experiments—it is a theoretical benefit based on Tensor Core specifications, not an empirical wall-clock speedup demonstrated in this paper. For the MXFP4 vs. NVFP4 comparison (Section 5), the paper measures how many additional tokens MXFP4 requires to match NVFP4's loss, establishing a training efficiency metric that accounts for convergence quality rather than per-step speed.

• Cross-validation / statistical protocol. The paper does not employ cross-validation or report statistical significance measures (confidence intervals, standard deviations, or error bars) for any results. The 10T-token 12B experiment represents a single training run per configuration due to computational cost—training 12B models multiple times with different seeds would be prohibitively expensive. The ablation studies (Figure 4) use a creative protocol to reduce cost: the 12B model is first trained with the full methodology to 3.43T tokens, then forked into separate runs where individual components are removed and training continues from that checkpoint to 4T tokens. This isolates the effect of each component without retraining from scratch, but it means the ablation results are conditional on the specific checkpoint state at 3.43T tokens—a component's importance earlier in training cannot be assessed. The 1.2B and 8B ablation experiments are full training runs from initialization, providing independent validation but at smaller scale. The paper does not discuss whether the findings are robust to different random seeds, weight initializations, or data orders.

Main Quantitative Results

12B Model: 10-Trillion-Token NVFP4 vs. FP8 Training

Validation loss trajectory (Figure 2). The NVFP4-trained model closely tracks the FP8 baseline throughout the entire 10T-token training run. During the stable phase of training (first 8T tokens, constant learning rate), the relative loss error of NVFP4 remains consistently below 1%. The paper states: "This indicates that the training dynamics of NVFP4 closely follows FP8, with only a small divergence appearing late in training." During the learning rate decay phase (final 2T tokens), the gap widens to slightly above 1.5%. The loss curves show the expected structure for the Warmup-Stable-Decay schedule: a flat stable phase, a change in slope at 8T tokens (learning rate decay onset), and a small jump at 9T tokens (dataset blend transition from Phase 2 to Phase 3). Critically, NVFP4 replicates all of these structural features, suggesting the quantization does not distort the fundamental optimization dynamics—the model responds to learning rate changes and data distribution shifts similarly to FP8.

Downstream task accuracy (Table 2, Figure 3). Despite the small loss gap, downstream task accuracies are "largely unaffected" (Section 3). The headline comparison is MMLU-Pro 5-shot: NVFP4 achieves 62.58%, nearly matching FP8's 62.62%—a difference of 0.04 percentage points. Across the full benchmark suite:

• General domain: NVFP4 outscores FP8 (69.82 vs. 68.99), a surprising slight advantage.
• Knowledge: MMLU 5-shot shows NVFP4 at 76.57 vs. FP8's 77.36 (gap of 0.79 points). AGIEval English CoT shows NVFP4 at 70.31 vs. FP8's 67.01 (NVFP4 outperforms by 3.3 points—a notable result that the paper does not discuss).
• Mathematics: GSM8k CoT shows NVFP4 at 92.27 vs. FP8's 89.08 (NVFP4 outperforms by 3.19 points), while MATH shows NVFP4 at 81.48 vs. FP8's 83.32 (gap of 1.84 points).
• Coding: This is the one domain where NVFP4 consistently underperforms. HumanEval+ shows 57.43 vs. 59.93 (gap of 2.5 points), and MBPP+ shows 55.91 vs. 59.11 (gap of 3.2 points). The paper attributes this to "noisy evaluations" and notes that "MBPP+ accuracy drops on the very final checkpoint evaluation and choosing another checkpoint could potentially lead to better accuracy."
• Multilingual: NVFP4 outperforms FP8 on both Global MMLU (74.94 vs. 74.00) and MGSM (85.53 vs. 81.87), with gaps of 0.94 and 3.66 points respectively.
• Commonsense Understanding: The aggregate score is 76.75 vs. 77.29 (gap of 0.54 points), with individual task scores showing mixed small differences: ARC Challenge is identical at 91.81, HellaSwag shows 83.09 vs. 83.83, OpenBookQA shows 47.40 vs. 47.60, PIQA shows 82.70 vs. 82.64, and Winogrande shows 78.77 vs. 80.58.

Key observation on task-level variance. The pattern of NVFP4 outperforming FP8 on some tasks (GSM8k, MGSM, AGIEval English) while underperforming on others (coding tasks, MMLU) suggests that the quantization does not uniformly degrade model quality—it introduces task-specific biases that could be positive or negative. The paper does not analyze this pattern or discuss whether it represents genuine capability differences or evaluation noise. The 3–4 point swings (GSM8k: +3.19, MGSM: +3.66, AGIEval: +3.30) are larger than would be expected from the "nearly matching" framing of the abstract, and the paper's treatment of these as unremarkable is a gap in the analysis.

Figure 3 detail. The time-series accuracy plots (MMLU Pro, MMLU, MATH 500, MBPP+) show NVFP4 and FP8 tracking closely throughout the 10T-token training run, with no systematic divergence over time. The curves are largely overlapping, with occasional crossings where NVFP4 temporarily outperforms FP8 before falling back or vice versa. This temporal tracking is important because it demonstrates that FP4 training does not cause progressive degradation—the model continues to improve at the same rate as FP8 throughout the full training horizon.

8B Model: NVFP4 vs. MXFP4 Training Efficiency (Section 5)

Training loss comparison (Figure 6a). On the 8B model trained for 1T tokens, NVFP4 achieves a relative loss error from BF16 of approximately −1.5% (consistent with the 12B result), while MXFP4 shows a relative error of approximately −2.5%. Both formats use the same training methodology (mixed precision with last 8 blocks in BF16, Hadamard transforms on Wgrad with d=16 for NVFP4 and d=32 for MXFP4 to match block size, 2D weight scaling, stochastic rounding on gradients). The 1 percentage point gap between the two FP4 formats is persistent throughout training, demonstrating that the format design differences (block size, scale factor precision, two-level scaling) translate directly to convergence quality even when all other methodology components are held constant.

Token efficiency equivalence (Figure 6b). The paper extends MXFP4 training beyond the 1T-token baseline to determine how many additional tokens are needed to match NVFP4's final validation loss. The result: MXFP4 requires approximately 1.36T tokens to match NVFP4's 1T-token loss—a 36% increase. The paper presents this as the central quantitative argument for NVFP4's advantage: "This translates to a considerable increase in training time for MXFP4, highlighting the benefits of NVFP4." The data points show NVFP4 at 1T tokens achieving a validation loss of approximately 1.263, while MXFP4 at 1T tokens reaches approximately 1.272, and MXFP4 at 1.36T tokens reaches approximately 1.263. The curve is roughly linear in the token range examined (1T to 1.5T), suggesting the 36% figure is measured rather than extrapolated.

Caveats on the comparison. This experiment uses a single model size (8B) and a single token horizon for the baseline (1T). The paper acknowledges this limitation: "Future studies should evaluate scaling laws for these formats on different parameter counts and token horizons." The 36% figure should be interpreted as a point estimate that may vary with model scale, training duration, and architecture. Additionally, the MXFP4 configuration uses d=32 Hadamard transforms (to match the MXFP4 block size) versus NVFP4's d=16—it is unclear whether this difference contributes to or mitigates the gap.

Ablation of Training Methodology Components on the 12B Model (Figure 4)

Experimental design. The 12B model is first trained with the full methodology (all four components) to 3.43T tokens. At that checkpoint, separate training runs are forked where one component is removed, and training continues for an additional 570B tokens (to 4T tokens total). The relative loss difference from FP8 is plotted for each ablation run and compared against the full-methodology baseline. The key aspects:

• Full methodology (NVFP4 baseline): Relative loss error remains approximately −0.5% to −1%, stable throughout the ablation window.
• Remove stochastic rounding: The relative error immediately worsens, reaching approximately −1.5% to −2% by 4T tokens. The degradation is gradual, consistent with bias accumulating over training steps.
• Remove Random Hadamard transforms: The error jumps to approximately −1.5% and shows volatility, suggesting instability from outlier-dominated quantization rather than gradual drift.
• Remove 2D weight scaling: The error immediately increases to approximately −1.5% and continues to trend worse, indicating sustained degradation from chain rule violations.
• Keep only last 4 blocks in BF16 (instead of last 8): The relative error remains approximately −0.5% to −0.75%, essentially overlapping with the full-methodology baseline. This is a positive finding: it suggests the 12B model could have been trained with fewer high-precision layers, increasing the FP4 fraction from 84% to roughly 92% of linear layers.

Interpretation. Every component removal except the reduction in high-precision layers causes measurable degradation. The paper states: "eliminating any of them leads to worse convergence." The finding that stochastic rounding, Hadamard transforms, and 2D scaling are each individually necessary for the 12B model, while the number of high-precision layers can potentially be reduced, is the core evidence for the "necessary system" claim—three of the four components are strictly required, and the fourth can be tuned for efficiency.

Important nuance on the ablation protocol. Because the ablations start from a checkpoint trained with the full methodology, they measure the effect of removing a component from a stable configuration, not the effect of never having the component. This is a weaker test than training from scratch without the component, because the model has already converged to a parameter state that is compatible with the full methodology. A component that is critical for early-training stability might appear less important in this protocol because the difficult early phase has already passed. This is acknowledged implicitly by the paper's statement that techniques are "essential for ensuring convergence and stability of the 12B model training over the 10T-token horizon"—the 3.43T-token starting point is already beyond the point where divergence would have occurred without the components.

Switching Precision Late in Training (Appendix D, Figure 7)

Experimental setup. Starting from the 12B model trained in full NVFP4 methodology, precision is switched at 8.2T tokens (just after learning rate decay begins at 8T) and training continues to 10T tokens. Four configurations are tested: (1) switch both forward and backward passes to BF16, (2) switch only the forward pass to BF16, (3) switch only the backward pass to BF16, (4) a late switch at 10T tokens (very end of training) for both forward and backward. Additionally, a run with 1D weight scaling is noted when switching the backward pass, since "doing so is marginally better than 2D weight scaling in such a setup."

Results.

• Full NVFP4 (no switch): Relative loss error is approximately −1.5% at 10T tokens.
• Switch both forward and backward at 8.2T: Error closes to near zero—effectively matching the FP8 baseline.
• Switch forward only at 8.2T: Error reduces to approximately −0.5%, recovering about two-thirds of the gap.
• Switch backward only at 8.2T: "No such improvement" compared to no switch—the error remains around −1.5%.
• Late switch at 10T: Error reduces but does not fully close, "presumably because the learning rate is too low for the weight updates to incorporate the higher-precision signal effectively."

Interpretation. The forward pass is the primary source of the FP4 loss gap. The paper states: "most of FP4 training's loss gap arises from quantizing tensors in the forward pass." Switching only the forward pass recovers most of the gap with minimal overhead—since forward-pass GEMMs are roughly one-third of total GEMM operations, this means approximately 6% of total training compute (one-third of the final 18%) runs in higher precision. The paper recommends this as a practical strategy for deployments where closing the loss gap matters.

Cumulative Effect of Stacking Techniques (Appendix E.1, Figure 8)

Experimental setup. On the 1.2B model trained for 1T tokens, the paper starts from a base configuration (NVFP4 on all layers, standard 1D scaling, nearest rounding, no Hadamard, no stochastic rounding) and adds techniques incrementally: first keeping the last 4 blocks in BF16, then adding 2D weight scaling, then adding Hadamard transforms on Wgrad, then adding stochastic rounding on gradients. Each addition further reduces the loss gap from BF16.

Results (from the figure).

• Base method (no techniques): Training diverges—not shown on the plot because it fails early.
• Last 4 in BF16 only: Relative loss error is approximately −1%, already close to BF16.
• + 2D weight scaling: Error reduces slightly.
• + Hadamard transforms: Error reduces further.
• + Stochastic rounding: Error reduces to its minimum, approximately −0.5%.

Key finding. "The relative benefit of each technique depends on the order in which the components are added." The marginal improvement from adding a technique is smaller when other techniques are already in place (e.g., stochastic rounding provides less benefit after Hadamard and 2D scaling are active than it would if added first). This interaction effect supports the "system of overlapping protections" interpretation: techniques address related failure modes, so having multiple protections in place creates redundancy that makes each individual component appear less critical.

Ablation Studies and Robustness Checks

Layer sensitivity to FP4 quantization (Appendix E.2, Figure 9). On the 1.2B model, keeping only the last 4 blocks in BF16 enables convergence (validation loss approaches BF16 baseline), while keeping the first 4 blocks without the last ones causes divergence. Keeping only the last 1 or last 2 blocks is insufficient—divergence still occurs. Keeping the first 2 and last 2 blocks converges, confirming that sensitivity is concentrated at the network's end. The paper speculates that weight gradients in final layers have the largest quantization errors due to accumulated backpropagation error from all preceding layers.

Stochastic rounding on different tensor types (Appendix E.3, Figure 10). On the 1.2B model: applying stochastic rounding to gradients (both Dgrad and Wgrad) achieves stable convergence with loss approaching BF16; applying it to activations or weights causes divergence; applying it to all backward-pass tensors (not just gradients) shows "little improvement over stochastic rounding of gradients only," confirming that the forward pass is the source of divergence when stochastically rounding non-gradient tensors. For the 12B model, stochastic rounding must be applied to gradients entering both Dgrad and Wgrad to achieve proper convergence, which is a stricter requirement than for the 1.2B model.

Random Hadamard transforms on different GEMMs (Appendix E.4.1, Figure 11). On the 1.2B model: applying Random Hadamard transforms to Wgrad inputs improves validation loss compared to no transforms; applying them to Fprop or Dgrad inputs degrades loss. The paper hypothesizes that the transform introduces additional quantization error, and the trade-off between outlier mitigation and added noise is favorable only for Wgrad. This finding is scale-dependent: "conclusions from smaller scales may not always hold for larger models" (Appendix E.4.2), and the paper speculates that Fprop and Dgrad might benefit from transforms at even larger scales.

Hadamard matrix size (Appendix E.4.2, Figure 12). On the 1.2B model, all matrix sizes from 2×2 to 128×128 show "virtually no difference" in validation loss. On the 12B model (forked from the full-methodology checkpoint at 3.4T tokens and switched to different sizes): 4×4 matrices cause an increase in loss, 16×16 maintains stable convergence, and 128×128 provides a "minor benefit" over 16×16. The paper chooses 16×16 as a balance, noting that "conclusions from smaller scales may not always hold."

Randomization in Hadamard transforms (Appendix E.4.3, Figure 13). On the 1.2B model, there is "no noticeable difference" between using a fixed random sign vector, no random sign vector, or a new random seed for every transform. On the 12B model (forked at 3.4T tokens): the absence of random sign vectors causes measurable degradation, while a single fixed seed vs. per-instance randomization shows no difference. The paper concludes that randomization is necessary at scale but a single fixed seed suffices.

Consistent representations between forward and backward passes (Appendix E.5, Figure 14). On the 1.2B model, testing different scaling configurations for weights: 1×16 scales along different dimensions in forward and backward (inconsistent) produces the worst loss (relative error approximately −3.5% to −4%); 1×16 scales along the same dimension (consistent, but impractical) achieves relative error approximately −1.5%; 16×16 block scales (practical, consistent) achieves intermediate performance (approximately −2.5% to −3%). For activations, the inconsistent case shows only mild degradation primarily during learning rate decay, confirming that activations are less sensitive to chain rule violations. The paper hypothesizes that "weights are more impacted than activations because errors induced from inconsistent weights materialize in the activation gradients, which flow through the model layers during backpropagation."

Switching precision for forward vs. backward pass (Appendix D, Figure 7). As discussed in the main results, switching only the forward pass to BF16 recovers most of the loss gap, while switching only the backward pass provides no improvement. This is presented in contrast to Chmiel et al. (2025).

Critical Assessment

Does the paper demonstrate that NVFP4 training achieves accuracy comparable to FP8 at production scale?

Yes, for the specific model and training configuration tested. The 12B model trained on 10T tokens shows MMLU-Pro accuracy of 62.58% vs. 62.62% for FP8 (Table 2), and validation loss tracks within ~1% relative error during stable training (Figure 2). These are strong results at a scale that is genuinely production-relevant—12B parameters and 10T tokens is a meaningful fraction of frontier-model training budgets, and the experiment runs to completion rather than extrapolating from early checkpoints.

However, the "comparable" claim has qualifications. First, the loss gap widens to >1.5% during learning rate decay, and some downstream tasks show non-trivial differences: coding tasks underperform by 2.5–3.2 percentage points (HumanEval+, MBPP+), while some math and multilingual tasks show NVFP4 outperforming FP8 by 3–4 percentage points (GSM8k: +3.19, MGSM: +3.66). The abstract frames the result as "MMLU-pro accuracy of 62.58%, nearly matching the 62.62% accuracy," which is true for that specific metric, but the full benchmark picture is more mixed. A reader focused on coding tasks would not conclude that NVFP4 "matches" FP8. The paper's attribution of coding-task differences to "noisy evaluations" and checkpoint selection is plausible but not demonstrated—no alternative checkpoint evaluation is shown.

Second, the FP8 baseline itself uses mixed precision (first block and last two blocks in BF16, Section A.1), so the comparison is not "pure FP8" vs. "FP4 with techniques"—it is "mostly-FP8 with some BF16" vs. "mostly-FP4 with some BF16 and several stabilizing techniques." A fairer comparison might give FP8 the same training methodology components (2D scaling, Hadamard transforms, stochastic rounding) and see whether the gap persists, but this experiment is not run. The paper implicitly assumes that FP8 does not need these techniques because it already converges well, but this assumption is untested.

Third, only one model architecture is tested at the primary scale. The 12B model uses a hybrid Mamba-Transformer architecture. Whether pure Transformer, mixture-of-experts, or dense architectures with different depth-to-width ratios would show the same stability is unknown. The 1.2B ablation model is a standard Transformer, suggesting some generalizability, but the scale is an order of magnitude smaller.

Does the ablation evidence support the claim that all four methodology components are necessary?

Partially, with an important caveat about the ablation protocol. Figure 4 shows that removing stochastic rounding, Hadamard transforms, or 2D weight scaling from a 3.43T-token checkpoint degrades convergence for the subsequent 570B tokens. This demonstrates that these components remain important through mid-to-late training. What it does NOT demonstrate is that these components are necessary for early training stability—the checkpoint already passed through the initial convergence phase with all components active. A component that is critical for the first few hundred billion tokens but unnecessary afterward would appear non-critical in this protocol. The paper's statement that "naively quantizing every linear layer to FP4 causes immediate training divergence" (Section 4.1) only applies to the mixed-precision component; it is not shown for the other three components individually.

The 1.2B model ablations partially fill this gap by training from scratch with different component combinations (Figures 8–14), and they confirm that removing components degrades convergence. However, the 1.2B model is much more forgiving—it converges with only high-precision layers and no other techniques (Figure 8, "Last 4 in BF16" line), while the 12B model allegedly requires all four. The extrapolation from 1.2B to 12B relies on the assumption that component importance increases monotonically with scale, which is plausible but not proven.

The finding on high-precision layer count deserves more scrutiny. Figure 4 shows that reducing from last 8 blocks to last 4 blocks in BF16 causes no degradation, and the paper treats this as evidence that fewer high-precision layers suffice. But this ablation starts at 3.43T tokens—it does not test whether training would have been stable from initialization with only 4 blocks in BF16. The 1.2B model ablations (Figure 9) do test this from scratch and show that last 4 blocks suffice for that model, providing some supporting evidence.

Does the NVFP4 vs. MXFP4 comparison (Section 5) establish a clear advantage for NVFP4?

Yes, for the 8B/1T-token configuration tested. Figure 6 shows NVFP4 achieving consistently better loss than MXFP4 under identical training methodology, and MXFP4 requiring 36% more tokens to match. The experiment is clean—same model, same data, same techniques, only the format differs.

However, the generalizability of the 36% figure is unknown. The paper explicitly states that "future studies should evaluate scaling laws for these formats on different parameter counts and token horizons" (Section 5). The experiment uses a single model size (8B), a single token budget (1T for the baseline), and a single architecture (hybrid Mamba-Transformer). Whether the gap widens, narrows, or stays constant at 70B or 175B scale is not addressed. Additionally, the MXFP4 configuration uses d=32 Hadamard transforms (matching the MXFP4 block size) while NVFP4 uses d=16—this is a sensible choice given the block structure, but it means the comparison is not purely "format A vs. format B"; it includes a secondary difference in the transform size. The paper does not report results for MXFP4 with d=16 or NVFP4 with d=32 to isolate the format effect from the transform-size effect.

Does the paper demonstrate that FP4 training delivers the claimed 2–3× throughput improvement?

No—and the paper does not claim to. This is explicitly scoped out: "this report is primarily concerned with the underlying algorithms and methodology rather than with runtime efficiency or system-level optimizations" (Section 1). The throughput numbers in Table 1 (4× on GB200, 6× on GB300 vs. BF16) are hardware specifications, not measured training throughput. Actual wall-clock speedup would depend on the fraction of training time spent in FP4 GEMMs vs. other operations (attention, normalization, communication, data loading), the overhead of the additional techniques (Hadamard transforms, 2D scaling quantization, stochastic rounding), and the cost of maintaining ~15% of layers in BF16. This is a significant gap between the paper's claimed impact ("delivering a 2–3× arithmetic throughput boost") and what is empirically validated. The paper establishes numerical feasibility—the necessary precondition for realizing hardware throughput—but does not measure realized throughput. A practitioner reading the abstract could reasonably assume the paper demonstrates end-to-end training speedup; it does not.

What experiments are missing that would strengthen the paper?

1. Comparison against prior FP4 training methods. The paper compares NVFP4 against FP8 and BF16 but not against MXFP4 trained with other published methodologies (Tseng et al., 2025b; Chmiel et al., 2025; Castro et al., 2025). This makes it impossible to determine whether NVFP4's advantage over MXFP4 (Section 5) comes from the format or from the training methodology—would MXFP4 with the same four-component methodology close the gap? Would the prior methods work at 12B scale if combined with NVFP4 format? Without these baselines, the paper cannot claim that NVFP4 is necessary for the training methodology; it can only claim that the methodology is sufficient for NVFP4.

2. Training-from-scratch ablations at 12B scale. All 12B-scale component removals start from a 3.43T-token checkpoint trained with the full methodology. Ablations from initialization would provide stronger evidence for the necessity of each component, though the computational cost is acknowledged as prohibitive.

3. Multiple random seeds or data orders. The primary 12B experiment is a single training run. Without replication, it is impossible to know whether the 0.04-point MMLU-Pro gap (62.62 vs. 62.58) is within run-to-run variance or represents a genuine precision effect. The 3–4 point swings on individual tasks (GSM8k, MGSM) could be noise that would average out over multiple runs.

4. Scaling law experiments. The paper reports one model size at the primary scale (12B) and one at the comparison scale (8B). A sweep of model sizes (e.g., 1B, 3B, 8B, 12B, 30B) would reveal whether the FP4-FP8 loss gap scales with model size—does it grow, shrink, or stay constant? This is directly relevant to the paper's implicit claim that the methodology will transfer to "future frontier models."

5. Measurement of actual wall-clock training throughput. Even a preliminary measurement on a single training step showing the fraction of time in FP4 GEMMs vs. overhead would calibrate the throughput claims and identify optimization bottlenecks for future work.

6. Analysis of the forward-pass bottleneck. The finding that forward-pass quantization causes most of the loss gap (Appendix D) is empirically demonstrated but not explained. Experiments measuring gradient signal-to-noise ratio or weight update alignment between FP4 and FP8 in forward vs. backward passes would strengthen this claim and provide mechanistic insight.

Summary

The paper's central empirical claim—that a specific combination of four training techniques enables stable 12B-scale FP4 pretraining over 10T tokens with accuracy comparable to FP8—is well-supported by the experiments shown. The validation loss tracking, downstream benchmark results, and ablation studies collectively make a convincing case that FP4 training at this scale is feasible with the described methodology. However, the paper's broader framing—that NVFP4 "represents a major step forward in narrow-precision LLM training algorithms" and "lays the foundation for faster and more efficient training of future frontier models"—relies on extrapolation from a single-model, single-scale experiment. The throughput claims are based on hardware specifications, not measured training speedups. The ablation evidence for component necessity is strong for mid-to-late training but weaker for early training stability. And the absence of comparisons against prior FP4 training methods leaves open the question of whether NVFP4 specifically, or the training methodology, is the primary driver of the results. These limitations do not undermine the paper's core contribution—demonstrating that FP4 pretraining at billion-parameter scale is possible—but they bound the strength of claims that can be drawn about necessity, optimality, or generalizability.

6. Limitations and Trade-offs

Throughput Claims Are Based on Hardware Specifications, Not Measured Training Speedups

The paper's central value proposition is that NVFP4 training delivers "a two- to three-fold boost in arithmetic performance and reducing memory usage by half compared to FP8" (Section 1). Table 1 quantifies this: 4× throughput over BF16 on GB200, 6× on GB300, and implied ~2× and ~3× over FP8 respectively. The abstract frames these as efficiency gains that make FP4 "the next logical step" in narrow-precision training.

The constraint: The paper explicitly disclaims empirical throughput measurement. Section 1 states: "this report is primarily concerned with the underlying algorithms and methodology rather than with runtime efficiency or system-level optimizations." Not a single wall-clock timing, throughput measurement (tokens/second), or memory utilization measurement appears in the paper. The 2–3× figures are specifications of Tensor Core math throughput—the theoretical maximum rate at which the hardware can perform FP4 multiply-accumulate operations—not measurements of end-to-end training throughput.

The consequence: The realized training speedup from FP4 will be substantially lower than the Tensor Core throughput numbers, potentially by a large and unknown factor. Several sources of overhead are present in the methodology but never quantified:

• Mixed-precision overhead: 16% of linear layers remain in BF16 for the 12B model (Section 4.1, Appendix A.1). These layers compute GEMMs at BF16 throughput, not FP4. Depending on where these layers sit in the network and how much computation they represent, the effective throughput is a weighted average of FP4 and BF16 rates.

• Hadamard transform overhead: The Random Hadamard transforms applied to Wgrad inputs require additional matrix multiplications. The paper notes the transform involves $mkd$ multiply-adds per tensor of size $m \times k$ with block size $d$ (Appendix C), and states this "can be implemented as batched matrix multiplications... and can be fused with other layers to reduce round-trips to device memory." But no measurement of this overhead is provided. At the chosen $d=16$, the transform cost is a small fraction of the GEMM itself—the paper claims this explicitly—but the absolute wall-clock cost on Blackwell hardware with FP4 Tensor Core acceleration for the GEMM but not for the transform is unknown.

• Two-level scaling overhead: Computing the global $amax_x$ for NVFP4's per-tensor FP32 scale requires "an extra pass through device memory: once to compute the global amax, and once to scale prior to conversion to FP4" (Appendix B.1). This memory pass is additional work that full-precision training does not incur. The paper notes that "the global scale could potentially span a smaller granularity (e.g., a row or block of elements) to avoid additional round-trips through device memory," flagging this as an optimization opportunity rather than a solved problem.

• Stochastic rounding hardware support: While Blackwell GPUs have native stochastic rounding instructions for FP4 conversion (Section 4.4), the latency of these instructions relative to nearest-rounding is not compared. If stochastic rounding is slower per operation than nearest rounding, applying it to gradients entering both Dgrad and Wgrad (as required for the 12B model, Appendix E.3) incurs a throughput penalty on all backward-pass GEMM inputs.

• Attention, normalization, and other non-GEMM operations: These remain in BF16 or FP32 (Section 4.1) and see no throughput improvement from FP4. The fraction of total training time spent in these operations depends on model architecture, sequence length, and hardware characteristics. For the hybrid Mamba-Transformer architecture with sequence length 8192 and batch size 736 (Appendix A.1), the non-GEMM fraction is not characterized.

Without quantifying any of these overheads, the headline throughput claims represent an upper bound that a practitioner cannot use to estimate actual training time reduction. The gap between theoretical Tensor Core throughput and realized end-to-end training throughput is exactly the engineering challenge that the paper defers to future work.

Evidence in the paper: The limitation is acknowledged through explicit scoping—Section 1 states the focus is on algorithms and methodology, not runtime efficiency. The Tensor Core specifications in Table 1 are presented as design motivation. No wall-clock measurements, no roofline analyses, and no profiling breakdowns appear anywhere. Appendix B.1 identifies the global scale memory pass as an overhead and suggests optimization. Appendix C notes that Hadamard transforms are "limited by memory traffic from reading the input tensor when using Tensor Cores" and can be fused with other layers.

Mitigation status: Not mitigated in this paper. The authors defer system-level optimization to future work. A practitioner evaluating whether to adopt NVFP4 training cannot determine expected wall-clock speedup from the paper alone and must replicate the methodology on their specific hardware and model configuration to measure realized throughput. This is a significant practical gap—the paper establishes numerical feasibility but not training-time feasibility—and it is the most important limitation for decision-making about deployment.

---

The 12B/10T-Token Experiment Is a Single Training Run with No Replication

The paper's primary validation—the 12B model trained on 10T tokens with NVFP4 and compared against an FP8 baseline—consists of exactly one training run per configuration. The cost of training 12B-parameter models on 10T tokens makes replication prohibitive, and the paper is transparent about the scale of the computational investment. However, this single-run design limits the strength of several key claims.

The constraint: No confidence intervals, standard deviations, standard errors, or any other statistical dispersion measure appears anywhere in the paper for the primary experiment. The 12B model is trained once with NVFP4 and once with FP8. The ablation studies that fork from the 3.43T-token checkpoint create multiple runs, but these share the identical pretraining history up to the fork point, so they are not independent replications of the full training process.

The consequence: Without replication, it is impossible to distinguish genuine precision-format effects from run-to-run variation caused by different random seeds, weight initializations, data orders, or hardware nondeterminism. Several observations in the results are consistent with noise rather than systematic effects:

• The MMLU-Pro gap: NVFP4 achieves 62.58% vs. FP8's 62.62% (Table 2)—a difference of 0.04 percentage points. This difference is reported as "nearly matching," which is a reasonable characterization, but the reader does not know whether the gap would be larger, smaller, or reversed in a second training run.

• Task-level swings: NVFP4 outperforms FP8 by 3.19 points on GSM8k (92.27 vs. 89.08), 3.66 points on MGSM (85.53 vs. 81.87), and 3.30 points on AGIEval English CoT (70.31 vs. 67.01) while underperforming by 2.50 points on HumanEval+ (57.43 vs. 59.93) and 3.20 points on MBPP+ (55.91 vs. 59.11). These are non-trivial differences—3+ points on standard benchmarks are often considered meaningful—but the paper attributes the coding underperformance to "noisy evaluations" (Section 3) without attributing the positive swings to anything. A practitioner cannot determine whether NVFP4 genuinely improves math performance or whether the observed swings are within the expected variance of a single training run.

• The validation loss gap: The relative loss error is described as "consistently below 1%" during stable training and "slightly above 1.5%" during learning rate decay (Section 3). Without variance estimates, the reader cannot assess whether the 1% vs. 1.5% difference is a genuine degradation or statistically indistinguishable given run-to-run noise.

The ablation results in Figure 4 face a related but distinct problem: because they fork from a single checkpoint at 3.43T tokens, differences between ablation runs reflect the effect of component removal conditional on the specific parameter state at that checkpoint. A component that caused divergence earlier in training (before 3.43T tokens) would not be detected by this protocol. The paper's claim that each component is "essential for ensuring convergence and stability of the 12B model training over the 10T-token horizon" (Section 4) is therefore based on inference from the 1.2B-scale training-from-scratch ablations (which do show degradation) combined with the 12B-scale mid-training removal ablations (which show degradation in the later phase), rather than on direct evidence that training from scratch without each component would fail at 12B scale.

Evidence in the paper: The paper reports no variance measures—no error bars, confidence intervals, or standard deviations in any figure or table. The training loss curves in Figure 2 are single lines per configuration. The downstream accuracy curves in Figure 3 are single lines. The ablation Figure 4 shows single lines per ablation condition. Section 5's MXFP4 vs. NVFP4 comparison (Figure 6) also uses single runs. The paper does not discuss the single-run limitation explicitly, though the computational infeasibility of replication at this scale is a reasonable implicit justification.

Mitigation status: Not addressed. The paper does not suggest that the 12B results be interpreted cautiously due to single-run variance, does not provide any estimate of expected run-to-run variation (e.g., from prior experience training Nemotron-H models at this scale), and does not discuss whether the 3–4 point task-level swings are within typical training variance for this model family. Future work could address this by reporting results across multiple seeds for a smaller model (e.g., the 8B or 1.2B) to establish the expected variance and determine whether the observed FP4/FP8 differences exceed it, but this is not done.

---

No Comparison Against Prior 4-Bit Training Methods

The paper evaluates NVFP4 training against FP8 and BF16 baselines and compares NVFP4 directly against MXFP4 in Section 5. It does not compare against any published FP4 training methodology from prior work—not the MXFP4 training approach of Tseng et al. (2025b), not the "FP4 All the Way" method of Chmiel et al. (2025), not the Quartet method of Castro et al. (2025), not the FP4 optimization work of Wang et al. (2025) or Zhou et al. (2025). These works are cited in the introduction and reference list but never used as experimental baselines.

The constraint: The paper's experimental design answers the question: "Can NVFP4 plus our four-component methodology match FP8 training?" It does not answer: "Is NVFP4 plus our methodology better than the best existing method for FP4 training?" These are different questions with different implications. If prior methods could achieve comparable results with MXFP4 or with a different methodology, then NVFP4's format advantages (Section 2) would be less consequential, and the methodology innovations would be the primary contribution. Conversely, if prior methods fail at 12B/10T-token scale while the paper's methodology succeeds with NVFP4, the format and methodology are both essential, and the contribution is stronger.

The consequence: The paper cannot isolate how much of its success comes from NVFP4's improved numerical properties (smaller block size, E4M3 block scales, two-level scaling) versus the training methodology (mixed precision, Hadamard transforms, 2D scaling, stochastic rounding). The MXFP4 vs. NVFP4 comparison in Section 5 partially addresses this—showing that MXFP4 with identical methodology requires 36% more tokens—but that experiment uses the paper's methodology for both formats. It does not test whether MXFP4 with Tseng et al. (2025b)'s methodology (which may include different techniques optimized for MXFP4's specific failure modes) would close or eliminate the gap. Format and methodology are confounded: the paper demonstrates that NVFP4 + paper's methodology works well, and that MXFP4 + paper's methodology works less well, but it does not demonstrate that NVFP4 is necessary for good FP4 training or that the paper's methodology is optimal for NVFP4.

Additionally, the absence of prior-method baselines means the paper cannot substantiate its implicit claim of novelty. The four techniques—mixed precision, Hadamard transforms, 2D scaling, stochastic rounding—are individually known from prior work. The paper argues that their combination at scale is novel, but without showing that prior combinations fail at this scale, the novelty claim rests on the ablation evidence showing that removing any one component degrades performance. While these ablations demonstrate that the components are individually important for the paper's specific configuration, they do not demonstrate that no other combination of existing techniques could achieve similar results.

Evidence in the paper: The related work discussion (Section 1, references) acknowledges the existence of prior FP4 training methods. Section 5 compares NVFP4 against MXFP4 under identical methodology. No experiment uses a published FP4 training recipe as a baseline. The paper does not discuss this gap or explain why such comparisons were not conducted (possible reasons include implementation complexity, format incompatibility, or the belief that the paper's methodology subsumes prior approaches).

Mitigation status: Not addressed. The paper neither compares against prior methods nor explains their absence. A practitioner familiar with the FP4 training literature cannot determine from this paper whether to adopt NVFP4 + this methodology versus continuing with MXFP4 + an existing methodology. The paper's contribution is most accurately described as "a specific combination of format and training techniques that works at scale," not "the best approach to FP4 training" or "the only approach that works at scale."

---

Generalizability Is Limited to a Single Architecture and Data Distribution

All primary experiments—the 12B/10T-token validation run, the 8B NVFP4 vs. MXFP4 comparison, and the majority of ablations—use the Nemotron-H hybrid Mamba-Transformer architecture (NVIDIA, 2025b,a). This architecture interleaves Mamba-2 blocks, self-attention blocks, and feed-forward blocks in a specific pattern (62 total blocks for the 12B model: 6 Self-Attention, 28 FFNs, 28 Mamba-2). The 1.2B ablation model uses a standard Transformer architecture (20 blocks, pure self-attention + FFN), providing some architectural diversity but at a scale two orders of magnitude smaller in parameters.

The constraint: The numerical sensitivity patterns that the paper maps—final layers being disproportionately sensitive to quantization, Hadamard transforms helping Wgrad but not Fprop/Dgrad, activations being less sensitive to chain rule violations than weights—were all discovered empirically on these specific architectures. There is no theoretical reason these patterns should be architecture-universal, and the paper does not claim they are. Different architectural choices could shift where numerical sensitivity concentrates:

• Pure Transformer models: Without Mamba-2 blocks (which use structured state-space computations with different gradient flow patterns), the sensitivity of final layers might differ. The 1.2B ablation model is a pure Transformer, and it does show qualitatively similar patterns (last 4 blocks are the critical ones, Figure 9), supporting some generalizability. But the 1.2B model is also much more forgiving overall—it converges with only high-precision layers and no other techniques—so the severity and necessity of each component may not transfer to larger pure Transformers.

• Mixture-of-Experts (MoE) architectures: The paper mentions MoE as future work (Section 6). In MoE models, the routing mechanism creates sparsity patterns where different experts process different tokens, potentially creating new outlier distributions in the expert weight gradients. The interaction between FP4 quantization and expert load balancing is unexplored.

• Different depth-to-width ratios: The 12B model has 62 blocks with model dimension 5120. A deeper, narrower model (more blocks, smaller hidden dimension) would have different gradient flow patterns and potentially different sensitivity to quantization in early vs. late layers. A shallower, wider model would have larger individual GEMMs where the FP4 block structure (16 elements per block) might behave differently relative to the tensor dimensions.

• Different training data distributions: The paper uses a specific curated data blend (Section A.1) designed for the Nemotron-H family. Data with different outlier characteristics—more code (which often has structured patterns that could interact with Hadamard transforms), more multilingual text (different token frequency distributions), or more synthetic data (potentially different activation patterns)—might produce different quantization error profiles.

The consequence: A practitioner training a different architecture—a pure dense Transformer, an MoE model, or a model with a very different depth-to-width ratio—cannot assume that the specific configuration recommended in Section 4 will transfer. The finding that only the last 4–8 blocks need high precision might not hold; different architectures might require high precision in different locations. The finding that Hadamard transforms are beneficial only for Wgrad might not hold; other architectures might benefit from transforms on Fprop or Dgrad, or might not need them at all. The finding that 2D weight scaling is necessary might be architecture-specific. The paper provides no diagnostic methodology for a practitioner to determine whether a particular architecture has different sensitivity patterns—it reports empirical results for one architecture family and suggests that "quantization error metrics could potentially serve as a mechanism to determine which linear layers should remain in higher precision" (Appendix E.2) without developing such a mechanism.

The single-dataset concern is less severe—the 10T-token corpus is diverse (web crawl, Wikipedia, math, code, academic, multilingual, synthetic SFT-style data), and the phased blending approach (Feng et al., 2024) is a realistic training data strategy. However, the downstream evaluation is concentrated on standard English-language benchmarks; the multilingual results (Global MMLU, MGSM) show some of the largest FP4/FP8 differences (NVFP4 outperforms FP8 by 3.66 points on MGSM), which could indicate either genuine improvement or dataset-specific sensitivity that would not replicate on a different training corpus.

Evidence in the paper: All primary experiments use the Nemotron-H hybrid architecture. The 1.2B ablation model uses a standard Transformer, and the patterns are qualitatively similar for that architecture (Appendix E). The paper acknowledges that future work should evaluate "additional architectures such as mixture-of-experts" (Section 6). No other architecture families are tested.

Mitigation status: Partially addressed through the 1.2B Transformer ablation model, which shows qualitatively consistent sensitivity patterns (last layers are the critical ones, stochastic rounding benefits gradients, Hadamard transforms on Wgrad help, 2D scaling matters). However, the scale gap (1.2B vs. 12B) and the paper's own observation that "conclusions from smaller scales may not always hold for larger models" (Appendix E.4.2) limit the strength of this mitigation. The paper does not provide theoretical reasoning for why the sensitivity patterns should be architecture-invariant, nor does it provide a methodology for practitioners to validate the patterns on their own architectures without replicating the full ablation suite.

---

The Methodology Was Validated Only on a Single Dataset (MATH Benchmark Equivalent, Downstream Evaluation Suite), and the Training Data Mixture Is Proprietary

The paper evaluates the 12B NVFP4-trained model on a comprehensive downstream benchmark suite (Table 2, Figure 3) spanning knowledge, math, code, multilingual, and commonsense reasoning tasks. However, the pretraining data itself—the 10T-token corpus—is described only as "a corpus of high-quality curated and synthetic dataset comprising of 10 trillion tokens based on NVIDIA (2025b), with data mixtures consisting of general web crawl data, wikipedia, math, code, academic data, crawl++, multilingual, and synthetic SFT-style data" (Appendix A.1).

The constraint: The exact composition, curation methodology, and quality filtering of the pretraining data are not disclosed. Paper references NVIDIA (2025b) and Feng et al. (2024) for details, but these are separate publications that a reader must consult to fully understand the training data. The phased data-blending approach (Phase 1: 70% diversity-focused, Phase 2: 20% high-quality, Phase 3: 10% final high-quality) is standard practice, but the specific datasets, their proportions, and their quality characteristics are not documented in this paper. This is common in industry technical reports but limits reproducibility.

The consequence: The interaction between FP4 quantization and training data quality is an unknown variable. If the training data is unusually clean and well-curated—as suggested by the description "high-quality curated and synthetic dataset"—then the activation and gradient distributions during training may be more well-behaved than they would be on noisier, less-filtered web-scale data. Outlier activations and gradients are known to arise from specific data patterns (rare tokens, malformatted text, extreme sequence lengths), and a curated dataset may naturally suppress these patterns. If so, the paper's demonstrated stability may depend partially on data quality, and practitioners training on less curated data might encounter different outlier distributions that the methodology does not handle as effectively.

Conversely, if the training data includes synthetic SFT-style data (as stated), the model may be exposed to more structured, pattern-regular text than would occur in purely natural data. Structured data formats (code, JSON, tables) can produce systematic activation patterns that interact with quantization differently than natural language. The paper's finding that coding tasks show the largest NVFP4 underperformance (HumanEval+: −2.5 points, MBPP+: −3.2 points) could indicate that FP4 quantization handles code-related computations less well, which would be especially relevant for practitioners training code-focused models.

Evidence in the paper: The data description is in Appendix A.1. The paper does not analyze the relationship between data composition and FP4 training stability. The downstream evaluation results are reported without discussion of how data-specific factors might influence them (beyond the coding-task speculation in Section 3).

Mitigation status: Not addressed. The data composition is proprietary, standard for NVIDIA technical reports, and the paper does not claim that the methodology is data-independent. A practitioner cannot determine from this paper whether their training data would produce similar FP4 stability. Future work could address this by training smaller models on publicly available datasets (e.g., Pile, C4, RedPajama) and comparing FP4 stability across data sources, but this is not done.

---

The Difficulty Estimation Cost (2048 Samples per Question) Is Not Amortized in Any Efficiency Calculation

Note: This limitation from the reference example does not apply to this paper. The NVFP4 paper does not use difficulty estimation—it is a training methodology paper, not an inference-time compute allocation paper. The ablation protocol uses a fixed training budget of 3.43T tokens before forking, which is a different kind of cost that Section 5's assessment already addresses under the single-run limitation. However, there is a parallel concern:

The Computational Cost of the Ablation Protocol Itself Is Extraordinary and the Paper Does Not Discuss the Trade-off

The paper's strongest evidence for the necessity of each methodology component comes from the 12B-scale ablation study (Figure 4): training the model with full methodology to 3.43T tokens, then forking separate runs where individual components are removed and training continues to 4T tokens. This requires training approximately six 12B-parameter configurations (full methodology + five ablations) for 570B tokens each, plus the shared 3.43T-token pretraining phase, totaling roughly 7T tokens of additional 12B-scale training beyond the primary 10T-token experiment. At the scale of the paper's experiments, this represents a substantial additional computational investment—likely millions of GPU-hours.

The constraint: The paper does not report the computational cost of the ablation experiments, does not discuss the carbon footprint, and does not frame the ablation protocol as a trade-off between experimental rigor and resource consumption. This is not a scientific limitation—the ablation design is sound—but a practical consideration for reproducibility. A research group attempting to validate or extend this methodology on a different architecture would need to invest comparable computational resources to determine whether all components are necessary for their specific setting. The paper provides no guidance on how to conduct cheaper ablations—for example, whether shorter training runs (e.g., 1T tokens instead of 3.43T) would reveal the same component importance, or whether the 1.2B-scale ablations in Appendix E are sufficient proxies for larger-scale sensitivity.

The consequence: The paper's methodology validation is tied to a specific computational budget that may be beyond the reach of many research groups. The 1.2B ablations provide some transferable evidence, but the paper repeatedly emphasizes that "conclusions from smaller scales may not always hold for larger models" (Appendix E.4.2), undermining their value as cheaper proxies. A practitioner who wants to apply the methodology to a 7B or 13B model—different architectures at different scales—cannot determine from the paper whether the 1.2B ablation results would generalize to their setting or whether they would need to replicate the full 12B-scale ablation protocol.

Evidence in the paper: The computational cost of experiments is never reported. The paper does not discuss the resource requirements for replicating the ablation protocol. The carbon footprint or energy consumption is not mentioned. This is standard for most ML papers but is notable given the paper's framing around efficiency—the introduction discusses the "massive investment of time, compute, and energy" required for frontier model training (Section 1), motivating FP4 as an efficiency improvement, but the paper's own experimental methodology incurs substantial compute overhead that is not contextualized.

Mitigation status: Not addressed. The paper provides no guidance on cheaper validation strategies, no power-analysis to determine minimum training horizon for detecting component importance, and no discussion of whether the 1.2B ablations should be considered sufficient proxies for larger-scale validation. The paper's own caution that smaller-scale conclusions may not hold for larger models (Appendix E.4.2) would apply to this concern, creating a tension: the paper warns against extrapolating from small models while relying on small-model ablations to support claims about large-model behavior.

---

Downstream Task Variance Is Inconsistently Interpreted

Across the benchmark suite in Table 2, NVFP4 shows performance that is sometimes better and sometimes worse than FP8, with swings up to 3.66 points in either direction. The paper interprets the negative swings (coding task underperformance) as likely evaluation noise—"MBPP+ accuracy drops on the very final checkpoint evaluation and choosing another checkpoint could potentially lead to better accuracy for this task" (Section 3)—but does not apply the same noise interpretation to the positive swings (GSM8k: +3.19, MGSM: +3.66, AGIEval English: +3.30). This asymmetric interpretation weakens the paper's claim that NVFP4 "achieves comparable accuracy to FP8 across most individual benchmarks" (Section 3), because the comparability appears to depend on whether differences are attributed to noise or to genuine format effects based on their direction.

The constraint: The paper does not provide a systematic framework for determining whether a task-level accuracy difference is statistically or practically significant. The single-run nature of the experiment makes this determination impossible without additional data. The paper's discussion treats the 0.04-point MMLU-Pro gap as evidence of matching, the 3+ point positive gaps as unremarkable, and the 3+ point negative gaps as suspect—a pattern that could be rationalized but is not argued from evidence.

The consequence: A practitioner evaluating NVFP4 for a coding-heavy application would see 2.5–3.2 point degradations on HumanEval+ and MBPP+ and have no way to determine from this paper whether these are genuine FP4 limitations or artifacts of a single training run. Conversely, a practitioner evaluating NVFP4 for multilingual applications would see 0.94–3.66 point improvements and similarly have no way to assess whether these are robust or spurious. The paper's framing—"NVFP4 achieves comparable accuracy"—is accurate for aggregate metrics (MMLU-Pro) but elides the task-level variance that matters for application-specific decisions.

Evidence in the paper: Table 2 reports per-task accuracies. The coding-task discussion in Section 3 attributes underperformance to noise. The math and multilingual improvements are not discussed. Figure 3 shows time-series accuracy curves where NVFP4 and FP8 cross repeatedly, consistent with run-to-run variance, but this is not analyzed.

Mitigation status: Partially mitigated by the time-series curves in Figure 3, which show that NVFP4 and FP8 track closely over the training horizon without systematic divergence. However, these curves show snapshots at different training checkpoints, not independent training runs, so they cannot distinguish noise from genuine differences. The paper's suggestion that a different checkpoint might improve coding accuracy is plausible but untested.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the frontier of narrow-precision training from a binary question ("can we train in FP4?") to a systems question ("what combination of techniques makes FP4 training stable?"). Before this work, the field had a collection of individually promising techniques—stochastic rounding for gradient bias, Hadamard transforms for outlier management, mixed precision for sensitive layers—but no demonstration that these techniques must be combined to achieve stability at production scale. The paper's central empirical result is not that any single technique enables FP4 training, but that removing any one of four components from a working configuration measurably degrades convergence at 12B scale (Figure 4). This recasts the problem from algorithmic invention to compositional engineering, and it changes how future research on even narrower formats should be structured.

The magnitude of this shift is a reframing, not a paradigm change. The individual techniques are not new; the paper makes no claims of algorithmic novelty for stochastic rounding, Hadamard transforms, 2D scaling, or mixed precision. What is new is the demonstration of necessity through ablation at a scale that practitioners care about. Prior FP4 training papers (Tseng et al., 2025b; Chmiel et al., 2025; Castro et al., 2025) demonstrated feasibility at hundreds of millions of parameters and hundreds of billions of tokens. This paper demonstrates it at 12 billion parameters and 10 trillion tokens—roughly an order of magnitude larger in both dimensions—and shows that the techniques become more critical, not less, as scale increases (Appendix E.4.2, E.4.3). The implication is that FP4 training is not "solved" in any fundamental sense; it is managed through a system of overlapping protections, and the required protections intensify with scale.

This reframing resolves a latent tension in the narrow-precision training literature. Prior work oscillated between optimism—"FP4 training can work" (Tseng et al., 2025b; Chmiel et al., 2025)—and implicit pessimism—the techniques were validated only at small scale, leaving open the possibility that FP4 training would break down on frontier-sized models. The paper reconciles these positions: FP4 training can work at scale, but only when the format is numerically adequate (NVFP4's E4M3 block scales and 16-element blocks vs. MXFP4's power-of-two scales and 32-element blocks) and the training methodology simultaneously addresses all major failure modes. The 36% token penalty for MXFP4 (Section 5, Figure 6b) quantifies the cost of format inadequacy; the ablation curves in Figure 4 quantify the cost of methodology omissions. Together, they establish that FP4 training lives in a narrow window of viability that requires coordinated choices across format design and training procedure.

A concrete consequence is that certain research directions become more attractive and others less so:

• More attractive: automated sensitivity mapping. The paper shows empirically that sensitivity is concentrated in specific locations (final layers, Wgrad GEMMs, gradient tensors), but discovers these locations through manual ablation. A research program that develops cheap, lightweight diagnostics for predicting a priori which layers, GEMMs, and tensors will be sensitive to FP4 quantization under a given architecture—without requiring 12B-scale ablation runs—would directly build on this paper's sensitivity map and make the methodology transferable to new architectures without heroic computational cost. The paper gestures at this: "quantization error metrics could potentially serve as a mechanism to determine which linear layers should remain in higher precision during training" (Appendix E.2).

• More attractive: verifier-free precision adaptation. The paper demonstrates that the forward pass is the primary source of the FP4 loss gap (Appendix D, Figure 7) and that switching only the forward pass to BF16 during learning rate decay recovers most of the gap. This suggests a dynamic precision approach: train in FP4 during the stable phase, monitor some signal of forward-pass quantization stress, and adaptively switch specific layers or operations to higher precision only when and where needed. The paper's fixed schedule (switch at 8.2T tokens) is a proof of concept; a learned or heuristic policy could squeeze more throughput from the same accuracy budget.

• More attractive: format co-design with training methodology. The NVFP4 vs. MXFP4 comparison is a single data point, but it establishes a methodology for evaluating format design choices: measure the token overhead required to match a baseline loss under identical training procedures. This could be extended to systematically explore the design space of 4-bit formats—block sizes, scale factor precisions, exponent/mantissa bit allocations—to find the Pareto frontier of numerical accuracy vs. hardware simplicity. The paper's finding that smaller blocks (16 vs. 32) and fractional-precision scales (E4M3 vs. UE8M0) provide measurable training benefits is a design principle that future format proposals should incorporate.

• Less attractive: single-technique FP4 training proposals. The paper's ablation evidence that removing any one component degrades convergence (Figure 4) makes it unlikely that a single new technique—a better rounding scheme, a smarter outlier transform, a different scaling strategy—will make FP4 training work at scale without the other components. Future work proposing a single FP4 training technique should be expected to demonstrate that it works in combination with—or replaces—the existing suite, not that it works in isolation on a toy model. The bar for claiming an FP4 training advance has been raised: the claim must be validated at billion-parameter scale and multi-trillion-token horizons, with ablations showing the new technique is necessary given the other components.

• Less attractive: FP4 training on pure MXFP4 without format improvements. The 36% token penalty for MXFP4 is a quantitative argument against investing significant effort in optimizing MXFP4 training methodology without addressing the format's inherent limitations. If the format itself imposes a ~36% efficiency tax, methodological improvements can at best recover that tax but cannot exceed the efficiency of a better format with equivalent methodology. The paper's result suggests that the community's focus should shift toward formats with fractional-precision block scaling (NVFP4 or future standards) rather than optimizing around MXFP4's power-of-two constraint.

Perhaps the most significant landscape change is lowering the perceived risk of FP4 adoption for training practitioners. Before this work, training a production model in FP4 was speculative: no public evidence existed that a 12B model could be trained on 10T tokens in 4-bit precision without catastrophic accuracy loss. The paper's MMLU-Pro comparison—62.58% vs. 62.62% for FP8 (Table 2)—provides a concrete, high-visibility datapoint that FP4 training does not sacrifice downstream capability on the most commonly cited LLM benchmark. This single number, validated at a scale that is relevant to real training budgets (12B parameters, 10T tokens), may do more to accelerate FP4 adoption than any algorithmic contribution because it directly addresses the risk-aversion that prevents practitioners from migrating away from FP8.

However, the paper also introduces a new risk awareness: that FP4 training's viability depends on format quality. The 36% MXFP4 penalty means that organizations with hardware that natively supports MXFP4 but not NVFP4 (non-NVIDIA Blackwell GPUs, future hardware implementing the OCP standard) may not achieve the same training efficiency. The paper thus creates a format-dependent value proposition: the efficiency gains are real on Blackwell hardware with NVFP4, but the path to FP4 training on other hardware depends on whether those platforms adopt NVFP4-like format features (fractional block scales, smaller blocks) or replicate MXFP4's limitations.

Follow-Up Research This Work Enables

Scaling laws for FP4 training across model sizes and token horizons. The paper demonstrates one point on the scaling curve: 12B parameters, 10T tokens, NVFP4 format, hybrid Mamba-Transformer architecture. The most urgent follow-up is to measure how the FP4-vs-FP8 loss gap scales with model size. Does the gap grow, shrink, or stay constant as we go from 1B to 10B to 100B parameters? The paper's evidence is mixed: the 1.2B model is forgiving (converges with only high-precision layers, Figure 8), while the 12B model requires all four techniques (Figure 4), suggesting the gap grows with scale. But this is two data points on different architectures (Transformer vs. hybrid Mamba-Transformer). A systematic sweep—training models at 1B, 3B, 8B, 12B, and 30B parameters on identical data for identical token horizons with the full methodology—would reveal whether FP4's viability is scale-dependent and whether the methodology needs to be strengthened for larger models (e.g., more high-precision layers, larger Hadamard matrices, additional techniques). A negative result—the gap widening super-linearly with scale—would bound FP4's applicability to models below some threshold size. A positive result—the gap stabilizing—would strengthen the case for FP4 on frontier models.

Automated layer sensitivity detection without full-scale ablation. The paper manually discovers that final layers are disproportionately sensitive to FP4 quantization (Appendix E.2, Figure 9) by exhaustively testing different configurations on the 1.2B model. This approach does not scale to new architectures. The follow-up is to develop a lightweight diagnostic that, given a few hundred training steps in BF16, predicts which layers will be sensitive to FP4 quantization. A concrete design: during an initial BF16 warmup phase, compute the per-layer quantization error (e.g., the mean squared error between the FP4-quantized and BF16 weight gradient) and use this as a predictor of training instability when that layer is quantized. Validate by training a 1.2B model with layers selected for high precision based on this metric and comparing against the paper's manual selection (last 4 blocks). If the automated selection matches or exceeds manual selection, the diagnostic can be applied to new architectures without human analysis. A strong version of this work would demonstrate the diagnostic on 3–4 different architectures (pure Transformer, MoE, state-space model variants) and show consistent accuracy.

FP4 training on publicly available datasets to decouple format effects from data quality. The paper's training data is a proprietary curated blend (Appendix A.1). A confounding factor in the paper's results is that the data quality may suppress outlier activations and gradients, making FP4 training easier than it would be on noisier web-scale data. The follow-up replicates the 1.2B ablation suite using a fully public dataset (e.g., Dolma, RedPajama-V2, or FineWeb-Edu) and compares the FP4 loss gap and component necessity against the paper's results. If the gap is larger on less curated data, it suggests FP4 training methodology needs to be data-aware—perhaps requiring more high-precision layers or larger Hadamard matrices for noisier data distributions. If the gap is unchanged, the paper's data quality concern is mitigated. This experiment is feasible at 1.2B scale (1T tokens) with academic compute budgets and would substantially strengthen or appropriately bound the paper's generalizability claims.

Dynamic precision scheduling during training. The paper's Appendix D result—that switching the forward pass to BF16 during learning rate decay recovers most of the loss gap—is a static schedule chosen by human inspection. A natural extension is to learn or design a dynamic policy that continuously adjusts per-layer or per-GEMM precision during training based on real-time signals. Concrete design: at each training step, measure the ratio of FP4 quantization error to gradient norm for each layer, and switch layers to higher precision when this ratio exceeds a threshold. Train a 1.2B model with this adaptive policy and compare against the paper's static mixed-precision configuration (last 4 blocks in BF16 throughout + forward-pass switch at decay). The hypothesis is that dynamic precision can achieve the same loss with fewer high-precision operations—early in training, more layers might need high precision; late in stable training, fewer might—recovering additional throughput beyond the paper's fixed allocation. A failure mode to anticipate: the decision overhead and precision-switching latency might negate throughput gains at small scale, so the experiment should measure both loss and wall-clock time.

Combining FP4 GEMMs with FP4 attention and communication. The paper keeps attention components, normalization, and tensor-parallel reductions in BF16 or FP32 (Section 4.1). Extending FP4 to these operations would increase the fraction of training compute that benefits from the 4–6× throughput boost. For attention specifically: the query-key and attention score-value batched GEMMs are currently in BF16. Quantizing these to NVFP4 would require addressing the numerical sensitivity of the softmax operation (which exponentiates attention scores—a small FP4 quantization error in the pre-softmax logits could produce large probability changes). A concrete design: apply Hadamard transforms to the query and key tensors before the attention GEMM, quantize to NVFP4, compute attention in higher precision, and measure whether the attention pattern is preserved. Train a 1.2B model with FP4 attention and compare attention pattern similarity (e.g., Jensen-Shannon divergence between FP4 and BF16 attention distributions) and downstream accuracy against the paper's BF16-attention baseline. The paper's Section 6 explicitly calls out "extending NVFP4 to attention and communication paths" as future work, making this a direct extension.

NVFP4 post-training (fine-tuning, RLHF, quantization-aware adaptation). The paper validates NVFP4 for pretraining only. Whether the format and methodology transfer to post-training phases—supervised fine-tuning, RLHF with a reward model, or continued pretraining on domain-specific data—is unexamined. Post-training typically uses smaller datasets, shorter training horizons, and lower learning rates, which could change sensitivity patterns. A concrete experiment: take the paper's 12B NVFP4-pretrained checkpoint and fine-tune it on a standard instruction dataset (e.g., OpenHermes, UltraChat) in both NVFP4 and BF16, measuring downstream benchmark accuracy and instruction-following quality (e.g., AlpacaEval, MT-Bench). If NVFP4 fine-tuning matches BF16, the pretrained checkpoint can serve as a drop-in base for the full training pipeline without precision switching. If it degrades, practitioners need a mixed-precision strategy for fine-tuning that may differ from the pretraining strategy.

Practical Applications and Downstream Use Cases

Frontier model pretraining at reduced cost and time. Organizations training models in the 10B–100B parameter range on multi-trillion-token corpora can directly apply the paper's methodology to reduce training time by an estimated 2–3× (based on the 4–6× Tensor Core throughput advantage of FP4 over BF16 from Table 1, moderated by the ~15% of layers in BF16 and non-GEMM overhead). A training run that currently takes 90 days on a GPU cluster using FP8 could, after system-level optimization of the methodology's overhead (Hadamard transforms, global scale computation), potentially complete in 30–45 days with NVFP4 on Blackwell hardware. The paper's 10T-token, 12B-scale validation is directly relevant to this regime—these are realistic dimensions for production models—and the MMLU-Pro matching (62.58% vs. 62.62%) provides the risk mitigation that would justify the engineering investment in porting a training codebase to FP4. The practical caveat is that organizations must be on Blackwell hardware (or future GPUs with native NVFP4 support); the paper's results do not transfer to previous-generation hardware without FP4 Tensor Cores.

Larger models within a fixed compute budget. For organizations with a fixed GPU budget and training time window, the paper's methodology enables training larger models than would be feasible in FP8. If a team has budget to train a 12B model in FP8 for 10T tokens, switching to NVFP4 could allow training a ~18B–24B model (50–100% more parameters) in the same wall-clock time, assuming the throughput gains translate proportionally and the methodology's mixed-precision fraction (15% BF16 layers) scales similarly. The paper doesn't validate this extrapolation—the scaling behavior of the FP4 loss gap with model size is unknown—but the 12B result is a credible anchor point. The downstream benchmark results suggest that a larger NVFP4-trained model might outperform a smaller FP8-trained model on some tasks (NVFP4 already shows gains on GSM8k and MGSM at matched size, Table 2), and the aggregate capability scores (General: 69.82 NVFP4 vs. 68.99 FP8) support the expectation that parameter-count gains would compound the format's efficiency benefits.

Energy-constrained or carbon-aware training. The paper's introduction frames the motivation partially in terms of energy: frontier model training requires "massive investment of... energy" (Section 1). For organizations operating under energy constraints—either due to datacenter power limits, carbon budgets, or electricity costs—FP4 training's ~50% memory reduction and 2–3× throughput improvement translate directly to lower total energy consumption per training run. If FP4 training reduces total GPU-hours by 2–3× (once system-level optimization is complete), it reduces the energy footprint proportionally. This is a direct operational benefit independent of training time: two organizations with the same GPU count but different energy budgets could both train the same model, with the FP4 team requiring half the electricity. The paper's validation that accuracy does not degrade (MMLU-Pro within 0.04 points) makes the energy efficiency argument compelling because there is no accuracy trade-off to weigh against the energy savings—the savings are "free" from a model quality perspective, conditional on the methodology being implemented correctly.

Edge and fine-tuning deployments where memory is the bottleneck. Although the paper validates only pretraining, the NVFP4 format's 50% memory reduction has immediate implications for fine-tuning and continued training scenarios where GPU memory capacity limits model size. A practitioner fine-tuning a 12B model with LoRA on a single GPU that barely fits the BF16 weights could, with NVFP4, potentially fine-tune a larger model (e.g., 24B) on the same hardware, since the weight storage is halved. The paper does not validate fine-tuning, but the pretraining stability results—showing that weights, activations, and gradients can all operate in FP4 without divergence—suggest that fine-tuning, which has similar computational patterns but smaller datasets and fewer steps, is likely to be at least as stable. This application is speculative without dedicated fine-tuning experiments but is a direct consequence of the memory reduction Table 1 quantifies (FP4 uses approximately half the memory of FP8). The paper's finding that forward-pass quantization is the primary bottleneck (Appendix D) suggests that fine-tuning, where forward-pass accuracy may be more important due to smaller datasets, might require the forward-pass-to-BF16 switch earlier or throughout, but this is untested.