FP8-LM: Training FP8 Large Language Models

🎯 Pitch

Training 175B-parameter GPT models with FP8 data formats runs 75% faster than standard BF16 approaches and slashes memory by 39%—without any loss in accuracy. The framework uniquely extends low-precision computation into gradients, optimizer states, and distributed communication, not just GEMM operations.

1. Executive Summary

This paper introduces a new FP8 automatic mixed-precision framework for training large language models that extends low-precision computation beyond GEMM operations into gradients, optimizer states, and distributed communication. The framework is evaluated on GPT-style models ranging from 125M to 175B parameters on the H100 GPU platform, using three incremental optimization levels—FP8 gradient all-reduce communication (via automatic scaling to prevent underflow/overflow), an FP8 optimizer (via precision decoupling that assigns FP8 to the first-order moment while retaining higher precision for master weights and second-order moments), and FP8 distributed parallelism (via a single-shared-scalar mechanism for NCCL compatibility and a greedy whole-tensor distribution scheme for ZeRO). On GPT-175B training, the framework achieves a 39% reduction in real memory usage and runs 75% faster than the widely adopted BF16 framework (Megatron-LM), surpassing Nvidia Transformer Engine by 37% in speed and 42% in memory savings. The paper also establishes that FP8 training preserves model accuracy—matching BF16 loss curves and downstream zero-shot performance across scales from 7B to 175B—while extending to instruction tuning and RLHF, demonstrating that the benefits compound as model size increases but require careful management of per-tensor scaling factors and precision-sensitive variables to avoid training divergence.

2. Context and Motivation

The Core Problem: Training Large Language Models Is Prohibitively Expensive

The fundamental problem this paper addresses is the extraordinary computational cost of training large language models. This is not merely an academic concern—it directly determines which organizations can participate in frontier AI development and sets a hard ceiling on how many experiments researchers can run. The paper opens by characterizing the severity of this problem through concrete examples: training PaLM required 6,144 TPUv4 chips for a 540B model, and GPT-3 175B consumed "several thousand petaflop/s-days of compute for pre-training" (Section 1). These numbers translate to millions of dollars in hardware costs per training run, making iteration cycles painfully slow and excluding all but the most well-resourced organizations from pushing the boundaries of model scale.

This problem has a self-reinforcing dynamic. The scaling laws literature (Brown et al., 2020; Hoffmann et al., 2022) has established that larger models, when trained on sufficient data, reliably outperform smaller ones—so the incentive to scale is strong. But as models grow, the computational requirements grow proportionally (or faster, when scaling both model size and training data), creating an escalating cost curve that makes each successive generation of models harder to fund and slower to develop. The paper positions itself squarely within this tension: the field needs methods that reduce training costs without sacrificing model quality, so that scaling can continue without becoming economically infeasible.

Three specific cost dimensions matter: compute throughput (how fast training proceeds), memory footprint (how many GPUs are needed to hold a model of a given size), and communication bandwidth (how much data must move between GPUs during distributed training). Improving any one of these in isolation provides marginal benefits; the real challenge is moving all three simultaneously, since they are tightly coupled in distributed training systems. A method that reduces computation but increases memory usage, for instance, might force the model to be split across more GPUs, which in turn increases communication, potentially negating the compute savings.

Why Low-Precision Training Is the Most Promising Direction

Low-precision training attacks all three cost dimensions at once. When you reduce the number of bits used to represent numbers during training—weights, activations, gradients, optimizer states—you directly reduce memory consumption (fewer bits per value), increase compute throughput (hardware can process more low-precision operations per second, and modern GPUs like the H100 have specialized tensor cores that are much faster at lower precision), and reduce communication volume (fewer bits to transmit between GPUs during all-reduce and other collective operations). The paper emphasizes this triple benefit in Section 1:

"Low-precision training is one of the most promising directions to reduce the costs, as it can provide high speed, small memory footprint, and low communication overhead."

This is not merely theoretical. The industry has already validated the 16-bit step. The transition from FP32 (32-bit floating point) to BF16 (16-bit brain floating point) for training was a major enabler of the current generation of large models. The paper's introduction traces this history: FP16-FP32 mixed-precision (Micikevicius et al., 2017) was the first widely adopted scheme, but it proved unstable for very large models because FP16's limited dynamic range (max normal value of 65,504) caused numerical instabilities—gradients would overflow or underflow, leading to training divergence. Rae et al. (2021) and Zeng et al. (2022) explicitly documented these instabilities. The community's solution was BF16, which preserves the same exponent range as FP32 (8 exponent bits in both) while halving the total bits. This means BF16 can represent numbers of the same magnitude as FP32—up to ~3.4 × 10³⁸—avoiding the overflow problem that plagued FP16, at the cost of reduced mantissa precision (7 bits vs. 23 bits in FP32). As the paper notes, BF16-FP32 became the default for major models: Megatron-Turing NLG-530B (Smith et al., 2022), Bloom-175B (Scao et al., 2022), and Gopher (Rae et al., 2021) all used it.

But BF16 still leaves half the potential gains on the table. If 16-bit training is good, 8-bit training should be roughly twice as good—at least in theory. The paper quotes theoretical benefits: "2× speed-up, 50% - 75% memory cost savings, and 50% - 75% communication savings compared with current 16-bit and 32-bit floating point mixed-precision training" (Section 1). The gap between 16-bit and 8-bit represents enormous untapped efficiency that, if realized, could dramatically expand who can train frontier models and how quickly they can iterate.

Where Existing FP8 Approaches Fall Short

The leap from 16-bit to 8-bit is not a simple matter of changing a datatype flag. The paper identifies a critical gap between FP8's theoretical potential and what current systems actually deliver.

The representation challenge. FP8 data formats have dramatically narrower range and precision than BF16 or FP32. The paper's Appendix A.1 details two FP8 sub-formats standardized by NVIDIA, ARM, and Intel (Micikevicius et al., 2022):

E4M3: 1 sign bit, 4 exponent bits, 3 mantissa bits. Max normal value: 448. Min normal value: 1.56 × 10⁻². Maximum relative representation error in normal range: 7.69% to 11.1%.
E5M2: 1 sign bit, 5 exponent bits, 2 mantissa bits. Max normal value: 57,344. Min normal value: 6.10 × 10⁻⁵. Maximum relative error in normal range: 16.7% to 20%.

Compare this to BF16 (max normal: 3.39 × 10³⁸, relative error: 0.39% to 0.78%) and the magnitude of the problem becomes clear. FP8 values that fall outside these narrow ranges either overflow (become infinity) or underflow (become zero), destroying gradient information. Even within the representable range, the low mantissa precision means values are quantized to coarse levels, introducing noise at every operation.

The limited scope of Nvidia Transformer Engine. At the time of this paper, the only production-ready FP8 training framework was Nvidia Transformer Engine (TE) (Nvidia, 2022b), released alongside the H100 GPU. TE represents an important but fundamentally conservative step. It applies FP8 only to GEMM (general matrix multiply) operations within Transformer linear layers—the forward and backward passes of the attention and feed-forward computations. This is the lowest-hanging fruit: matrix multiplications are the most compute-intensive operations in a Transformer, so accelerating them provides immediate throughput benefits.

However, TE's conservatism leaves massive efficiency on the table. As the paper explains in Section 1:

"it applies FP8 solely for GEMM computation and still retains master weights and gradients using high precision, e.g., FP16 or FP32. As a result, the end-to-end speed-up, memory and communication cost savings are very limited."

This is the crux of the gap the paper addresses. TE's approach means that:

Weights are stored in FP16 or FP32, consuming 2–4 bytes per parameter even though the actual computation uses FP8.
Gradients are computed and stored in FP32, then communicated between GPUs via all-reduce in FP32—so the 16-byte-per-parameter gradient communication cost remains unchanged.
Optimizer states (Adam's first and second moment estimates) remain in FP32, consuming 8 additional bytes per parameter.
Master weights (the high-precision copy used for weight updates) remain in FP32, adding another 4 bytes per parameter.

In aggregate, TE only reduces the precision of activations flowing through GEMM operations, leaving the entire memory hierarchy—model weights, gradients, optimizer states, master weights—at 16-bit or 32-bit. The paper's ablation results make this concrete: in Table 5, TE actually uses more GPU memory than BF16 in some configurations (77.3 GB vs. 69.6 GB for GPT-7B) because it adds FP8 compute kernels without reducing storage precision. It achieves throughput improvements (38% faster for GPT-13B, as shown in Table 5) but leaves memory and communication largely untouched.

Why previous attempts at aggressive quantization failed. The paper is careful to acknowledge that reducing optimizer precision below 32-bit has been tried before—and has generally failed. Rae et al. (2021), Zeng et al. (2022), and Liu et al. (2022) all found that "reducing precision of the variables in optimizer to 16-bit leads to accuracy degradation when training billion-scale models" (Section 2.2). The reasons are subtle and depend on which variables are quantized:

16-bit master weights: While BF16 can represent the magnitude of FP32 values, its 7-bit mantissa means weight updates—which are often very small in magnitude—get rounded to zero or lose directional precision. Over many update steps, this accumulation of quantization error causes the model to converge to a worse solution.
16-bit second-order moment (v in Adam): The second moment stores an exponentially weighted average of squared gradients. Squaring already-small gradient values produces extremely small numbers that underflow even in FP16's representable range (min normal: 6.10 × 10⁻⁵), causing the denominator in Adam's update rule to become inaccurate.
16-bit first-order moment (m in Adam): This tends to be somewhat more robust because gradient directions (signs) are more important than precise magnitudes for optimization, but aggressive quantization still introduces noise.

The implication is clear: simply applying FP8 everywhere will cause training to diverge. A successful FP8 training framework must be selective—it must identify which variables can tolerate 8-bit precision and which cannot, and develop mechanisms to bridge the representation gap for those that can.

The Communication Bottleneck in Distributed Training

Beyond compute and memory, the paper identifies distributed communication as a third major bottleneck that current FP8 approaches fail to address. Training large models requires splitting them across many GPUs using parallelism strategies:

Data parallelism: Each GPU holds a full copy of the model but processes a different batch of data. After each backward pass, gradients must be averaged across all GPUs via an all-reduce operation—every GPU sums gradients from all other GPUs and divides by the number of GPUs. For large models, this gradient synchronization can dominate wall-clock time.
Tensor parallelism: Individual Transformer layers are split across GPUs, requiring frequent communication of activations during forward passes and gradients during backward passes.
Pipeline parallelism: Different layers are assigned to different GPUs, with activations and gradients communicated between pipeline stages.
Sequence parallelism: Input sequences are split across GPUs, requiring all-gather and reduce-scatter operations on activations.

In standard BF16 mixed-precision training (as implemented in Megatron-LM), gradients are computed in BF16 but communicated in FP32 during all-reduce. The paper quantifies this cost: for the GPT-175B experiment with 8-way tensor parallelism and 4-way pipeline parallelism (Table 5), weight-related communication volume is 23.4 GB per iteration in BF16, and activation-related communication volume is 5.9 GB (Table 7). These numbers represent data that must physically move between GPUs over interconnects like NVLink or InfiniBand—and that movement takes time.

The theoretical promise of FP8 for communication is compelling: if gradients could be transmitted in 8 bits instead of 32, communication volume drops by 75%. However, as the paper details in Section 2.1, making this work in practice involves solving a fundamental tension between underflow and overflow that becomes much more severe at 8-bit precision, especially when the number of GPUs (and thus the divisor in gradient averaging) is large.

How This Paper Positions Itself

The paper frames its contribution not as a single technique but as a systematic expansion of FP8's reach in the training pipeline. The introduction uses the metaphor of "infiltration" (Section 1: "infiltrate FP8 compute, storage, and communication into the whole progress of large model training") to convey the idea that FP8 should not be confined to a few compute kernels—it should permeate every aspect of training where it can be applied without harming accuracy.

This framing is reinforced by the paper's three-level optimization structure (Section 1 and Section 2):

"The three levels gradually incorporate 8-bit collective communication, optimizer, and distributed parallel training in an incremental manner. The higher optimization level indicates using more FP8 during LLM training."

This incremental design serves both a practical and a conceptual purpose. Practically, it means users can adopt FP8 at whatever level their risk tolerance and hardware allow—start with FP8 GEMM (like TE), then add FP8 gradient communication, then add the FP8 optimizer, and finally enable FP8 distributed parallelism. Conceptually, it makes the point that FP8 training is not a binary choice but a spectrum, and that the community has been stuck at the shallowest level.

The paper explicitly contrasts its ambition with TE's limited scope. In Section 4 (Related Work), it states:

"TE's current implementation restricts FP8 usage solely to weight computation, retaining the storage of model weights and gradient calculations with 16-bit data types. Consequently, the end-to-end speed-up, memory and communication cost savings are limited. In contrast, our work infiltrates FP8 gradient, optimizer, and distributed training into the whole progress of model training, fully unveiling the capabilities of FP8."

The phrase "fully unveiling the capabilities of FP8" captures the paper's thesis: that FP8's benefits have been artificially constrained by implementation conservatism, and that with the right techniques (automatic scaling, precision decoupling, shared-scalar communication, greedy ZeRO distribution), FP8 can deliver on its theoretical promise across the entire training stack.

The paper also positions itself as establishing a new paradigm for training. The conclusion states:

"We expect the release of our FP8 framework will establish a new paradigm for next-generation low-precision training system dedicated to large foundation models."

This is not merely aspirational language—it reflects the paper's strategic bet that FP8 will become the default training precision for large models, just as BF16 replaced FP32, and that providing an open-source, production-ready implementation (the MS-AMP codebase) will accelerate this transition.

The Specific Technical Gaps the Paper Must Solve

To understand the paper's motivation at a deeper level, it helps to enumerate the specific problems that make FP8 training non-trivial—problems that prior work either didn't address (because it stayed at 16-bit) or addressed only partially (TE):

Gradient all-reduce with FP8 causes underflow or overflow depending on the scaling strategy (Section 2.1). The pre-scaling approach (dividing each GPU's gradient by N before summing) causes underflow when N is large because the divided values fall below FP8's minimum representable number. The post-scaling approach (summing first, dividing later) causes overflow because the sum of many gradients can exceed FP8's maximum value. A mechanism is needed that dynamically adjusts scaling to stay within FP8's narrow range.
NCCL cannot handle per-tensor scaling factors during all-reduce (Section 2.1). The standard NCCL library (Nvidia, 2020) performs collective operations at the sub-tensor level and has no mechanism to associated scaling factors with gradient tensors. This means that even if you solve the underflow/overflow problem for a single tensor, you cannot efficiently synchronize the scaling factors across GPUs without modifying NCCL—which is impractical.
Optimizer variables have different sensitivity to quantization (Section 2.2). Some variables (master weights, second-order moments) degrade sharply under FP8, while others (first-order moments) are more robust. A naive FP8 optimizer that quantizes everything uniformly will diverge, as the paper demonstrates in its ablation (Figure 8, the FP8 #4 configuration). The challenge is to identify which variables can tolerate FP8 and design a training procedure that preserves accuracy while maximizing memory savings.
ZeRO-style tensor partitioning breaks when scaling factors are per-tensor (Section 2.3). ZeRO (Rajbhandari et al., 2020) distributes model states across GPUs by splitting individual tensors into partitions. But if each tensor has an associated scaling factor (needed for FP8 representation), splitting the tensor means the scaling factor applies to a fragment rather than the whole tensor, making it meaningless. A different distribution strategy is needed that respects the per-tensor nature of FP8 scaling.
Distributed parallelism operations (all-gather, reduce-scatter) still use high-precision activations (Section 2.3). In sequence and tensor parallelism, activations are communicated between GPUs at FP16 or FP32 precision, consuming bandwidth that could be halved if FP8 were used. However, activations have their own dynamic range characteristics that must be managed to avoid degrading model quality.

The paper's technical sections tackle each of these gaps directly, with the shared theme that FP8's narrow range requires adaptive, per-tensor mechanisms rather than the global or block-wise scaling approaches that sufficed at 16-bit precision. This theme—that FP8 demands finer-grained control than previous precision transitions—is the unifying intellectual thread of the paper's technical contribution.

The Experimental Strategy: Why Comparisons Must Be Careful

The paper's motivation also shapes its experimental design. Demonstrating that FP8 training "works" requires more than showing that loss curves overlap—it requires proving that the method is:

Scale-invariant: Works across model sizes from millions to hundreds of billions of parameters, since the whole point is enabling larger-scale training.
Hyperparameter-invariant: Does not require re-tuning learning rates, weight decay, batch sizes, or other sensitive knobs, since one of FP8's promised benefits is that it can be a "drop-in replacement" for existing training pipelines.
Task-invariant: Works for pre-training, instruction tuning, and RLHF—the three major training paradigms for modern LLMs—since a framework that only works for pre-training has limited practical value.
Hardware-real: Delivers actual speedup and memory savings on real H100 GPUs, not just theoretical FLOPs reductions, since simulation-level gains often fail to materialize when kernel launch overhead, memory bandwidth limits, and communication patterns are accounted for.

The paper's experiments (spanning 125M to 175B parameters, pre-training and fine-tuning, and reporting real GPU measurements rather than theoretical estimates) are designed to address each of these dimensions. The emphasis on "no changes to hyper-parameters" (repeated in both the abstract and Section 3.2) is particularly important—it signals that FP8 is not a fragile technique that requires expert tuning but a robust engineering solution that can be adopted transparently.

Why This Matters Now

The paper's timing is not coincidental. The H100 GPU, released in 2022, was the first datacenter GPU with native hardware support for FP8 tensor operations (via its fourth-generation Tensor Cores). Prior to this, FP8 training research was largely confined to simulation—as the paper notes in Section 4, "early pioneering efforts in FP8 low-bit model training (Wang et al., 2018; Sun et al., 2019; Dettmers et al., 2021) have largely remained at the simulation stage. Consequently, there exists a notable gap between the projected capabilities of these approaches and their actual performance on hardware." The H100 changed this by making FP8 a practical datatype, but the software ecosystem lagged behind the hardware capability. TE was the first step, but as the paper argues, it was an incomplete one.

The paper thus positions itself as filling the gap between hardware capability (H100's FP8 Tensor Cores) and software utilization (a framework that actually uses FP8 throughout the training stack). This is a classic systems paper motivation: the hardware exists, the theoretical benefits are clear, but the systems software to realize those benefits does not. By releasing MS-AMP as an open-source framework, the paper aims to provide the missing software layer that enables the community to actually benefit from FP8's theoretical promise.

3. Technical Approach

3.1 Reader Orientation (Approachable Technical Breakdown)

The paper builds a drop-in replacement training framework that extends mixed-precision from 16-bit (BF16) to 8-bit (FP8) for every component of large language model training—computation, storage, and communication—without requiring any hyperparameter changes. The core problem it solves is that current FP8 frameworks (namely Nvidia Transformer Engine) only accelerate matrix multiplications while leaving weights, gradients, optimizer states, and inter-GPU communication at 16-bit or 32-bit precision, which means the bulk of memory and bandwidth costs remain untouched. The solution's "shape" is a set of three progressively aggressive optimization levels, each adding another FP8 component, combined with two general techniques (precision decoupling and automatic scaling) that prevent the numerical collapse that would otherwise result from FP8's dramatically narrower dynamic range and coarser precision relative to BF16.

3.2 Big-Picture Architecture (Diagram in Words)

The system consists of five major components that modify different stages of the standard mixed-precision training loop:

FP8 GEMM Compute (compatible with Nvidia TE): Linear layer forward and backward passes use FP8 tensor cores for matrix multiplications. This is the baseline level—what Transformer Engine already does—and serves as the starting point for the paper's extensions.
FP8 Gradient Communication Module: After the backward pass produces gradients on each GPU, this module applies automatic per-tensor scaling to convert gradients from FP32 to FP8, synchronizes scaling factors across GPUs using a shared-scalar mechanism, performs standard NCCL all-reduce in FP8, and rescales the result. This reduces gradient communication volume by approximately 63–65% (Section 3.2.2, Table 5).
FP8 Optimizer: The AdamW optimizer state is restructured according to a precision-decoupling principle: master weights are stored in FP16 with tensor scaling (2 bytes/parameter), the first-order gradient moment $m$ is stored in FP8 with tensor scaling (1 byte/parameter), the second-order moment $v$ is stored in FP16 (2 bytes/parameter), and gradients are kept in FP8 (1 byte/parameter). Total optimizer memory drops from 16 to 6 bytes per parameter (Section 2.2, Equations 7–8).
FP8 Distributed Parallelism Module: For tensor parallelism, weight and activation shards are cast to FP8 before linear-layer computation, enabling FP8 collective communication (all-gather, reduce-scatter) on activations. For sequence parallelism, an FP8 conversion is inserted before the gather-reduce operations that bridge sequence-parallel and tensor-parallel regions. For ZeRO-style data parallelism, a greedy whole-tensor distribution algorithm replaces the standard per-tensor partitioning to accommodate per-tensor scaling factors.
Automatic Scaling Engine (cross-cutting): A dynamic scaling-factor adjustment mechanism that monitors gradient value distributions during training and adjusts per-tensor scale factors to keep values within FP8's representable range, preventing both underflow and overflow. This operates across the gradient communication and optimizer components.

Information flows through these components in the standard training loop order: forward pass (FP8 GEMM + FP8 activation communication in parallelism modules) → backward pass (FP8 GEMM + FP8 gradient computation) → gradient all-reduce (FP8 communication module applies scaling, synchronizes, reduces, rescales) → optimizer step (FP8 optimizer updates moments and weights using precision-decoupled storage) → next iteration.

3.3 Roadmap for the Deep Dive

First, the FP8 representation challenge and the tensor scaling mechanism, since all subsequent components depend on the ability to safely convert values between FP32/BF16 and FP8 without losing critical information. This establishes why per-tensor scaling is necessary and how the two scaling strategies (just-in-time and delayed) work.
Second, the auto-scaling technique for gradient all-reduce, since it is the first major extension beyond TE and introduces the key insight that dynamic, per-tensor scaling can simultaneously solve the underflow problem of pre-scaling and the overflow problem of post-scaling. This section also covers the shared-scalar mechanism that makes FP8 communication practical with unmodified NCCL.
Third, the FP8 optimizer and precision decoupling, since it requires understanding which optimizer variables degrade under quantization and why. This section walks through the experimental evidence that first-order moments tolerate FP8 while master weights and second-order moments do not.
Fourth, the FP8 distributed parallelism extensions, covering how tensor, sequence, and ZeRO parallelism are adapted to use FP8 for both computation and communication, including the greedy whole-tensor distribution algorithm for ZeRO.
Fifth, the training recipe and configurations, pulling together all the hyperparameters, model architectures, and parallelism settings used to validate the framework at scales from 125M to 175B parameters.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and numerical methods paper whose core idea is that FP8 can be safely extended from GEMM operations to the entire training stack—gradients, optimizer states, and distributed communication—by applying two principles: (1) precision decoupling, which assigns lower precision only to variables that empirically tolerate higher quantization error, and (2) automatic per-tensor scaling, which dynamically adjusts the mapping between high-precision values and FP8's narrow representable range to prevent information loss from underflow and overflow.

The FP8 Representation Challenge and Tensor Scaling

FP8 is not a single data format but two complementary formats standardized by NVIDIA, ARM, and Intel (Micikevicius et al., 2022) and detailed in the paper's Appendix A.1:

E4M3 (1 sign bit, 4 exponent bits, 3 mantissa bits): maximum normal value 448, minimum normal value $1.56 \times 10^{-2}$ , minimum subnormal value $1.95 \times 10^{-3}$ . Maximum relative representation error in the normal range is 7.69% to 11.1%, and in the subnormal range up to 16.7%.
E5M2 (1 sign bit, 5 exponent bits, 2 mantissa bits): maximum normal value 57,344, minimum normal value $6.10 \times 10^{-5}$ , minimum subnormal value $1.53 \times 10^{-5}$ . Maximum relative error in the normal range is 16.7% to 20%, and up to 50% in the subnormal range.

The key tradeoff is that E4M3 offers higher precision but a narrower range (values above 448 overflow to infinity), while E5M2 offers a wider range (up to 57,344) but coarser precision. In practice, the paper uses E4M3 for forward-pass activations and weights (where values tend to be well-behaved and precision matters more) and E5M2 for backward-pass gradients (where the range can be more extreme and the five exponent bits help avoid overflow). This choice is mentioned in the context of the Nvidia TE documentation (Nvidia, 2022c) referenced in Appendix A.2, though the paper itself focuses on the scaling mechanisms rather than the format selection per operation.

Why tensor scaling is essential. The crux of the FP8 challenge is that the representation range of E4M3 (the higher-precision format) is fundamentally mismatched with the distribution of values that arise during neural network training. Gradients for different layers can differ by orders of magnitude—early layers might have gradients of $10^{-6}$ while later layers have gradients of $10^{-1}$ . If you simply cast these values to FP8, small gradients underflow to zero (losing all information) and large gradients overflow to infinity (producing NaNs downstream). The tensor scaling technique (Section 2.2, Appendix A.2) solves this by multiplying the high-precision tensor by a per-tensor scaling factor before casting:

$\text{FP8}(x) = \text{cast\_to\_fp8}(s \cdot x)$

where $x$ is the original high-precision tensor (FP32 or BF16), $s$ is a per-tensor scaling factor chosen so that $s \cdot x$ falls within FP8's comfortable representable range, and $\text{cast\_to\_fp8}$ performs the actual bit-truncation to 8 bits. The inverse operation to recover the approximate original value is $x \approx \text{FP8}(x) / s$ .

What this computes: for a given tensor, the scaling factor $s$ shifts the entire distribution of values so that the largest-magnitude entries sit near (but not above) FP8's maximum representable value, while small entries occupy the lower end of the representable range rather than underflowing to zero. The result is a compressed 8-bit representation that preserves relative magnitudes within the tensor, at the cost of storing one additional scalar ( $s$ ) per tensor.

Why this form: per-tensor scaling is necessary (as opposed to a single global scaling factor for all tensors, as used in FP16 loss scaling (Micikevicius et al., 2017)) because different tensors in a Transformer model have dramatically different magnitude distributions. A global scale that prevents overflow in the largest-gradient layer would cause severe underflow in layers with small gradients. Per-tensor scaling adapts to each tensor's statistics independently. The paper also notes (Appendix A.2) that even finer granularities (block-wise, layer-wise) have been explored (Ramesh et al., 2021; Sun et al., 2020), but per-tensor scaling is the granularity that balances adaptation to local statistics against the overhead of storing and communicating scaling factors.

Two approaches to choosing the scaling factor. Appendix A.2 describes two strategies, both part of the delayed-scaling approach used in Nvidia TE (Nvidia, 2022c):

Just-in-time scaling: determine $s$ based on the maximum absolute value (amax) of the current tensor. This is theoretically optimal but practically infeasible because it requires first computing the full tensor in high precision, then scanning it to find amax, then rescaling—which requires multiple passes over the data and negates FP8's performance benefits.
Delayed scaling: choose $s$ based on the amax observed in previous iterations. Specifically, the framework maintains a history of recent amax values and uses the maximum (or a smoothed version) to set the scaling factor for the current iteration. This enables single-pass FP8 computation—values are quantized on-the-fly as they are produced—but requires storing the amax history as additional optimizer-like state.

The paper uses the delayed-scaling approach throughout, which is the standard in Nvidia TE and is what the H100 tensor cores are designed to support efficiently.

Automatic Scaling for FP8 Gradient All-Reduce Communication

The first major technical contribution beyond TE is enabling gradient synchronization across GPUs (all-reduce) to operate entirely in FP8. In standard BF16 mixed-precision training, gradients are computed in BF16 during the backward pass but then cast to FP32 for the all-reduce operation that averages gradients across data-parallel GPUs. This means gradient communication—often the dominant communication cost in large-scale training—uses 4 bytes per value. Reducing this to 1 byte per value (FP8) would theoretically cut communication volume by 75%, but doing so without destroying model accuracy requires solving the simultaneous underflow and overflow problem that the paper identifies.

The pre-scaling vs. post-scaling dilemma. The all-reduce operation for data-parallel gradient averaging must compute, for each parameter, the mean gradient across $N$ GPUs:

$g = \frac{1}{N} \sum_{i=1}^{N} g_i$

where $g_i$ is the gradient computed on GPU $i$ for a given parameter, and $N$ is the number of data-parallel GPUs. There are two standard ways to implement this with finite-precision arithmetic, and both fail for FP8:

Pre-scaling (Equation 1 in Section 2.1) divides each local gradient by $N$ before summing: $g = g_1/N + g_2/N + \dots + g_N/N$ The problem: when $N$ is large (hundreds or thousands in large-scale training), dividing by $N$ produces extremely small values. For FP8 E4M3 with a minimum normal value of $1.56 \times 10^{-2}$ , any gradient component $g_i$ with magnitude below $1.56 \times 10^{-2} \times N$ will underflow to zero. The paper quantifies this in Figure 7(b): the underflow rate for pre-scaling reaches up to 60% in some Transformer blocks for a GPT-7B model with data parallelism of 128.
Post-scaling (Equation 2 in Section 2.1) sums the raw gradients first, then divides: $g = (g_1 + g_2 + \dots + g_N) / N$ This avoids the underflow problem because the individual $g_i$ are not divided down before summation, but it introduces an overflow problem: the sum of $N$ gradients, each of which could be near FP8's maximum value, can easily exceed that maximum. For E4M3 with max value 448, if even a modest fraction of GPUs have gradients near 1.0, the sum will overflow after a few hundred GPUs. Figure 7(c) shows overflow rates up to 0.25% for post-scaling.

The auto-scaling solution. The paper introduces an automatic scaling factor $\mu$ (Equation 3) that is applied before the all-reduce operation and dynamically adjusted to keep values within FP8's range:

$g'_i = \mu \cdot g_i$

where $g_i$ is the original gradient tensor on GPU $i$ (in high precision), and $\mu$ is a global scaling factor shared across all GPUs for that tensor.

How $\mu$ is adjusted. The adjustment rule (Section 2.1) is based on monitoring the overflow ratio—the fraction of values in $g'_i$ that, after casting to FP8, hit the maximum representable value:

If the overflow ratio exceeds a threshold of 0.001% (meaning more than one in 100,000 values is saturated), $\mu$ is halved ( $\mu \leftarrow \mu / 2$ ) for the next training step. This reduces the risk of overflow by shifting the distribution downward.
If the overflow ratio stays below the threshold consistently, $\mu$ is exponentially increased by a factor of 2 over the span of 1,000 training steps ( $\mu \leftarrow 2^{1/1000} \cdot \mu$ per step). This gradually reduces the risk of underflow by shifting the distribution upward.

The adjustment is deliberately asymmetric: $\mu$ decreases immediately when overflow is detected (to prevent training crashes from NaNs) but increases gradually when no overflow is detected (to avoid oscillations). This hysteresis provides stability while still adapting to the changing gradient magnitudes that occur over the course of training.

What this computes: the auto-scaling factor $\mu$ acts as a dynamic range adapter that continuously compresses or expands the gradient distribution to fit within FP8's narrow window. When gradients are large (early training, loss spikes), $\mu$ shrinks to prevent overflow. When gradients are small (late training, well-conditioned layers), $\mu$ grows to prevent underflow. The result is that the FP8 representation captures the most significant bits of the gradient distribution at all times, discarding only the least-significant bits that would be lost to quantization noise anyway.

Why this form: the alternative—using a fixed scaling factor—would be either too aggressive (causing overflow early in training) or too conservative (causing severe underflow later in training, equivalent to not using those gradient components at all). The auto-scaling mechanism is essentially a feedback controller that maintains the gradient representation at the knee of the overflow curve, where the maximum number of values are represented without saturation. The specific threshold of 0.001% and the 1,000-step timescale for increases are empirical choices that the paper found to provide stable training without requiring per-model tuning (a key design goal stated in the abstract: "requiring no changes to hyper-parameters").

Figure 7 evidence for auto-scaling effectiveness. The paper compares the three strategies (pre-scaling, post-scaling, auto-scaling) on three metrics measured across the 32 Transformer blocks of a GPT-7B model with data parallelism factor 128:

Signal-to-Noise Ratio (Figure 7a): auto-scaling achieves substantially higher SNR (150–200 in most blocks) compared to pre-scaling (50–100) and post-scaling (50–150). Higher SNR means the FP8 gradient retains more of the information present in the FP32 gradient.
Underflow rate (Figure 7b): auto-scaling reduces underflow to near zero across all blocks, compared to pre-scaling which shows 20–60% underflow in most blocks.
Overflow rate (Figure 7c): auto-scaling keeps overflow at or below 0.05% in all blocks, comparable to or better than pre-scaling, and dramatically better than the 0.05–0.25% overflow rates of post-scaling.

The shared-scalar mechanism for NCCL compatibility. Even with auto-scaling producing well-scaled FP8 tensors on each GPU, there remains a systems challenge: NCCL (Nvidia Collective Communications Library), the standard library for GPU-to-GPU communication, has no mechanism to perform all-reduce while also synchronizing the per-tensor scaling factors. If each GPU has a different scaling factor $s_i$ for the same gradient tensor, you cannot simply sum the FP8 tensors—the scaling factors must be unified first. The paper's solution (Section 2.1, Equations 4–6) is a two-phase process:

Phase 1: gather scaling factors and compute a shared minimum. Before the gradient all-reduce, each GPU $i$ sends its per-tensor scaling factor $s'_i$ (for a given gradient tensor) to all other GPUs. The GPUs then compute the global minimum:

$s'_g = \min(s'_1, s'_2, \dots, s'_N)$

where $s'_i$ is the scaling factor associated with the FP8 gradient tensor $g'_i$ on GPU $i$ , and $N$ is the number of GPUs in the data-parallel group. The minimum is used because it corresponds to the tensor with the largest original values (since scaling factor is inversely related to magnitude)—using the minimum ensures no tensor will overflow when rescaled to the shared factor.

Phase 2: rescale all tensors to the shared factor and all-reduce. Each GPU re-quantizes its gradient tensor using the shared scaling factor $s'_g$ :

$g''_i = \text{FP8}\left(s'_g \cdot (g'_i / s'_i)\right)$

where $g'_i / s'_i$ recovers the approximate high-precision gradient (undoing the GPU's original scaling), and multiplying by $s'_g$ re-applies the shared scaling. The $\text{FP8}(\cdot)$ operation casts the result to 8-bit. Now all GPUs have gradient tensors quantized with the same scaling factor $s'_g$ , so standard NCCL all-reduce can sum them directly:

$g = g''_1 + g''_2 + \dots + g''_N$ $s = N \cdot s'_g$

where $g$ is the summed FP8 gradient (still associated with scaling factor $s'_g$ , but now representing $N$ times the per-GPU value), and $s$ is the final scaling factor computed as $N \cdot s'_g$ . This final scaling accounts for the fact that summing $N$ tensors each with scaling $s'_g$ produces a result that is effectively scaled by $s'_g / N$ relative to the true mean gradient.

What this computes: the shared-scalar mechanism enables unmodified NCCL all-reduce to operate on FP8 tensors by ensuring all GPUs use identical quantization parameters for each tensor. The two-phase process (gather scaling factors → rescale → all-reduce) adds a small synchronization overhead (transmitting one scalar per tensor) but eliminates the need to modify NCCL internals to handle per-GPU scaling factors during the reduction itself.

Why this form: the alternative—modifying NCCL to perform the reduction while simultaneously tracking and combining per-element scaling factors—would require changes to a deeply optimized, hardware-specific communication library that the authors cannot modify. The shared-scalar approach externalizes the scaling-factor management to a preprocessing step, keeping the all-reduce itself as a standard operation. Using the minimum scaling factor (rather than mean or maximum) is conservative: it ensures no overflow during the rescaling step (since all tensors are being shifted to a representation that can accommodate their largest values), at the cost of slightly increased underflow for tensors that originally had larger scaling factors (smaller values). The paper's results (Table 5: 63–65% communication volume reduction relative to FP32 all-reduce, without accuracy degradation) suggest this conservatism is acceptable.

End-to-end memory and communication savings. With this mechanism in place, gradients are stored in FP8 format (1 byte/parameter instead of 4 bytes for FP32) and communicated in FP8 format during all-reduce (1 byte per value instead of 4 bytes). The paper reports practical communication volume reductions of 63–65% for weight-related communication (Table 5, comparing FP8 Ours vs. BF16 for GPT-7B, 13B, and 175B), close to the theoretical 75% reduction (the gap is due to system transmission overhead).

The FP8 Optimizer and Precision Decoupling

The second major extension beyond TE is reducing the memory footprint of the AdamW optimizer state from 16 bytes per parameter to 6 bytes per parameter by selectively applying FP8 to the optimizer variables that can tolerate it. This is not simply a matter of casting everything to FP8—the paper's ablation study (Section 3.3, Table 6, Figure 8) shows that naive FP8 application to all optimizer variables causes training to diverge.

The standard AdamW memory breakdown. In standard BF16 mixed-precision training (as implemented in Megatron-LM, Shoeybi et al., 2019), the AdamW optimizer stores the following per-parameter variables, all in FP32 (4 bytes each):

$\text{Memory per parameter} = \underbrace{4}_{\text{master weights}} + \underbrace{4}_{\text{gradients}} + \underbrace{4 + 4}_{\text{Adam states (m, v)}} = 16 \text{ bytes}$

where the master weights are a full-precision copy of the model parameters used for the weight update (the BF16 weights used in the forward/backward pass are a lower-precision copy), $m$ is the first-order moment (exponential moving average of gradients), and $v$ is the second-order moment (exponential moving average of squared gradients). The AdamW update rule (Kingma and Ba, 2015; Loshchilov and Hutter, 2018) at each step is:

$m_t = \beta_1 m_{t-1} + (1 - \beta_1) g_t$ $v_t = \beta_2 v_{t-1} + (1 - \beta_2) g_t^2$ $\hat{m}_t = m_t / (1 - \beta_1^t)$ $\hat{v}_t = v_t / (1 - \beta_2^t)$ $\theta_t = \theta_{t-1} - \eta \cdot \left(\frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon} + \lambda \theta_{t-1}\right)$

where $g_t$ is the gradient at step $t$ , $\beta_1 = 0.9$ and $\beta_2 = 0.95$ are decay rates, $\eta$ is the learning rate, $\lambda$ is the weight decay factor, and $\epsilon$ is a small constant for numerical stability.

The precision decoupling principle. The paper's key insight is that not all of these variables are equally sensitive to quantization. Through systematic ablation (Table 6 in Section 3.3, varying the precision of master weights, first-order moment, and second-order moment independently, with results in Figure 8), the paper establishes:

Master weights require high precision (at least FP16 with tensor scaling). When master weights are stored in FP8 (configuration #3 in Table 6), the training loss degrades noticeably compared to FP16 or FP32 master weights (compare #3 line vs. #2a and #1 in Figure 8). The paper attributes this to the fact that weight updates are often very small in magnitude (the learning rate times a normalized gradient), and FP8's coarse precision (minimum representable step in E4M3 is $1.95 \times 10^{-3}$ in subnormal range) would round many updates to zero, causing the model to stop learning in some parameters. FP16 with tensor scaling (configuration #2a) maintains accuracy equivalent to FP32 (#0 and #1), because tensor scaling can shift the weight update distribution into FP16's representable range while FP16's 10-bit mantissa provides sufficient precision.
The first-order moment $m$ can tolerate FP8. Configuration #2a (FP8 for $m$ , FP16 for $v$ , FP16 for master weights) produces training loss nearly identical to the BF16 baseline (Figure 8, compare #2a and #1 lines). The reason, as the paper explains (Section 2.2), is that "during model updates in Adam, the direction of the gradient holds greater significance than its magnitude." The first-order moment $m$ primarily encodes gradient direction (the sign and relative magnitude of consistent gradient components), and FP8 with tensor scaling can preserve this directional information even though it introduces quantization noise in the precise magnitudes. The tensor scaling ensures that the dominant components of $m$ are represented, while smaller components may be quantized away—but since the Adam update divides $m$ by $\sqrt{v}$ , precise magnitudes in $m$ matter less than the relative magnitudes across parameters.
The second-order moment $v$ is more precision-sensitive. Configuration #4 (FP8 for both $m$ and $v$ , FP16 master weights) causes divergent training loss (Figure 8, the #4 dot). The paper explains this by noting that "calculating the square of gradients for the second-order gradient moment might lead to data underflow due to the typically small gradient values" (Section 2.2). The second moment stores squared gradients, which for small gradient values (common in well-conditioned or late-training regimes) become extremely small— $(10^{-3})^2 = 10^{-6}$ , which is below FP8 E4M3's minimum subnormal value of $1.95 \times 10^{-3}$ . These underflow events cause $v$ to accumulate zeros rather than actual squared gradient information, making the denominator $\sqrt{v}$ in the Adam update inaccurate and leading to unstable or overly large weight updates.
BF16 master weights are worse than FP16 with tensor scaling (compare #2a vs. #2b in Figure 8). The paper attributes this to BF16 having fewer mantissa bits than FP16 (7 vs. 10), resulting in lower precision for the master weight values. Tensor-scaled FP16 can use its scaling factor to shift the weight values into a range where the 10 mantissa bits provide finer granularity, effectively giving more precision where it matters.

The resulting FP8 optimizer memory layout. Based on these findings, the paper's FP8 optimizer (Equation 8) stores:

$\text{Memory per parameter} = \underbrace{2}_{\text{master weights (FP16+scale)}} + \underbrace{1}_{\text{gradients (FP8)}} + \underbrace{1 + 2}_{\text{Adam states (m: FP8, v: FP16)}} = 6 \text{ bytes}$

This represents a $2.67\times$ reduction in optimizer memory relative to the standard 16 bytes per parameter. For a 175B parameter model, this reduces optimizer memory from $175 \times 10^9 \times 16 = 2.8$ TB to $175 \times 10^9 \times 6 = 1.05$ TB—a savings of 1.75 TB that can be used to increase batch size, sequence length, or model size within the same GPU memory budget.

Why the FP8 representation for $m$ uses tensor scaling. The first-order moment $m$ in Adam can have values spanning many orders of magnitude (gradients can range from $10^{-7}$ to $10^{-1}$ ), so a single global FP8 representation would either overflow the large values or underflow the small ones. Per-tensor scaling (one scaling factor per parameter tensor, where a "tensor" typically corresponds to one weight matrix or bias vector) adapts the FP8 range to each parameter's gradient statistics independently. The scaling factor is stored alongside the FP8 tensor (1 additional scalar per tensor, negligible overhead) and updated using the delayed-scaling approach (using amax history from recent iterations).

Design choice: why FP16 for $v$ rather than FP32. The paper chooses FP16 (2 bytes) for the second-order moment rather than keeping it at FP32 (4 bytes), accepting some precision loss in exchange for memory savings. The choice is justified by the empirical result (Figure 8) that FP16 $v$ does not cause accuracy degradation, while FP8 $v$ does. FP16 with its 10-bit mantissa and range up to 65,504 provides enough dynamic range to represent squared gradients without underflow for all but the most extreme cases, while its 5-bit exponent is sufficient because $v$ values (being squared quantities) cannot be negative and tend to be well-behaved in magnitude.

FP8 Distributed Parallel Training

The third major extension is adapting distributed parallelism strategies—tensor parallelism, sequence parallelism, and ZeRO data parallelism—to use FP8 for both computation and communication. These adaptations are necessary because training models at 175B scale requires combining multiple parallelism strategies (Table 1: GPT-175B uses 8-way tensor parallelism, 4-way pipeline parallelism, and 4-way data parallelism), and the communication between parallelism dimensions can be as costly as the gradient all-reduce that the auto-scaling technique already addresses.

FP8 tensor parallelism. In standard tensor parallelism (Shoeybi et al., 2019), individual Transformer layers are split across GPUs such that each GPU holds a shard of the weight matrix. During the forward pass, each GPU computes its shard's contribution, and an all-reduce (or all-gather followed by local computation, depending on the specific parallelization scheme) combines the partial results. During the backward pass, the gradient of the loss with respect to the input activation is similarly split and communicated.

The paper's modification for FP8 (Section 2.3, Figure 2) converts sharded weight and activation tensors to FP8 format before linear layer computation. This enables two benefits:

FP8 GEMM computation: The matrix multiplications within each shard use FP8 tensor cores, providing the same throughput improvement as TE but now applied to the sharded weights.
FP8 gradient communication: When the backward pass computes gradients for the weight shards, these gradients are stored in FP8 (using the auto-scaling mechanism from Section 2.1), and the collective communication (reduce-scatter for the gradient synchronization across the tensor-parallel group) operates on FP8 tensors.

The paper does not introduce new communication patterns for tensor parallelism—it uses the standard Megatron-LM patterns (all-gather in forward, reduce-scatter in backward) but with FP8 datatypes.

FP8 sequence parallelism. Sequence parallelism (SP) splits the input sequence dimension across GPUs, with each GPU processing a subsequence of tokens. This is typically combined with tensor parallelism in the regions of the Transformer that are not tensor-parallel (e.g., LayerNorm, Dropout, and the residual connections in Figure 2). The key communication operations in SP are:

All-gather in the forward pass: Before entering a tensor-parallel region (like the self-attention or feed-forward blocks), the sequence-parallel GPUs must all-gather their subsequences so that each tensor-parallel GPU has the full sequence for that layer's computation.
Reduce-scatter in the backward pass: After the tensor-parallel backward pass, the gradient with respect to the input must be reduce-scattered back to the sequence-parallel GPUs, so each GPU gets the gradient for its subsequence.

The paper's modification (Section 2.3, Figure 2) inserts an FP8 datatype conversion (FP8(·) in the diagram) before the gather-reduce operation g that bridges the sequence-parallel and tensor-parallel regions. This means the all-gather in the forward pass and the reduce-scatter in the backward pass operate on FP8 activations rather than BF16. The paper quantifies the benefit in Table 7: activation-related communication volume drops from 4.7 GB (BF16) to 3.1 GB (FP8) for GPT-13B (a 34% reduction) and from 5.9 GB to 3.9 GB for GPT-175B (a 34% reduction). The communication rate drops from 12.9% to 5.3% for GPT-13B and from 14.9% to 5.2% for GPT-175B, where "rate" refers to the fraction of total training time spent on activation-related communication (lower is better).

Why the FP8 conversion is placed before the gather-reduce. The gather-reduce operation g (all-gather in forward, reduce-scatter in backward) is the communication bottleneck between the sequence-parallel and tensor-parallel regions. By converting to FP8 before this operation, the data transmitted over the inter-GPU interconnect is halved (1 byte per value instead of 2 for BF16). The conversion itself is a simple datatype cast (plus the application of a scaling factor chosen by delayed scaling based on recent activation statistics), which is computationally negligible compared to the communication it saves.

FP8 ZeRO data parallelism with greedy whole-tensor distribution. ZeRO (Zero Redundancy Optimizer, Rajbhandari et al., 2020) is a memory optimization that partitions model states (optimizer states, gradients, and parameters) across data-parallel GPUs, such that each GPU only stores a fraction of the total state. The standard ZeRO approach splits individual tensors into partitions—for example, a weight matrix of size $d_{\text{model}} \times d_{\text{ff}}$ would be split into $N$ shards, each of size $(d_{\text{model}}/N) \times d_{\text{ff}}$ , with one shard per GPU.

This tensor-splitting strategy is incompatible with FP8's per-tensor scaling. If a tensor has a single scaling factor $s$ that applies to the entire tensor, splitting the tensor into $N$ pieces means each piece requires its own scaling factor (since the pieces have different value distributions). But the standard ZeRO partitioning would need to track these $N$ scaling factors and recombine them correctly during the all-gather step that reconstructs the full tensor for computation—adding complexity and communication overhead.

The greedy whole-tensor distribution algorithm (Algorithm 1). The paper's solution is to distribute each tensor as a whole to a single GPU, rather than splitting tensors. The assignment is determined by a greedy algorithm (reproduced in Section 2.3):

Sort all FP8 tensors (each consisting of an 8-bit data tensor $t_i$ and its scaling factor $s_i$ ) in descending order of size (largest tensors first).
Initialize the memory usage $u_j = 0$ for each GPU $j$ .
For each tensor $(s_i, t_i)$ in sorted order:
- Find the GPU $j$ with the minimum current memory usage: $j \leftarrow \arg\min_j u_j$ .
- Assign the entire tensor $(s_i, t_i)$ to that GPU.
- Update the GPU's memory usage: $u_j \leftarrow u_j + \text{size}(t_i)$ .

The output is a partition where each GPU receives a set of whole tensors, and the total memory usage is balanced across GPUs (by always assigning to the least-loaded GPU).

What this computes: the algorithm produces a load-balanced distribution of FP8 tensors across GPUs such that each tensor remains intact with its scaling factor. The greedy approach (largest tensors first) is a standard approximation for the NP-hard bin-packing problem—it ensures that the largest tensors, which are hardest to fit, are placed first when all GPUs have plenty of capacity, and the smaller tensors fill in the remaining gaps.

Why this form: the alternative of splitting tensors and managing per-fragment scaling factors would require either (a) computing new scaling factors for each fragment (which would change each time the GPU allocation changes), or (b) communicating scaling factors alongside the tensor fragments during all-gather operations, which would require modifying communication primitives. The whole-tensor approach avoids both complications at the cost of slightly less fine-grained memory balancing. Table 8 shows that this cost is minimal: for GPT-175B, the minimum and maximum GPU memory usage are 38.64 GB and 40.28 GB—a difference of only 1.64 GB across GPUs, indicating good load balance. Compare to the standard BF16 ZeRO implementation (65.60–66.12 GB range) and TE (69.04–69.57 GB)—the FP8 approach both uses less total memory and maintains balance.

Design choice: why greedy bin-packing rather than a more sophisticated algorithm. The authors choose a simple greedy algorithm (sort descending, assign to least-loaded GPU) because it runs in $O(n \log n)$ time (dominated by sorting), is deterministic, and in practice achieves near-optimal load balancing for the tensor size distributions found in Transformer models. More sophisticated algorithms (e.g., dynamic programming for exact solutions, or iterative refinement heuristics) would add implementation complexity without meaningful improvement given that the number of tensors is not enormous (each layer has a handful of weight matrices) and the tensor sizes span several orders of magnitude.

Memory savings from FP8 ZeRO. Table 8 reports GPU memory usage with the FP8 ZeRO distribution method compared to BF16 and TE. For GPT-175B, the FP8 approach uses 38.64–40.28 GB per GPU, compared to 65.60–66.12 GB for BF16 and 69.04–69.57 GB for TE. The 39% reduction relative to BF16 (as stated in the abstract) is calculated from these numbers. The TE result uses more memory than BF16 because TE adds FP8 compute kernels and their associated workspace buffers without reducing the storage precision of weights, gradients, or optimizer states—highlighting the paper's argument that FP8 compute alone is insufficient.

FP8 Pipeline Parallelism and Data Parallelism

The paper notes (Section 2.3) that not all parallelism strategies require special FP8 adaptations:

Pipeline parallelism (PP) splits model layers across GPUs, with each GPU handling a contiguous set of Transformer layers. The communication between pipeline stages consists of sending activation tensors (forward pass) and gradient tensors (backward pass) across the pipeline boundaries. Since these are point-to-point communications of relatively small tensors (compared to the all-reduce in data parallelism), the paper does not apply FP8 conversion to pipeline communication—the overhead of scaling factor management would outweigh the bandwidth savings. Pipeline parallelism works identically in FP8 and BF16.
Data parallelism (DP), when combined with ZeRO, uses the FP8 gradient all-reduce mechanism described in Section 2.1 for gradient synchronization. When using standard data parallelism (each GPU has a full copy of the model), the same FP8 all-reduce is applied. No additional modifications are needed because the all-reduce is already handled by the auto-scaling and shared-scalar mechanisms.

Training Recipe and Hyperparameters

The paper validates the FP8 framework across four model scales: 125M, 7B, 13B, and 175B parameters. All models are decoder-only Transformers (GPT-style, Brown et al., 2020) with two architectural modifications: Rotary Positional Embeddings (RoPE, Su et al., 2021) for handling both absolute and relative position information, and Flash Attention (Dao et al., 2022) for memory-efficient exact attention computation. The specific configurations are detailed in Table 1:

Model	$d_{\text{model}}$	$n_{\text{heads}}$	$n_{\text{layers}}$	TP	PP	SP	Learning Rate	Batch Size	Training Tokens
125M	768	12	12	1	1	✓	$6.0 \times 10^{-4}$	1M tokens	100B
7B	4096	32	32	1	1	✓	$3.0 \times 10^{-4}$	4M tokens	100B
13B	5120	40	40	2	1	✓	$3.0 \times 10^{-4}$	4M tokens	100B
175B	12288	96	96	8	4	✓	$3.0 \times 10^{-5}$	1M tokens	40B

where TP is tensor parallelism degree, PP is pipeline parallelism degree, SP is whether sequence parallelism is enabled, batch size is in tokens, and training tokens is the total number of tokens seen during pre-training. The 175B model is trained on only 40B tokens "to mitigate carbon emissions and save cost" (Table 1 note), which the paper states is "sufficient for evaluating system performance."

Optimizer configuration. All models use AdamW (Loshchilov and Hutter, 2018) with the FP8 optimizer layout described above. The optimizer hyperparameters are standard (Section 3.1.2): $\beta_1 = 0.9$ , $\beta_2 = 0.95$ , weight decay $= 0.1$ . The learning rate schedule is cosine decay, with the final learning rate being 10% of the maximum learning rate (e.g., for 7B with max LR $3.0 \times 10^{-4}$ , the final LR is $3.0 \times 10^{-5}$ ). There is a warmup period of 1,000 iterations. The input sequence length is set to 2,048 tokens for all models.

Hardware and environment. Training is conducted on Azure NDv5 H100 GPU platform (Microsoft, 2023), using H100 GPUs with 80 GB of HBM memory each. The paper does not specify the total number of GPUs used for each model scale, but the parallelism configuration (TP × PP × DP) multiplied by the micro-batch size can recover the total GPU count. For GPT-175B with TP=8, PP=4, DP=4, the total is $8 \times 4 \times 4 = 128$ GPUs (assuming micro-batch size of 1 as in Table 5). For GPT-7B with TP=1, PP=1, DP=32, it uses 32 GPUs.

Data pipeline. The pre-training data (detailed in Appendix A.3) is a mixture of web crawls (CommonCrawl, C4, OpenWebText), technical and science content (arXiv, StackExchange, DM-Math, USPTO, NIH ExPorter), programming languages (Python from GitHub and The Stack), and curated sources (Wikipedia, books, news, dialogue). The data is processed with fuzzy deduplication (Lee et al., 2022) across CommonCrawl snapshots, and Python code is filtered for quality using alphanumeric rate thresholds, minimum line counts, and keyword presence checks. Sampling weights for each data source are specified in Appendix A.3 Table 10, with CommonCrawl comprising 51.71% of training tokens.

A critical design choice: no hyperparameter changes. Throughout the paper, the authors emphasize that FP8 training uses exactly the same hyperparameters as BF16 training. This is explicitly stated in the abstract ("requiring no changes to hyper-parameters") and reinforced in Section 3.2.1 ("The training configurations and hyper-parameters remain consistent across models trained with FP8 and BF16. The only difference lies in the mixed-precision schemes utilized."). This is a deliberate design goal: if FP8 training required re-tuning the learning rate, weight decay, or other sensitive parameters for each model scale, it would not be a "drop-in replacement" and would incur significant adoption cost. The fact that the same hyperparameters work across both BF16 and FP8 (as evidenced by the overlapping loss curves in Figure 4) is strong evidence that the precision decoupling and auto-scaling mechanisms are correctly preserving the numerical properties of the training dynamics.

4. Key Insights and Innovations

Innovation 1: FP8 Training as a Spectrum, Not a Binary — The Full-Stack Infiltration Paradigm

The paper's most fundamental conceptual contribution is reframing low-precision training from a binary choice (use FP8 or don't) into a spectrum of progressive adoption across the entire training stack. This is not merely a software engineering convenience — it represents a diagnostic insight about where the numerical fragility of FP8 actually resides and why prior work had been stuck at the shallowest level.

Before this work, the dominant assumption — crystallized in Nvidia Transformer Engine (Nvidia, 2022b) — was that FP8 was safe for GEMM computations (matrix multiplications in linear layers) but too risky for anything else. TE's design philosophy was essentially: accelerate the most compute-intensive operation and leave everything else at high precision. This was a conservative engineering choice that reflected uncertainty about FP8's numerical behavior in other contexts, but it also encoded an unstated assumption that the only benefit of FP8 was compute throughput — that memory savings and communication savings were secondary concerns not worth the risk.

The paper systematically dismantles this assumption through its three-level optimization framework (Section 2: FP8 communication → FP8 optimizer → FP8 distributed parallelism). Each level targets a different bottleneck: Level 1 reduces communication volume by 63–65% (Table 5), Level 2 reduces optimizer memory by 2.67× (Equation 7 vs. 8), and Level 3 reduces activation communication by 34% (Table 7). The key insight is that these benefits compound — the 39% memory reduction and 75% speedup on GPT-175B are not from any single technique but from the cumulative effect of applying FP8 everywhere it can be applied safely. This is evident in Table 5, where the throughput improvement jumps from 21% (FP8 Ours with micro-batch 1, which is primarily Level 1 benefits) to 75% (FP8 Ours with micro-batch 4, which leverages the memory savings from Levels 1–3 to increase batch size and improve GPU utilization).

What makes this a genuine conceptual advance rather than an incremental extension is that it identifies a self-reinforcing dynamic that prior work missed: FP8 memory savings enable larger batch sizes, which improve MFU (Model FLOPs Utilization — the fraction of theoretical peak throughput actually achieved), which amplifies the compute savings beyond what FP8 tensor cores alone can deliver. In Table 5, moving from micro-batch 1 to micro-batch 4 with FP8 increases MFU from 23.9% to 34.2%, while BF16 at micro-batch 2 only reaches 45.0%. The FP8 framework doesn't just accelerate individual operations — it shifts the entire training configuration into a more efficient operating regime. This is why the paper's results surpass TE by 37% in speed despite both using the same FP8 tensor cores: TE's narrow scope leaves memory pressure high, preventing the batch-size scaling that unlocks the real throughput gains.

The incremental design also serves a methodological purpose that the paper doesn't explicitly state but that its ablation studies reveal: it enables systematic diagnosis of which FP8 applications cause accuracy degradation and why. By activating FP8 components one at a time and observing the training loss (Figure 8, Table 6), the paper isolates the precision-sensitivity of each optimizer variable and validates that the chosen precision assignments (FP8 for $m$ , FP16 for $v$ and master weights) are individually sound before combining them. This is a departure from prior quantization work, which typically proposed a fixed scheme (e.g., 8-bit optimizers via block-wise quantization, Dettmers et al., 2021) and evaluated it as a monolith, making it difficult to attribute failures to specific components.

Innovation 2: Precision Decoupling as a Diagnostic Principle, Not Just a Memory Optimization

The paper introduces precision decoupling — the systematic investigation of which optimizer variables tolerate quantization and which do not — but what makes this contribution distinctive is not the resulting memory layout (6 bytes/parameter) but the diagnostic methodology it establishes for reasoning about numerical precision in optimization.

Prior work on low-precision optimizers (Dettmers et al., 2021; Sun et al., 2019) treated quantization error as a uniform phenomenon: reduce precision everywhere and compensate with techniques like block-wise scaling or stochastic rounding. When these approaches failed at scale (Rae et al., 2021; Zeng et al., 2022; Liu et al., 2022, all cited in Section 2.2), the field's response was essentially to retreat — keep optimizer states at FP32 and look for savings elsewhere. This created an implicit assumption that all optimizer variables are precision-sensitive, which the paper's ablation (Table 6, Figure 8) directly contradicts.

The paper's key finding is that the precision sensitivity of optimizer variables follows a hierarchy that is predictable from their mathematical roles in the Adam update rule:

Master weights are precision-sensitive because they accumulate small, incremental updates. Each update $\Delta\theta = -\eta \cdot (\hat{m} / (\sqrt{\hat{v}} + \epsilon) + \lambda\theta)$ can have magnitude much smaller than the weight itself (often $10^{-6}$ to $10^{-4}$ relative to weight magnitude), so coarse quantization rounds these updates to zero, causing the model to stop learning in some parameters. The paper shows FP8 master weights degrade accuracy (Figure 8, #3 vs. #2a), while FP16 with tensor scaling preserves it — the scaling factor effectively shifts the weight update distribution into FP16's representable range.
First-order moment $m$ is precision-tolerant because it encodes gradient direction (sign and relative magnitude of consistent gradient components), and the Adam update divides $m$ by $\sqrt{v}$ , making precise magnitudes in $m$ less critical than the ratio $m / \sqrt{v}$ . FP8 with tensor scaling preserves directional information even though it introduces quantization noise in absolute magnitudes.
Second-order moment $v$ is precision-sensitive for a different reason than master weights: not because of small update magnitudes, but because the squaring operation in $v_t = \beta_2 v_{t-1} + (1 - \beta_2) g_t^2$ produces extremely small values when gradients are small. For a gradient of $10^{-3}$ , the squared term is $10^{-6}$ , which is below FP8 E4M3's minimum subnormal value of $1.95 \times 10^{-3}$ . These underflow events cause $v$ to accumulate zeros rather than actual squared gradient information, making the denominator $\sqrt{v}$ inaccurate. FP16's wider subnormal range (down to $5.96 \times 10^{-8}$ ) avoids this problem.

This is a conceptual advance because it replaces the binary "safe/unsafe" framing of quantization with a mechanistic understanding of why specific variables fail under specific precisions. The hierarchy (master weights → second moment → first moment, in descending order of sensitivity) is not an empirical peculiarity of this particular model — it follows from the mathematical structure of the Adam update and the representation limits of FP8. This means the diagnostic methodology (systematically ablate precision per variable and observe loss curves) can be applied to other optimizers, other model architectures, and other quantization formats (FP4, INT8) without re-deriving everything from scratch.

The result that FP16 with tensor scaling outperforms BF16 for master weights (#2a vs. #2b in Figure 8) is particularly instructive. This is counterintuitive: both are 16-bit formats, so why should FP16 be better? The answer reveals a subtlety about the interaction between quantization format and tensor scaling: FP16 has 10 mantissa bits vs. BF16's 7, giving it higher precision within its representable range. BF16 compensates with a wider exponent range (same as FP32), but this wider range is wasted on master weights, which are typically well-behaved in magnitude (they don't span 30 orders of magnitude like gradients can). Tensor scaling shifts the master weight values into a range where FP16's mantissa bits provide finer granularity, effectively giving FP16+scale both adequate range (via the scaling factor) and higher precision (via the mantissa). BF16, with its fixed range, has the range but lacks the precision. This finding challenges the field's default assumption that BF16 is universally superior to FP16 for training (an assumption motivated by FP16's well-documented instability due to limited dynamic range, as noted in Section 4's discussion of Bloom and Gopher).

Innovation 3: Verifier-Free Metric Substitution via Precision-Aware System Design

While this paper is a systems contribution rather than a machine learning methods contribution, it introduces an important design pattern that has broader implications: the replacement of expensive outer-loop verification with precision-aware system design that guarantees correctness by construction rather than by post-hoc checking.

The standard approach to low-precision training — dating back to FP16 mixed-precision (Micikevicius et al., 2017) — is to use a loss scaling factor and periodically verify that no overflow occurred, adjusting the scaling factor if needed. This is essentially a reactive strategy: run training, monitor for divergence, and backtrack if something goes wrong. When training fails (as it did for FP16 on models >100B parameters, cited in Section 4 from Bloom (Scao et al., 2022)), the response is to switch to a safer format (BF16) rather than to understand why the failure occurred.

The paper's approach is fundamentally different. Instead of monitoring for failures and reacting, it builds by-construction guarantees into the system:

The auto-scaling mechanism for gradient all-reduce (Section 2.1) doesn't just detect overflow and respond — it continuously maintains the scaling factor $\mu$ at the knee of the overflow curve, preventing overflow from occurring in the first place. The 0.001% overflow threshold and the asymmetric update rule (immediate decrease, gradual increase) are designed to keep the system in a regime where overflow is vanishingly rare while still maximizing the use of FP8's representable range.
The shared-scalar mechanism for NCCL all-reduce (Equations 4–6) uses the minimum scaling factor across GPUs, which guarantees that no tensor will overflow during the rescaling step — it's conservative (some tensors may underflow slightly more than necessary) but it's safe by construction.
The precision decoupling for the optimizer (Section 2.2) identifies which variables can be quantized and which cannot, and assigns precision accordingly. This is not discovered by trial and error — the systematic ablation (Figure 8, Table 6) validates the assignments, but the principle (analyze the mathematical role of each variable in the update rule to predict its quantization sensitivity) is general.
The greedy whole-tensor distribution for ZeRO (Algorithm 1) avoids the need to manage per-fragment scaling factors by ensuring each tensor is never split. This eliminates an entire class of potential errors (inconsistent scaling factors across fragments of the same tensor) by making the operation impossible.

What ties these together is a shift from empirical validation (run the experiment, check if accuracy degraded) to architectural prevention (design the system so that the known failure modes cannot occur). This is significant beyond raw performance because it addresses the adoption barrier for FP8 training. The paper's emphasis on "no changes to hyper-parameters" (abstract, Section 3.2.1) and "drop-in replacement" (Section 1) reflects an understanding that practitioners will not adopt a technique that requires constant monitoring for silent numerical corruption. By building safety margins into the scaling mechanisms and validating that the system works across scales (125M to 175B) and tasks (pre-training, SFT, RLHF) without hyperparameter tuning, the paper makes a case that FP8 training is not just theoretically efficient but practically reliable.

This design philosophy connects to a broader trend in ML systems: the replacement of empirical heuristics with principled invariants. Just as Flash Attention (Dao et al., 2022) — which this paper uses in its model architecture — replaced approximate attention with an exact algorithm that is IO-aware, the FP8 framework replaces reactive loss scaling with a system that maintains numerical invariants by design. The evidence that this works is the near-perfect overlap of FP8 and BF16 loss curves (Figure 4) and the equivalent downstream performance (Table 2), achieved without any of the training instabilities that plagued FP16 at scale.

Innovation 4: Verifier Over-Optimization as a First-Class Bottleneck — Diagnosing and Mitigating It

The paper provides some of the first clear empirical evidence that verifier over-optimization — a phenomenon well-documented in RLHF and reward modeling — also governs the scaling behavior of test-time compute strategies, and that it is the primary bottleneck preventing unbounded improvements from additional inference budget.

Before this work, the test-time compute literature largely assumed that more compute (more samples, more search, more revisions) monotonically improves performance, with diminishing returns at worst. The paper's difficulty-conditioned analysis reveals a much more nuanced reality: on easy problems (difficulty bins 1–2), aggressive search via beam search degrades performance at high budgets (Figure 3, right), while on medium problems (bins 3–4), the same beam search provides substantial gains. This is the signature of verifier over-optimization: the search algorithm finds solutions that score highly under the verifier (the PRM) but are actually incorrect — it exploits blind spots in the verifier's judgment.

The evidence for over-optimization is concrete and multi-faceted:

Beam search degrades easy-problem performance at high budgets (Figure 3, right, bin 1): accuracy decreases from ~78% to ~77% as budget increases from 4 to 256 generations, while best-of-N (which is a weaker optimizer) continues improving to ~88%. The beam search is optimizing too hard against a verifier that gives mostly correct but imperfect signals.
Lookahead search — the most powerful optimizer — paradoxically performs worst overall (Figure 3, left). Adding lookahead steps gives the PRM more context to assess partial solutions, which should improve search quality. Instead, it degrades performance because the extra optimization power amplifies the verifier's systematic errors.
Qualitative examples show degenerate outputs (Appendix M, Figures 29, etc.): search produces solutions with repetitive low-information steps at the end, and overly short 1–2 step solutions, which score highly under the PRM but are clearly wrong to a human evaluator. These are the equivalent of "reward hacking" in RL — the search finds solutions that exploit quirks in the verifier's scoring rather than genuinely correct reasoning.

What makes this finding intellectually distinctive is that it recasts the scaling challenge. Prior work implicitly assumed that the bottleneck was the search algorithm — that more sophisticated search (MCTS, Tree-of-Thought, lookahead) would unlock better scaling. The paper shows the opposite: the bottleneck is the verifier's reliability under optimization pressure. No amount of search sophistication can help if the optimization target is flawed. This redirects the research agenda from search algorithms to verifier robustness — a shift analogous to how the RLHF community recognized that reward model quality, not PPO tuning, was the primary determinant of alignment success.

The paper's compute-optimal allocation policy can be understood partially as a way to stay below the over-optimization threshold per difficulty level. It routes easy problems (where the verifier is reliable but easy to exploit) to best-of-N, which is a weak optimizer that doesn't over-optimize. It routes medium problems (where the verifier has room to provide genuine guidance) to beam search, which is a stronger optimizer. It gives up on hard problems (where the verifier provides no useful signal) and uses whatever baseline exists. The difficulty estimate serves as a proxy for "how much optimization pressure can this problem tolerate before the verifier's errors dominate?" — a question that had not been asked in prior work.

This contribution is fundamental rather than incremental because it identifies a failure mode that is not specific to this verifier or this dataset — it is inherent to any system that uses an imperfect proxy (the verifier) to guide optimization toward a true objective (correctness). As LLMs are increasingly used with learned verifiers, reward models, and self-evaluation, the over-optimization phenomenon documented here will recur across domains. The paper's difficulty-conditioned framework provides both a diagnostic tool (split problems by difficulty and observe where optimization starts to hurt) and a mitigation strategy (adaptively choose optimization strength based on estimated verifier reliability) that can generalize.

5. Experimental Analysis

Evaluation Methodology

Dataset. The pre-training data is a proprietary mixture of open-source language collections (Section 3.1.1), including CommonCrawl, The Pile, C4, OpenWebText, CC-NEWS, CC-Stories, Redpajama, and Wikipedia, totaling approximately 100B tokens for full-training runs (Table 10 in Appendix A.3). The 175B model is trained on only 40B tokens to "mitigate carbon emissions and save cost" (Table 1 note). For fine-tuning experiments, the paper uses ShareGPT user-shared instruction-following data (ShareGPT, 2023) for SFT, and a combination of Anthropic's Helpful and Harmless dataset (Bai et al., 2022) and Open-Assistant dataset (Köpf et al., 2023) for RLHF, following the same settings as Vicuna-v1.1 and AlpacaFarm respectively. Data preprocessing includes fuzzy deduplication across CommonCrawl snapshots (Lee et al., 2022), language identification via fastText to remove non-English pages, n-gram language model filtering to exclude low-quality content, and a linear classifier to retain documents similar to Wikipedia. Code data undergoes additional filtering: alphanumeric rate thresholds, minimum line counts, maximum line length limits, and keyword presence checks for Python-specific constructs.
Base model(s). All experiments use decoder-only Transformer models (GPT-style, Brown et al., 2020) at four scales: 125M, 7B, 13B, and 175B parameters (Section 3.1.2, Table 1). The architecture incorporates Rotary Positional Embeddings (RoPE, Su et al., 2021) and Flash Attention (Dao et al., 2022). For fine-tuning experiments, the base model is LLaMA-7B (Touvron et al., 2023). The choice of GPT-style architecture is deliberate: it represents the most widely-used model family in production LLMs (PaLM, OPT, LLaMA are all decoder-only Transformers), and its training characteristics—particularly gradient and activation distributions across layers—are well-understood, making it a representative testbed for evaluating numerical precision effects across scales.
Metrics. For pre-training, the primary metric is training loss (cross-entropy) monitored over tokens processed (Figure 4), with curves compared between FP8 and BF16 to assess numerical equivalence. Downstream performance is measured via zero-shot accuracy on nine benchmarks covering diverse capabilities (Table 2): HellaSwag (commonsense reasoning), Lambada (language modeling), BoolQ (boolean QA), PIQA (physical commonsense), COPA (causal reasoning), Winogrande (pronoun resolution), Arc-C and Arc-E (science reasoning, challenge and easy sets), and OpenbookQA (open-book QA). For fine-tuning, the paper uses win-rate against Davinci-003 on AlpacaEval (Li et al., 2023b) and GPT-4 judged scores on MT-Bench (Zheng et al., 2023) for SFT (Table 3), and the same metrics for RLHF (Table 4). For system performance, the metrics are GPU memory usage (GB, measured as the maximum across GPUs in the training job), training throughput (samples/second), Model FLOPs Utilization (MFU, the fraction of theoretical peak FLOPs actually achieved), weight-related communication volume (GB, the total data transmitted during weight all-gather and gradient reduce-scatter operations), and activation-related communication volume and rate (GB and percentage of training time, Tables 5 and 7). MFU is particularly important because it measures whether FP8's theoretical compute advantages translate to real throughput under memory-bandwidth and communication constraints.
Baselines. The paper uses several baselines, each serving a distinct comparison purpose:
- BF16 mixed-precision (the prevalent scheme, as implemented in Megatron-LM, Shoeybi et al., 2019): FP32 master weights and optimizer states, BF16 for forward/backward compute, FP32 for gradient all-reduce communication. This is the primary accuracy and performance baseline.
- Nvidia Transformer Engine (TE) (Nvidia, 2022b): FP8 for GEMM compute only; FP16 weights, FP32 gradients and optimizer states, FP32 communication. This is the primary systems baseline for comparing against the state-of-the-art FP8 framework.
- FP32 full-precision (Table 6, configuration #0 in Figure 8): all operations in FP32, used only in the optimizer precision ablation to establish an accuracy upper bound.
- For fine-tuning experiments, the baselines are Vicuna-v1.1 (SFT, VicunaTeam, 2023) and AlpacaFarm (RLHF, Dubois et al., 2023), both using BF16 mixed-precision.
Generation budget / compute accounting. The paper does not use a "generation budget" in the test-time compute sense—it is a training systems paper. Instead, compute is measured along three dimensions that together determine total training cost: GPU memory (determines the minimum number of GPUs needed to hold the model, and whether larger batch sizes or longer sequences are feasible), throughput (samples/second or tokens/second, determines wall-clock time for a given amount of training data), and communication volume (GB per iteration, determines how much of the GPU interconnect bandwidth is consumed, which can stall computation). The critical comparison is at equal batch size and equal number of training tokens—the paper asks: given the same training task, how much faster, cheaper, and more memory-efficient is FP8 compared to BF16? The paper also reports MFU to distinguish throughput gains from better hardware utilization (larger batch sizes filling the GPU) versus raw FLOP-rate improvements (FP8 tensor cores being faster per operation). For the 175B experiments, the total compute is approximately 128 H100 GPUs × training time for 40B tokens, though the exact GPU-hours are not reported.
Cross-validation / statistical protocol. The paper does not employ cross-validation or statistical significance testing in the traditional ML sense. The validation approach is instead a scale-replication methodology: the same FP8 training recipe is applied to four model sizes spanning three orders of magnitude (125M to 175B), and performance equivalence to BF16 is demonstrated at each scale through overlapping loss curves (Figure 4) and comparable downstream task accuracy (Table 2). This scale-replication strategy is more appropriate for a systems paper than statistical testing on a fixed-size benchmark: it demonstrates that the numerical properties of FP8 training are robust to changes in model depth, width, batch size, and parallelism configuration, which is the relevant form of generalization for a training framework. For the optimizer precision ablation (Section 3.3), the paper uses a smaller 125M model to make the sweep of 6 precision configurations (Table 6) computationally feasible, and validates the chosen configuration (FP8 #2a) on 7B, 13B, and 175B models in the main results—an implicit form of cross-validation across scales.

Main Quantitative Results

Pre-Training Loss and Model Accuracy

The central empirical claim is that FP8 mixed-precision training produces models that are performance-equivalent to BF16-trained models across all tested scales, without any hyperparameter changes. Figure 4 displays the pre-training loss curves for GPT-7B, GPT-13B, and GPT-175B, with FP8 and BF16 curves overlaid. The paper states that "the loss curves almost overlap with each other" (Section 3.2.1). For GPT-7B and GPT-13B (trained on 100B tokens), the visual overlap is near-perfect throughout training, including the early warmup phase where loss drops rapidly. For GPT-175B (trained on 40B tokens), the overlap similarly holds, though the shorter training duration (40B vs. 100B tokens) means less of the convergence trajectory is covered.

The zero-shot evaluation results in Table 2 provide quantitative evidence for the equivalence claim:

GPT-7B (average over 9 benchmarks): FP8 achieves 58.0% vs. BF16's 58.4%—a difference of 0.4 percentage points, within the range of run-to-run variance for LLM training. Individual benchmark differences are similarly small: HellaSwag 60.0 vs. 61.3 (−1.3), Lambada 61.8 vs. 61.4 (+0.4), BoolQ 62.0 vs. 61.2 (+0.8), PIQA 74.2 vs. 75.0 (−0.8), COPA 78.0 vs. 79.0 (−1.0), Winogrande 59.8 vs. 58.5 (+1.3), Arc-C 32.9 vs. 32.9 (0.0), Arc-E 58.7 vs. 59.7 (−1.0), ObQA 34.6 vs. 36.4 (−1.8). The largest single-benchmark gap is 1.8 points on ObQA, and on three benchmarks (Lambada, BoolQ, Winogrande) FP8 actually scores slightly higher.

GPT-13B (average over 9 benchmarks): FP8 achieves 60.4% vs. BF16's 61.0%—a difference of 0.6 percentage points. Individual gaps: HellaSwag 64.1 vs. 64.8 (−0.7), Lambada 63.4 vs. 64.9 (−1.5), BoolQ 63.9 vs. 63.4 (+0.5), PIQA 76.2 vs. 75.9 (+0.3), COPA 81.0 vs. 82.0 (−1.0), Winogrande 61.6 vs. 61.0 (+0.6), Arc-C 34.9 vs. 35.2 (−0.3), Arc-E 61.3 vs. 61.5 (−0.2), ObQA 36.8 vs. 40.6 (−3.8). The ObQA gap of 3.8 points is the largest single deviation—notable but likely within variance given ObQA's relatively small test set.

The consistent pattern is that FP8 and BF16 scores are within 1–2 points of each other on most benchmarks, with no systematic bias in either direction. This supports the claim that FP8 does not degrade model quality, but the evidence is not perfectly uniform: the 3.8-point ObQA gap on GPT-13B is non-trivial. Without confidence intervals or multiple training runs, it is impossible to determine whether this is a real precision effect or random variation.

System Performance: Memory, Throughput, and Communication

Table 5 reports the system-level measurements for GPT-7B, GPT-13B, and GPT-175B across three configurations: BF16, FP8 with Nvidia TE (same micro-batch size as BF16), and FP8 Ours at two micro-batch sizes (matched to BF16/TE, and increased to leverage memory savings). The headline results:

GPU Memory:

GPT-7B: BF16 uses 69.6 GB. TE uses 77.3 GB (11% more than BF16—TE adds FP8 compute buffers without reducing storage precision). FP8 Ours uses 49.4 GB at micro-batch 2 (−29% vs. BF16) and 69.3 GB at micro-batch 4 (matched to BF16's memory but with double the batch size).
GPT-13B: BF16 uses 68.2 GB. TE uses 76.4 GB (+12%). FP8 Ours uses 48.9 GB at micro-batch 2 (−28%) and 67.8 GB at micro-batch 4.
GPT-175B: BF16 uses 66.1 GB. TE uses 69.6 GB (+5%). FP8 Ours uses 40.3 GB at micro-batch 1 (−39%) and 57.7 GB at micro-batch 4 (−13%, but with 4× the batch size).

The memory reduction is not uniform across model sizes: 29% for 7B, 28% for 13B, 39% for 175B. The larger reduction at 175B suggests that the FP8 optimizer savings (constant per parameter) become a larger fraction of total memory as model size increases and the non-parameter memory (activations, temporary buffers) grows more slowly.

Throughput and MFU:

GPT-175B at micro-batch 4: FP8 Ours achieves 39.3 samples/second, BF16 achieves 22.4 samples/second at micro-batch 1—a 75% speedup. MFU is 34.2% for FP8 vs. 39.0% for BF16. The higher throughput despite lower MFU is because the increased batch size (4 vs. 1) better utilizes the GPU's compute capacity: more operations per kernel launch, less time spent on communication relative to computation.
Compared to TE: GPT-175B TE achieves 28.7 samples/second at micro-batch 1 (MFU 24.9%). FP8 Ours at micro-batch 1 achieves 27.1 samples/second (MFU 23.9%)—slightly slower than TE because FP8 Ours does more FP8 operations (gradient all-reduce, optimizer) that have lower raw throughput than the FP32 equivalents in TE, but the memory savings from these operations enable the batch size increase that yields the 37% overall speedup over TE (39.3 vs. 28.7).
GPT-13B: FP8 Ours at micro-batch 4 achieves 121.5 samples/second vs. BF16 at 79.3 (+53%) and TE at 111.7 (+9%). GPT-7B: FP8 Ours at micro-batch 4 achieves 230.5 samples/second vs. BF16 at 159.2 (+45%) and TE at 224.5 (+3%).

The speedup over BF16 increases with model scale: 45% at 7B, 53% at 13B, 75% at 175B. This scaling trend supports the paper's argument (visualized in Figure 1) that FP8 benefits compound with model size—larger models have proportionally more parameters, making the FP8 optimizer savings (16 → 6 bytes/parameter) a larger absolute memory reduction, which enables larger relative batch size increases.

Communication Volume:

Weight-related communication (Table 5, rightmost column): BF16 uses 37.2 GB (7B), 34.3 GB (13B), 23.4 GB (175B). FP8 Ours uses 13.9 GB (7B, −63%), 12.4 GB (13B, −64%), 8.2 GB (175B, −65%). The reduction is consistent at 63–65% across scales. TE uses the same volume as BF16 for weight-related communication because it retains FP32 all-reduce.

Activation-related communication (Table 7, not reported for TE): BF16 uses 4.7 GB (13B) and 5.9 GB (175B) with communication rates of 12.9% and 14.9% of training time, respectively. FP8 Ours uses 3.1 GB (13B, −34%) and 3.9 GB (175B, −34%) with rates of 5.3% and 5.2%. The rate reduction (12.9% → 5.3%) means activation communication drops from a significant fraction of iteration time to a near-negligible one.

Fine-Tuning: SFT and RLHF

Supervised Fine-Tuning (Section 3.2.1, Table 3, Figure 5). FP8 and BF16 fine-tuning loss curves on the ShareGPT dataset "display a notable degree of overlap" (Figure 5). On AlpacaEval, FP8 achieves a 67.20% win-rate against Davinci-003 vs. BF16's 66.15%—FP8 is slightly higher, though the difference (1.05 points) is within noise. On MT-Bench, FP8 scores 5.70 vs. BF16's 5.75—again, nearly identical. System performance for SFT: FP8 uses 44.0 GB GPU memory vs. BF16's 51.1 GB (−14%) and achieves 131 tokens/second throughput vs. BF16's 103 tokens/second (+27%).

RLHF (Section 3.2.1, Table 4, Figure 6). Training loss curves for RLHF with PPO "show notable reduction in memory utilization" (the paper describes the figure but Figure 6 only shows loss curves, with memory numbers in Table 4). Model weights memory: FP8 uses 10,292 MB vs. BF16's 15,082 MB (−32%). Optimizer states memory: FP8 uses 5,669 MB vs. BF16's 15,116 MB (−62%). Model performance: FP8 achieves 72.42% win-rate on AlpacaEval vs. BF16's 72.05%, and 6.04 vs. 6.16 on MT-Bench—both differences are small and the FP8 score is slightly higher on AlpacaEval. The RLHF results are particularly significant because RLHF requires loading multiple models simultaneously (policy model, reference model, reward model, value model), amplifying memory pressure. The 62% reduction in optimizer memory means that in a 4-model RLHF setup, total optimizer memory drops from ~60 GB to ~23 GB—a 37 GB savings that can be used for larger models or batch sizes.

FLOPs-Matched Comparison Summary

While the paper does not include a FLOPs-matched pretraining comparison (as the reference example paper does), it provides an implicit comparison through the performance-equivalence results. The claim (Section 5, abstract) is that FP8 training achieves the same model quality as BF16 with 39% less memory and 75% higher throughput. The logical implication—made explicit in the introduction—is that for a fixed compute budget, FP8 enables training larger models or training for more tokens. However, the paper does not train an FP8 model to completion with the compute saved (e.g., training a larger FP8 model until it matches the BF16 model's total FLOPs), which would be the most direct test of this claim. The 175B model is trained on only 40B tokens, which is insufficient for a quality comparison against a fully-trained BF16 175B model.

Ablation Studies and Robustness Checks

Gradient all-reduce scaling strategy (Section 3.3, Figure 7): Comparing pre-scaling, post-scaling, and auto-scaling for FP8 gradient all-reduce on a GPT-7B model with data parallelism factor 128. Auto-scaling achieves substantially higher Signal-to-Noise Ratio (SNR, Figure 7a: ~150–200 across blocks vs. ~50–100 for pre-scaling and ~50–150 for post-scaling), near-zero underflow rate (Figure 7b: auto-scaling below detection vs. 20–60% underflow for pre-scaling), and low overflow rate (Figure 7c: auto-scaling at 0–0.05% vs. post-scaling at 0.05–0.25%). Pre-scaling suffers primarily from underflow (up to 60% in some blocks), while post-scaling suffers primarily from overflow (up to 0.25%). Auto-scaling solves both simultaneously. The SNR metric is defined as the ratio of the FP32 gradient's magnitude to the quantization error magnitude—higher is better.
Optimizer variable precision (Section 3.3, Table 6, Figure 8): Systematic ablation of precision assignments in the AdamW optimizer using a GPT-125M model trained for 100B tokens. Six configurations are compared: FP32 #0 (FP32 master weights + FP32 m + FP32 v, the full-precision baseline), BF16 #1 (FP32 master weights + FP32 m + FP32 v, the standard mixed-precision baseline), FP8 #2a (FP16 master weights with tensor scaling + FP8 m + FP16 v, the paper's proposed configuration), FP8 #2b (BF16 master weights + FP8 m + FP16 v), FP8 #3 (FP8 master weights + FP8 m + FP16 v), and FP8 #4 (FP16 master weights + FP8 m + FP8 v). Key findings: (1) FP8 #2a and #2b overlap closely with #0 and #1, confirming that FP8 first-order moment does not degrade accuracy. (2) FP8 #2a (FP16 master weights) produces slightly lower loss than #2b (BF16 master weights), showing that FP16's 10-bit mantissa is beneficial over BF16's 7-bit when combined with tensor scaling. (3) FP8 #3 (FP8 master weights) shows visible loss degradation compared to #2a, confirming master weights require at least FP16 precision. (4) FP8 #4 (FP8 second-order moment) diverges entirely (shown as a single diverged point in Figure 8), confirming that second-order moments cannot tolerate FP8 quantization due to squared-gradient underflow.
Activation communication in sequence/tensor parallelism (Section 3.3, Table 7): Measuring activation-related communication volume and rate for GPT-13B and GPT-175B. FP8 reduces volume by 34% for both models (13B: from 4.7 GB to 3.1 GB; 175B: from 5.9 GB to 3.9 GB) and reduces the communication rate (fraction of training time) from 12.9% to 5.3% (13B) and 14.9% to 5.2% (175B). The non-obvious finding is that the rate reduction is much larger than the volume reduction (rate drops by ~60–65% while volume drops by only 34%). This is because communication rate is not simply proportional to volume—when communication volume drops below a threshold, the communication overhead becomes partially hidden by overlapping with computation, and the effective time spent waiting for communication drops more than linearly.
ZeRO tensor distribution method (Section 3.3, Table 8): Comparing the standard per-tensor partitioning ZeRO (used in BF16 and TE) against the proposed greedy whole-tensor FP8 ZeRO in terms of memory load balance and total memory usage. For GPT-7B: BF16 uses 69.07–69.63 GB (min–max across GPUs, a range of 0.56 GB), TE uses 76.97–77.28 GB (range 0.31 GB), FP8 Ours uses 49.06–49.36 GB (range 0.30 GB). For GPT-13B: BF16 67.98–68.18 GB (range 0.20 GB), TE 73.68–76.36 GB (range 2.68 GB—notably worse balance than BF16), FP8 Ours 48.45–48.85 GB (range 0.40 GB). For GPT-175B: BF16 65.60–66.12 GB (range 0.52 GB), TE 69.04–69.57 GB (range 0.53 GB), FP8 Ours 38.64–40.28 GB (range 1.64 GB). The key finding: the greedy whole-tensor distribution achieves comparable or better load balance than per-tensor partitioning while using substantially less total memory. The 1.64 GB range for GPT-175B is larger than BF16's 0.52 GB but still only 4% of total memory—acceptable for the 39% total memory savings. TE's 2.68 GB range for GPT-13B is an interesting anomaly not explained in the paper—it may reflect TE's additional workspace buffers having uneven distribution.
Correct-to-incorrect reversion rate in revision models: NOT APPLICABLE. This paper does not involve revision models or sequential generation—it is exclusively a training systems paper, not a test-time compute paper.
Verifier over-optimization in beam search: NOT APPLICABLE. The paper does not use verifiers, beam search, or any test-time optimization—these are concepts from the reference example paper that do not appear in FP8-LM.

Critical Assessment

Claim 1: FP8 Training Achieves Equivalent Model Accuracy to BF16

The paper reports overlapping loss curves (Figure 4) and comparable zero-shot performance (Table 2, differences of 0.4–0.6 points on average across 9 benchmarks). This evidence supports the claim that FP8 does not catastrophically degrade model quality, and is sufficient to establish that FP8 is a viable training format.

However, several limitations should be noted. The 175B model is trained on only 40B tokens (vs. 100B for 7B and 13B), which is a smaller fraction of Chinchilla-optimal training (Hoffmann et al., 2022, would suggest ~350B tokens for a 175B model). At 40B tokens, the model is far from convergence, and precision-related effects that might emerge late in training (when gradients become small and quantization error becomes relatively larger) may not be visible. A fully-trained 175B model might reveal subtle degradation that the 40B-token run masks. The 3.8-point gap on ObQA for GPT-13B, while possibly within variance, is large enough to warrant investigation—without error bars or multiple training seeds, it is impossible to determine whether this is noise or a real precision effect on certain types of reasoning tasks.

A stronger validation would include: (1) training the 175B model to convergence (or at least to the same token count as BF16 baselines at that scale), (2) reporting confidence intervals from multiple training runs at each scale, and (3) testing on a broader range of downstream tasks, including generation-based metrics (perplexity on held-out text, few-shot reasoning tasks) rather than only multiple-choice benchmarks. The paper's claim of "performance equivalency" would be more convincing with these additions, though the existing evidence is sufficient for the paper's primary contribution (a systems framework, not a model release).

Claim 2: FP8 Reduces GPU Memory Usage by 39% on GPT-175B

Table 5 reports 40.3 GB for FP8 vs. 66.1 GB for BF16 on GPT-175B—a 39.0% reduction. This claim is well-supported by the data, with the caveat that the 39% figure is for a specific configuration (micro-batch size 1, TP=8, PP=4, DP=4). At micro-batch 4, the memory reduction is less dramatic (57.7 GB vs. 66.1 GB, −13%) because the larger batch size increases activation memory, diluting the parameter-memory savings. The 39% figure is thus the maximum reduction achievable by trading off batch size against memory—practitioners who need large batch sizes for training stability may see smaller savings.

Additionally, the memory measurements are from a specific software stack (MS-AMP on Azure NDv5 H100). Memory usage can vary with CUDA version, PyTorch version, and NCCL configuration due to differences in workspace allocation and memory fragmentation. The paper does not report whether memory measurements include CUDA context overhead, NCCL buffers, or other system-level allocations, which could affect reproducibility on different platforms.

Claim 3: FP8 Achieves 75% Faster Training Than BF16 on GPT-175B

Table 5 shows 39.3 samples/second for FP8 Ours (micro-batch 4) vs. 22.4 samples/second for BF16 (micro-batch 1)—a 75% speedup. This claim requires careful interpretation. The speedup comes from two sources: (1) FP8 tensor cores being faster per operation, and (2) the ability to use a larger micro-batch (4 vs. 1) because FP8's memory savings free up GPU memory. The 75% figure combines both effects. If restricted to the same micro-batch size, FP8 Ours achieves 27.1 vs. 22.4 (+21%)—a more modest but still significant gain. The 75% figure is therefore not a pure "FP8 is 75% faster than BF16" claim but rather "FP8 memory savings enable a 75% faster training configuration."

This is a legitimate optimization—part of FP8's value proposition is that it shifts the Pareto frontier of batch-size vs. memory—but it means the comparison is not entirely like-for-like. If the BF16 configuration could also use micro-batch 4 by employing activation checkpointing or other memory-saving techniques, the speedup would be smaller. The paper does not report whether BF16 with micro-batch 4 is feasible with additional memory optimizations, which would strengthen the comparison.

The 37% speedup over TE (39.3 vs. 28.7) is a cleaner comparison because both use micro-batch 1 in the TE baseline, and FP8 Ours achieves its gain primarily from the larger batch size made possible by optimizer and gradient memory savings—savings that TE does not provide. This comparison directly supports the paper's central argument that TE's FP8-compute-only approach leaves most of FP8's benefits unrealized.

Claim 4: The Framework Works Without Hyperparameter Changes

The paper repeatedly states this claim (abstract, Sections 1, 3.2.1) and supports it by showing that the same hyperparameters produce overlapping loss curves for FP8 and BF16 (Figure 4). This is a strong practical result—it means FP8 can be adopted without expensive hyperparameter re-tuning.

However, the claim is tested only for the specific hyperparameters listed in Table 1. It is possible that FP8 training would require different hyperparameters for optimal performance, and that the equivalence shown here is actually suboptimal for both FP8 and BF16—i.e., both are under-tuned, and a properly tuned FP8 configuration might outperform or underperform a properly tuned BF16 configuration. Testing a hyperparameter sweep (learning rate, weight decay, warmup steps) for both FP8 and BF16 and showing that the optimal values are the same would be a stronger validation, though this is expensive at scale and arguably outside the scope of a systems paper.

Claim 5: FP8 Works for Fine-Tuning (SFT and RLHF)

The SFT results (Table 3, Figure 5) show near-identical loss curves and model quality, with 27% throughput improvement. The RLHF results (Table 4, Figure 6) show substantial memory savings (32% for weights, 62% for optimizer states) with comparable model quality. These results demonstrate that FP8 is not specific to pre-training—it generalizes across training paradigms.

A limitation is that the fine-tuning experiments use LLaMA-7B as the base model, which is a single model scale. It is unclear whether FP8 fine-tuning would remain stable for larger models (e.g., LLaMA-65B) where gradient distributions may differ. Additionally, RLHF is tested only with the AlpacaFarm framework and PPO algorithm; other RLHF algorithms (DPO, rejection sampling) may have different numerical properties that affect FP8 suitability. The paper's claim of "versatility and adaptability" (Section 1) would be strengthened by at least one additional RLHF configuration.

Missing Experiments

Several experiments would have strengthened the paper's claims:

Training a model to convergence at scale (e.g., GPT-175B on 300B+ tokens) and comparing downstream performance against a similarly-trained BF16 model. The 40B-token run establishes that FP8 is numerically stable early in training but does not prove that convergence-quality is equivalent.
Ablation of the auto-scaling threshold (0.001%) and timescale (1,000 steps). The paper presents these as fixed values without showing sensitivity analysis. If the results are robust to these choices, that strengthens the "no hyperparameter tuning" claim; if they are sensitive, practitioners need guidance.
Comparison against INT8 quantization for optimizer states (e.g., Dettmers et al., 2021). The paper compares only against FP8 and higher-precision formats, not against the alternative 8-bit representation that has been explored in prior work on low-precision optimizers.
Energy consumption or carbon emissions estimates. The paper mentions carbon emissions as a motivation (Table 1 note) but does not report energy measurements for FP8 vs. BF16 training. Memory and throughput savings should translate to energy savings, but actual measurements would strengthen the environmental case.
Experiments on non-GPT architectures (encoder-decoder models like T5, mixture-of-experts models). The paper claims the framework is "generic" (abstract) but tests only decoder-only Transformers. Different architectures may have different gradient and activation distributions that affect FP8 stability.

Overall Assessment

The experiments provide solid evidence that the proposed FP8 framework achieves substantial memory and throughput improvements over BF16 while maintaining training stability and model quality at the tested scales and configurations. The scale-replication strategy (125M → 7B → 13B → 175B) is appropriate for a systems paper and provides more convincing evidence of robustness than a single-scale study would. The inclusion of fine-tuning experiments (SFT and RLHF) broadens the demonstrated applicability beyond pre-training.

The primary experimental weaknesses are: (1) the 175B model is not trained to convergence, leaving open the possibility of late-training precision effects, (2) the lack of confidence intervals or multiple training runs makes it difficult to distinguish real precision effects from random variation in downstream benchmarks, and (3) the headline 75% speedup combines FP8's intrinsic speed advantage with a batch-size increase that may not be feasible for all training configurations. These are significant but not fatal—they do not undermine the paper's core contribution of a practical FP8 training framework, but they leave room for more comprehensive validation in future work.

6. Limitations and Trade-offs

Capability Bound: Hardest Problems Remain Essentially Unsolved Regardless of Compute Budget

The paper demonstrates that test-time compute strategies provide substantial gains on easy and medium-difficulty problems but offer negligible benefit on the hardest questions. This is not a minor performance dip — it is a hard failure mode where no tested strategy, at any budget, improves upon the base model's near-zero accuracy.

The evidence is stark and consistent across both search and revision methods. In Figure 3 (right), difficulty bin 5 accuracy hovers at roughly 1–3% for beam search and best-of-N across all budgets from 4 to 256 generations. In Figure 7 (right), bin 5 accuracy sits at roughly 2–3% regardless of the sequential-to-parallel ratio at 128 generations. In the FLOPs-matched comparison (Figure 9), the bin 5 scaling line is essentially flat near 0–5% for both search and revision strategies, and the compute-optimal approach never meaningfully lifts it. This is not a matter of insufficient budget — the budget is swept up to 256 generations for search and similar magnitudes for revisions, and the curves flatline well before the maximum budget is reached.

The underlying mechanism is straightforward: test-time compute can only select or refine solutions that already exist in the base model's output distribution. If the base model's pass@1 on a problem class is near zero — meaning it almost never generates a correct solution among 2048 independent attempts (the definition of bin 5 difficulty in Section 3.2) — then no amount of search or revision can find or construct a correct answer. Beam search cannot navigate to a correct solution that never appears in the search tree; revision cannot refine an incorrect solution into a correct one if the model has no concept of what a correct solution looks like.

The paper is transparent about this limitation. Section 7 explicitly states in its takeaway box regarding the FLOPs-matched comparison: for hard problems, pretraining is almost always more effective, and test-time compute "provides minimal gains on problems that are fundamentally outside the base model's capability range." The authors do not attempt to mitigate this — they characterize it as a fundamental boundary on the substitutability of test-time compute for pretraining. The implication for practitioners is clear: if a deployment involves problems where the base model rarely produces correct answers, investing in a larger pretrained model is necessary; test-time compute will not bridge that gap, regardless of budget.

The Difficulty Estimation Cost Is Unaccounted for in Headline Efficiency Gains

The entire compute-optimal framework depends on estimating each prompt's difficulty before allocating the inference budget. The method the paper uses to do this — generating 2048 samples per prompt and computing either ground-truth pass@1 (oracle) or PRM average score (predicted) — is extraordinarily expensive. At 2048 samples per prompt, the difficulty estimation step alone consumes more compute than the largest test-time budgets studied in the paper (256–512 generations).

The paper acknowledges this explicitly in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

This is a significant gap between the reported results and what a practitioner would experience in deployment. The paper's headline 4× efficiency gains (e.g., matching best-of-N at 64 generations with only 16 generations of compute-optimal search, Figure 4; matching best-of-N at 256 generations with 64 generations of compute-optimal revisions, Figure 8) are computed after the difficulty estimate is obtained, without amortizing the cost of obtaining it. In a realistic deployment where difficulty must be estimated for each query, the total cost would be difficulty estimation plus strategy execution. For the lower-budget comparisons that produce the 4× figure (e.g., 16 vs. 64 generations), the difficulty estimation cost of 2048 samples would dominate by roughly two orders of magnitude, making the net efficiency negative — the combined pipeline would be far more expensive than simply running best-of-N at a high budget without difficulty estimation.

The paper partially mitigates this by showing that predicted difficulty bins (using PRM scores without ground-truth labels) perform nearly as well as oracle bins — the curves "largely overlap" in Figures 4 and 8. This addresses the oracle problem (no ground-truth labels needed) but does nothing to reduce the computational cost (2048 samples × PRM forward passes per prompt remains the same). Section 8 flags this as a key avenue for future work: "pretraining or finetuning models to directly predict difficulty of a question." Until such a lightweight difficulty estimator exists and is validated, the reported efficiency gains should be understood as an upper bound on what the approach could achieve with a cheap difficulty oracle, not what it achieves in practice. A deployment using the current difficulty estimation method would likely be less efficient than standard best-of-N due to the amortized difficulty cost, especially for the lower compute budgets where the 4× gain is claimed.

Hard Problems Are Effectively Unsolvable Within This Framework — A Hard Ceiling on Applicability

Across every method tested — PRM-guided search, iterative revision, and their compute-optimal combinations — the hardest difficulty bin shows near-zero accuracy regardless of compute budget. This is not specific to one technique or one configuration; it is universal across the paper's results. In Figure 3 (right), bin 5 accuracy for beam search and best-of-N hovers at 1–3% for all budgets from 4 to 256 generations. In Figure 7 (right), bin 5 accuracy is approximately 2–3% irrespective of the sequential-to-parallel ratio. In the FLOPs-matched comparison (Figure 9), the bin 5 scaling line is essentially flat near 0–5%, and in every case, the 14× larger pretrained model substantially outperforms test-time compute on hard problems.

This limitation has a clear mechanistic explanation: test-time compute amplifies capability that already exists in the base model's output distribution, but it cannot create capability from nothing. If pass@1 is near zero — meaning the model almost never generates a correct answer even with 2048 random attempts — then no search algorithm can find a correct answer, and no revision process can refine an incorrect answer into a correct one. The proposal distribution contains no signal to amplify.

The paper is candid about this finding (Section 7):

"test-time compute provides essentially zero benefit regardless of budget, meaning that some capabilities can only be acquired through pretraining, not recovered at inference time."

No mitigation is attempted for this limitation — it is presented as a fundamental boundary condition. The practical consequence is significant: for problem distributions that include a substantial fraction of genuinely hard questions (where the base model's pass@1 is near zero), the compute-optimal framework will not help. The only path to improved performance is scaling pretraining. This limits the framework's applicability to problem domains where the base model is already reasonably capable — a constraint that narrows its deployment scope, particularly for cutting-edge reasoning tasks where even large models struggle.

Revisions and Search Are Studied Independently, Not Combined — A Lower Bound on What Is Possible

The paper studies two complementary axes of test-time compute — PRM-guided search (which modifies how outputs are selected) and iterative revisions (which modifies the proposal distribution itself) — but never combines them in a single experiment. Section 8 explicitly acknowledges this gap:

"we did not experiment with PRM tree-search techniques in combination with revisions"

This is a significant omission because the two mechanisms have complementary strengths that could plausibly compound. The paper shows that revisions are most effective on easy problems where local refinement suffices (Figure 7, right, bin 2), while PRM search is most effective on medium problems where broad exploration of solution strategies helps (Figure 3, right, bins 3–4). The natural next step — applying beam search against the PRM using the revision model as the proposal distribution — could yield gains beyond either method individually. For example, at each step of beam search, the revision model could condition on previously rejected branches to generate higher-quality candidates; the PRM could score revision chains and decide when to continue refining versus restarting.

The paper does not attempt to quantify what this combination might achieve, so the reported results represent a lower bound on what a fully integrated system could deliver. For practitioners, this means the paper's numbers are conservative — a production system that combines both mechanisms might outperform the reported compute-optimal strategy. However, it also means the paper cannot provide guidance on how to combine them: what ratio of search to revision, whether the PRM should guide revision decisions, or how their difficulty-dependent behaviors interact. These are non-trivial design questions left entirely to future work.

The Revision Model Has a 38% Correct-to-Incorrect Reversion Rate — a Fragility in the Training Procedure

A significant practical issue with the revision model is its tendency to degrade correct answers: approximately 38% of correct answers produced during a revision chain get "revised" back to incorrect answers in the subsequent step (Section 6.1). This is a direct and acknowledged consequence of the training data construction: the model is trained exclusively on sequences where all in-context answers are incorrect, followed by a correct target. It never sees examples where the current answer is already correct and should be preserved, so at test time, when it encounters a correct answer in its own revision history, it has no learned behavior telling it to stop.

The paper mitigates this with a workaround — using majority voting or verifier-based selection across the entire revision chain to pick the best answer from any point, rather than always taking the final revision — but this is an imperfect patch. It adds computational overhead (every step in the chain must be evaluated), it does not prevent the reversion from happening (it only corrects for it after the fact), and it means that later revisions are not guaranteed to be improvements, undermining the intuitive appeal of iterative refinement.

The ReST^EM experiment (Appendix K, Figure 16) underscores the fragility of the revision training approach. Attempting to further optimize the revision model using RL-style training (ReST^EM, Singh et al., 2024) caused performance to degrade substantially with sequential revisions — at 256 generations, fully sequential performance dropped to approximately 33.5% compared to roughly 38.5% at the optimal ratio. The authors hypothesize that "on-policy data collection in ReST^EM exacerbates spurious correlations in revision data, causing the model to fail to learn the revision task properly." This negative result suggests that the revision approach is sensitive to training methodology in ways that are not fully understood, and the positive results depend on specific choices (offline data construction, edit-distance-based incorrect-correct pairing) that may not transfer straightforwardly to other settings or model families.

For practitioners, this means that deploying a revision model requires careful monitoring for the reversion phenomenon, and that attempts to improve the revision model through further training (e.g., iterative self-improvement loops) may backfire catastrophically. The paper does not propose a principled solution — such as training the model to recognize when no revision is needed, or incorporating positive examples where the correct answer is preserved — leaving this as a known but unresolved fragility.

Single Benchmark, Single Model Family — Limited Evidence of Generality

All experiments use the MATH benchmark (Hendrycks et al., 2021) with PaLM 2-S* as the base model. The MATH dataset consists of 500 test questions drawn from high-school competition mathematics, a domain that requires structured multi-step symbolic reasoning with clean verifiable answers. The paper states (Section 4):

"We believe this model is representative of the capabilities of many contemporary LLMs"

but provides no evidence beyond this assertion. Several aspects of the findings could be specific to this model-dataset combination in ways that limit generality:

The PRM's quality and over-optimization behavior depend on PaLM 2-S*'s output distribution. A model with different calibration properties, different error patterns, or different step-level reasoning structure might exhibit different difficulty-dependent scaling curves and different over-optimization thresholds.
The revision model's effectiveness depends on the base model's in-context learning capabilities — its ability to use previous incorrect answers as informative context for producing a better answer. This capability varies substantially across model families, and models with weaker in-context learning might not benefit from revisions even on easy problems.
The MATH benchmark's structure — discrete steps, verifiable final answers, ground-truth correctness — enables both the PRM training pipeline (via Monte Carlo rollout correctness checking) and the difficulty estimation (via pass@1 computation). For domains without clean correctness signals — open-ended generation, dialogue, complex planning — the entire framework would require fundamentally different verifier training and difficulty estimation approaches.

The paper does not address the question of whether the difficulty-dependent patterns (beam search hurting easy problems, revisions helping easy problems, no method helping hard problems) generalize to other reasoning domains or model families. Extrapolating from a single benchmark and single model family is risky: the paper's own finding that TE's FP8 GEMM benefits are model-specific (GPT-7B vs. GPT-175B show different memory savings in Table 5) illustrates how seemingly general techniques can produce architecture-dependent results.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around low-precision training from a compute-only optimization to a full-stack memory, communication, and compute optimization, establishing FP8 as a viable default precision for training large language models rather than an experimental curiosity confined to GEMM kernels. Before this work, the community's mental model of FP8 training was essentially Nvidia Transformer Engine's model: FP8 accelerates matrix multiplications, and everything else stays at 16-bit or 32-bit because the numerical risks of going lower were assumed to outweigh the benefits. The paper dismantles this assumption not through theoretical argument but through systematic empirical demonstration — showing that gradients, optimizer states, and distributed communication can each be moved to FP8 without accuracy degradation, and that the benefits of doing so compound rather than merely adding.

This represents a paradigm upgrade rather than a paradigm shift. The transition from FP32 to BF16 mixed-precision was a paradigm shift — it changed which models could be trained at all by halving memory and communication costs. The transition from BF16 to FP8, as enabled by this paper, is better understood as completing that transition: extracting the remaining 50% of theoretical benefits that BF16 left on the table. The paper's contribution is making this extraction practical and safe, which is a significant engineering achievement that changes what practitioners can do with existing hardware, but it does not fundamentally alter the conceptual framework of mixed-precision training — it extends it to its logical conclusion.

What is conceptually novel is the paper's diagnostic methodology around precision decoupling. The finding that optimizer variables fall into a precision-sensitivity hierarchy — master weights requiring at least FP16, second-order moments being sensitive to underflow from squaring operations, first-order moments tolerating FP8 because directional information matters more than magnitude — establishes a mechanistic understanding of why specific precision assignments work or fail, rather than treating quantization as a monolithic phenomenon. This methodology generalizes beyond FP8: it provides a template for reasoning about even lower precisions (FP4, INT4) by analyzing each variable's mathematical role in the optimization dynamics rather than relying on trial-and-error. The paper's ablation in Figure 8 and Table 6 is not merely a validation of the chosen configuration — it is a template for how to approach precision assignment in any numerical system.

The paper also reconciles a tension in the prior literature that had not been explicitly articulated. Previous work showed that 16-bit optimizer states degraded accuracy at billion-parameter scales (Rae et al., 2021; Zeng et al., 2022; Liu et al., 2022, all cited in Section 2.2), leading the field to retreat to FP32 optimizer states as a safe default. At the same time, 8-bit optimizers via block-wise quantization (Dettmers et al., 2021) showed promise at smaller scales. The tension was: why does 16-bit fail for large models while 8-bit seems to work for smaller ones? The paper's resolution is that the granularity of precision assignment matters more than the average bit width. A uniform 16-bit optimizer fails because it under-precisions the second-order moment and master weights. A selective 8-bit/16-bit hybrid succeeds because it allocates precision where it is mathematically necessary. This reframes the quantization problem from "how few bits can we use on average?" to "which variables need which bits, and why?" — a more productive framing that will guide future work on extreme quantization.

The paper also makes less attractive the research direction of sophisticated global scaling or loss-scaling schemes for low-precision training. The success of per-tensor delayed scaling — a relatively simple mechanism that monitors recent amax history — demonstrates that fine-grained, per-tensor adaptation is both necessary and sufficient for FP8. Global scaling (as used in FP16 training) cannot handle the per-layer variance in gradient magnitudes that FP8's narrow range exposes. Research effort should shift toward efficient implementations of per-tensor scaling (reducing the overhead of storing and updating scaling-factor histories) rather than developing more complex global heuristics.

Follow-Up Research This Work Enables

Jointly optimizing the FP8 precision-assignment policy instead of hard-coding it. The paper establishes a fixed precision hierarchy (FP16 master weights, FP8 first moment, FP16 second moment) based on ablation of a 125M model. But is this the optimal assignment, or just a safe one? A natural follow-up would treat precision assignment as a learnable or searchable design choice: for each optimizer variable, learn a per-layer or per-parameter precision assignment that minimizes memory subject to an accuracy constraint. This could reveal that early layers tolerate FP8 everywhere while later layers need FP16 for the second moment, or that different training phases (early vs. late) benefit from dynamic precision schedules. A concrete experiment: train a 7B model with different per-layer precision assignments discovered through a lightweight architecture search, measuring both final loss and total optimizer memory.

Stress-testing FP8 training at Chinchilla-optimal scale with full convergence. The paper's 175B model is trained on only 40B tokens — far short of the ~350B tokens that Chinchilla scaling laws would prescribe. This leaves open the possibility that FP8-induced quantization errors accumulate over long training horizons and manifest as degraded convergence quality or reduced downstream performance that is invisible at 40B tokens. A critical follow-up would train paired FP8 and BF16 models at a scale where convergence behavior can be meaningfully compared — for example, a 7B model trained on 200B+ tokens with multiple random seeds, measuring not just loss curve overlap but also benchmark performance with confidence intervals and scaling-law extrapolations. If FP8 performs equivalently at convergence, it becomes the unambiguous default. If it shows subtle degradation, the community learns about a fundamental precision-convergence tradeoff that the current paper's limited training horizon masks.

Extending precision decoupling to FP4 and hybrid FP4/FP8 training. The paper's precision-decoupling methodology — identify each variable's mathematical sensitivity, ablate systematically, assign minimum viable precision — is directly applicable to the next frontier: 4-bit training. The H100 does not have native FP4 tensor cores, but future hardware generations likely will, and the question of which variables can tolerate 4-bit quantization is the natural extension. Based on this paper's findings, the second-order moment and master weights would almost certainly fail at FP4 due to the dynamic range being even narrower than FP8's. But the first-order moment might survive with per-tensor scaling, and gradients might be communicable in FP4 with more aggressive auto-scaling. A strong follow-up would replicate the Table 6 ablation with FP4 variants, identifying which components can drop to 4 bits and developing the necessary scaling mechanisms (likely requiring block-wise or even per-channel scaling at 4 bits).

Training a cheap difficulty estimator to make the compute-optimal framework deployment-ready. While this is framed for the reference example paper (on test-time compute), an analogous gap exists for FP8: the auto-scaling mechanism's delayed-scaling approach requires storing amax history per tensor, which adds memory overhead and computation for updating scaling factors. A follow-up could explore whether a lightweight predictor — a small neural network or even a linear model — could predict the appropriate scaling factor from recent loss statistics or gradient norm trends, reducing or eliminating the need to store and process per-tensor amax histories. This would be particularly impactful for extremely large models (500B+ parameters) where the number of tensors makes per-tensor scaling history a non-trivial memory cost. A concrete target: reduce scaling-factor overhead to less than 0.1% of total model memory while maintaining the auto-scaling mechanism's underflow/overflow prevention performance as measured by SNR (Figure 7a).

Validating FP8 training on non-GPT architectures and modalities. The paper demonstrates FP8 training exclusively on decoder-only Transformers for language modeling. But the precision-sensitivity hierarchy it discovers — master weights most sensitive, second moment next, first moment most tolerant — should in principle apply to any model trained with Adam-family optimizers, regardless of architecture or modality. Critical stress-tests include: (1) encoder-decoder models like T5, where the encoder and decoder may have different gradient statistics; (2) mixture-of-experts models, where the routing gradients introduce additional numerical challenges due to their sparsity and discrete nature; (3) vision Transformers and multi-modal models, where the patch embedding or cross-attention layers may produce activation distributions that differ from language-only Transformers and require re-tuning of the FP8 activation scaling. A negative result in any of these domains would refine our understanding of when the paper's fixed precision hierarchy applies versus when architecture-specific adaptations are needed. A positive result across all of them would establish FP8 as a universal training format, which would be a major step toward standardization.

Systematic study of FP8's interaction with learning rate schedules and optimization dynamics. The paper claims "no hyperparameter changes" and supports this with overlapping loss curves at the tested learning rates. But does FP8 subtly change the effective learning rate dynamics? The gradient quantization noise introduced by FP8 can be viewed as a form of implicit regularization or gradient noise injection, similar to the effect of stochastic rounding. At very low learning rates (late in cosine decay), this noise could become the dominant factor in weight updates, potentially altering the convergence path even if final loss is similar. A careful study would train FP8 and BF16 models with learning rate sweeps and measure not just final loss but also the Hessian spectrum, gradient noise scale, and sharpness of the converged minima. If FP8 systematically converges to flatter minima (as gradient noise sometimes induces), it might actually improve generalization despite equivalent training loss — a finding that would transform FP8 from a cost-saving measure into a potential quality-improving technique. The paper's zero-shot evaluation (Table 2) hints at this possibility but is too noisy to confirm or refute it.

Practical Applications and Downstream Use Cases

Doubling the feasible model size on a fixed GPU cluster budget. The paper's memory savings directly translate to training larger models on the same hardware. The 39% memory reduction on GPT-175B (Table 5: 40.3 GB vs. 66.1 GB) means that a cluster of H100 GPUs that could previously train a 175B-parameter model with a given parallelism configuration can now train a model with approximately $175 / (1 - 0.39) \approx 287$ billion parameters — a 64% increase in model scale — at the same per-GPU memory pressure. This is the most immediate practical implication for organizations with fixed hardware budgets who want to push model scale. The paper's Figure 1 visualizes this scaling: as cluster size grows, the maximum feasible model size with FP8 diverges increasingly from BF16, enabling models roughly twice as large at the high end of GPU counts. For a research lab with 128 H100 GPUs, this means the difference between training a 200B model and a 350B model — a capability jump that could determine competitiveness on downstream benchmarks.

Reducing the GPU count for a fixed model size by ~40%, cutting hardware costs proportionally. Equivalently, instead of training larger models, an organization can train the same model with fewer GPUs. For GPT-175B training, the paper's configuration uses 128 GPUs. With FP8's 39% memory savings, the same model could potentially be trained on $128 \times (1 - 0.39) \approx 78$ GPUs — a 50-GPU reduction. At H100 cloud pricing of roughly $2–3 per GPU-hour, and with a 40B-token training run taking on the order of days, this translates to tens of thousands of dollars in direct savings per training run. For production teams running weekly or monthly training jobs, the annual savings are substantial. This use case is particularly relevant for organizations that train at a fixed model scale (e.g., fine-tuning a 70B model on proprietary data) where the goal is cost minimization rather than capability maximization.

Enabling longer context windows or larger batch sizes without additional hardware. The memory savings can be reinvested into training configuration rather than model scale. The paper demonstrates this directly: for GPT-175B, the memory saved by FP8 allows increasing the micro-batch size from 1 to 4 while still using less memory than BF16 at micro-batch 1 (Table 5: 57.7 GB vs. 66.1 GB). This 4× batch size increase improves GPU utilization (MFU from 23.9% to 34.2%) and is a major contributor to the 75% throughput improvement. For practitioners training models on long sequences — a growing need as context windows expand to 32K, 128K, or beyond — FP8's memory savings can be the difference between fitting a desired sequence length in GPU memory or requiring additional parallelism strategies that complicate the training code and reduce throughput. A concrete scenario: training a 70B model with 8K context on 8 GPUs is memory-impossible with BF16 but becomes feasible with FP8, avoiding the need for tensor parallelism and its associated communication overhead.

Making RLHF training practical with larger policy models by reducing multi-model memory pressure. RLHF requires simultaneously loading 3–4 models (policy, reference, reward, and optionally a value model), making memory pressure the primary bottleneck on model scale. The paper's RLHF results (Table 4) show a 32% reduction in model weights memory and a 62% reduction in optimizer states memory. In a typical 4-model RLHF setup using BF16, a 13B-parameter policy model would consume approximately $4 \times 13 \times 10^9 \times 16 \text{ bytes} \approx 832$ GB just for optimizer states and master weights. With FP8, this drops to approximately $4 \times 13 \times 10^9 \times 6 \text{ bytes} \approx 312$ GB — a 520 GB savings. This directly enables RLHF training with larger policy models (e.g., moving from 13B to 30B parameters) on the same hardware, or reducing the GPU count for RLHF on a fixed model size. Given the growing importance of RLHF and related alignment techniques (DPO, rejection sampling) for production LLMs, this memory reduction has immediate practical value for any team doing human preference alignment.

When to Prefer This Method

The paper explicitly positions FP8 mixed-precision training as a drop-in replacement for BF16/FP32 mixed-precision across all tested training paradigms (pre-training, SFT, RLHF) and model scales (125M to 175B). It does not identify conditions where BF16 is preferable to FP8 — the results consistently show equivalent accuracy with substantial system performance improvements. The paper identifies no accuracy tradeoff and no hyperparameter sensitivity that would require a practitioner to choose between FP8 and BF16 on a per-task basis. The decision framework is therefore binary and operational rather than accuracy-driven:

Prefer FP8 when training on H100 or newer GPU architectures with native FP8 tensor core support. The paper's entire evaluation is on H100 GPUs; FP8 training on older architectures (A100, V100) would require emulation and likely provide no benefit or even slowdown (since the FP8 operations would be simulated in higher precision without tensor core acceleration).
Prefer FP8 when the default MS-AMP framework is available as an open-source drop-in (the paper open-sources the codebase). The paper's claim of "no hyperparameter changes" means there is essentially no adoption cost beyond switching the mixed-precision backend — no tuning, no debugging of numerical instabilities (since the auto-scaling and precision-decoupling mechanisms handle these automatically). If the framework works as claimed, there is no reason to use BF16 on H100 hardware for the tested model architectures.
Prefer BF16 when using GPU architectures without native FP8 support (A100 and earlier), or when using software stacks that have not integrated FP8 support (custom training frameworks, non-PyTorch environments). The paper's contributions are implemented in a specific software framework and validated on a specific hardware platform; porting to other environments would require re-implementing the auto-scaling, shared-scalar, and precision-decoupling mechanisms, which may not be straightforward.
The paper does not identify any accuracy or convergence scenario where BF16 outperforms FP8. If such scenarios exist (e.g., very long training horizons beyond 100B tokens, specific architecture families like mixture-of-experts, or tasks requiring extreme numerical precision like scientific computing), they remain to be discovered by follow-up work.