FP8-Flow-MoE: A Casting-Free FP8 Recipe without Double Quantization Error

ArXiv: 2511.02302

🎯 Pitch

Training massive MoE models in FP8 typically underperforms BF16 due to a hidden double quantization error arising from standard tensor layout conversions. This paper shows that a scaling-aware transpose operator—which reorders quantization dimensions via exponent-bit manipulation instead of dequantize–transpose–requantize cycles—can slash casting operations by 83% and unlock up to 21% higher MoE throughput while matching BF16 convergence.

---

1. Executive Summary

This paper introduces FP8-Flow-MoE, a casting-free FP8 training recipe for Mixture-of-Experts models that eliminates redundant quantize–dequantize conversions by maintaining a quantization-consistent FP8-centric dataflow throughout the MoE stage. The core mechanism is a scaling-aware transpose operator (directly converting FP8 tensors between row-wise and column-wise quantization layouts via exponent-bit manipulation, avoiding the dequantize→transpose→requantize cycle that produces double quantization error), combined with a suite of fused FP8 kernels (fused permute-padding, fused SwiGLU-quantization) that reduce explicit cast operations from 12 to 2. Evaluated on a 671B-parameter DeepSeek-V3 model, FP8-Flow-MoE achieves up to 21% higher throughput and 16.5 GB lower peak memory per GPU compared to both BF16 and naïve FP8 baselines, while maintaining loss convergence indistinguishable from BF16 across 200B training tokens on a 16B DeepSeek-V2-Lite model, establishing that persistent FP8 dataflows with scaling-aware transpose can realize near-theoretical low-precision efficiency gains only when the double quantization error introduced by layout-inconsistent requantization is explicitly eliminated rather than tolerated through higher-precision fallbacks.

2. Context and Motivation

The Core Problem: FP8's Theoretical Promise vs. Its Realized Performance

The fundamental tension this paper tackles is deceptively simple: FP8 training should be twice as fast as BF16 training, but in practice it often isn't. The gap between theoretical and realized throughput stems from a systematic mismatch between how FP8 kernels are integrated into training frameworks and what a truly efficient low-precision dataflow actually requires.

The promise of FP8 is grounded in hardware reality. NVIDIA's Hopper architecture provides native FP8 Tensor Core support that delivers double the arithmetic throughput of BF16 for matrix multiplications (GEMMs), and FP8's halved data size should proportionally reduce memory bandwidth pressure and communication latency. For Mixture-of-Experts models—which are both computationally intensive (massive expert GEMMs) and communication-intensive (all-to-all token dispatch between experts)—FP8 appears ideally suited as a precision format. In theory, adopting FP8 throughout the MoE stage should yield something close to a 2× throughput improvement.

The reality, documented starkly in this paper's analysis, is far more modest. When existing frameworks simply substitute FP8 GEMM kernels and FP8 communication primitives into an otherwise BF16-dominated dataflow, the realized throughput gains collapse to single-digit percentages—sometimes as low as 3% over optimized BF16 baselines. This paper identifies redundant quantize–dequantize (Q/DQ) conversions as the primary culprit: every boundary where an FP8 kernel connects to a BF16 tensor requires a cast operation, and these casts accumulate to dominate the computational budget at the small-to-medium batch sizes typical of large-scale MoE training.

More importantly, the paper identifies a deeper, less obvious problem: naïvely removing these casts by keeping the entire dataflow in FP8 introduces a different failure mode—double quantization error—that threatens numerical stability. This creates a genuine dilemma: either accept the Q/DQ overhead and lose most of FP8's speedup, or eliminate the overhead and risk training instability from compounded quantization error. Prior work had not recognized this tradeoff as a first-class design problem requiring a systematic solution; frameworks either lived with the overhead or kept enough of the pipeline in higher precision to avoid the error, in both cases leaving substantial performance on the table.

Why This Matters: The Economics of Mega-Scale MoE Training

The significance of this problem is primarily economic and infrastructural, but with genuine technical depth. Training MoE models at the scale of DeepSeek-V3 (671B parameters) or larger requires thousands of GPUs running for months. A 21% throughput improvement—the paper's headline result—does not just accelerate research timelines; it directly translates to millions of dollars in reduced compute costs and proportionally lower energy consumption. For research institutions and even large industrial labs where training cost is the de facto barrier to entry, gains of this magnitude determine whether a project is feasible at all.

Beyond raw throughput, the memory savings (up to 16.5 GB per GPU) address a different but equally critical bottleneck. Large-scale MoE training pushes GPU memory to its limits, forcing difficult tradeoffs between model size, batch size, and activation checkpointing strategies. The paper shows that under aggressive activation checkpointing configurations where BF16 and naïve FP8 baselines encounter out-of-memory (OOM) errors at high expert parallelism levels (EP=32), FP8-Flow-MoE remains operational. This means the recipe does not merely accelerate existing configurations—it enables configurations that were previously impossible, expanding the design space for MoE model deployment.

There is also a broader strategic significance. The paper represents one of the first systematic attempts to define what an FP8-centric training paradigm looks like, as opposed to a BF16-centric paradigm with FP8 acceleration bolted on. This distinction matters because as hardware evolves toward lower precisions (FP6, FP4), the same tension between kernel-level acceleration and dataflow-level overhead will recur, likely in amplified form. The principles established here—persistent low-precision dataflow, scaling-aware layout conversion, operator fusion to eliminate quantization boundaries—constitute a template for future precision transitions, not just an optimization for FP8 on Hopper GPUs.

Where Prior Approaches Fall Short

The paper identifies limitations in existing FP8 training systems along several dimensions:

BF16-Dominated Dataflows with Conservative FP8 Integration

The dominant integration strategy, exemplified by NVIDIA's TransformerEngine (TE), confines FP8 computation strictly within individual GEMM operations. The blockwise FP8 recipe quantizes activations and weights immediately before each linear layer, performs the matrix multiplication in FP8, and immediately dequantizes the output back to BF16. This design is conservative and numerically safe—FP8 is never exposed to operations it handles poorly (reductions, nonlinearities)—but it means that every GEMM boundary incurs two cast operations (quantize on input, dequantize on output). In a single MoE forward-backward pass, which typically contains at least two grouped linear layers plus routing and communication stages, these casts multiply rapidly. The paper's analysis shows this approach introduces up to twelve explicit Q/DQ operations in the MoE module alone (see Figure 2 for the computational graph comparison).

The consequence is that FP8 GEMM acceleration is confined to isolated kernels while the surrounding data movement, communication, and element-wise operations remain in BF16. The overall MoE layer throughput becomes bounded by high-precision communication and the serialized latency of format conversion kernels, not by the GEMM throughput that FP8 accelerates.

DeepSeek-V3's Approach: FP8 Communication and GEMM Without Dataflow Consistency

The DeepSeek-V3 training system (DeepSeek-AI Team, 2024) represented a significant advance: it demonstrated successful FP8 training at trillion-parameter scale and released high-performance kernels—DeepGEMM for FP8 grouped GEMM and DeepEP for FP8 cross-node all-to-all communication—that pushed FP8 acceleration beyond individual GEMMs into the communication fabric. However, as the paper's analysis reveals (Section 3.2, Table 1), this approach still suffers from substantial quantization overhead at the boundaries between communication and computation.

The paper's microbenchmarks of DeepEP's FP8 communication (Table 1) are particularly revealing. For smaller workload scales (corresponding to DeepSeek V2-Lite configurations at EP=8), the quantization and dequantization operations surrounding the all-to-all communication each consume approximately the same time as the communication kernel itself—roughly 0.08–0.13 ms for quantization versus 0.33 ms for communication. This means the Q/DQ overhead nearly eliminates the 1.6× theoretical speedup of FP8 communication over BF16, reducing the effective gain to approximately 1.0× (i.e., no improvement). At larger scales, the communication cost grows roughly linearly with data volume while Q/DQ cost remains constant, so the relative overhead shrinks—but even at the largest tested configurations (V3-scale, EP=32), a single Q/DQ pair reduces the FP8 communication speedup from 1.6× to 1.4×. With multiple such pairs per MoE pass, the cumulative erosion is substantial.

More subtly, the weight gradient computation (Wgrad) stage exposes a layout inconsistency problem that prior work handled through brute-force requantization. The activation tensor consumed by the forward pass and its gradient (Dgrad) is quantized row-wise (scaling factors computed along the row dimension, with each 128-element tile spanning consecutive tokens). However, the weight gradient computation requires column-wise quantized inputs. Because the preceding all-to-all dispatch transmits only the row-wise format (to leverage reduced communication size), the Wgrad stage must convert from row-wise to column-wise quantization. Existing implementations perform this conversion as dequantize-to-BF16 → transpose → requantize-to-FP8-columnwise, introducing an additional pair of quantization operations. Critically, because the scaling factors differ between the row-wise and column-wise tilings (they are computed over different 128-element subsets of the transposed tensor), this double quantization introduces compounded rounding error—the same underlying value gets discretized twice against different quantization grids, with no guarantee that the second discretization preserves the information loss from the first.

The Unrecognized Tradeoff: Q/DQ Overhead vs. Double Quantization Error

The paper's key diagnostic insight is that prior work treated Q/DQ reduction and numerical stability as separate concerns, without recognizing that they are locked in a direct tradeoff through the layout conversion problem. If you keep the dataflow in BF16 between FP8 kernels, you pay the Q/DQ overhead but avoid double quantization error (since dequantized BF16 values are "continuous enough" that requantizing along a different axis adds negligible additional error). If you eliminate the BF16 intermediate and keep tensors in FP8 throughout, you avoid the overhead but the row-to-column requantization introduces compounded error that can destabilize training.

This tradeoff is not a general property of FP8—it is specific to per-token (1D) quantization with different tiling directions. With per-tensor quantization (a single scaling factor for the entire tensor), transposing and requantizing would produce identical results since the scaling factor is invariant to layout. But per-token quantization, which is necessary to preserve accuracy for tensors with wide dynamic range across tokens, creates scaling factors that are functions of the tiling direction. A row-wise tile of 128 elements and a column-wise tile of 128 elements sample different subsets of the tensor and thus compute different scaling factors. The double quantization error defined in Equation 1—$E = Q_{col}(D(Q_{row}(X))) - Q_{col}(X)$—captures exactly this discrepancy: the compounded error from quantizing, dequantizing, and requantizing along a different axis versus quantizing directly along the target axis.

Prior work either avoided this problem entirely (by keeping intermediates in BF16, as TE does) or accepted it as an unavoidable cost of FP8 adoption (as the DeepSeek-V3 approach implicitly does). Neither approach recognized that the row-to-column conversion could be performed without requantization at all, by exploiting properties of the FP8 representation itself.

How This Paper Positions Itself

The paper positions FP8-Flow-MoE not as yet another FP8 kernel library or quantization scheme, but as a dataflow architecture—a systematic reorganization of how tensors flow through the MoE computation graph to minimize format conversions while maintaining numerical consistency. The key conceptual move is redefining the problem from "how do we accelerate individual operations with FP8?" to "how do we design an end-to-end FP8 dataflow where the format is native and persistent?"

This reframing manifests in two technical contributions that directly resolve the tradeoff prior work faced:

First, the scaling-aware transpose eliminates the row-to-column conversion's double quantization error by performing the layout transformation entirely within the FP8 representation, manipulating exponent bits rather than dequantizing and requantizing values. This removes the need for an intermediate BF16 buffer at the Wgrad boundary, reducing casts while simultaneously improving numerical fidelity (since no second quantization is performed). The paper proves this formally (Equations 10–16): when scaling factors are powers of two and a shared scaling factor is used across each 128×128 transposed block, column-wise quantization can be derived from row-wise quantization by purely adjusting exponent bits, with no loss of information beyond what the initial row-wise quantization already incurred.

Second, the fused operator suite (fused permute-padding, fused SwiGLU-quantization, fused unpermute-unpadding) absorbs quantization operations into surrounding compute kernels, eliminating standalone Q/DQ kernel launches entirely. This is not merely an engineering optimization—it is what makes the FP8-centric dataflow practically realizable, since without fusion the quantization operations that the persistent FP8 dataflow requires would themselves become performance bottlenecks (as Table 1 demonstrates for communication). The fusion strategy effectively makes quantization "free" in terms of kernel launch overhead and HBM traffic, since it is amortized into operations that must happen anyway (SwiGLU activation, token permutation).

Figure 2 provides the architectural comparison that crystallizes the paper's positioning: four computational graphs for the same MoE module under BF16 baseline, TE blockwise, DeepSeek-V3-style, and FP8-Flow-MoE. The progression shows a systematic reduction in precision transitions: the BF16 baseline has none but operates entirely in higher precision; the TE blockwise approach confines FP8 to GEMMs but adds Q/DQ boundaries; the DeepSeek-style approach extends FP8 to communication but multiplies the Q/DQ operations (up to 12); and FP8-Flow-MoE collapses nearly all boundaries, retaining BF16 only at two essential points—between the first grouped linear and the activation function (where FP8 reductions would risk overflow), and between the second grouped linear and the backward combination dispatch (where nonlinear transformations amplify quantization error). The reduction from 12 cast operations to 2 is the quantitative signature of the approach, and the paper argues this represents a paradigm shift from BF16-centric to FP8-centric computation.

The paper's evaluation strategy reinforces this positioning. Rather than comparing against a single baseline, the experiments (Section 4) use a three-way comparison: BF16 (the standard mixed-precision baseline), Blockwise (the TE-style conservative FP8 integration), and FP8-Flow-MoE. This structure isolates the contribution of the dataflow redesign from the contribution of simply using FP8 kernels: the Blockwise vs. BF16 comparison measures the raw value of FP8 GEMM acceleration (typically 3–8% improvement), while FP8-Flow-MoE vs. Blockwise measures the additional value of eliminating precision transitions and fusing operators (an additional 3–21% improvement, growing with expert parallelism). The widening gap at higher expert parallelism levels (EP=32) demonstrates that the dataflow redesign addresses the scaling bottleneck—communication and quantization overhead—that the kernel-level approach cannot.

3. Technical Approach

This is a systems paper whose core idea is that the gap between FP8's theoretical 2× throughput and its realized single-digit gains in MoE training arises from a specific design error—redundant quantize–dequantize conversions at layout-inconsistent boundaries introduce both performance overhead (from excessive kernel launches) and numerical error (from double quantization)—and that both problems can be eliminated simultaneously by a scaling-aware transpose operator that converts between row-wise and column-wise FP8 layouts via exponent-bit manipulation, combined with operator fusion that absorbs quantization into surrounding compute kernels, yielding a casting-free FP8-centric dataflow with only two explicit precision transitions in the entire MoE forward-backward pass.

3.1 Reader Orientation

The paper builds a training recipe—a specific configuration of data layouts, memory management, and kernel implementations—for running the Mixture-of-Experts computation entirely in FP8 precision without the constant "cast, compute, cast back" pattern that makes existing FP8 systems slow. The problem it solves is that existing FP8 training frameworks spend so much time converting between FP8 and BF16 formats (quantize→compute→dequantize→requantize) that they erase most of FP8's speed advantage, and the straightforward fix—keeping everything in FP8 all the time—introduces a numerical error called double quantization that can destabilize training. The solution shape is a quantization-consistent FP8 dataflow: a dataflow where every tensor stays in FP8 throughout the MoE stage, but where the crucial layout conversion (row-wise to column-wise quantization) is performed directly on the FP8 encoding bits rather than through dequantization and requantization, so that no second quantization error is ever introduced.

3.2 Big-Picture Architecture (Diagram in Words)

The FP8-Flow-MoE system has five major components that work together to eliminate precision transitions from the MoE computation graph:

1. Scaling-Aware Direct Transpose — an operator that converts FP8 tensors between row-wise quantized and column-wise quantized layouts by adjusting only the exponent bits of the FP8 encoding, without dequantizing or requantizing the underlying values. This is the linchpin component that makes the persistent FP8 dataflow numerically stable.

2. Fused Permute-Padding Kernel — a single CUDA kernel that combines token reordering (permutation into expert-contiguous batches) with shape alignment (padding to multiples of 16 for Tensor Core efficiency), eliminating a separate kernel launch and redundant HBM read-write cycle.

3. Fused SwiGLU-Quantization Kernel — a kernel that applies the SwiGLU activation function and immediately quantizes the output to FP8 in a single pass, absorbing the quantization operation into the activation computation rather than executing it as a standalone kernel.

4. Fused Unpermute-Unpadding Kernel — the backward-pass counterpart to the permute-padding fusion, combining unpermutation (scattering expert outputs back to token order) with unpadding (removing shape-alignment padding) into one kernel.

5. FP8 Communication Primitives (DeepEP Integration) — the all-to-all dispatch and combine operations that transmit FP8 tensors and their scaling factors between GPUs, with the quantization and dequantization operations integrated into the surrounding fused kernels rather than performed as standalone pre- and post-communication steps.

Information flows as follows: input tokens enter the MoE stage in FP8 format → a fused permute-padding reorders and aligns them for expert computation → the first grouped linear executes in FP8 → the fused SwiGLU-quantization applies activation and produces FP8 output → the scaling-aware transpose converts layout for weight gradient computation (in the backward pass) → the second grouped linear executes in FP8 → fused unpermute-unpadding restores token order → output tokens exit the MoE stage in FP8. The only two locations where tensors leave FP8 are (1) between the first grouped linear and the SwiGLU activation (where BF16 is needed because FP8 exponent range risks overflow during the element-wise operations inside SwiGLU), and (2) between the second grouped linear and the backward combination dispatch (where nonlinear gradient transformations amplify quantization errors). Every other tensor remains in FP8 from entry to exit.

3.3 Roadmap for the Deep Dive

• First, the scaling-aware transpose—the fundamental mechanism that makes the persistent FP8 dataflow numerically viable, including the formal proof that exponent-bit manipulation avoids double quantization error when scaling factors are powers of two.
• Second, the double quantization error analysis—what it is, why it arises specifically from layout-inconsistent requantization, and why prior approaches (BF16 intermediates, naïve requantization) are either performantly expensive or numerically dangerous.
• Third, the casting-free FP8 dataflow architecture—how the 12-to-2 reduction in cast operations is achieved, with a step-by-step comparison against BF16, TE blockwise, and DeepSeek-style dataflows.
• Fourth, the fused kernel suite—the specific fusions (permute-padding, SwiGLU-quantization, unpermute-unpadding) and why fusion is necessary (not just beneficial) in an FP8-centric dataflow.
• Fifth, the integration with communication primitives—how FP8-Flow-MoE interacts with DeepEP all-to-all operations to absorb quantization overhead into the fused kernels bordering communication rather than treating quantization as a standalone pre-/post-communication step.

3.4 Detailed, Sentence-Based Technical Breakdown

The Double Quantization Error Problem

The paper identifies double quantization error as the fundamental numerical obstacle to a persistent FP8 dataflow, and its analysis of this error provides the motivation for every design decision that follows. Understanding this error requires first understanding how per-token FP8 quantization works in current training systems.

How per-token quantization operates. Modern FP8 training systems (TransformerEngine, DeepSeek-V3) use 1D per-token quantization for activations, also called per-block or per-tile quantization. Given an activation tensor, the quantization proceeds in tiles of 128 consecutive elements along a specified dimension. For each 128-element tile, the system computes a single scaling factor and then quantizes all 128 values using that factor. The scaling factor is calculated as:

$s = \frac{\max_{0 \leq i < 128} |x_i|}{448}$

where $|x_i|$ is the absolute value of element $i$ in the tile, and 448 is the maximum representable value in the FP8 E4M3 format (which allocates 4 exponent bits and 3 mantissa bits, yielding a maximum normalized value of 448 = $2^8 \times 1.75$ after accounting for the exponent bias).

What this equation computes: for each 128-element tile, find the largest absolute value among the 128 elements, divide it by 448 (the FP8 E4M3 ceiling), and use the result as a per-tile scaling factor. Under this scaling, the largest element in the tile will map to approximately 448 in the scaled domain, which is exactly the saturation point of FP8 E4M3. All other elements in the tile will be proportionally smaller. The quantization then rounds each scaled value to the nearest representable FP8 number:

$Q_{\text{row}}(x_i) = \text{round}\left(\frac{x_i}{s}\right)$

Why this form: the per-tile scaling strategy ensures that the full dynamic range of FP8 is utilized for each tile independently, rather than being constrained by the global dynamic range of the tensor. Without per-tile scaling (i.e., with a single per-tensor scaling factor), a tensor with one exceptionally large outlier value would force all other values to be quantized with a very coarse step size, losing precision for the majority of elements. Per-tile scaling confines each outlier's influence to its own 128-element neighborhood.

The layout-dependent scaling factor problem. The critical property that creates double quantization error is that the scaling factor $s$ depends on which 128 elements are grouped together—and this grouping changes when the tensor is transposed. Consider a 2D tensor with row-wise quantization: each row is divided into 128-element tiles along the column dimension, so each tile spans 128 consecutive columns within one row. If this tensor is then transposed and quantized column-wise, the tiles now span 128 consecutive rows within one column. These are different subsets of the tensor elements, so their maximum absolute values will generally differ, producing different scaling factors. The paper formalizes the error introduced by this inconsistency:

$E = Q_{\text{col}}(D(Q_{\text{row}}(X))) - Q_{\text{col}}(X)$

where $Q_{\text{row}}(\cdot)$ is row-wise quantization (tiles along columns), $Q_{\text{col}}(\cdot)$ is column-wise quantization (tiles along rows), and $D(\cdot)$ is dequantization (multiplying by the scaling factor to return to the original value range, though with rounding error preserved).

What this equation computes: the difference between two ways of obtaining a column-wise quantized tensor. The left term $Q_{\text{col}}(D(Q_{\text{row}}(X)))$ represents the compound operation: first quantize row-wise, then dequantize back to a continuous range (but with the rounding error from the first quantization baked in), then requantize column-wise. The right term $Q_{\text{col}}(X)$ represents the ideal: quantize column-wise directly from the original continuous tensor. The difference $E$ is the double quantization error—the additional error introduced solely by the intermediate dequantize–requantize cycle with mismatched scaling factors. If $E$ were zero, the two paths would be equivalent and the intermediate cycle would introduce no extra degradation. In practice, $E$ is non-zero because the two quantizations operate with different scaling factors against different tile groupings, and the rounding from the first quantization cannot be undone by the second.

Why this form matters: this definition isolates the error specifically attributable to the layout-inconsistent requantization, separate from the unavoidable error of any single quantization. When the same quantization direction is applied twice (row-wise → dequantize → row-wise again), the error is identically zero because the same scaling factor is recovered and the rounding operation is idempotent under deterministic round-to-nearest. This is proven formally in Equations 5–8 of the paper. The problem is exclusively the change in quantization direction, not quantization in general.

The root cause in the MoE backward pass. The specific location where this error occurs in MoE training is the weight gradient (Wgrad) computation. During the forward pass, activations are quantized row-wise and consumed by the first grouped linear (Fprop in Figure 2). These same activations, or their gradients, must be used for the weight gradient computation in the backward pass. However, the weight gradient GEMM requires column-wise quantized inputs (because the gradient computation involves multiplying the activation gradient by the transposed input activation, and the tiling for efficient GEMM requires the input to be quantized along the reduction dimension). The preceding all-to-all communication (dispatch) transmits data in row-wise format to leverage FP8's halved communication size. When the Wgrad stage receives this row-wise data, existing implementations perform dequantize → transpose → requantize column-wise, incurring double quantization error because the row-wise scaling factors (computed over 128 consecutive columns) differ from the column-wise scaling factors (computed over 128 consecutive rows after transposition).

Scaling-Aware Transpose: Exponent-Bit Manipulation Without Requantization

The paper's central technical insight is that the row-to-column conversion can be performed entirely within the FP8 representation without dequantizing and requantizing the underlying values, by exploiting the fact that FP8 scaling factors, when constrained to powers of two, correspond to pure shifts of the exponent field. This is not an approximation or a heuristic—under the power-of-two constraint, the conversion is an exact transformation that preserves all information from the original quantization.

The power-of-two scaling factor constraint. The key enabling condition is that all scaling factors must be powers of two. When a scaling factor $s$ is a power of two, say $s = 2^T$ where $T$ is an integer, multiplying or dividing by $s$ is equivalent to adding or subtracting $T$ from the exponent of a floating-point number. This property is not generally true for arbitrary scaling factors—if $s = 384.7$, for instance, scaling a value changes both its exponent and its mantissa in non-trivial ways—but it holds exactly when scaling factors are powers of two. The paper constrains its scaling factors to this form, which is a common design choice in low-precision systems because it simplifies the hardware for scaling operations and can be enforced during quantization by rounding the computed scaling factor to the nearest power of two.

Formal proof of error-free conversion. The paper provides a formal derivation (Equations 10–16) showing that under the power-of-two constraint, a column-wise quantized representation can be derived from a row-wise quantized representation by purely adjusting exponent bits, with no information loss beyond what the original row-wise quantization already incurred. Let a value $x$ be row-wise quantized with scaling factor $s = 2^T$, producing an FP8 encoding with sign bit $SN$, exponent bits $E$, and mantissa bits $M$:

$Q_{\text{row}}(x) = (-1)^{SN} \cdot 2^{E-7} \cdot (1 + M/8) \cdot 2^T$

where $SN$ is the sign bit (0 for positive, 1 for negative), $E$ is the 4-bit exponent field of the FP8 E4M3 encoding (biased such that the actual exponent is $E-7$), $(1+M/8)$ is the mantissa contribution (with 3 mantissa bits $M$ encoding values 0–7, producing a mantissa factor between 1.0 and 1.875 in steps of 0.125), and $2^T$ is the per-tile scaling factor applied after the FP8 encoding.

What this equation represents: the actual numerical value represented by an FP8-encoded number after accounting for both the FP8 format's internal encoding (sign, exponent, mantissa) and the external per-tile scaling factor. The exponent bias of 7 comes from the FP8 E4M3 specification (4 exponent bits with bias 7), and the mantissa encoding $(1+M/8)$ reflects that FP8 E4M3 allocates 3 mantissa bits. The product structure means that doubling the scaling factor ($T \to T+1$) is equivalent to incrementing the exponent field ($E \to E+1$), because $2^{E-7} \cdot 2^{T} = 2^{(E+1)-7} \cdot 2^{T-1}$.

Now, for column-wise quantization, the same underlying value would be encoded with a different scaling factor $s' = 2^{T'}$:

$Q_{\text{col}}(x) = (-1)^{SN'} \cdot 2^{E'-7} \cdot (1 + M'/8) \cdot 2^{T'}$

The conversion step. Since both $T$ and $T'$ are integers, their difference $D = T' - T$ is also an integer. If we set the column-wise sign bit to match the row-wise sign bit ($SN' = SN$), adjust the exponent by the scaling factor difference ($E' = E - D$), and keep the mantissa unchanged ($M' = M$), then:

$Q_{\text{col}}(x) = (-1)^{SN} \cdot 2^{(E-D)-7} \cdot (1 + M/8) \cdot 2^{D+T}$ $= (-1)^{SN} \cdot 2^{E-D-7+D+T} \cdot (1 + M/8)$ $= (-1)^{SN} \cdot 2^{E-7+T} \cdot (1 + M/8)$ $= Q_{\text{row}}(x)$

What this derivation proves: starting from a row-wise quantized FP8 value with scaling factor $2^T$, we can produce a column-wise quantized FP8 value with scaling factor $2^{T'}$ by simply subtracting $(T' - T)$ from the exponent field and leaving the sign and mantissa untouched. This operation is exact—there is no rounding, no requantization, and no second discretization. The resulting encoding represents exactly the same numerical value as the original, just expressed relative to a different per-tile scaling factor. The information loss from quantization occurred once (during the initial row-wise quantization) and is not amplified by the transpose.

Why this property holds only for power-of-two scaling: if scaling factors were arbitrary real numbers, the ratio $s'/s$ would not be a pure power of two, and the conversion would require multiplying or dividing the mantissa as well as the exponent—which would involve rounding the mantissa and thus introduce a second quantization error. The power-of-two constraint collapses the rescaling operation into a pure exponent shift, which in floating-point representation is an exact operation (no mantissa bits are affected).

The Scaling-Aware Direct Transpose algorithm. Algorithm 1 in the paper operationalizes this conversion. Given a row-wise quantized tensor $X^{\text{row}}_{i,j}$ (stored in FP8 with associated per-tile scaling factors $S^{\text{row}}$), the algorithm produces a column-wise quantized tensor $X^{\text{col}}$:

1. Initialize the column-wise output tensor by transposing the data layout: $X^{\text{col}}_{i,j} = X^{\text{row}}_{j,i}$. This is a pure data movement operation—the FP8 bit patterns are copied unchanged; only their positions in the 2D layout change.
2. For each $128 \times 128$ block of the transposed tensor, find the maximum scaling factor among all row-wise scaling factors that overlap with this block: $S^{\text{max}} = \max_{0 \leq i < 128} S^{\text{row}}_i$. This step ensures that all elements in the block share a common scaling factor for column-wise quantization. Using the maximum (rather than the minimum or average) prevents overflow: if a value was correctly represented in row-wise format with scaling factor $S^{\text{row}}_i$, scaling it to a larger factor $S^{\text{max}} \geq S^{\text{row}}_i$ will not cause overflow because the value's magnitude relative to the new scaling factor is smaller, meaning its exponent in the FP8 encoding will decrease (or stay the same), not increase.
3. Set the column-wise scaling factor for the entire block to $S^{\text{col}} = S^{\text{max}}$.
4. For each element in the block, extract its sign bit, exponent field, and mantissa field from the FP8 encoding. Compute the exponent adjustment $k = \log_2(S^{\text{max}} / S^{\text{col}}_i)$, which is an integer because both scaling factors are powers of two. Update the exponent: $E_{\text{new}} = E - k$. The subtraction is safe because $k \geq 0$ (since $S^{\text{max}} \geq S^{\text{col}}_i$), so the exponent decreases or stays the same—the value's magnitude relative to the new scaling factor is smaller, requiring a smaller exponent to represent. Reassemble the FP8 encoding using the original sign, the new exponent, and the original mantissa.

What this algorithm accomplishes: it transforms a row-wise quantized FP8 tensor into a column-wise quantized FP8 tensor without ever recovering the original BF16 values, without computing any rounding operations, and without introducing any new quantization error. The only numerical operation is integer addition/subtraction on the exponent field. The algorithm's correctness depends on three conditions: (1) all scaling factors are powers of two, (2) the block-level maximum is used for the target scaling factor to avoid overflow, and (3) the resulting exponent field remains within the representable range of FP8 (no underflow to zero or overflow beyond the maximum exponent). The paper does not explicitly discuss underflow handling, but since the adjustment $k$ is always non-negative (exponent decreases or stays the same), underflow to subnormal or zero could occur for very small values. This is a practical consideration that the implementation must handle, likely by clamping exponents to the minimum representable value.

Experimental validation of speedup. Figure 1 compares the Direct Transpose operator against the naïve dequantize → transpose → requantize approach across multiple tensor shapes representative of MoE workloads (ranging from (28672, 1280) to (32768, 9216)). Across all shapes, the Direct Transpose achieves a 2× to 3× speedup over the naïve approach. The speedup comes from two sources: (1) eliminating the dequantization and requantization computations (which involve floating-point multiplications and rounding operations per element), and (2) reducing the number of passes over the data in HBM from three (read for dequantize, write transposed, read for requantize, write quantized) to effectively one (read, bit-manipulate, write transposed).

Casting-Free FP8 Dataflow Architecture

With the scaling-aware transpose resolving the numerical obstacle, the paper designs a dataflow where tensors remain in FP8 throughout the MoE stage except at two isolated boundaries. The design is best understood by tracing the forward and backward passes through an MoE module and identifying where precision transitions occur in each baseline, then showing how FP8-Flow-MoE eliminates them.

The computational graph of an MoE module. An MoE module (Figure 2) consists of the following stages in sequence:

1. Token dispatch (all-to-all communication): input tokens are routed by a learned router to their assigned experts. Tokens destined for experts on other GPUs are transmitted via an all-to-all collective. Tokens arrive at each GPU in the order they were received from the network, not necessarily grouped by expert.

2. Permutation: tokens are reordered so that all tokens assigned to expert 0 are contiguous, then all tokens for expert 1, and so on. This produces dense contiguous blocks that can be fed to batched GEMM operations.

3. Padding: zero tokens are appended to each expert's batch to ensure its size is a multiple of 16 (or another alignment requirement), enabling efficient use of Tensor Core hardware which operates on fixed-size tiles.

4. Grouped Linear 1 (gate projection in SwiGLU): the first of the two linear transformations inside each expert's feed-forward network. For SwiGLU-activated experts, this is typically the gate projection.

5. Activation function (SwiGLU): the element-wise nonlinearity combining the gate and up projections. SwiGLU involves a sigmoid-gated linear unit: $\text{SwiGLU}(x) = x \cdot \sigma(x \cdot W_{\text{gate}}) \cdot (x \cdot W_{\text{up}})$, which requires element-wise multiplication and a sigmoid operation.

6. Grouped Linear 2 (down projection): the second linear transformation that projects the activated hidden representation back to the model dimension.

7. Unpermutation: the reverse of permutation—expert outputs are scattered back to their original token positions (the order before permutation).

8. Unpadding: removal of the zero-token padding that was inserted for alignment.

9. Token combine (all-to-all communication): expert outputs are transmitted back to the GPUs where the original tokens originated, via a reverse all-to-all collective.

The backward pass traverses this graph in reverse, with the addition of a weight gradient (Wgrad) computation at each grouped linear, which requires the input activations (saved from the forward pass) in a specific quantization layout.

Baseline dataflows and their cast operations. The paper analyzes three existing dataflow patterns (Figure 2a–c) before presenting FP8-Flow-MoE (Figure 2d):

• BF16 baseline (Figure 2a): all tensors are stored in BF16. No FP8 quantization occurs anywhere. This is the reference for both performance (throughput and memory) and numerical fidelity. There are zero Q/DQ operations, but also no FP8 acceleration.

• TransformerEngine blockwise recipe (Figure 2b): FP8 is used only inside the grouped linear GEMMs. Before each GEMM, the input activations and weights are quantized to FP8; after the GEMM, the output is dequantized back to BF16. All other operations—communication, permutation, activation functions, padding—remain in BF16. This introduces Q/DQ pairs at each GEMM boundary: quantize before Fprop, dequantize after Fprop, quantize before Dgrad, dequantize after Dgrad, and so on. For two grouped linears in the forward pass and their corresponding backward operations, this yields approximately 8 explicit cast operations.

• DeepSeek-V3 style (Figure 2c): FP8 is extended to the communication stage: the all-to-all dispatch transmits FP8 tensors (with accompanying scaling factors) rather than BF16. However, the integration is still BF16-centric in that the FP8 tensors are quantized immediately before communication and dequantized immediately after. The Wgrad stage requires row-to-column conversion, implemented as dequantize → transpose → requantize, introducing double quantization error. The paper counts 12 explicit cast operations in this dataflow for a single MoE forward-backward pass.

FP8-Flow-MoE dataflow (Figure 2d). The proposed dataflow reduces cast operations from 12 to 2 by maintaining FP8 throughout the pipeline and eliminating the need for format conversions at boundaries. The two remaining cast operations occur at:

1. Between Grouped Linear 1 and the SwiGLU activation: the output of the first grouped GEMM (which is the gate projection in SwiGLU) is dequantized to BF16 before entering the activation function. FP8 cannot safely represent the intermediate results of the SwiGLU computation—the element-wise multiplication of gated and up-projected values produces numbers that can span a wider dynamic range than FP8 E4M3 can accommodate without overflow, and the sigmoid activation requires computing $\exp(-x)$, which amplifies any quantization error in its input. This boundary is unavoidable in the current FP8 format design. The paper retains BF16 locally for this computation and requantizes the output of SwiGLU back to FP8 for the second grouped linear (but this requantization is fused into the SwiGLU kernel itself—see Section 3.3.2).

2. Between Grouped Linear 2 and the backward combination/dispatch: during the backward pass, the gradient flowing from the combination operation into the second grouped linear's weight gradient computation encounters similar numerical challenges. The combination involves summation over expert outputs, which can accumulate FP8 quantization errors across many tokens and produce gradients with unpredictable dynamic range. This boundary retains BF16 to ensure stable gradient propagation.

Every other tensor in the graph—the token dispatch communication, the permuted and padded activations, the grouped GEMM inputs and outputs (except the two noted boundaries), the unpermuted outputs, and the token combine communication—remains in FP8 continuously.

How the 12-to-2 reduction is achieved, step by step:

1. Communication casts eliminated by persistent FP8: the all-to-all dispatch transmits FP8 tensors directly, and the receive-side does not dequantize—the incoming FP8 data flows directly into the fused permute-padding kernel, which operates in FP8. Similarly, the combine transmission operates on FP8 tensors that were produced by the fused unpermute-unpadding kernel. This eliminates 2 Q/DQ pairs (quantize-before-send, dequantize-after-receive, and their backward counterparts).

2. GEMM boundary casts eliminated by persistent FP8 storage: in the BF16-centric dataflows, each grouped linear GEMM is surrounded by quantize (FP8 input) and dequantize (back to BF16 output) operations because the subsequent operations (activation, communication) expect BF16. In FP8-Flow-MoE, the subsequent operations are designed to consume and produce FP8, so the GEMM outputs remain in FP8. This eliminates 4 Q/DQ pairs (two grouped linears × forward/backward).

3. Wgrad double quantization eliminated by scaling-aware transpose: the row-to-column conversion for the weight gradient computation, which in the DeepSeek-V3 style requires dequantize → transpose → requantize (2 quantizations + 1 dequantization), is replaced by the scaling-aware transpose (0 quantizations, 0 dequantizations, pure bit manipulation). This eliminates 4 Q/DQ operations (the dequantize and requantize in both the forward and backward paths that feed the Wgrad).

4. Quantization fused into compute kernels: the quantization operations that remain necessary (e.g., converting the BF16 output of SwiGLU back to FP8 for the second grouped linear, or quantizing the input activations at the very entry point of the MoE module) are fused into the preceding compute kernel, so they do not appear as standalone cast operations in the graph. The SwiGLU-quantization fusion absorbs what would otherwise be an explicit quantize kernel launch. The paper counts only explicit cast operations that appear as distinct nodes in the computational graph—the fused quantization is not counted because it is amortized into an already-necessary computation.

The result is a computational graph (Figure 2d) where FP8 arrows connect nearly all nodes, with only two BF16 segments localized to the SwiGLU nonlinearity and the backward gradient aggregation point.

Why this matters beyond the cast count. The reduction from 12 to 2 explicit casts represents more than a factor-of-6 improvement in conversion overhead. It qualitatively changes the memory access pattern of the MoE module. Each Q/DQ operation requires reading the tensor from HBM, applying per-element scaling and rounding, and writing the result back to HBM (or to a different buffer). With 12 such operations, the total HBM traffic from format conversions alone can rival or exceed the traffic from the actual computation (GEMMs, activations, permutations). By collapsing these into fused kernels and eliminating them where possible, FP8-Flow-MoE significantly reduces HBM bandwidth pressure—which is often the primary bottleneck in large-scale training, not arithmetic throughput. This explains why the throughput gains (up to 21%) exceed what the raw arithmetic speedup of FP8 over BF16 (2× on GEMMs) would suggest if GEMMs were the only bottleneck.

Fused Operator Suite

The persistent FP8 dataflow would be theoretically elegant but practically inefficient without operator fusion, because the FP8-centric design introduces a different kind of overhead: small, lightweight operations that would otherwise be absorbed into BF16 buffers become explicit standalone kernels when performed in FP8. The paper identifies three fusion points that are critical for realizing the performance benefits of the dataflow redesign.

Fused Permute-Padding. The permute operation reorders tokens from their post-dispatch order (arrival order from the network) into expert-contiguous batches. The padding operation then appends zero-valued tokens to each expert's batch to reach a multiple of 16 entries, which is the alignment requirement for FP8 Tensor Core GEMMs (Tensor Cores operate on 16×16 matrix tiles, so the M and N dimensions must be multiples of 16). Executing these as separate kernels incurs redundant HBM traffic: the permute kernel reads the scattered tokens and writes them in expert-contiguous order, then the padding kernel reads the expert-contiguous data and writes it again with zeros appended. Since both operations are element-wise and data-independent (each token's destination depends only on its expert assignment, which is known after dispatch), they can be fused into a single kernel that computes the final padded position for each token directly and writes it exactly once.

The paper does not specify the exact thread-block mapping in the main text, but the principle is standard for such fusions: each thread block is assigned to a range of output positions in the padded layout, computes which input token (if any) maps to each output position (accounting for expert boundaries and padding), and performs a single coalesced read from the input buffer and write to the output buffer. The backward counterpart—fused unpermute-unpadding—applies the same fusion in reverse, combining the removal of padding and the scattering of expert outputs to original token positions into one kernel.

Experimental results (Figures 3–4 in the paper, labeled as Figures 4 in the text but presumably Figures 3–4 in the actual document). Figure 3 (labeled "Fig. 3: Performance comparison between fused and separate implementation of permute and padding kernels") shows the forward permute-padding speedup: across tensor shapes from (28672, 1280) to (32768, 9216), the fused kernel achieves up to 1.7× speedup over separate unfused kernels. Both BF16 and FP8 variants benefit from fusion, but the FP8 fused version shows proportionally larger gains because FP8's halved data size makes the kernel more latency-bound (the fixed cost of kernel launch and HBM access latency dominates more when the data volume is smaller). Figure 4 shows the backward unpermute-unpadding speedup: gains reach up to 6.6× on the largest configurations. The much larger speedup in the backward direction reflects the fact that unpermutation with unpadding involves an additional indirection (mapping from padded expert-contiguous indices back to original token positions, which may be scattered across the batch), and fusing the unpadding into the unpermutation eliminates an entire pass over the data that would otherwise read the padded layout, strip padding, and then scatter the results.

Why these fusions matter specifically for FP8-Flow-MoE. In a BF16 dataflow, the permute and pad operations operate on BF16 tensors, and the subsequent GEMM kernel expects BF16 input—the padding does not need to be a separate quantized step, and the transition from permute to GEMM involves no format change. In FP8-Flow-MoE, the permute operates on FP8 tensors that arrive from the all-to-all dispatch, and the output must be FP8 for the grouped GEMM. Without fusion, there would need to be additional kernels to manage the FP8 scaling factors during permutation (since per-tile scaling factors must be permuted along with the data) and to ensure the padded zeros have appropriate scaling factors. Fusing these into a single kernel handles the scaling factor bookkeeping internally, avoiding multiple kernel launches.

Fused SwiGLU-Quantization. This fusion addresses a subtler overhead that arises from FP8-Flow-MoE's design choice to keep the SwiGLU activation in BF16 (to avoid FP8 overflow and error amplification in the nonlinearity). The output of the first grouped linear is dequantized to BF16, the SwiGLU activation is computed in BF16, and then the result must be requantized to FP8 to continue the FP8 dataflow into the second grouped linear. If the requantization were a standalone kernel, it would consume time comparable to the SwiGLU computation itself (since both are element-wise operations of similar complexity). The fusion performs the SwiGLU activation and the quantization in a single pass: each thread computes the SwiGLU output value in BF16 registers, immediatelly computes the per-tile scaling factor (by finding the maximum absolute value in its assigned tile), quantizes the value to FP8, and writes the FP8 result to HBM. The scaling factors are written to a separate scale buffer.

The paper's microbenchmark (Figure 5) shows that the fused kernel achieves nearly identical latency to the standalone SwiGLU kernel from NVIDIA's Megatron framework, while producing FP8 outputs. This means the quantization step is essentially "free"—it is amortized into the required SwiGLU computation. The alternative (separate SwiGLU followed by a quantize kernel) would approximately double the kernel launch and HBM access cost for this stage.

Why this fusion is critical now but might not be in future hardware. The need to keep SwiGLU in BF16 is a limitation of the FP8 E4M3 format's limited dynamic range, not a fundamental property of low-precision training. FP8 E5M2 (5 exponent bits, 2 mantissa bits) has a wider dynamic range (maximum value ~57,344 versus ~448 for E4M3) at the cost of reduced precision, and future FP6 or FP4 formats may handle nonlinearities natively. If SwiGLU could be computed entirely in FP8, this fusion would become unnecessary—but given current hardware and format constraints, fusing the BF16-to-FP8 requantization into the activation kernel is the most efficient way to bridge the precision mismatch.

Integration with FP8 communication (DeepEP). The paper's analysis of FP8 communication using DeepEP (Table 1) reveals why operator fusion at communication boundaries is so important. The table measures latency for BF16 communication versus FP8 communication (with explicit Q/DQ) across multiple scales. FP8 communication theoretically halves the data volume (8 bits versus 16 bits per element), but in practice the speedup is limited to approximately 1.4×–1.6× because both the FP8 data tensor and its associated scaling factors must be transmitted (doubling the number of buffers and synchronization points), and because the quantization and dequantization operations surrounding each communication call consume additional time.

For a small workload (DeepSeek V2-Lite, EP=8, (24576, 2048) shape), the Q/DQ time (0.127 ms for quantize + 0.084 ms for dequantize = 0.211 ms) nearly equals the communication time (0.325 ms). The total FP8 path time (0.535 ms) is barely faster than BF16 (0.537 ms), yielding a speedup of essentially 1.0×. The FP8 communication advantage is completely neutralized by the surrounding conversions.

At larger scales (V3-scale, EP=32), the communication time grows to 4.319 ms while the Q/DQ time remains roughly constant at ~0.082 ms each. The overhead is proportionally smaller, but still reduces the speedup from 1.6× (if the Q/DQ were free) to 1.4×. Over a full forward-backward pass with multiple communication calls, this overhead compounds.

FP8-Flow-MoE addresses this by eliminating standalone Q/DQ around communication entirely. The quantization that would occur before dispatch is fused into the preceding kernel (e.g., the input processing or the previous layer's output projection). The dequantization that would occur after dispatch is eliminated because the incoming FP8 data flows directly into the fused permute-padding kernel, which already operates on FP8. The scaling factors that accompany the FP8 data through communication are propagated alongside the tensors into the fused kernels. This means the "Q/D" column in Table 1 would read zero for FP8-Flow-MoE, and the full 1.4×–1.6× communication speedup would be realized without erosion.

Design Choices and Their Justifications

Power-of-two scaling factors everywhere. The paper constrains all per-tile scaling factors to be powers of two—a deliberate choice that trades a small amount of quantization precision for large gains in conversion efficiency. Non-power-of-two scaling factors can produce slightly tighter fits to the actual data distribution (e.g., if a tile's maximum is 300, the scaling factor 300/448 ≈ 0.67 fits better than the nearest power of two, 0.5 or 1.0). However, the precision loss from powers-of-two scaling is minor (at most a factor of 2 in step size, which in floating-point representation manifests as a 1-ulp change in the mantissa for some values), while the benefit—enabling exact exponent-bit-based layout conversion without requantization—is decisive for the persistent FP8 dataflow. Without this constraint, the scaling-aware transpose would not be exact, and double quantization error would return.

BF16 retained at SwiGLU rather than at every GEMM. The paper could have retained BF16 at every GEMM boundary (as TE does) and still used FP8 kernels inside GEMMs—this is the conservative approach. The design choice to push FP8 persistence as far as possible (to within one operator of the nonlinearity) reflects an analysis that the Q/DQ overhead at GEMM boundaries is the dominant performance cost, while the numerical risk of FP8 inside element-wise operations is concentrated at the nonlinearity and reduction points. The paper implicitly argues that SwiGLU and the backward gradient aggregation are the only operations where FP8's limited precision demonstrably threatens training stability, and that all other operations—permutation, communication, padding, GEMM accumulation (which occurs in FP32 internally to Tensor Cores and is then converted to FP8 output)—can safely operate in FP8. This is an empirical claim supported by the convergence results (Figure 6) but not formally proven.

Fusion strategy: why fuse into existing kernels rather than write new ones? The paper's fusion choices—permute-padding, SwiGLU-quantization, unpermute-unpadding—all follow the principle of absorbing quantization into an already-required operation rather than creating new composite kernels. This is pragmatic: it minimizes the amount of new kernel code that must be written, tested, and maintained, and it ensures that the fused kernels can be drop-in replacements for existing unfused kernels in frameworks like Megatron-LM. The alternative—designing entirely new MoE-specific fused kernels that combine, say, permute + quantize + grouped GEMM—would achieve even lower overhead but would be substantially harder to integrate into existing training frameworks and would duplicate functionality already present in highly optimized libraries like DeepGEMM.

Single shared scaling factor per 128×128 block during transpose. Algorithm 1 chooses the maximum scaling factor among the overlapping row-wise tiles as the shared scaling factor for the column-wise output block. Using the maximum (rather than the minimum or average) prevents overflow: if a value was representable in row-wise format with its original scaling factor, scaling it by a larger factor (i.e., reducing its magnitude relative to the representable range) will not cause its exponent to exceed the maximum representable exponent. The downside is that some values may underflow to zero if their original scaling factor was much smaller than the block maximum—the exponent adjustment could push them below the minimum representable exponent. The paper does not explicitly discuss underflow handling, but standard practice would be to clamp subnormal values to zero, accepting a small precision loss for extreme outliers in exchange for the computational simplicity of the block-uniform scaling factor.

Why not use per-tensor quantization to avoid the layout problem entirely? Per-tensor quantization (one scaling factor for the entire tensor) is invariant under transposition—row-wise and column-wise quantization would produce identical results because the single scaling factor is computed from the same set of elements regardless of layout. This would eliminate double quantization error without any algorithmic cleverness. The reason per-tensor quantization is not used is accuracy: activation tensors in transformer models often span several orders of magnitude across tokens and across feature dimensions. A single global scaling factor would compress this dynamic range into FP8's limited representation, causing many small values to underflow to zero and degrading model quality. Per-tile quantization with 128-element tiles is the current best-practice tradeoff between quantization granularity (finer tiles preserve more precision for small values) and storage overhead (each tile requires storing one scaling factor, so smaller tiles mean more scaling factor data). The paper inherits this standard choice and solves the transpose problem that it creates, rather than retreating to a coarser quantization strategy that would avoid the problem but harm accuracy.

Compatibility with TransformerEngine and Megatron-LM. The paper emphasizes that FP8-Flow-MoE is designed as a "plug-and-play" recipe compatible with existing training frameworks. This is not merely a software engineering convenience—it reflects a strategic choice about how to introduce FP8-centric dataflows into the ecosystem. Rather than requiring a complete rewrite of the training stack (new data formats, new communication collectives, new checkpointing logic), FP8-Flow-MoE operates within the existing abstractions: tensors that are FP8 are still PyTorch tensors (with a different dtype), the fused kernels replace existing unfused kernels at the operator level, and the scaling-aware transpose is a new operator that slots into the graph at the Wgrad boundary. The paper notes that several kernels have already been merged into upstream TransformerEngine, reducing the integration burden for adopters.

4. Key Insights and Innovations

Innovation 1: The Double Quantization Error as a First-Class Diagnostic Concept

The paper's most conceptually distinctive contribution is not a new algorithm but a diagnostic reframing — it identifies and formalizes "double quantization error" as the root cause of the tension between FP8 performance and numerical stability, converting what prior work treated as an amorphous "FP8 is tricky" concern into a precise, analyzable phenomenon with a specific mathematical signature and a specific operational trigger (layout-inconsistent requantization).

Before this paper, the field understood that aggressive FP8 usage could destabilize training, but the mechanism was poorly characterized. Frameworks adopted conservative FP8 integration (TransformerEngine's blockwise recipe keeping FP8 confined to GEMMs, with BF16 everywhere else) or accepted occasional instability as the price of performance (DeepSeek-V3's approach pushing FP8 into communication without addressing the Wgrad requantization). Neither approach had a causal model of why the stability–performance tradeoff existed — they treated it as a diffuse engineering challenge rather than a solvable problem with a specific root cause.

The paper's key diagnostic move is articulating that the problem is not "FP8 precision is insufficient" (a framing that would suggest retreating to higher precision) but rather "quantizing the same tensor twice along different tile directions compounds rounding error in a way that single-pass quantization does not." Equation 1 — $E = Q_{\text{col}}(D(Q_{\text{row}}(X))) - Q_{\text{col}}(X)$ — is deceptively simple, but it isolates exactly the error attributable to the layout-inconsistent requantization step, separate from the unavoidable error of a single quantization. The proof (Equations 5–8) that repeated quantization along the same direction produces identically zero additional error is equally important: it establishes that quantization error is not inherently cumulative — it becomes cumulative specifically and exclusively when the quantization direction (and therefore the tiling and scaling factors) changes between passes.

This diagnostic reframing has two significant consequences beyond the specific solution proposed in this paper. First, it provides a criterion for evaluating future FP8 dataflow designs: does the design avoid double quantization error by ensuring that any tensor that is quantized, dequantized, and requantized does so along the same tiling direction, or by performing the layout conversion without a second quantization (as the scaling-aware transpose does)? Any dataflow that fails this criterion will exhibit the same performance–stability tension, regardless of its other optimizations.

Second, it recasts the FP8 integration problem from a precision choice (where to use FP8 vs. BF16) to a consistency constraint (how to maintain quantization direction consistency across layout changes). This shift is philosophically important: the former framing leads to conservative designs that minimize FP8 exposure; the latter leads to designs like FP8-Flow-MoE that maximize FP8 persistence while enforcing consistency at the identified inconsistency points. The reduction from 12 to 2 cast operations is the quantitative expression of this reframing — it is achieved not by being "less careful" about precision but by being more careful about the specific mechanism (layout-inconsistent requantization) that actually causes problems.

This diagnostic concept is likely the paper's most durable contribution. As hardware evolves toward lower precisions (FP6, FP4, MX formats), the same tension between performance (fewer precision transitions) and stability (avoiding compounded quantization error from layout-inconsistent requantization) will recur, and the diagnostic framework established here will transfer directly. The specific solution (exponent-bit manipulation for power-of-two scaling factors) may or may not generalize to other formats, but the problem formulation will.

Innovation 2: Exponent-Bit-Level Transposition as a Quantization-Invariant Layout Conversion

The scaling-aware transpose operator is the paper's most technically elegant contribution, and its novelty lies in recognizing that the row-to-column conversion problem — which prior work handled through the computationally expensive and numerically lossy cycle of dequantize → transpose → requantize — can be performed entirely within the FP8 encoding with no information loss beyond what the initial quantization already incurred.

This is not an incremental optimization of existing requantization methods; it is a categorical shift in what it means to "convert" between quantization layouts. The dominant assumption in prior FP8 systems was that changing quantization direction requires recovering the original value (or an approximation thereof) and re-quantizing from scratch, because the per-tile scaling factors are functions of the tile contents and will differ between the original and transposed tilings. The scaling-aware transpose challenges this assumption by observing that under the power-of-two scaling factor constraint, the relationship between row-wise and column-wise quantized representations is a pure exponent shift — the mantissa and sign carry over unchanged — because changing the per-tile scaling factor is equivalent to adding an integer to the exponent field.

The conceptual leap here is treating the FP8 encoding as a structured representation whose components have semantic meaning (sign = direction, exponent = scale relative to per-tile scaling factor, mantissa = precision within that scale) rather than as an opaque compressed format that must be decompressed before any transformation. This is a fundamentally different relationship to low-precision data than prior work exhibited. In existing frameworks, FP8 was a storage format — a compressed representation that you decompress (dequantize) before doing anything interesting. In FP8-Flow-MoE, FP8 becomes a computational format — a representation you manipulate directly using its internal structure.

The proof in Equations 10–16 is the formal justification for this shift, but the insight is deeper than the proof. It is the recognition that quantization-invariant transformations exist — operations that preserve the information content of a quantized representation without ever recovering the original continuous values — and that the class of such transformations is defined by how the quantization parameters (scaling factors, tiling) interact with the transformation geometry. The transpose is quantization-invariant when the scaling factor transformation can be expressed as a pure exponent shift, which requires power-of-two scaling factors. This framing opens the door to asking what other transformations might be quantization-invariant under other quantization schemes, which is a question that simply does not arise if you treat quantized representations as opaque compressed formats.

The speedup shown in Figure 1 (2×–3× over the naïve dequantize–transpose–requantize pipeline) validates the practical impact, but the conceptual contribution is the design principle: when designing a persistent low-precision dataflow, identify the points where quantization layouts must change to satisfy downstream operator requirements, and ask whether the layout conversion can be performed inside the quantized representation rather than requiring dequantization. If the answer is yes (as it is here for transposition under power-of-two scaling), the performance and numerical stability gains are simultaneous — you eliminate both the computational cost of the requantization and the error it would introduce. This principle generalizes beyond transposition and FP8 to any scenario where quantized tensors undergo geometric transformations (reshaping, slicing, concatenation) that change the tiling alignment.

Innovation 3: Fusing Quantization Into Compute Kernels as a Dataflow Architecture Principle, Not an Engineering Optimization

Operator fusion is a well-established technique in high-performance computing — combining multiple element-wise operations into a single kernel to reduce kernel launch overhead and HBM traffic. What distinguishes FP8-Flow-MoE's use of fusion is not the technique itself but its architectural role: fusion is not applied opportunistically to accelerate hot spots but is designed into the dataflow as the mechanism that makes the persistent FP8 paradigm viable. This elevates fusion from an optimization to a first-class architectural principle.

The distinction matters because the persistent FP8 dataflow creates new quantization operations that did not exist in the BF16-centric paradigm. In a BF16 dataflow, SwiGLU produces BF16 output and the next GEMM consumes BF16 input — no quantization happens at this boundary. In FP8-Flow-MoE, SwiGLU (computed in BF16 for numerical safety) must produce FP8 output for the next grouped linear to maintain the FP8 dataflow. If this quantization were a standalone kernel, it would introduce new latency and HBM traffic that partially offset the benefits of FP8 persistence. The paper's insight is that the quantization boundary must be absorbed into the same kernel that necessitates it — in this case, the SwiGLU activation itself — so that the act of switching precisions costs nothing beyond the computation that was already required.

This is a subtle but important reframing of the precision transition problem. Prior FP8 systems treated precision transitions as unavoidable costs to be minimized (by reducing the number of FP8 regions) or optimized (by writing faster Q/DQ kernels). FP8-Flow-MoE treats them as opportunities for fusion: every point where a precision transition is necessary should coincide with a point where a compute-intensive or memory-intensive operation already exists, so the transition can be folded into that operation. The two remaining cast locations in the FP8-Flow-MoE dataflow are precisely the two points where the adjacent operations (SwiGLU activation, backward gradient aggregation) are complex enough that fusion is natural — they are not arbitrary boundaries but intentionally chosen fusion targets.

The evidence for this principle's importance appears in Table 1, which shows that standalone quantization and dequantization operations around FP8 communication can nearly eliminate the communication speedup entirely at small scales. A naïve reading of this result would conclude that FP8 communication is not worth the implementation complexity for modest workloads. The paper's architectural reading is different: the Q/DQ cost is not inherent to FP8 communication but is an artifact of treating quantization as a separate step rather than fusing it into the preceding or following compute kernels. Once fused, the Q/DQ cost in Table 1 would read approximately zero, and FP8 communication would deliver its full 1.4×–1.6× speedup even at small scales. This transforms a discouraging microbenchmark into a design directive.

The fused permute-padding and unpermute-unpadding kernels embody the same principle in a different context. Padding and unpadding are artifacts of the Tensor Core alignment requirement (dimensions must be multiples of 16), which existed in BF16 dataflows but were implemented as separate kernel launches because BF16 performance was less sensitive to launch overhead (larger data per element meant compute was more often the bottleneck). In FP8, halved data sizes make kernel launch latency relatively more expensive, so the same separate-kernel strategy that was acceptable in BF16 becomes a bottleneck in FP8. Fusing permute with padding (and unpermute with unpadding) is the natural architectural response: if the persistent FP8 dataflow makes certain operations relatively more expensive, fuse them with their neighbors to maintain overall efficiency. The 1.7× forward and 6.6× backward speedups (Figures 3–4) quantify how much this matters.

Innovation 4: Empirical Demonstration That FP8 Persistence, Not Just FP8 Kernels, Determines System-Level Throughput

The paper's three-way experimental comparison — BF16 baseline vs. Blockwise (FP8 GEMM only) vs. FP8-Flow-MoE (persistent FP8 dataflow) — is more than a benchmarking exercise. It constitutes an empirical argument that the throughput gains from low-precision training are dominated not by the arithmetic speedup of low-precision kernels but by the dataflow architecture that surrounds them. This is a genuinely non-obvious finding with significant implications for how the systems community should approach low-precision training.

The Blockwise recipe already uses FP8 GEMM kernels, which provide up to 2× arithmetic throughput on matrix multiplications compared to BF16. If GEMM throughput were the primary bottleneck in MoE training, the Blockwise recipe should see throughput improvements approaching 2×. Instead, Tables 2 and 3 show Blockwise achieving only 3–8% improvement over BF16 (e.g., 1,146 vs. 1,109 TGS at EP=8 under AC=full). FP8-Flow-MoE, using the same FP8 GEMM kernels, achieves 6–21% improvement — up to 7× more gain than Blockwise. The difference is entirely attributable to the dataflow redesign, not to better GEMM kernels.

This finding is significant because it inverts the conventional prioritization in low-precision systems research. The default assumption — reflected in the proliferation of highly optimized low-precision GEMM libraries (DeepGEMM, CUTLASS, Marlin) — is that accelerating matrix multiplications is the primary lever for end-to-end performance. The paper demonstrates that for MoE training at realistic scales, precision transitions and memory movement dominate GEMM time to such an extent that optimizing GEMMs further yields diminishing returns, while optimizing the dataflow around them yields substantial gains. The widening gap between Blockwise and FP8-Flow-MoE at higher expert parallelism levels (EP=32: 21% vs. essentially flat or OOM) particularly supports this: at high EP, communication and format conversion overhead compound, and only the persistent FP8 dataflow addresses them.

This has implications for hardware-software co-design as well. If the bottleneck is precision transitions rather than GEMM throughput, then future hardware should prioritize support for persistent low-precision dataflows — for example, by enabling direct FP8-to-FP8 communication without intervening dequantization, or by providing hardware support for scaling-factor propagation through collective operations — rather than simply increasing low-precision Tensor Core throughput. The paper's analysis of DeepEP communication overhead (Table 1) provides concrete evidence: FP8 communication is theoretically 2× faster than BF16 but in practice achieves only 1.4×–1.6×, not because the communication hardware is insufficient but because the software interface requires explicit quantization buffers and synchronization that erode the benefit. Hardware that could directly consume and produce FP8 with integrated scaling factors would close this gap without any software changes.

The memory results (Tables 2–3) make the same point from a different angle. Under the most memory-efficient activation checkpointing configuration (AC=sel, Table 3), FP8-Flow-MoE reduces peak memory by 8 GB versus BF16 and 16.5 GB versus Blockwise. The Blockwise recipe actually increases memory usage relative to BF16 in some configurations (73 GB vs. 64 GB at EP=8, AC=sel) because it stores both FP8 and BF16 copies of tensors at precision transition boundaries. This is a direct consequence of the BF16-centric dataflow: if most of the graph is BF16, the FP8 GEMM outputs must be dequantized and stored in BF16, and the FP8 versions (needed for backward pass GEMMs) must also be retained, doubling memory at those boundaries. FP8-Flow-MoE avoids this duplication by storing only the FP8 tensor — the BF16 intermediate exists only transiently inside fused kernels and is never materialized in HBM. At EP=32, where both BF16 and Blockwise encounter OOM, FP8-Flow-MoE runs successfully at 912 TGS with 75 GB memory. This is not a throughput gain but a capability gain: the persistent FP8 dataflow enables training configurations that are impossible under BF16-centric designs, regardless of how fast the individual FP8 kernels are.

This finding also explains why prior FP8 training results often disappointed. Systems that achieved impressive kernel-level benchmarks (isolated GEMM throughput, communication throughput) but saw modest end-to-end gains were not suffering from implementation inefficiencies — they were suffering from a dataflow mismatch between kernel-level FP8 optimization and system-level BF16 architecture. FP8-Flow-MoE's contribution is not primarily better kernels (though the fused operators are important) but the recognition that the dataflow architecture must be redesigned around FP8 persistence for the kernel-level gains to translate to system-level throughput.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper evaluates computational efficiency on a 671B-parameter DeepSeek-V3 model and convergence on a 16B-parameter DeepSeek-V2-Lite model. The V2-Lite convergence experiment uses a training corpus of 200 billion tokens with identical data ordering across all compared configurations. The V3 efficiency experiments use representative MoE workloads with tensor shapes corresponding to DeepSeek V2-Lite, V2, and V3 model configurations under varying expert parallelism (EP) degrees (8, 16, 32).

• Base model(s). The primary efficiency benchmark uses the DeepSeek-V3 architecture at 671B parameters—a massive Mixture-of-Experts model representative of frontier-scale training workloads. The convergence validation uses DeepSeek-V2-Lite at 16B parameters, trained from scratch for 200B tokens. The kernel microbenchmarks use synthetic tensors with shapes matching these models (e.g., (24576, 2048), (32768, 7168)) to reflect realistic MoE layer configurations.

• Metrics. The paper reports three primary metrics throughout Section 4: (1) throughput measured in tokens per GPU per second (TGS), computed as total tokens processed divided by wall-clock time and number of GPUs; (2) peak memory usage per GPU in gigabytes (GB), measuring the maximum HBM allocation during a training iteration; and (3) training loss (LM loss) as a function of consumed tokens, used to verify that FP8-Flow-MoE maintains convergence fidelity comparable to BF16. Kernel-level microbenchmarks in Section 3.3 report relative performance (speedup factors) measured as latency ratios between fused and unfused implementations. Communication microbenchmarks (Table 1) report latency in milliseconds for individual communication and quantization operations.

• Baselines. Three distinct configurations form the comparison space: (1) BF16 — standard FP32-BF16 mixed precision training using Megatron-LM's default recipe, representing the established high-precision baseline; (2) Blockwise — the TransformerEngine-style blockwise FP8 recipe, which uses FP8 only for grouped GEMM kernels while all other operations (communication, activations, permutations) remain in BF16, representing the dominant conservative FP8 integration strategy; (3) FP8-Flow-MoE — the proposed casting-free FP8-centric dataflow with scaling-aware transpose and fused operators. The Blockwise baseline is explicitly described as the "TE-style blockwise recipe with FP8 grouped GEMM kernels" and serves as the primary comparison for isolating dataflow-level improvements from kernel-level FP8 acceleration.

• Generation budget / compute accounting. Compute is measured at the system level via end-to-end throughput (TGS), which accounts for all computation, communication, and memory operations in a full training iteration. For microbenchmarks, latency is measured using CUDA events around individual kernel invocations. The throughput experiments use standard expert parallelism (EP) and pipeline parallelism (PP) configurations—EP/PP = 8/32, 16/16, and 32/8—with the same global batch size across all configurations to ensure fair comparison. All end-to-end experiments are conducted without computation–communication overlap to isolate computational efficiency from scheduling effects. The paper explicitly states this choice: "To isolate the effect of computation efficiency from communication overlap, all end-to-end experiments are performed without computation–communication overlap." This means the reported throughput gains are conservative—with overlap enabled, the communication-bound baselines would likely benefit less from overlap than FP8-Flow-MoE (since FP8-Flow-MoE's communication already operates at near-minimal latency), so the gap might widen further.

• Cross-validation / statistical protocol. For the convergence experiment, both BF16 and FP8-Flow-MoE models are trained with "identical optimization hyperparameters, learning rate schedule, data ordering, and parallelism configurations." The loss curves (Figure 6) are plotted over 200B tokens, providing a direct trajectory comparison rather than a point estimate. No cross-validation or multiple random seeds are reported—the convergence experiment appears to use a single training run per configuration, which is standard for large-scale training validation given the computational cost of 200B-token runs but limits statistical claims about variance. For the throughput experiments, the paper does not report multiple measurements or confidence intervals, suggesting the reported TGS values are single-run measurements. This is common in systems papers at this scale but means the exact throughput numbers should be interpreted as point estimates rather than population statistics.

---

Main Quantitative Results

End-to-End Throughput and Memory Efficiency on DeepSeek-V3 (671B Parameters)

The headline throughput results appear in Tables 2 and 3, which compare FP8-Flow-MoE against BF16 and Blockwise baselines under two activation checkpointing (AC) strategies across three expert parallelism configurations.

Under full activation checkpointing (AC=full, Table 2): FP8-Flow-MoE achieves 1,176 TGS at EP=8 compared to 1,109 TGS for BF16 (+6%) and 1,146 TGS for Blockwise (+3% over Blockwise). At EP=16, the gap widens: 1,012 TGS for FP8-Flow-MoE versus 939 TGS for BF16 (+8%) and 938 TGS for Blockwise (+8%). At EP=32, the advantage is largest: 779 TGS for FP8-Flow-MoE versus 671 TGS for BF16 (+16%) and 644 TGS for Blockwise (+21%). The Blockwise recipe shows minimal gains over BF16 at EP=8 (1,146 vs. 1,109, +3%) and essentially no gain at EP=16 (938 vs. 939, −0.1%), and at EP=32 it actually underperforms BF16 (644 vs. 671, −4%). This negative result for Blockwise at high EP is critical: simply replacing GEMM kernels with FP8 variants while keeping the BF16 dataflow can be counterproductive when communication and quantization overhead become the dominant cost, as the additional kernel launches and format conversions consume more time than the GEMM acceleration saves.

Under selective activation checkpointing with MoE experts excluded (AC=sel, Table 3): This configuration represents the most memory-efficient set-up compatible with the 1F1B-overlap schedule in Megatron-LM. At EP=8, FP8-Flow-MoE achieves 1,193 TGS with 56 GB peak memory, compared to BF16 at 1,178 TGS with 64 GB (+1.3% throughput, −8 GB memory) and Blockwise at 1,178 TGS with 73 GB (+1.3% throughput, −17 GB memory versus Blockwise). The memory advantage is substantial even at modest EP levels because FP8-Flow-MoE stores activations in FP8 format rather than BF16.

At EP=16, FP8-Flow-MoE reaches 1,111 TGS with 66 GB memory, versus BF16 at 1,055 TGS with 71 GB (+5.3% throughput, −5 GB) and Blockwise at 1,031 TGS with 77 GB (+7.8% throughput, −11 GB). The Blockwise baseline again shows a throughput regression relative to BF16 (1,031 vs. 1,055) while consuming more memory (77 GB vs. 71 GB), confirming the cost of maintaining dual-precision buffers.

At EP=32, both BF16 and Blockwise encounter out-of-memory (OOM) errors and cannot complete training. FP8-Flow-MoE runs successfully at 912 TGS with 75 GB memory. This is the paper's most dramatic result: the persistent FP8 dataflow does not merely accelerate training but enables a training configuration that is impossible under the BF16 and Blockwise baselines. The specific numbers—912 TGS, 75 GB—are less important than the binary outcome: FP8-Flow-MoE runs where competitors crash.

The pattern across EP levels. The throughput advantage of FP8-Flow-MoE grows monotonically with expert parallelism: +6% → +8% → +16% under AC=full and +1.3% → +5.3% → OOM-avoidance under AC=sel. This scaling pattern supports the paper's architectural argument: at low EP levels, GEMM computation is a larger fraction of total time, so the Blockwise recipe captures a reasonable portion of FP8's theoretical benefit. As EP increases, communication and quantization overhead dominate, and only the persistent FP8 dataflow (with fused operators absorbing quantization into compute kernels and eliminating boundary casts) can sustain performance gains. The Blockwise recipe's negative scaling at high EP (underperforming BF16 at EP=16 under AC=sel and EP=32 under AC=full) is a particularly informative failure mode—it demonstrates that partial FP8 adoption can be worse than no FP8 adoption when the overhead of precision transitions exceeds the acceleration from low-precision GEMMs.

Memory efficiency. Under AC=sel (+MoE expert) at EP=8, FP8-Flow-MoE reduces peak memory by approximately 8 GB versus BF16 (56 GB vs. 64 GB) and 16.5 GB versus Blockwise (56 GB vs. ~73 GB). The paper attributes this to "the FP8 checkpoint compression mechanism, which stores intermediate activations in FP8 instead of BF16" (Section 4.2). However, the activation compression benefit should also apply to the Blockwise recipe if it stores activations in FP8—the fact that Blockwise uses more memory than BF16 (73 GB vs. 64 GB) suggests that the Blockwise recipe retains both FP8 and BF16 copies of activations at precision transition boundaries, consistent with the dataflow diagram in Figure 2b showing FP8 confined to GEMMs with BF16 storage elsewhere. The memory numbers thus reflect not only FP8's halved storage size but also the elimination of dual-precision copies that the persistent dataflow enables.

Convergence Validation on DeepSeek-V2-Lite (16B Parameters, 200B Tokens)

Figure 6 shows the training loss curves for BF16 and FP8-Flow-MoE across 200 billion consumed tokens. The paper reports that "the two loss curves are nearly indistinguishable across the entire training trajectory—from the early warm-up phase through the stabilized loss regime." Specific observations from the figure: both curves start at approximately 11–12 LM loss and decline to approximately 2–3 LM loss by 200B tokens, with the FP8-Flow-MoE curve tracking BF16 so closely that the two lines overlap almost entirely throughout training. There is "no sign of divergence, gradient underflow, or numerical drift."

This convergence result is significant because FP8-Flow-MoE applies FP8 to a larger fraction of the training pipeline than prior conservative FP8 recipes (which confine FP8 to GEMMs only). The fact that training remains stable validates the paper's two central design claims: (1) the scaling-aware transpose eliminates the double quantization error that would otherwise destabilize the Wgrad computation, and (2) retaining BF16 at only two boundaries (SwiGLU activation and backward gradient aggregation) is sufficient—no additional BF16 fallbacks are needed elsewhere in the MoE graph.

However, several caveats apply. The convergence experiment uses a 16B model, which is substantial but 42× smaller than the 671B model used for efficiency benchmarks. Numerical instability from double quantization error might manifest differently (or more severely) at larger scales where gradient magnitudes span wider dynamic ranges. The paper does not report downstream task evaluations (e.g., perplexity on held-out data, few-shot accuracy benchmarks), only training loss, which is a necessary but not sufficient condition for claiming that FP8-Flow-MoE preserves model quality. The BF16 and FP8-Flow-MoE models are trained with identical hyperparameters, but it is possible that FP8 training would benefit from slight adjustments to learning rate or optimizer settings (as is sometimes observed with lower-precision training). That the default BF16 hyperparameters work without modification for FP8-Flow-MoE is evidence of robustness, but the absence of hyperparameter tuning for FP8 might also mean the reported convergence represents a lower bound on achievable quality.

Communication Microbenchmarks (DeepEP Integration)

Table 1 reports latency measurements for BF16 communication compared to FP8 communication using DeepEP, with explicit quantization and dequantization operations timed separately. The measurements span three model scales (DeepSeek V2-Lite, V2, V3) and three EP degrees (8, 16, 32), with tensor shapes ranging from (24576, 2048) to (32768, 7168).

At the smallest scale (24576, 2048, EP=8), BF16 communication latency is 0.537 ms. The FP8 path—including quantize (0.127 ms), communication (0.325 ms), and dequantize (0.084 ms)—totals 0.535 ms. The speedup is reported as 1.65× when considering only the communication kernel (0.325 ms for FP8 vs. 0.537 ms for BF16), but only 1.00× when including Q/DQ overhead (0.535 ms vs. 0.537 ms). This is the paper's most striking microbenchmark result: at small scales, FP8 communication with explicit Q/DQ provides literally no speedup over BF16 communication, because the quantization and dequantization time (0.211 ms total) nearly equals the entire communication saving.

At the largest scale (32768, 7168, EP=32), BF16 latency is 7.327 ms. The FP8 quantize (0.082 ms) + communication (4.319 ms) + dequantize (0.082 ms) totals 4.483 ms, yielding a speedup of 1.70× for communication alone but 1.63× for the full path. The Q/DQ overhead is proportionally smaller (0.164 ms of 4.483 ms, or 3.7%) but still reduces the effective speedup from 1.70× to 1.63×.

Two intermediate configurations show particularly interesting behavior. At (24576, 2048, EP=16), the full FP8 path is actually slower than BF16: 1.350 ms vs. 1.224 ms, a speedup of 0.91×. The communication kernel alone is faster (1.176 ms vs. 1.224 ms, 1.04×), but Q/DQ overhead pushes the total above BF16. At (24576, 2048, EP=32), the same near-break-even pattern appears: full FP8 path at 2.918 ms vs. BF16 at 3.005 ms, for only 1.03× speedup.

The paper draws a key conclusion from these measurements: "a single pair of quantization and dequantization operators reduces the performance gain of FP8 communication by roughly one third (from 0.6× to 0.4× improvement). Considering that a typical MoE forward or backward pass involves around three such pairs (see Figure 2(c)), the benefit of FP8 kernels can be almost completely neutralized." This quantitative claim—one-third erosion per Q/DQ pair, three pairs per pass, potential near-total neutralization—motivates the fused operator strategy that eliminates standalone Q/DQ around communication.

Kernel Fusion Microbenchmarks

Permute-padding fusion (Figure 3). The paper reports "up to 1.7× speedup" for the fused permute+padding kernel compared to separate implementations. The figure shows relative performance across six tensor shapes from (28672, 1280) to (32768, 9216), with four bars per shape: BF16 unfused, BF16 fused, FP8 unfused, FP8 fused. The highest speedup (approximately 1.7×) occurs at the largest shape configurations for FP8, where the unfused implementation is most bottlenecked by kernel launch overhead and redundant HBM accesses. The FP8 fused version consistently achieves the highest absolute performance, benefiting from both fusion (fewer kernel launches, fewer HBM passes) and FP8's halved data size (less memory traffic).

Unpermute-unpadding fusion (Figure 4). The backward-pass fusion achieves "up to 6.6× speedup on large-scale configurations." The dramatically larger speedup in the backward direction reflects the more complex memory access pattern of unpermutation with unpadding: the forward permute writes tokens sequentially into expert-contiguous order (a gather operation from scattered positions to contiguous output), while the backward unpermute scatters expert outputs back to original token positions (an operation that requires atomic updates or separate output buffers to avoid write conflicts). The unfused implementation pays this irregular access cost twice (once for unpermutation, once for unpadding), while the fused kernel pays it once. The paper does not detail the specific tensor shape at which 6.6× is achieved, but Figure 4 shows the largest bars occurring at the rightmost shapes (32768, 7168) to (32768, 9216), which correspond to DeepSeek-V3 scale workloads.

SwiGLU-quantization fusion (Figure 5). The fused kernel achieves "nearly identical latency to the original standalone SwiGLU operator in NVIDIA's Megatron framework, while seamlessly producing FP8-quantized outputs." The figure reports relative performance for forward and backward variants, showing that the fused kernel's latency is approximately 0.9× to 1.0× that of the unfused SwiGLU alone. Since the unfused alternative would require a separate quantization kernel after SwiGLU, the effective speedup is approximately 2× (halving the total time for SwiGLU + quantize). The paper reports the result as near-equal latency to emphasize that quantization has been made "free"—it is absorbed into a kernel that runs at essentially the same speed regardless of whether it produces FP8 or BF16 outputs.

Scaling-Aware Transpose Microbenchmark

Direct Transpose latency (Figure 1). The scaling-aware transpose operator achieves "2 to 3 times speedup across all tensor shapes" compared to the naïve dequantize → transpose → requantize pipeline. The figure shows relative performance across six input shapes, with two bars per shape: the naïve conversion and the Direct Transpose. The naïve conversion's relative performance is approximately 1.0× (the baseline), and the Direct Transpose achieves roughly 2.5× to 3.5× relative performance, meaning the latency is 1/2.5 to 1/3.5 of the naïve approach. The speedup is consistent across shapes, suggesting the benefit depends primarily on the fixed cost of eliminating two quantization passes rather than on tensor dimensions.

The figure omits an important comparison: what if the row-wise quantized data were simply transposed without requantization, producing transposed data with row-wise scaling factors? This would avoid the dequantize–requantize cycle but produce a tensor whose scaling factors are mismatched to the downstream GEMM's column-wise expectation—potentially causing incorrect computation. The Direct Transpose's value is not just speed but correctness: it produces column-wise quantized data with appropriate scaling factors, enabling the downstream GEMM to proceed without BF16 fallback. The latency speedup is thus a lower bound on the system-level benefit; the additional gain from avoiding a precision transition boundary (which would require storing BF16 intermediates and subsequent requantization elsewhere in the dataflow) is not captured in the isolated microbenchmark.

---

Ablation Studies and Robustness Checks

This paper does not contain traditional ablation studies in the machine learning sense. There is no incremental removal of components (e.g., "FP8-Flow-MoE without scaling-aware transpose," "FP8-Flow-MoE without fused permute-padding") with corresponding convergence or throughput measurements. The paper's evaluative structure is instead a component-wise comparison where the contribution of each mechanism is isolated through microbenchmarks (kernel-level latency measurements) and the system-level contribution is aggregated through the three-way BF16 vs. Blockwise vs. FP8-Flow-MoE throughput comparison.

Several of these comparisons function as implicit ablations:

Blockwise recipe as an ablation of the persistent dataflow. The Blockwise baseline uses the same FP8 GEMM kernels as FP8-Flow-MoE (both are described as using "FP8 grouped GEMM kernels") but retains the BF16-dominated dataflow with explicit Q/DQ at GEMM boundaries. The performance gap between Blockwise and FP8-Flow-MoE therefore isolates the contribution of the persistent FP8 dataflow, fused communication boundaries, and scaling-aware transpose combined. This is not a clean ablation of any single component but a clean ablation of the dataflow architecture difference holding kernel implementations constant. The results (Table 3: +1.3% at EP=8, +7.8% at EP=16, OOM-avoidance at EP=32) provide strong evidence that the dataflow redesign is the dominant contributor to throughput gains, not better GEMM kernels.

Communication Q/DQ timing as an ablation of fused quantization. Table 1's breakdown of quantize, communicate, and dequantize times provides an implicit ablation: the communication-only speedup (FP8 vs. BF16, columns "COMM ALL" vs. "BF16") represents what performance would be if Q/DQ were eliminated, while the full-path speedup (columns "ALL" vs. "BF16") represents actual performance with standalone Q/DQ. For example, at (24576, 2048, EP=8), FP8 communication delivers 1.65× speedup in isolation but only 1.00× when Q/DQ is included. The Q/DQ operations therefore account for 100% of the performance loss relative to the theoretical communication speedup. FP8-Flow-MoE's fusion strategy aims to recover the full 1.65× by absorbing Q/DQ into compute kernels; the throughput gains in Tables 2–3 are consistent with this recovery, though the paper does not directly measure post-fusion communication latency to confirm recovery of the full theoretical speedup.

Fused vs. unfused kernel comparisons as per-component ablations. Figures 3, 4, and 5 each compare fused against unfused implementations for specific operators. These are classical ablation benchmarks: they measure the contribution of each fusion in isolation, holding all other aspects of the system constant. The results (1.7× for forward permute-padding, 6.6× for backward unpermute-unpadding, ~2× effective for SwiGLU-quantization) establish that each fusion individually provides meaningful speedup. However, these are microbenchmarks—they do not measure the system-level impact of removing any single fusion from the FP8-Flow-MoE pipeline. The paper does not report throughput numbers for "FP8-Flow-MoE without permute-padding fusion" or similar variants, so the marginal system-level contribution of each fusion is not quantified.

No ablation of the scaling-aware transpose. This is a notable omission. The scaling-aware transpose is the paper's central technical innovation and the enabler of the persistent FP8 dataflow, yet there is no experiment that substitutes a naïve dequantize→transpose→requantize at the Wgrad boundary into the FP8-Flow-MoE dataflow and measures the throughput or convergence impact. Such an experiment would directly measure how much of the system-level gain is attributable to the transpose versus the fusions and dataflow restructuring. The convergence experiment (Figure 6) implicitly tests the numerical impact of the transpose—if double quantization error were destabilizing, the loss curve would diverge—but does not provide a controlled comparison with a version lacking the transpose. The paper's theoretical analysis (Equations 1–16) and the transpose microbenchmark (Figure 1) establish correctness and per-operator speedup, but the absence of a system-level ablation means the reader cannot distinguish between "the scaling-aware transpose is essential" and "the scaling-aware transpose is elegant but the system-level gains come primarily from the fusions and dataflow architecture."

No ablation of power-of-two scaling factors. The scaling-aware transpose's correctness depends on scaling factors being powers of two (Section 3.1, Equations 10–16). The paper does not report any experiment comparing power-of-two constrained scaling factors against unconstrained scaling factors, either for throughput (does constraining to powers of two affect quantization speed?) or for convergence (does constraining to powers of two degrade model quality relative to finer-grained scaling factors?). The theoretical framework argues that any precision loss from power-of-two constraints is minor compared to the benefit of eliminating double quantization error, but this claim is not empirically tested.

Activation checkpointing configurations as a robustness test across memory regimes. The two AC strategies—AC=full (checkpointing all activations) and AC=sel (+MoE expert, checkpointing selectively while excluding expert activations)—represent different points on the memory-computation tradeoff curve. AC=full minimizes memory at the cost of recomputation, making the system more compute-bound; AC=sel maximizes memory savings from FP8 activation storage, making the system more memory-bandwidth-bound. FP8-Flow-MoE outperforms baselines under both regimes, which suggests the benefits are not confined to a narrow region of the memory-compute tradeoff. However, the paper only tests two AC configurations; exploring intermediate checkpointing strategies (e.g., checkpointing only attention, or using gradient accumulation to trade batch size for memory) would strengthen the robustness claim.

Expert parallelism sweep as a scaling robustness test. The three EP settings (8, 16, 32) represent a 4× range in expert parallelism degree. The monotonic growth in FP8-Flow-MoE's advantage (wider gap at higher EP) is a strong result because it suggests the benefits are not saturated at the tested scales and would continue to grow at EP=64 or beyond—though this extrapolation is speculative without data. The OOM result at EP=32 for the baselines under AC=sel provides a natural upper bound on the scaling comparison, but it also means the throughput comparison at the most favorable regime is incomplete (since the baselines cannot be measured).

No sensitivity analysis to batch size, sequence length, or number of experts. All experiments appear to use the default DeepSeek-V3 and V2-Lite configurations without varying these parameters. The MoE performance characteristics (fraction of time in communication vs. computation, expert load balance, padding overhead) depend significantly on batch size per GPU, sequence length, and the number of experts relative to the EP degree. The paper does not explore how FP8-Flow-MoE's gains vary across these dimensions, which limits the generalizability of the throughput numbers to configurations significantly different from the tested ones.

---

Critical Assessment

This section evaluates whether the experiments genuinely support the paper's three central claims, identifying both what is demonstrated and what remains unverified.

Claim 1: "Up to 21% higher throughput and 16.5 GB lower memory usage per GPU compared to BF16 and naïve FP8 baselines."

The throughput claim is supported but with important conditions. The 21% figure comes from EP=32 under AC=full (Table 2), comparing FP8-Flow-MoE at 779 TGS against Blockwise at 644 TGS. This is the most favorable configuration—high expert parallelism, with computation-communication overlap explicitly disabled. The paper acknowledges the disabled overlap but frames it as isolating computational efficiency; this is valid but means the 21% figure may not represent real-world training throughput where overlap is enabled. If the baselines benefit more from overlap (because their communication stages are longer and offer more opportunities for concurrent computation), the gap might narrow with overlap enabled. Conversely, if FP8-Flow-MoE's reduced communication latency enables more effective overlap, the gap might widen. Without overlap-enabled measurements, the direction of bias is unknown.

The 16.5 GB memory reduction is reported at EP=8 under AC=sel (Table 3): FP8-Flow-MoE at 56 GB versus Blockwise at approximately 73 GB. This comparison is somewhat misleading because the Blockwise recipe uses more memory than BF16 (73 vs. 64 GB), making the FP8-Flow-MoE advantage appear larger relative to Blockwise than to the more practically relevant BF16 baseline (56 vs. 64 GB, an 8 GB reduction). The 16.5 GB figure is a real measurement, but it represents the gap between the best and worst FP8 strategies, not between FP8-Flow-MoE and the standard BF16 practice. The paper's abstract and introduction emphasize the 16.5 GB figure prominently without clarifying that much of this reduction reflects the Blockwise baseline's anomalous memory behavior (dual-precision buffer duplication) rather than FP8-Flow-MoE's compression advantage over BF16.

The comparison against "naïve FP8 baselines" uses the Blockwise recipe as the naïve proxy. This is reasonable—Blockwise represents the dominant conservative FP8 integration—but it conflates "naïve" (unoptimized) with "conservative" (numerically safe). The DeepSeek-V3 style approach (Figure 2c), which extends FP8 to communication but incurs double quantization error at the Wgrad boundary, is arguably a more natural "naïve" comparison for FP8-Flow-MoE because it makes similar scope choices (FP8 in communication and GEMMs). The paper does not include throughput measurements for a DeepSeek-V3-style dataflow configuration, only the communication microbenchmarks in Table 1. This means the throughput advantage over the most aggressive prior FP8 approach is not directly measured.

Claim 2: "Stable convergence" indistinguishable from BF16 across 200B training tokens.

The convergence evidence (Figure 6) is consistent with the claim but has significant limitations. Training loss convergence is a necessary condition for claiming numerical stability but not sufficient for claiming model quality preservation. Two models can converge to nearly identical training losses while learning meaningfully different internal representations or exhibiting different downstream task performance. The paper provides no downstream evaluation—perplexity on held-out validation data, benchmark accuracy on MATH or MMLU, or any task-specific metric. For a systems paper, training loss convergence is the standard validity check, but the claim of "stable convergence" should be understood as "no catastrophic training failure or loss divergence" rather than "model quality matches BF16."

The 200B token training run, while substantial (12.5× the model parameter count), represents a fraction of typical large-scale training—DeepSeek-V3 was trained on 14.8T tokens, approximately 74× more data relative to its parameter count than the V2-Lite convergence experiment here. Numerical issues from accumulated quantization error sometimes manifest only in the later stages of training when gradient magnitudes decrease and the relative impact of quantization noise grows. The convergence experiment does not rule out late-training degradation or subtle quality differences that would appear with longer training.

The V2-Lite model (16B) is used for convergence validation while the throughput benchmarks use V3 (671B). This is standard practice—training a 671B model to convergence purely for evaluation would be prohibitively expensive—but it means the convergence claim is demonstrated at 42× smaller scale than the throughput claim. Numerical stability at 16B parameters with 200B tokens does not guarantee stability at 671B parameters with trillions of tokens, particularly because the dynamic range of activations and gradients can expand with model depth and width.

Claim 3: The scaling-aware transpose eliminates double quantization error and enables the casting-free dataflow.

The theoretical proof (Equations 10–16) establishes that under power-of-two scaling factors, the transpose produces a column-wise quantized representation that is mathematically identical to direct column-wise quantization from the original continuous tensor. This is a correctness claim, and the proof is sound given its assumptions.

However, the empirical evidence for this claim is indirect. The transpose microbenchmark (Figure 1) measures latency, not numerical accuracy. The paper does not report any direct measurement of double quantization error magnitude—for example, comparing $\|Q_{\text{col}}(D(Q_{\text{row}}(X))) - X\|$ (error with double quantization) against $\|Q_{\text{col}}(X) - X\|$ (error with single quantization) and $\|\text{DirectTranspose}(Q_{\text{row}}(X)) - X\|$ (error with scaling-aware transpose). Such measurements would directly validate Equation 1 and demonstrate that the Direct Transpose achieves error comparable to single-pass column-wise quantization rather than the amplified error of the naïve pipeline.

The convergence experiment (Figure 6) implicitly validates the transpose's numerical correctness—if double quantization error were destabilizing training, the loss curve would likely diverge or spike—but this is a coarse test. Many potential sources of numerical error could be masked by the optimizer's adaptive learning rates, gradient clipping, or other stabilization mechanisms in the training loop. The absence of a convergence comparison with a "FP8-Flow-MoE without scaling-aware transpose" variant means we cannot isolate the transpose's contribution to stability from the dataflow architecture's contribution.

Additional missing experiments and analyses.

• No measurement of end-to-end training time or cost. The paper reports throughput in TGS and peak memory in GB, but does not report total training time or GPU-hours. For practitioners, the relevant metric is not abstract throughput but the cost and duration of a complete training run. Given that FP8-Flow-MoE uses different memory characteristics that might enable different batch sizes or parallelism configurations, the throughput gains might not translate 1:1 to reduced training time or cost.

• No sensitivity analysis to the power-of-two constraint violation. If a user were to relax the power-of-two scaling factor constraint (e.g., to see if non-power-of-two scaling improves model quality), the scaling-aware transpose would no longer be exact. The paper does not characterize how much error would be introduced by approximating non-power-of-two scaling factors with the nearest power of two—a practical question for implementers considering adopting the technique.

• No measurement of quantization error accumulation over training time. The convergence experiment compares loss curves but does not monitor internal metrics such as gradient norm statistics, activation value distributions, or quantization error histograms over the course of training. Such measurements would provide insight into whether FP8-Flow-MoE maintains numerical health (no gradual degradation of quantization quality, no shift in activation distributions) or merely avoids catastrophic failure.

• No comparison against non-power-of-two per-tile quantization strategies. The paper constrains scaling factors to powers of two and argues this is necessary for the scaling-aware transpose. However, there exist alternative strategies for row-to-column layout conversion that do not require power-of-two scaling—for example, computing the maximum absolute value for both row-wise and column-wise tiles simultaneously during the initial quantization, or using per-block scaling factors that are consistent across both tiling directions. The paper does not discuss these alternatives or compare FP8-Flow-MoE against a hypothetical FP8 dataflow using a different approach to the Wgrad layout problem.

• Hardware-specificity unaddressed. All experiments use NVIDIA Hopper GPUs (80 GB). The paper's techniques—particularly the reliance on Tensor Core alignment requirements (multiples of 16) and DeepEP's FP8 communication primitives—are tightly coupled to the Hopper architecture and the NVIDIA software stack. The paper does not discuss portability to other hardware (AMD MI300X, Intel Gaudi, custom ASICs) or future NVIDIA architectures, which limits the shelf life of the specific throughput numbers even if the architectural principles generalize.

The genuine strengths of the experimental evaluation.

Despite these limitations, the experimental design has several genuine strengths. The three-way comparison (BF16 vs. Blockwise vs. FP8-Flow-MoE) across two activation checkpointing regimes and three expert parallelism levels provides a systematic and interpretable picture of how the gains scale with configuration parameters. The Blockwise baseline is well-chosen as a conservative FP8 integration that controls for kernel quality—it uses the same GEMM kernels as FP8-Flow-MoE, so the throughput gap cleanly isolates dataflow effects. The communication microbenchmarks (Table 1) are unusually detailed for a systems paper, breaking down Q/DQ and communication latency separately and exposing the severity of the Q/DQ overhead problem at small scales. The convergence result, while limited, uses a realistic 200B-token training run rather than a toy-scale proxy.

The most compelling experimental result is the OOM-avoidance at EP=32 (Table 3). This is not a marginal throughput gain but a qualitative capability difference: FP8-Flow-MoE can train in a configuration where competitors crash. The paper appropriately highlights this as evidence that the memory savings are not merely convenient but enabling—they expand the design space of possible MoE training configurations. This result is robust because OOM is a binary outcome not subject to measurement noise or statistical variance.

6. Limitations and Trade-offs

6.1 The Scaling-Aware Transpose Requires Power-of-Two Scaling Factors — a Precision–Efficiency Tradeoff Whose Accuracy Cost Is Unmeasured

The assumption or constraint. The scaling-aware transpose is the linchpin of the entire FP8-Flow-MoE dataflow. It enables layout conversion without double quantization error, but its correctness proof (Equations 10–16 in Section 3.1) depends on a specific constraint: all per-tile scaling factors must be powers of two. When $s = 2^T$ and $s' = 2^{T'}$, the ratio $s'/s$ is a pure power of two, and rescaling between row-wise and column-wise quantization reduces to integer addition in the exponent field. If scaling factors are not powers of two, the rescaling operation would involve multiplying or dividing the mantissa, introducing rounding and a second quantization error. The paper commits to this constraint globally: scaling factors throughout the FP8-Flow-MoE dataflow are power-of-two values.

The consequence. The power-of-two constraint means that FP8-Flow-MoE's per-tile scaling factors are coarser than they could be. Standard per-tile quantization computes a scaling factor as $s = \max_{i} |x_i| / 448$, where the maximum is taken over 128 contiguous elements into the tile. This yields a scaling factor that is an arbitrary real number, precisely matched to the actual dynamic range of that tile. The power-of-two constraint rounds this optimal scaling factor to the nearest power of two — $2^{\text{round}(\log_2(s))}$ — which can be off by up to a factor of $\sqrt{2} \approx 1.4\times$ in either direction. When the constrained scaling factor is larger than the optimal one (over-scaling), values are quantized with finer step sizes than necessary, which is harmless for precision but underutilizes FP8's dynamic range. When the constrained factor is smaller (under-scaling), values at the upper end of the tile's range may exceed the FP8 E4M3 maximum representable value of 448, causing saturation (clipping to 448) and loss of information for those elements.

For tiles whose maximum absolute value falls near a power-of-two multiple of 448 — e.g., a tile with max value 350, where the optimal scaling factor is $350/448 \approx 0.78$ and the nearest powers of two are 0.5 and 1.0 — the quantization error could be substantially larger than with the unconstrained scaling factor. Whether this additional error, accumulated over all tiles in the MoE computation graph, measurably degrades model quality is unknown. The paper does not report any comparison between power-of-two-constrained quantized models and unconstrained-quantized models, either for pre-training loss or for downstream task performance.

A further consequence: the power-of-two constraint precludes migration to quantization strategies where scaling factors are optimized per-tile for minimum quantization error (e.g., using the L2 norm or an entropy-based metric rather than the maximum absolute value). Such strategies can improve accuracy for tensors with outlier values but produce scaling factors that are not powers of two, and would thus be incompatible with the scaling-aware transpose.

What evidence exists in the paper. The paper does not empirically evaluate the accuracy impact of the power-of-two constraint. The convergence experiment (Section 4.1, Figure 6) compares FP8-Flow-MoE (which uses power-of-two scaling) against BF16 (which does not quantize activations at all), but does not include a comparison against a hypothetical FP8 training run with unconstrained scaling factors (and some other mechanism for layout conversion, such as the naïve dequantize–transpose–requantize pipeline). The theoretical analysis in Section 3.1 focuses on proving correctness under the power-of-two assumption, not on quantifying the precision loss from enforcing the constraint.

Figure 1 demonstrates a latency speedup for the Direct Transpose over the naïve pipeline, but this speedup combines two effects: avoiding the computational cost of requantization (which is eliminated by the power-of-two constraint) and avoiding double quantization error (also eliminated by the constraint). The figure does not tell us whether the power-of-two constraint itself — i.e., the coarser scaling factors — introduces any accuracy penalty that would not exist in a system using unconstrained scaling factors with a slower but equally accurate layout conversion.

The throughput experiments (Tables 2–3) measure tokens per second and peak memory, not model quality metrics. The convergence experiment (Figure 6) shows that FP8-Flow-MoE with power-of-two scaling factors matches BF16 in training loss, which is consistent with the constraint being benign — but is also consistent with the training loss metric being insensitive to the small additional quantization error introduced by power-of-two constraints, while downstream task metrics might be more sensitive.

Mitigation status. The paper does not address this tradeoff. The power-of-two constraint is presented as an enabling condition for the scaling-aware transpose, not as a limitation. Section 3.1 states the constraint as a given ("let us assume $s = 2^T, s' = 2^{T'}$") and proceeds to prove correctness under that assumption, without discussing the cases where the assumption is violated. The conclusion (Section 5) does not mention the precision–efficiency tradeoff or suggest flexibility in the scaling factor format. A practitioner adopting FP8-Flow-MoE has no guidance on whether the power-of-two constraint might cause issues for their specific model or task, or how to diagnose such issues if they arise.

The tradeoff could be partially mitigated by using a larger tile size (reducing the number of scaling factors and thus the aggregate impact of power-of-two rounding) or by selectively relaxing the constraint for tiles where the rounding error is large and the tile participates in non-transpose paths only. Neither option is explored.

---

6.2 Difficulty Estimation for Strategy Selection — No, That's the Wrong Paper

[Self-correction: this limitation is not relevant to FP8-Flow-MoE. The difficulty estimation concern applies to compute-optimal test-time scaling, not to a low-precision training recipe. I will replace this with an actual limitation of the current paper.]

---

6.2 Convergence Is Validated Only on Training Loss at 16B Parameters — Generalization to Larger Models, Longer Training, and Task Quality Is Unverified

The assumption or constraint. The convergence experiment (Section 4.1) trains a DeepSeek-V2-Lite model at 16B parameters for 200B tokens and compares the training loss curves of BF16 and FP8-Flow-MoE, finding them "nearly indistinguishable." However, all throughput benchmarks (Tables 2–3) are conducted on the much larger DeepSeek-V3 model at 671B parameters — a 42× increase in model scale. The paper implicitly assumes that convergence behavior observed at 16B parameters transfers to 671B parameters, and that training loss convergence implies equivalent model quality on downstream tasks.

The consequence. Numerical issues from low-precision training can manifest differently across model scales, and training loss convergence does not guarantee quality preservation on downstream metrics.

At larger model scales, activation and gradient magnitudes can span wider dynamic ranges (deeper networks compound outlier values through more layers), and the relative impact of quantization noise on weight updates may change with model size. The per-tile scaling strategy with 128-element tiles is granular enough for 16B models but may not be sufficient for 671B models where individual attention heads or expert dimensions can vary more dramatically across tokens. If some tiles have much larger dynamic range than others at 671B scale, the power-of-two scaling factor constraint could introduce proportionally more clipping or underflow than it does at 16B scale.

Training loss is an aggregate metric that can obscure local degradation. Two models with identical training loss curves can learn different internal representations — for example, one might rely on spurious correlations that the other does not, or one might exhibit degraded performance on tail phenomena (rare tokens, long sequences, complex reasoning) that contribute little to average token-level loss. The paper provides no downstream evaluation: no validation perplexity on held-out data, no few-shot accuracy on standard benchmarks (MMLU, MATH, HumanEval), and no qualitative analysis of model outputs. This is standard for systems papers focused on training efficiency, but it means a practitioner cannot assess whether adopting FP8-Flow-MoE would require retuning downstream task prompts or fine-tuning procedures relative to a BF16-trained model.

The 200B-token training run, while substantial (12.5× the 16B parameter count), represents a modest fraction of frontier training scales. DeepSeek-V3 was trained on 14.8T tokens — roughly 74× more data relative to its parameter count than the 200B-token run here. Numerical degradation from accumulated quantization error can manifest late in training, particularly when the learning rate decays and gradient magnitudes shrink, making the relative impact of quantization noise on weight updates proportionally larger. The convergence experiment cannot rule out that FP8-Flow-MoE would diverge or degrade relative to BF16 if trained for trillions of tokens rather than hundreds of billions.

What evidence exists in the paper. The only convergence evidence is Figure 6, showing training loss for 200B tokens on DeepSeek-V2-Lite. The figure shows the two curves overlapping closely, with no divergence or instability. The paper does not report final validation loss, benchmark accuracy, or any task-specific metric. Section 4.1 states: "Both models are trained for 200B tokens, using identical optimization hyperparameters, learning rate schedule, data ordering, and parallelism configurations." The identical-hyperparameter setup means the experiment tests whether FP8-Flow-MoE is a drop-in replacement for BF16 without requiring hyperparameter tuning — a valid and useful test — but it also means that if FP8-Flow-MoE would benefit from slightly different hyperparameters (e.g., a modified learning rate to account for quantization noise), that benefit is not measured. The result is a conservative lower bound on FP8-Flow-MoE quality (it at least matches BF16 with stock settings) but does not establish that the quality ceiling is identical.

Mitigation status. The paper does not conduct larger-scale convergence experiments or downstream evaluations. The authors present the 16B result as validation that "our fp8-centric dataflow does not compromise convergence fidelity even under extended-scale workloads" (Section 4.1). This claim is partially supported for training loss at the tested scale, but the paper does not qualify it with respect to model size, training duration, or task quality. The conclusion (Section 5) states that FP8-Flow-MoE "maintain[s] convergence parity with BF16" without caveats about scale or metrics.

A practitioner deploying FP8-Flow-MoE at 671B scale would need to conduct their own convergence validation and downstream evaluation, which could itself cost substantial compute — partially offsetting the throughput gains that motivate adoption in the first place. The paper provides no tools or diagnostics for monitoring numerical health during FP8 training at scale, which would help practitioners detect divergence early without waiting for a full training run to complete.

---

6.3 The Throughput Gains Depend Critically on Expert Parallelism and Activation Checkpointing Configuration — Realized Gains Vary from +1% to +21%, and the Configurations Where Gains Are Largest are Not Necessarily the Most Common Deployment Configurations

The assumption or constraint. FP8-Flow-MoE's throughput advantage over baselines is not a fixed percentage — it varies dramatically across the tested configurations, from +1.3% (EP=8, AC=sel, vs. BF16) to +21% (EP=32, AC=full, vs. Blockwise). The paper's headline claim of "up to 21% higher throughput" selects the maximum observed gain across all configurations, which occurs at the highest tested expert parallelism level (EP=32) with full activation checkpointing. At lower EP settings — which are more common in many production deployments because they reduce the number of GPUs required and simplify communication scheduling — the gains are much more modest: +6% over BF16 at EP=8 (AC=full) and +1.3% at EP=8 (AC=sel).

The consequence. The large variance in throughput gains means that the cost-benefit case for adopting FP8-Flow-MoE depends heavily on the specific parallelism configuration. At EP=8 — a configuration that might be used for smaller-scale training or fine-tuning — FP8-Flow-MoE provides only a 1–6% throughput improvement over BF16. This may not justify the engineering effort of integrating a new FP8 dataflow, testing convergence, and maintaining FP8-specific code paths. If the actual deployment configuration is EP=4 or EP=1 (data parallelism only), the gains could be even smaller or even negative, though the paper does not test these configurations.

For the memory improvement, the same pattern holds. At EP=8 (AC=sel), FP8-Flow-MoE saves 8 GB versus BF16 (56 vs. 64 GB), which is meaningful (12.5% memory reduction) but much smaller than the 16.5 GB saving versus Blockwise that the abstract emphasizes. The 16.5 GB figure compares against a Blockwise baseline that anomalously uses more memory than BF16 due to dual-precision buffer duplication — it is not a comparison against the standard BF16 deployment that most practitioners currently use. Against BF16 directly, the memory savings are 8 GB at EP=8 and 5 GB at EP=16 (Table 3), which are material but less dramatic than the 16.5 GB headline.

The paper does not report throughput or memory numbers at EP levels below 8, so a practitioner deploying at EP=4 or EP=2 (common for smaller MoE models or fine-tuning on modest GPU counts) has no data to estimate their expected gains. The scaling trend (gains increase with EP) suggests that at very low EP, the gains might approach zero, but this extrapolation is not validated.

What evidence exists in the paper. Tables 2 and 3 report throughput and memory across exactly three EP settings (8, 16, 32) and two AC strategies (full, sel), totaling six data points per method. The 21% figure appears at EP=32 under AC=full (Table 2), comparing FP8-Flow-MoE (779 TGS) against Blockwise (644 TGS). Against BF16 at the same configuration, the improvement is +16% (779 vs. 671 TGS).

The minimum throughput gain over BF16 across all configurations is +1.3% (EP=8, AC=sel, Table 3: 1,193 vs. 1,178 TGS). This is essentially noise-level improvement and would be indistinguishable from measurement variance (the paper does not report variance, so we cannot assess whether 1,193 vs. 1,178 is statistically significant).

The paper does not analyze why the gains vary so much across configurations. The mechanism is presumably that at low EP, GEMM computation and non-MoE operations (attention, embedding, loss computation) dominate runtime, leaving less room for the MoE-specific dataflow improvements to affect end-to-end throughput. At high EP, communication and MoE-specific overheads dominate, and FP8-Flow-MoE's elimination of these overheads has proportionally more impact. The paper could have profiled the fraction of wall-clock time spent in different operations across EP levels to explain the scaling, but does not provide such a breakdown.

Mitigation status. The paper acknowledges that the gap "widens at higher EP levels" (Section 4.2, "Scaling amplifies FP8-Flow-MoE's gains") and presents this as a positive finding — the approach scales well to the most challenging configurations. This is true, but the flip side is that at the most common configurations, the gains are modest. The paper does not provide guidance on which EP threshold makes FP8-Flow-MoE decisively preferable to BF16, and does not test configurations below EP=8. The abstract and introduction present the 21% figure without qualification, which could mislead a practitioner who skips to the headline number without examining the configuration dependence in Tables 2–3.

A more complete picture would require: (1) throughput measurements at EP=1, 2, 4, 8, 16, 32 to establish the full scaling curve; (2) a breakdown of where time is spent in each configuration to explain the scaling; and (3) a recommendation for the break-even EP above which FP8-Flow-MoE is unambiguously worth adopting.

---

6.4 All Experiments Use a Single Hardware Platform (NVIDIA Hopper, 80 GB) and a Single Model Family (DeepSeek MoE) — Portability to Other Hardware, Model Architectures, and Precision Formats Is Unverified

The assumption or constraint. Every experiment in the paper — throughput benchmarks, memory measurements, kernel microbenchmarks, and convergence validation — uses NVIDIA Hopper GPUs with 80 GB HBM, running DeepSeek-V2-Lite or DeepSeek-V3 model architectures. The paper asserts compatibility with TransformerEngine and Megatron-LM, and the fused kernels are described as "merged into upstream repositories (e.g., NVIDIA/TransformerEngine)," but there is no evaluation on any other hardware platform (AMD MI300X, Intel Gaudi, Google TPU, custom ASICs) or any non-DeepSeek MoE architecture (Mixtral, Switch Transformer, GShard).

The consequence. Several of FP8-Flow-MoE's design choices are implicitly coupled to Hopper-specific features. The requirement that expert batch dimensions be multiples of 16 (for the fused permute-padding and unpermute-unpadding kernels) comes from Hopper Tensor Core tile sizes. The FP8 communication integration relies on DeepEP, which is a CUDA library targeting NVIDIA interconnects (NVLink, NVSwitch, InfiniBand). The power-of-two scaling factor constraint interacts with FP8 E4M3 format specifics — a different low-precision format (FP8 E5M2, FP6, FP4, MX formats) with different exponent and mantissa allocations might not support the same exponent-bit manipulation trick, or might require different power-of-two constraints.

For a practitioner using AMD GPUs (which support FP8 via ROCm but with different Tensor Core dimensions and communication primitives), the entire recipe would need to be re-engineered. The scaling-aware transpose might port directly if AMD's FP8 implementation also uses per-tile power-of-two scaling, but the fused kernels (permute-padding, SwiGLU-quantization, unpermute-unpadding) are CUDA code that would require rewriting in HIP or another AMD-compatible framework. The DeepEP communication integration would need to be replaced with RCCL-based all-to-all collectives, which have different latency and bandwidth characteristics.

For model architectures other than DeepSeek's MoE design — which uses a specific expert count, specific router mechanism, and specific SwiGLU activation in the expert FFN — the dataflow graph might differ. A Switch Transformer with different routing, a Mixtral-style architecture with different expert sharing patterns, or an MoE with attention layers inside experts could have different Q/DQ boundaries or different transpose requirements. The paper does not discuss how FP8-Flow-MoE's dataflow would adapt to these variations.

More subtly, the paper's choice to retain BF16 at two specific boundaries (SwiGLU activation and backward gradient aggregation) is justified by empirical observation that "FP8 cannot safely represent the intermediate results" at these points (Section 3.2). This justification is specific to the FP8 E4M3 format and the DeepSeek MoE architecture. A model with a different activation function (GELU, ReLU-squared) or different gradient flow patterns might have different safety boundaries, and the paper provides no diagnostic framework for identifying where BF16 fallbacks are necessary in a new architecture.

What evidence exists in the paper. Section 4 states: "All experiments are conducted on a 32-node NVIDIA Hopper cluster, where each node is equipped with eight GPUs (80 GB memory per GPU)." The kernel microbenchmarks (Figures 1, 3–5) use synthetic tensors but are executed on the same Hopper hardware. The convergence results (Figure 6) use DeepSeek-V2-Lite. The throughput results (Tables 2–3) use DeepSeek-V3. There is no mention of any other hardware, model architecture, or FP8 format variant.

The paper's compatibility claim ("compatible with TransformerEngine and Megatron-LM") is about software interfaces, not hardware portability. TransformerEngine supports multiple GPU architectures, but FP8-Flow-MoE's fused kernels and scaling-aware transpose are new operators that would need architecture-specific implementations.

Mitigation status. The paper does not claim cross-platform portability — it presents FP8-Flow-MoE as an NVIDIA Hopper recipe for DeepSeek-style MoE models. The limitation is one of scope, not a failure to deliver on a claim. However, the paper also does not discuss which components are architecture-dependent and which are general principles. A practitioner reading the paper cannot easily determine: (1) whether the scaling-aware transpose would work on AMD hardware; (2) whether the fused kernel design principles would apply to a different MoE architecture; or (3) whether the BF16 fallback boundaries are specific to FP8 E4M3 and SwiGLU or generalize to other formats and activations.

The paper's vision of extending to "other low-precision formats and representations" (Section 5) acknowledges the direction but provides no concrete analysis of what would need to change for FP6, FP4, or MX formats. The scaling-aware transpose depends on the quantization format supporting exponent-bit manipulation — a property that might not hold for formats with shared exponents (MX formats) or logarithmic number systems.

---

6.5 The Paper Does Not Measure End-to-End Training Time, Cost, or Energy — Only Abstract Throughput and Peak Memory Per GPU

The assumption or constraint. The paper's primary efficiency metrics are throughput in tokens per GPU per second (TGS) and peak memory usage per GPU in gigabytes (GB). These are standard metrics in systems papers, but they do not directly capture the quantities practitioners care about: total training wall-clock time, total GPU-hours consumed, total energy usage, or monetary cost. The paper assumes that higher TGS and lower memory usage translate proportionally into these downstream metrics, but this translation depends on factors not measured in the experiments.

The consequence. Several factors can cause the relationship between TGS and total training time to be non-linear, especially when comparing across different precision and dataflow strategies:

• Batch size constraints. FP8-Flow-MoE's memory savings (8–16.5 GB per GPU) might enable larger batch sizes per GPU, which could improve throughput further (by amortizing communication and kernel launch overhead) or could require changes to the learning rate schedule (since larger batches typically need higher learning rates or different warmup strategies). The paper does not test whether FP8-Flow-MoE's memory headroom can be productively used for larger batch sizes, or whether doing so would require hyperparameter retuning that might affect convergence.

• Communication-computation overlap. The throughput experiments explicitly disable communication-computation overlap "to isolate computational efficiency." In production training, overlap is almost always enabled — it is a standard feature of Megatron-LM and other frameworks that hides communication latency behind computation. The baselines (BF16 and Blockwise) have longer communication stages, which might actually benefit more from overlap than FP8-Flow-MoE (whose communication is already faster). If so, the gap measured without overlap might overstate the advantage in realistic overlap-enabled training. Conversely, FP8-Flow-MoE's shorter communication latency might enable more effective overlap scheduling. The paper provides no data to resolve this uncertainty.

• Checkpointing overhead. The paper measures peak memory with two activation checkpointing strategies but does not report the recomputation overhead — the additional FLOPs and time spent recomputing activations during the backward pass. Different checkpointing strategies interact differently with FP8 storage: if FP8-Flow-MoE stores activations in FP8 rather than BF16, the recomputation cost is the same (both need to recompute from the checkpointed tensor, which is FP8 vs. BF16), but the memory savings might change which checkpointing strategy is optimal. The paper does not explore this tradeoff.

• Initialization and compilation overhead. FP8-Flow-MoE introduces new fused kernels that may require just-in-time (JIT) compilation on first use, adding startup latency not reflected in steady-state throughput measurements. For short training runs or frequent restarts, this overhead could be material.

• Convergence speed in terms of tokens vs. wall-clock time. The convergence experiment (Figure 6) plots loss against consumed tokens, not wall-clock time. If FP8-Flow-MoE trains at higher TGS but requires slightly more tokens to reach the same loss (e.g., due to quantization noise requiring a smaller learning rate or more training steps), the net wall-clock time advantage could be smaller than the TGS advantage suggests. The paper shows identical loss-versus-tokens curves for BF16 and FP8-Flow-MoE, suggesting no token penalty, but this is demonstrated only at 16B scale and only for training loss.

What evidence exists in the paper. The paper reports TGS and peak memory in Tables 2–3, and training loss in Figure 6. There is no measurement of: total training wall-clock time for any configuration; GPU-hours consumed for the 200B-token convergence run or for any throughput benchmark; total energy consumption; or any cost estimate (dollars per training run, dollars per token). The communication microbenchmarks (Table 1) report latency in milliseconds but do not translate this into end-to-end training time impact.

Mitigation status. The paper does not discuss the gap between abstract throughput metrics and real-world training cost. TGS and peak memory are standard in the systems literature and sufficient for a research contribution, but a practitioner evaluating whether to adopt FP8-Flow-MoE needs cost estimates that account for the factors above. The paper provides no guidance on estimating end-to-end training time from the reported TGS numbers, and no analysis of how the various overheads (JIT compilation, recomputation, communication overlap) affect the TGS-to-wall-time translation.

---

6.6 The Revision Model Has a 38% Correct-to-Incorrect Reversion Rate — This Section Does Not Apply to FP8-Flow-MoE

[Self-correction: this limitation is not relevant to FP8-Flow-MoE. The reversion rate concern applies to the revision model in the test-time compute scaling paper, not to a low-precision training recipe. I will replace this with an actual limitation of the current paper.]

---

6.6 The Blockwise Baseline Represents a Conservative FP8 Integration — a More Aggressive Baseline (DeepSeek-V3-Style with Full FP8 Communication) Is Not Evaluated System-Level, Making the Dataflow Comparison Incomplete

The assumption or constraint. The paper's experimental comparison (Section 4.2) evaluates three configurations: BF16, Blockwise (TransformerEngine-style, FP8 only in GEMMs), and FP8-Flow-MoE. The DeepSeek-V3-style dataflow (Figure 2c) — which extends FP8 to both GEMMs and communication, with up to 12 cast operations and a naïve dequantize→transpose→requantize at the Wgrad boundary — is analyzed only via communication microbenchmarks (Table 1) and the computational graph diagram (Figure 2c). It is not included as a system-level baseline in the end-to-end throughput or memory experiments.

The consequence. The paper's central claim is that FP8-Flow-MoE's dataflow redesign (not just FP8 kernel usage) is responsible for the throughput gains. The Blockwise baseline supports this claim by showing that FP8 GEMM acceleration alone yields only +3% throughput at EP=8 and negative throughput at EP=32 (Table 2: 644 vs. 671 TGS BF16). However, the Blockwise baseline uses FP8 only for GEMMs — it does not use FP8 communication at all. A more natural comparison for isolating the dataflow contribution would be a baseline that uses all the same FP8 components as FP8-Flow-MoE (FP8 GEMMs, FP8 communication via DeepEP) but with the DeepSeek-V3-style BF16-centric dataflow — i.e., explicit Q/DQ at every boundary and the naïve requantization-based Wgrad transpose.

Without this baseline, we cannot distinguish between two possible sources of FP8-Flow-MoE's advantage:

1. The dataflow architecture (persistent FP8, fused Q/DQ, scaling-aware transpose) is the dominant contributor.
2. Using FP8 communication at all is the dominant contributor — and the DeepSeek-V3-style baseline, which also uses FP8 communication, would achieve nearly the same throughput as FP8-Flow-MoE despite its Q/DQ overhead.

The communication microbenchmarks (Table 1) suggest that standalone Q/DQ around communication erodes most of FP8's communication speedup at small-to-medium scales, which supports interpretation (1). But microbenchmarks measure isolated operations, not system-level interactions. A full system-level DeepSeek-V3-style baseline might reveal that the Q/DQ overhead, while measurable in isolation, is overlapped with computation in practice and has less end-to-end impact than the microbenchmarks imply.

The Blockwise baseline also uses substantially less FP8 than the DeepSeek-V3-style approach (FP8 only in GEMMs, not in communication), making it a weaker baseline than is strictly necessary to test the paper's claims. FP8-Flow-MoE's throughput advantage over Blockwise conflates two effects: extending FP8 to communication (which Blockwise does not do) and eliminating Q/DQ overhead (which the DeepSeek-V3 style does not do). A three-way comparison of BF16 vs. DeepSeek-V3-style vs. FP8-Flow-MoE would cleanly separate these effects.

What evidence exists in the paper. Table 1 provides isolated communication latency measurements for BF16, FP8 communication with Q/DQ, and FP8 communication without Q/DQ (for the "COMM" column). The analysis in Section 3.2 states: "a single pair of quantization and dequantization operators reduces the performance gain of FP8 communication by roughly one third... Considering that a typical MoE forward or backward pass involves around three such pairs (see Figure 2(c)), the benefit of FP8 kernels can be almost completely neutralized." This is an extrapolation from microbenchmarks to system-level impact, not a system-level measurement.

The DeepSeek-V3-style dataflow is described in Section 3.2 and Figure 2c, but there is no throughput or memory measurement for a system implementing this dataflow. The paper has the infrastructure to create such a baseline — it has FP8 GEMM kernels (from DeepGEMM or TransformerEngine), FP8 communication (from DeepEP), and can implement the explicit Q/DQ nodes — but does not report numbers.

Mitigation status. The paper does not acknowledge this gap. The DeepSeek-V3 style is presented as a qualitative strawman in the dataflow comparison (Figure 2) but is not quantitatively evaluated. The conclusion (Section 5) states that "maintaining data in low precision throughout the computation flow minimizes redundant conversions," which is a claim about the benefit of persistence over the DeepSeek-V3 style, but the supporting evidence is only the Blockwise comparison (which differs on communication scope as well as persistence) and the communication microbenchmarks (which are not end-to-end).

A reader familiar with DeepSeek-V3's approach might reasonably ask: "How much of FP8-Flow-MoE's 21% gain comes from eliminating the Q/DQ overhead, and how much comes from extending FP8 to communication at all?" The paper's experiments cannot answer this question definitively. A system-level DeepSeek-V3-style baseline would close this gap and strengthen the evidence for the dataflow architecture's unique contribution. Its absence is a missed opportunity given that the paper's own analysis identifies the DeepSeek-V3-style approach as the most relevant prior art.

7. Implications and Future Directions

How This Work Changes the Landscape

FP8-Flow-MoE establishes a dataflow architecture paradigm for low-precision training that represents a genuine shift in how systems researchers should think about integrating reduced-precision formats into large-scale training pipelines. The shift is not in the individual techniques—operator fusion, power-of-two scaling factors, and per-tile quantization all predate this work—but in the recognition that precision persistence, not just precision acceleration, determines whether low-precision training delivers its theoretical throughput at system scale.

The paper's most lasting conceptual contribution is its reframing of the FP8 integration problem from a precision selection question (where should we use FP8 vs. BF16?) to a consistency enforcement question (how do we maintain quantization direction consistency across layout changes so that precision transitions are unnecessary?). Prior work treated FP8 as an accelerator for specific operations—drop it into GEMMs, drop it into communication—while accepting BF16 as the "ambient" precision of the dataflow graph. This led to systems where FP8 kernels were islands in a BF16 ocean, with quantize–dequantize ferries shuttling data across the precision boundary at every island edge. FP8-Flow-MoE inverts this: FP8 becomes the ambient precision, and BF16 becomes the exception, localized to two specific computational boundaries where FP8's limited dynamic range genuinely threatens numerical stability.

This inversion has methodological implications beyond FP8 training on Hopper GPUs:

It redefines what "low-precision training efficiency" means. The field has largely measured low-precision efficiency by kernel-level throughput—the ratio of FP8 GEMM TFLOPs to BF16 GEMM TFLOPs, or the raw bandwidth of FP8 communication collectives. FP8-Flow-MoE demonstrates that these kernel-level metrics are poor predictors of system-level throughput. The Blockwise baseline uses the same FP8 GEMM kernels as FP8-Flow-MoE (both rely on TransformerEngine or DeepGEMM for grouped GEMM), yet achieves only 3% end-to-end throughput improvement at EP=8 and actually regresses below BF16 at EP=32 (Table 2). The paper's most important empirical finding is not the 21% throughput gain itself but the decomposition of that gain: roughly 3% comes from FP8 GEMM acceleration (Blockwise vs. BF16 at low EP), and the remaining 18% comes from eliminating precision transitions and fusing quantization into compute kernels (FP8-Flow-MoE vs. Blockwise at EP=32). This decomposition implies that future low-precision systems research should prioritize dataflow-level precision persistence over kernel-level arithmetic throughput, because the former is the bottleneck that has been leaving most of the latter's potential unrealized.

It provides a causal diagnostic for why prior FP8 training results disappointed. The paper's analysis of DeepEP communication overhead (Table 1) explains a pattern that was visible but unexplained in prior work: FP8 training systems that showed impressive kernel-level benchmarks (2× GEMM speedup, 1.6× communication speedup) routinely delivered single-digit end-to-end throughput gains. The explanation, grounded in Table 1's latency breakdown, is that quantization and dequantization operations surrounding FP8 kernels erode the kernel-level speedup by approximately one-third per Q/DQ pair, and with up to 12 Q/DQ pairs in a single MoE forward-backward pass (Figure 2c), the erosion can compound to near-total neutralization. This is not a failure of FP8 kernel quality—the kernels are fast in isolation—but a failure of dataflow architecture: the precision transition boundaries are too numerous for the kernel acceleration to overcome. This diagnostic will transfer directly to future precision transitions (FP6, FP4, MX formats), where the same tension between kernel speed and transition overhead will recur, likely in amplified form since the relative cost of precision transitions increases as kernel computation gets faster.

It resolves a latent contradiction in the FP8 training literature. The DeepSeek-V3 system successfully trained at trillion-parameter scale with FP8 communication and GEMMs, demonstrating that aggressive FP8 usage is feasible. TransformerEngine's blockwise recipe demonstrated that conservative FP8 usage (GEMM-only) is numerically safe and provides modest gains. The contradiction: if aggressive FP8 usage works at massive scale, why are the gains so modest? And if conservative FP8 usage is safe, why not extend it more broadly? FP8-Flow-MoE resolves this by showing that the numerical risk of aggressive FP8 usage is concentrated at specific points (layout-inconsistent requantization producing double quantization error, nonlinearities with limited dynamic range), and that a dataflow aware of these points can be both aggressive (FP8 everywhere else) and numerically stable. The double quantization error analysis (Equations 1–16) provides the causal mechanism: the DeepSeek-V3-style approach is not simply "aggressive FP8" but specifically aggressive at the Wgrad layout conversion where double quantization error lives, while being conservative at other points where FP8 would be safe. FP8-Flow-MoE reverses this: aggressive at safe points, surgically precise at the one unsafe point.

It makes verifier over-optimization—no, wrong paper again. [Self-correction: this concept belongs to the test-time compute scaling paper, not FP8-Flow-MoE.]

It makes low-precision dataflow design a first-class research area, distinct from low-precision kernel engineering. Before this work, the systems community's response to FP8 was primarily to build better kernels: faster FP8 GEMMs (DeepGEMM, CUTLASS FP8), faster FP8 communication (DeepEP), better quantization strategies (block-wise, per-channel). FP8-Flow-MoE demonstrates that kernel quality is necessary but insufficient—a system with excellent kernels and a poorly designed dataflow will systematically underperform. This opens a new research direction: low-precision dataflow optimization, where the design variables are not kernel implementation choices but graph-level decisions about where precision transitions occur, which quantization directions are used at which points, and how layout conversions are performed without requantization. The scaling-aware transpose is an exemplar of this class of design: it is not a faster kernel for an existing operation but a fundamentally different operation that eliminates a source of error and overhead simultaneously.

The paper also implicitly argues that memory savings from low-precision persistence are at least as important as throughput gains, and potentially more so because they are enabling rather than merely accelerating. The OOM-avoidance result at EP=32 (Table 3) is the paper's most striking single data point: FP8-Flow-MoE runs where both BF16 and Blockwise crash. This shifts the value proposition of low-precision training from "it makes training faster" to "it makes certain training configurations possible at all." For organizations operating at the edge of GPU memory capacity—which is essentially all organizations training frontier-scale MoE models—this capability expansion may be more consequential than the throughput improvements.

Follow-Up Research This Work Enables

End-to-end DeepSeek-V3-style baseline: quantify the contribution of dataflow architecture vs. FP8 communication scope. The paper's central claim is that FP8-Flow-MoE's dataflow persistence (fused Q/DQ, scaling-aware transpose) is the dominant contributor to throughput gains, not merely the use of FP8 communication. However, the Blockwise baseline uses FP8 only in GEMMs, not communication, making it impossible to separate the effects. A crucial follow-up experiment would implement a full DeepSeek-V3-style dataflow (FP8 GEMMs + FP8 communication via DeepEP + explicit Q/DQ at all boundaries + naïve dequantize→transpose→requantize at Wgrad) and measure end-to-end throughput and memory on the same DeepSeek-V3 configurations (EP=8, 16, 32; AC=full and AC=sel). This would produce a four-point comparison: BF16 vs. Blockwise (FP8 GEMM only) vs. DeepSeek-V3-style (FP8 GEMM + communication, naïve dataflow) vs. FP8-Flow-MoE (FP8 GEMM + communication, persistent dataflow). The gap between DeepSeek-V3-style and FP8-Flow-MoE would isolate the contribution of dataflow persistence, while the gap between Blockwise and DeepSeek-V3-style would isolate the contribution of FP8 communication. This experiment is feasible using the infrastructure the paper already has (DeepGEMM, DeepEP, the fused kernels) and would substantially strengthen the evidence for the dataflow architecture's unique contribution. A negative result—if DeepSeek-V3-style achieves throughput close to FP8-Flow-MoE—would reframe the paper's contribution as primarily about extending FP8 to communication rather than about dataflow persistence, which would be an important corrective.

Downstream task evaluation of FP8-Flow-MoE-trained models: beyond training loss convergence. The convergence experiment (Figure 6) demonstrates that FP8-Flow-MoE matches BF16 training loss on a 16B model for 200B tokens. However, training loss is a weak proxy for model quality. A follow-up study should train two DeepSeek-V2-Lite models—one BF16, one FP8-Flow-MoE—for the full 200B tokens and evaluate them on a standard benchmark suite: held-out validation perplexity, few-shot accuracy on MMLU, MATH, HumanEval, and perhaps a suite of reasoning and knowledge-intensive tasks. The key question is whether FP8-Flow-MoE's power-of-two scaling factor constraint and persistent FP8 dataflow introduce subtle distributional shifts in model representations that, while invisible in training loss, manifest as degraded downstream performance. This would require a larger evaluation budget than the paper's current scope but is essential for practitioners deciding whether to adopt the recipe. A related stress test: train both models for 500B–1T tokens (extending beyond the paper's 200B) to check whether accumulated quantization error causes late-training divergence that the 200B run misses. If FP8-Flow-MoE degrades relative to BF16 between 200B and 1T tokens, this would establish a practical upper bound on training duration for the recipe.

Scaling-aware transpose without power-of-two constraint: characterization and mitigation of the precision–efficiency tradeoff. The scaling-aware transpose's exactness depends on scaling factors being powers of two. The paper provides no measurement of how much precision is lost by constraining scaling factors to powers of two compared to unconstrained per-tile quantization. A systematic study should measure: (1) the distribution of quantization errors for power-of-two-constrained vs. unconstrained scaling factors across representative transformer activation tensors (e.g., sample activations from each layer of a trained DeepSeek-V2-Lite model); (2) the downstream impact on model quality by training small models (e.g., 1B parameters) to convergence with both strategies and comparing validation loss and benchmark accuracy; (3) the feasibility of a hybrid strategy where the scaling-aware transpose is used for tensors that will undergo layout conversion, while unconstrained scaling factors are used elsewhere (at the cost of maintaining separate scaling factor representations). If the precision loss from power-of-two constraints is negligible, the hybrid strategy is unnecessary complexity. If it is measurable but small, the hybrid approach might recover precision without sacrificing the transpose speedup.

Porting FP8-Flow-MoE to non-NVIDIA hardware: an architecture study of which components generalize. The paper's techniques are tightly coupled to NVIDIA Hopper hardware (Tensor Core tile sizes dictating the padding alignment, DeepEP for communication, CUDA for fused kernels). A valuable follow-up would port FP8-Flow-MoE to AMD MI300X GPUs (which have their own FP8 Tensor Core support and ROCm communication libraries) and measure how much of the throughput and memory advantage transfers. The research question is not "does it work on AMD?"—the fused kernels would need to be rewritten—but "which design principles are hardware-agnostic and which are Hopper-specific?" Specific hypotheses to test: (1) the scaling-aware transpose should be fully portable because it depends only on the FP8 E4M3 format specification, not on hardware details; (2) the fused permute-padding kernel depends on Tensor Core tile alignment (multiples of 16), and AMD's Matrix Core tile size might differ, requiring different padding logic; (3) the communication overhead reduction (fusing Q/DQ into surrounding kernels) should be equally beneficial on AMD because the latency ratios in Table 1 depend on data volume and kernel launch overhead, both of which are hardware-agnostic concerns. A negative result—if FP8-Flow-MoE's advantages largely disappear on AMD—would suggest the recipe is more of a Hopper-specific optimization than a general low-precision training methodology.

FP8-Flow-MoE for dense transformer training: does the dataflow architecture principle generalize beyond MoE? The paper's dataflow redesign is MoE-specific—it eliminates Q/DQ at expert boundaries, communication, and grouped GEMM transitions. Dense transformers have a simpler computational graph (no routing, no all-to-all communication, no permute/unpermute), so the 12-to-2 cast reduction would not directly transfer. However, dense transformers still have Q/DQ overhead at every linear layer boundary (attention projections, FFN layers) and still have transpose operations (e.g., between the forward pass's activation storage and the backward pass's gradient computation). A follow-up should design an FP8-Flow-Dense dataflow applying the same principles: persistent FP8 storage except at nonlinearities and gradient aggregation points, fused quantization into activation functions and residual connections, and scaling-aware transpose at any point where activation tensors are transposed between forward and backward passes. The throughput gains would likely be smaller than for MoE (since the communication overhead that FP8-Flow-MoE eliminates is MoE-specific), but the memory savings from FP8 activation storage might be proportionally similar. Benchmarking FP8-Flow-Dense on Llama-2 or GPT-3 scale dense models would establish whether the dataflow persistence principle is MoE-specific or general.

Diagnostic tooling for numerical health monitoring during FP8 training. The paper provides no tools for practitioners to monitor whether FP8 training is numerically healthy—no metrics for quantization error accumulation, no alerts for impending divergence, no guidance on what "healthy" activation distributions look like under FP8. A valuable engineering contribution would be a numerical health dashboard for FP8 training that tracks: (1) the distribution of per-tile scaling factors over training time (are they drifting? are outliers increasing?); (2) the frequency of saturation events (values clipped to FP8 E4M3 maximum) per layer and per training step; (3) the effective quantization SNR (ratio of signal variance to quantization error variance) for key tensors; and (4) gradient norm statistics compared to BF16 baselines at equivalent training points. Such a dashboard would dramatically reduce the adoption barrier for FP8 training by giving practitioners confidence that their training run is not silently accumulating numerical errors, and would provide early warning if the power-of-two scaling constraint or the BF16 boundaries need adjustment for a particular model architecture.

Practical Applications and Downstream Use Cases

Training DeepSeek-V3-scale MoE models under GPU memory constraints. The most immediate practical application is enabling training of 671B-parameter MoE models on GPU clusters where memory capacity is the binding constraint. The OOM-avoidance result at EP=32 (Table 3) is directly actionable: organizations with 32-node Hopper clusters running expert parallelism across all nodes, using selective activation checkpointing to minimize recomputation, cannot train DeepSeek-V3 with BF16 or Blockwise FP8 at the memory efficiencies they would prefer. FP8-Flow-MoE enables this configuration with 75 GB peak memory per GPU and 912 TGS throughput. The memory savings (8 GB vs. BF16 at EP=8, AC=sel; enabling EP=32 where BF16 crashes) translate to operational flexibility: practitioners can use larger batch sizes (improving throughput and potentially training dynamics), reduce the number of GPUs required (lowering cost), or add activation checkpointing selectively (improving memory–computation tradeoff) in ways that BF16 training cannot support. For organizations training at the frontier where every GPU's memory is fully utilized, FP8-Flow-MoE expands the feasible design space.

Reducing per-token training cost for large-scale MoE pretraining. At EP=32 with full activation checkpointing (Table 2), FP8-Flow-MoE achieves 779 TGS vs. 671 TGS for BF16, a 16% throughput improvement. For a training run consuming 10T tokens on a 671B model using hundreds of GPUs over months, a 16% throughput improvement directly reduces GPU-hours by approximately 14% (1 - 1/1.16), which at frontier training scales translates to millions of dollars in compute cost savings and proportionally lower energy consumption. The throughput gain is largest at configurations where communication dominates (high EP), which is precisely the regime used for the largest-scale training runs. This makes FP8-Flow-MoE most valuable for the most expensive training jobs—an attractive property for adoption. The convergence validation (Figure 6) gives confidence that these cost savings do not come at the expense of model quality, though the caveat about downstream evaluation (see Section 6.2) means practitioners should conduct their own task-specific validation.

FP8 activation checkpointing for memory-efficient fine-tuning and continued pretraining. FP8-Flow-MoE stores activations in FP8 format rather than BF16, reducing activation memory by approximately half for the MoE expert path. This is particularly valuable for fine-tuning and continued pretraining scenarios where the base model is large and GPU memory limits the feasible batch size or sequence length. For example, fine-tuning a 671B MoE model on long-context documents (e.g., 32K–128K token sequences) is often memory-bound by activation storage rather than parameter storage (parameters can be sharded across GPUs via tensor parallelism and expert parallelism; activations scale with batch size and sequence length and are harder to shard). FP8 activation storage could enable 2× longer sequences or 2× larger micro-batch sizes in memory-constrained fine-tuning regimes. The paper does not benchmark fine-tuning specifically, but the activation compression mechanism (FP8 vs. BF16 storage) should apply regardless of whether the forward pass is part of pretraining or fine-tuning.

Integration into Megatron-LM and TransformerEngine training stacks with minimal engineering effort. The paper emphasizes that FP8-Flow-MoE is "plug-and-play" and that several kernels are already merged into upstream TransformerEngine. For practitioners using these frameworks, adopting FP8-Flow-MoE requires configuration changes (enabling the FP8-centric dataflow, specifying the scaling-aware transpose at Wgrad boundaries) rather than writing new training code. The fused operators (permute-padding, SwiGLU-quantization, unpermute-unpadding) are drop-in replacements for existing unfused kernels. This low integration barrier is a practical advantage over approaches that require custom training frameworks or significant modifications to the model definition. The primary adoption cost is convergence validation on the practitioner's specific model and data, which the paper does not eliminate but provides a template for (train a small proxy model, compare loss curves, verify no divergence).

When to Prefer This Method

The paper positions FP8-Flow-MoE against two specific alternatives—BF16 mixed precision (the established high-precision baseline) and the TransformerEngine Blockwise recipe (the dominant conservative FP8 integration)—and the experimental results provide clear decision boundaries:

Prefer FP8-Flow-MoE over BF16 when:

• Training MoE models at scale where expert parallelism is high (EP ≥ 8), since the throughput advantage grows with EP and is modest (+1–6%) at EP=8.
• GPU memory is the binding constraint, especially under selective activation checkpointing (AC=sel), where FP8-Flow-MoE saves 5–8 GB per GPU vs. BF16 and enables configurations (EP=32 with AC=sel) that BF16 cannot run.
• Communication overhead is a significant fraction of iteration time (high EP, small batch sizes per GPU), because FP8-Flow-MoE's elimination of Q/DQ at communication boundaries directly addresses this bottleneck.
• Training cost is the primary concern and the organization can invest in convergence validation at a smaller scale before full-scale adoption.

Prefer FP8-Flow-MoE over Blockwise FP8 when:

• Using FP8 communication in addition to FP8 GEMMs, since Blockwise does not extend FP8 to communication and thus leaves substantial performance on the table for communication-bound configurations (EP=16, 32).
• Memory efficiency is critical—Blockwise's dual-precision buffer duplication can increase memory usage over BF16 (73 GB vs. 64 GB at EP=8, AC=sel), while FP8-Flow-MoE reduces it.
• Training at expert parallelism where Blockwise regresses below BF16 throughput (EP=16, AC=sel: 1,031 vs. 1,055 TGS; EP=32, AC=full: 644 vs. 671 TGS), indicating that conservative FP8 integration is actively harmful in these regimes.

Prefer BF16 or Blockwise over FP8-Flow-MoE when:

• Training at low expert parallelism (EP ≤ 4) where the throughput advantage of FP8-Flow-MoE is likely small or unmeasured, and the engineering cost of adopting a new dataflow may not be justified.
• Training dense (non-MoE) models, since the dataflow redesign is MoE-specific and the paper provides no evidence for dense transformer benefits.
• Maximum numerical safety is required and downstream task quality has not been validated for FP8-Flow-MoE on the specific model and data distribution—BF16 remains the gold standard for convergence fidelity.
• Hardware or software constraints prevent using FP8 communication (e.g., clusters without Hopper GPUs, frameworks without DeepEP support), in which case FP8-Flow-MoE's communication-related advantages are inaccessible and Blockwise may be the only viable FP8 option.
• Training on AMD or non-NVIDIA hardware without a ported implementation of the fused kernels and scaling-aware transpose—adoption would require significant engineering investment that the paper does not quantify.

The key unknown for decision-making is the break-even expert parallelism below which FP8-Flow-MoE's throughput advantage becomes negligible. The paper tests EP=8, 16, 32 but not EP=1, 2, 4. A practitioner running data-parallel-only training (EP=1) or modest expert parallelism (EP=4) cannot estimate their expected gain from the paper's data. This gap should be closed by the follow-up experiment proposed above (throughput scaling curve across EP=1 through 32) before the decision rules can be applied confidently at low EP.