FP8-RL: A Practical and Stable Low-Precision Stack for LLM Reinforcement Learning

ArXiv: 2601.18150

🎯 Pitch

RL-trained LLMs can generate rollouts in FP8 with up to 44% higher throughput and match BF16 training quality—but only if you add importance-sampling correction. Without correction, the low-precision generation drifts enough to degrade accuracy, showing that the mismatch between the FP8 rollout policy and the high-precision trainer is a real stability threat, not just a theoretical concern.

1. Executive Summary

This report introduces FP8-RL, a practical low-precision rollout stack that accelerates reinforcement learning for LLMs by applying FP8 quantization to both linear layers and the KV-cache during generation, implemented within the veRL ecosystem with support for FSDP, Megatron-LM, vLLM, and SGLang. The system addresses two core challenges unique to RL—dynamic weight synchronization at every policy update step (blockwise W8A8 quantization with per-step weight re-quantization and loading into the inference engine) and train-inference mismatch from low-precision rollouts (token-level truncated importance sampling with clipping threshold C=2)—across dense models (Qwen3-8B-Base) and MoE models (Qwen3-30B-A3B-Base) trained with DAPO on the AIME24 benchmark under long-context generation (20K-token maximum responses). FP8 W8A8 linear rollout delivers 10–20% speedup for the 8B dense model and 30–50% for the 30B MoE model, while adding KV-cache quantization yields up to 44% cumulative throughput improvement, with learning curves closely tracking BF16 baselines when importance-sampling correction is enabled, establishing that FP8 rollouts can match full-precision training quality only when paired with explicit mismatch mitigation—without TIS, accuracy degrades noticeably—and that end-to-end FP8 (training + rollout) further reduces distribution drift but does not eliminate it.

2. Context and Motivation

The Core Problem: RL Rollouts Are the Bottleneck, and We Can't Just Use Off-the-Shelf FP8

The fundamental problem this paper addresses is that reinforcement learning for large language models is bottlenecked by generation (rollout), and existing low-precision inference solutions cannot be applied naïvely because RL introduces unique engineering and algorithmic challenges not present in static deployment.

To understand why this matters, we need to first understand the structure of an LLM RL training loop. In standard RL fine-tuning (using algorithms like PPO, DAPO, or GRPO), each training step proceeds as follows: the current policy model generates completions for a batch of prompts (the rollout phase), a reward model scores those completions, advantages are computed, and then the policy is updated via gradient-based optimization (the training phase). This cycle repeats for thousands of steps. The critical property is that the rollout phase must use the most recent policy weights—the model that generates completions must be identical to the model being optimized. In RL terms, this is an on-policy requirement.

The rollout phase involves autoregressive generation, which is computationally expensive for two reasons: first, the quadratic complexity of attention with respect to sequence length, and second, the memory footprint of the key-value (KV) cache, which stores intermediate attention states for every token in the sequence to avoid recomputation during incremental decoding. The paper cites Seer [1], which reports that rollout can consume approximately 80% of total iteration time in synchronous LLM RL settings. This means that even if training were instantaneous, the end-to-end RL loop would still be dominated by the time spent generating text. Consequently, accelerating rollout is the highest-leverage intervention for improving overall throughput and shortening experimentation cycles.

FP8 quantization—representing floating-point numbers with 8 bits rather than the standard 16 (BF16/FP16)—is a mature technology that reduces both compute cost and memory traffic during inference. It is already supported in widely used serving engines like TensorRT-LLM [2], vLLM [3], and SGLang [4]. On modern NVIDIA hardware (Hopper and Blackwell architectures), FP8 tensor cores can perform matrix multiplications roughly twice as fast as their BF16 counterparts while consuming half the memory bandwidth for weight and activation transfers. For static inference—where a model is loaded once and serves many requests—FP8 is essentially a free lunch: quantize the model offline, load it into the inference engine, and enjoy higher throughput at negligible accuracy loss.

But RL is not static inference. The paper identifies two fundamental barriers that prevent off-the-shelf FP8 inference from working in an RL pipeline:

Challenge 1: Dynamic weight synchronization. In RL, the policy weights change every training step. This means the rollout engine cannot simply load a pre-quantized model once and serve indefinitely. Instead, after every training update, the new weights must be retrieved from the training backend, re-quantized to FP8, and loaded into the inference engine—a process that must happen at the granularity of individual RL steps. This is an engineering challenge because modern RL stacks typically decouple training and inference into separate components (e.g., FSDP or Megatron-LM for training, vLLM or SGLang for inference), often running on different processes or even different machines. Building a robust, low-latency pipeline that synchronizes freshly quantized weights at step-level frequency across these decoupled systems is non-trivial, and prior work had not provided a turnkey solution.

Challenge 2: Train-inference mismatch from low-precision rollouts. Even if we solve the engineering challenge of dynamic synchronization, there is a deeper algorithmic problem. The RL training objective assumes that the rollouts are sampled from the exact policy being optimized: the gradient estimator (e.g., the PPO clipped surrogate objective) computes probability ratios $\pi_\theta(a|s) / \pi_{\theta_{\text{old}}}(a|s)$, where $\pi_{\theta_{\text{old}}}$ is the behavior policy that generated the data. If rollouts are generated by an FP8-quantized version of the policy (call it $\pi_\theta^{\text{FP8}}$) while the trainer optimizes the full-precision policy $\pi_\theta$, the data is effectively off-policy. The quantized model produces a slightly different token distribution than the full-precision model due to accumulated quantization error—weights are rounded, activations are truncated, and attention computations use lower-precision arithmetic. This distribution shift introduces bias into the gradient estimates, which prior work has shown can destabilize training and even cause collapse [5, 6, 7]. The paper frames this as a train-inference mismatch: the training algorithm believes it is optimizing $\pi_\theta$ with on-policy data, but the data actually comes from a different distribution $\pi_\theta^{\text{FP8}}$, violating a core assumption of the algorithm.

Why This Problem Matters (Real-World Impact)

The significance of this problem extends beyond academic curiosity. RL post-training (particularly RLHF and its variants like DAPO, GRPO, and RLOO) has become a critical stage in developing state-of-the-art LLMs. Models like GPT-4, Claude, Gemini, and DeepSeek-R1 all employ RL fine-tuning to improve instruction following, reasoning, safety alignment, and tool use. As models grow larger and context lengths increase (driven by reasoning tasks that require chain-of-thought over thousands of tokens), the rollout phase becomes proportionally more expensive.

Consider a concrete scenario drawn from the paper's experiments: training a 30B-parameter MoE model with 20K-token maximum responses. Each training step requires generating $32 \times 3 \times 16 = 1536$ responses (32 prompts, $\times 3$ for the multi-pass DAPO protocol, $\times 16$ responses per prompt for sample diversity), each up to 20K tokens long. The total tokens generated per step can reach tens of millions. At BF16 precision, the KV-cache for such long sequences can exhaust GPU memory, forcing the inference engine to preempt and restart generation—wasting computation and reducing throughput. The paper reports that even for the smaller 8B dense model at 20K context, BF16 KV-cache causes "frequent request preemptions" that "wasted computation and throttled throughput." Making rollout faster directly shortens the iteration cycle, enabling faster experimentation, more training steps within a fixed budget, and ultimately better models.

Furthermore, the economic argument is compelling. If rollout consumes 80% of iteration time, a 44% speedup in rollout (the paper's best combined result) translates to roughly a 35% reduction in total training wall-clock time. For organizations running thousand-GPU RL training jobs, this represents millions of dollars in saved compute and dramatically shortened development cycles. The paper positions FP8 rollout as an efficiency lever that can be adopted without sacrificing model quality—a key requirement for production adoption—provided the mismatch is properly corrected.

Where Existing Approaches Fall Short

The paper identifies several gaps in prior work across three areas: FP8 quantization for LLMs, RL systems for LLMs, and train-inference mismatch mitigation.

FP8 Quantization for LLMs (Gap: No RL-Specific Workflow)

FP8 formats were formally specified by Micikevicius et al. [12], who proposed the E4M3 (4 exponent bits, 3 mantissa bits) and E5M2 encodings. FP8-LM [13] demonstrated that these formats could be applied to weights, gradients, and optimizer states during pretraining and fine-tuning, achieving 2× throughput speedup. DeepSeek-V3 [8] reported the first successful industrial-scale FP8 training using fine-grained blockwise quantization ($1 \times 128$ tiles for activations, $128 \times 128$ blocks for weights), setting a new benchmark for low-precision training.

On the inference side, FP8 deployment is well-established in serving frameworks (vLLM [3], TensorRT-LLM [2], SGLang [4]), which combine reduced-precision computation with optimized attention kernels, paged KV-cache management, and efficient runtime scheduling.

However, none of these prior efforts addressed the RL use case. FP8 training work assumes a static model being trained once. FP8 inference work assumes a pre-quantized model serving many requests. Neither provides the dynamic weight synchronization pipeline needed when the model changes every step. Neither addresses the train-inference mismatch that arises when quantized rollouts feed into a full-precision trainer. The paper fills this gap by building a complete RL-specific FP8 rollout workflow that handles both the engineering (synchronization) and algorithmic (mismatch correction) challenges.

RL Systems for LLMs (Gap: No Native FP8 Rollout Support)

The landscape of RL systems for LLMs has evolved rapidly. Early frameworks like DeepSpeed-Chat [14] used synchronous execution, leading to poor GPU utilization. More recent systems decouple training and inference:

• OpenRLHF [15]: Uses Ray to distribute actor, critic, reward, and reference models across GPUs, integrating vLLM for accelerated generation.
• veRL [9]: Introduces a flexible programming model allowing seamless integration of training backends (FSDP, Megatron-LM) with inference backends (vLLM, SGLang).
• NeMo-RL [11]: Builds on NVIDIA's NeMo ecosystem, providing tight Megatron-LM integration and native FP8 training support.
• ROLL [16]: Focuses on hardware-aware workload mapping for large clusters.
• slime [17]: A lightweight framework integrated with SGLang, optimizing for MoE routing consistency.
• AReaL [18]: An asynchronous architecture that decouples generation and training to avoid idle waiting.

While these frameworks provide the infrastructure for modern LLM RL, none offered native support for FP8 rollouts prior to this work. They either defaulted to BF16/FP16 precision for generation or required manual, error-prone integration of FP8 inference engines with training backends. The paper explicitly builds on veRL and NeMo-RL, adding FP8 rollout as a first-class feature with configuration-level enablement, making FP8 acceleration accessible through a simple configuration flag rather than requiring bespoke engineering per-project.

Train-Inference Mismatch in LLM RL (Gap: No Systematic Solution for FP8-Induced Mismatch)

The broader problem of train-inference mismatch—where the rollout policy differs from the training policy due to numerical or implementation differences—has been recognized as a critical challenge in LLM RL. Several approaches have been proposed:

• Importance sampling (IS) [5, 7, 19]: Reweight gradient contributions using the likelihood ratio $\pi_\theta(a|s) / \pi_{\text{rollout}}(a|s)$ to correct for off-policy data. Truncated or masked variants (TIS/MIS) control variance by clipping extreme weights.
• Rollout Router Replay (R3) [20]: For MoE models, records which experts were selected during inference and replays those exact routing decisions during training, eliminating expert-selection mismatch.
• Bitwise consistency [21]: Aligns kernels and enforces determinism across parallel configurations so that training and inference produce numerically identical outputs.
• FP16 over BF16 [22]: Qi et al. argue that BF16 rounding error accumulates during autoregressive decoding, and switching to FP16 consistently across training and inference substantially reduces divergence.

These works establish that mismatch is real and damaging, but none specifically address the mismatch introduced by FP8 quantization. FP8 introduces a fundamentally different type of mismatch than the kernel-level or precision-level discrepancies studied in prior work. Quantization error from FP8 is systematic and weight-dependent: it affects every token's generation probability, and the magnitude depends on the quantization scheme, block size, and the specific weight distribution. Moreover, FP8 mismatch compounds with sequence length, since each generated token conditions on previous tokens that may have been sampled from a slightly shifted distribution. The paper contributes a systematic evaluation of how FP8-induced mismatch manifests in long-context RL training (measuring mismatch KL divergence), and validates that existing importance-sampling corrections (token-level TIS) are sufficient to mitigate it at the quantization levels studied.

How This Paper Positions Itself

The paper does not claim to invent fundamentally new quantization formats or RL algorithms. Its contribution is integration and validation: building a practical, production-ready system that makes FP8 rollout work in real RL pipelines, then rigorously demonstrating that it preserves training quality across model architectures, precision configurations, and correction strategies.

The positioning can be understood along three axes:

Engineering contribution (enabling dynamic FP8 in RL). The paper designs and implements a complete synchronization pipeline (Figure 1) that handles the full cycle: retrieve BF16 weights from training backend → blockwise quantize to FP8 → load into inference engine → generate rollouts → feed results back to trainer. This is implemented in veRL with support for multiple training backends (FSDP, Megatron-LM) and inference engines (vLLM, SGLang), and is configurable via a single command-line flag. The paper also extends FP8 quantization to the KV-cache, introducing two calibration paradigms (inference-side and trainer-side, Figure 6) to handle per-step QKV scale recalibration under dynamic weights. This is non-trivial engineering that removes the friction of adopting FP8 in RL workflows.

Algorithmic validation (showing FP8 works, with conditions). The paper conducts a comprehensive empirical study comparing FP8 configurations against BF16 baselines on learning curves (validation accuracy, reward, response length, mismatch KL) rather than just throughput numbers. The key finding—that FP8 rollout without correction degrades accuracy (Figure 2, green curve), but FP8 rollout with token-level TIS closely tracks BF16 (Figure 2, blue vs. orange curves)—establishes a clear operational prescription: always pair FP8 rollout with importance-sampling correction. This finding is replicated across dense models (Section 2.2.2), MoE models (Section 2.2.3), and KV-cache quantization (Section 2.3.2), lending it robustness.

Performance characterization (showing where gains come from). Rather than reporting a single speedup number, the paper disaggregates gains: linear W8A8 provides 10–20% on dense models and 30–50% on MoE models (Figures 3, 5), KV-cache FP8 adds 38% (Figure 8), and the combination reaches 44%. The paper explains why MoE models benefit disproportionately (higher arithmetic intensity, larger memory footprint reduction, freed KV-cache capacity reducing preemptions) and why KV-cache quantization is so impactful for long-context generation (doubling effective cache capacity eliminates preemption-driven throughput collapse). This analysis helps practitioners anticipate where FP8 will provide the most value for their specific workloads.

The paper also positions itself as a foundation for future work, explicitly noting that the current study is limited to FP8 (E4M3) and suggesting exploration of more aggressive formats like NVFP4, scaling to larger models, and extending to multi-turn and agentic RL scenarios. It positions end-to-end FP8 (training + rollout) as the next logical step, demonstrating initial results showing reduced mismatch compared to rollout-only FP8 (Section 2.4).

In summary, this paper addresses a clear practical gap—FP8 inference exists, RL systems exist, but nobody had made them work together reliably—by providing both the engineering infrastructure and the empirical evidence that FP8 rollout is safe and effective when combined with standard importance-sampling correction.

3. Technical Approach

3.1 Reader Orientation (Approachable Technical Breakdown)

The paper builds a practical engineering system—not a new algorithm or quantization formula—that retrofits existing FP8 inference technology to work reliably inside the dynamic, step-by-step weight-updating loop of LLM reinforcement learning. The core problem this system solves is that off-the-shelf FP8 inference assumes a static model (quantize once, serve forever), but RL changes the model weights every training step, which breaks both the engineering pipeline (how do you re-quantize and reload weights at step frequency?) and the algorithmic assumptions (the training objective expects on-policy data, but quantized rollouts are effectively off-policy). The "shape" of the solution is a three-phase synchronization workflow plus a token-level statistical correction that reweights biased samples back toward the true policy distribution.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components, arranged in a cyclical pipeline with a correction module that wraps the training update:

1. Training Backend (FSDP or Megatron-LM): Maintains the master BF16/FP16 weights of the policy model. After each training step, it exposes the updated weights to the synchronization module. This is the source of truth for what the policy should be.

2. FP8 Quantization and Synchronization Module: Retrieves BF16 weights from the training backend, applies blockwise FP8 quantization (producing quantized weights and per-block scaling factors), and loads them into the inference engine. For KV-cache quantization, it also recalibrates QKV scales (either on the inference side or the trainer side). This module runs at the start of every rollout phase.

3. Inference Engine (vLLM or SGLang): Receives FP8-quantized weights and scaling factors, then generates rollouts (autoregressive text completions) from the current policy for a batch of prompts. It supports W8A8 linear layers (with dynamic activation quantization) and optionally FP8 KV-cache storage. This is the "rollout policy" $\pi_\theta^{\text{FP8}}$ that actually produces the data.

4. Rollout Correction Module (Token-Level TIS): During the subsequent training step, when computing policy gradient estimates, it computes the importance weight $w(a|s) = \pi_\theta(a|s)/\pi_\theta^{\text{FP8}}(a|s)$ for each token and clips it to $[-C, C]$ with $C=2$. This reweights the contribution of each token to the loss, correcting for the fact that the data came from the quantized policy rather than the true policy.

Information flows cyclically: Training Backend → (weight retrieval) → Quantization Module → (quantized weights + scales) → Inference Engine → (generated rollouts) → (reward computation, advantage estimation) → Rollout Correction Module → (corrected gradients) → Training Backend. The cycle repeats for thousands of RL steps, with weights re-quantized and re-synchronized at every step.

3.3 Roadmap for the Deep Dive

• First, the quantization scheme itself (Section 3.4.1): The E4M3 format, blockwise granularity ($128 \times 128$ blocks), which layers get quantized and which don't, and the distinction between static weight quantization and dynamic activation quantization. This is the foundation—everything else depends on understanding what FP8 quantization actually does to the model.

• Second, the dynamic weight synchronization pipeline (Section 3.4.2): The three-phase workflow (initialization, weight synchronization, inference), how quantized weights flow from trainer to inference engine at step frequency, and why this is non-trivial in decoupled training-inference stacks. This is the core engineering contribution.

• Third, the train-inference mismatch and importance-sampling correction (Section 3.4.3): Why FP8 rollouts create off-policy data, the importance weight formula, truncated importance sampling (TIS) with the clipping threshold $C=2$, and why token-level correction is used rather than sequence-level. This is the algorithmic lynchpin that makes FP8 rollout work.

• Fourth, KV-cache quantization (Section 3.4.4): Why long-context RL rollouts are memory-bound on KV-cache, the challenge of per-step QKV scale recalibration under dynamic weights, and the two calibration paradigms (inference-side via forced recalibration, trainer-side via calibration on training data). This extends FP8 beyond linear layers to the memory bottleneck.

• Fifth, end-to-end FP8 RL (Section 3.4.5): Extending FP8 to the training side as well, the rationale (reduced mismatch, training acceleration), and the comparison between rollout-only FP8 and end-to-end FP8. This pushes toward the logical endpoint of FP8 throughout the RL loop.

• Sixth, hyperparameter and configuration summary (Section 3.4.6): All concrete numbers, settings, and software versions used in the experiments, collected in one place for reproducibility.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems engineering and empirical validation paper whose core idea is that FP8 inference can be made to work in the dynamic RL setting by (a) building a robust per-step weight quantization and synchronization pipeline, and (b) applying token-level truncated importance sampling to correct for the off-policy bias introduced by quantization error, with the correction being necessary (not optional) for maintaining training quality.

3.4.1 Blockwise FP8 Quantization for Linear Layers

Quantization Format and Granularity

The paper adopts the E4M3 FP8 format for all quantized computations. In this format, each 8-bit floating-point number allocates 4 bits to the exponent and 3 bits to the mantissa (the fractional part). This provides a dynamic range of approximately $[-448, 448]$—meaning the largest representable magnitude before overflow is about 448, and the smallest normalized positive value is about $2^{-6} \approx 0.0156$. This range is sufficient for neural network weights and activations in practice because deep networks rarely produce values outside this range, and the 3-bit mantissa provides enough precision (about 1 part in 8, or roughly 12.5% relative error at worst) for the forward pass of most architectures.

The quantization is blockwise with block size $B = 128 \times 128$. This means that rather than computing a single scaling factor for an entire weight matrix (per-tensor quantization) or a single scaling factor per row/column (per-channel quantization), the weight matrix is partitioned into $128 \times 128$ tiles, and each tile gets its own independent scaling factor. The formal description is:

$\widehat{W}_{ij} = \text{round}\left(\frac{W_{ij}}{\text{scale}_{ij}}\right) \in \text{FP8}$

where $W \in \mathbb{R}^{m \times n}$ is the original BF16 weight matrix, $\text{scale}_{ij}$ is a scalar derived from the maximum absolute value within the $128 \times 128$ block containing position $(i, j)$, and $\widehat{W}_{ij}$ is the quantized FP8 value.

What it computes: For each weight element $W_{ij}$ in the original BF16 matrix, the formula divides by its block's scaling factor and rounds to the nearest representable FP8 value. The scaling factor is computed so that the largest absolute value in the block maps to the maximum representable FP8 magnitude (approximately 448). This ensures that the full FP8 dynamic range is utilized for each block, minimizing quantization error in regions with small weights while preventing overflow in regions with large weights.

Why this form: Per-tensor quantization—using one scaling factor for an entire weight matrix—would force the scaling factor to accommodate the single largest-magnitude weight anywhere in the matrix. If most weights are small but a few are large, the scaling factor becomes large, and the small weights get quantized to zero or near-zero values, causing significant information loss. Blockwise quantization with $128 \times 128$ tiles is a compromise: it provides fine enough granularity that each block's weight distribution is relatively homogeneous (avoiding the outlier-dominated-scaling problem) while remaining coarse enough to be hardware-efficient on modern tensor cores, which can process $128 \times 128$ tile operations efficiently. This block size follows the approach validated at industrial scale in DeepSeek-V3 [8], which used $128 \times 128$ blocks for weights and $1 \times 128$ tiles for activations.

Static vs. Dynamic Quantization

The paper distinguishes between two quantization modes based on when the quantization happens and what is quantized:

• Static quantization (weights): Weights are quantized once during the synchronization phase, before the rollout batch begins. Since the weights are fixed throughout a given rollout step (the policy doesn't change mid-generation), the quantized weights can be loaded into the inference engine and reused for all tokens across all sequences in the batch. This avoids repeated quantization overhead during generation.

• Dynamic quantization (activations): Activations—the intermediate values produced during each forward pass—vary for every input token and every layer. They cannot be pre-quantized because they depend on the specific prompt and the autoregressive generation history. Instead, they are quantized on-the-fly during each forward pass. The inference engine computes scaling factors dynamically based on the observed activation values, quantizes them to FP8, performs the matrix multiplication in FP8, and then dequantizes the output back to BF16/FP16 for subsequent layers.

This distinction is critical for understanding where the compute savings come from: the expensive matrix multiplications (QKV projections, attention output, MLP layers) use FP8 weights (pre-quantized) and FP8 activations (dynamically quantized), executing on FP8 tensor cores at roughly double the throughput of BF16. The surrounding operations (nonlinearities, layer norm, residual connections, attention softmax) remain in higher precision to avoid accuracy degradation from quantizing numerically sensitive operations.

Quantization Scope (Which Layers Get Quantized)

The paper applies W8A8 quantization to a specific subset of linear layers, chosen to maximize compute savings while avoiding sensitivity to quantization error:

Quantized layers (FP8 W8A8):

• Attention projections: q_proj, k_proj, v_proj (the linear transformations that produce query, key, and value tensors from the input hidden states) and o_proj (the output projection that maps concatenated attention head outputs back to the hidden dimension).
• MLP layers: gate_proj, up_proj, down_proj (the three linear transformations in a gated MLP block, where gate_proj and up_proj produce intermediate activations that are element-wise multiplied, and down_proj projects back to the hidden dimension).
• MoE expert layers: fc1, fc2 (the two linear layers within each mixture-of-experts feed-forward network; these are the most compute-intensive components in MoE models, making their quantization particularly impactful).

Excluded layers (kept in BF16/FP16):

• Embedding layers: The input token embedding lookup table. Quantizing embeddings can distort the initial token representations, and the lookup operation is memory-bandwidth-bound rather than compute-bound, so FP8 would provide limited speedup.
• Normalization layers: LayerNorm, RMSNorm, and similar operations. These are numerically sensitive because they compute mean and variance statistics that affect all subsequent computations; quantizing them can introduce instability.
• Output projection (lm_head): The final linear layer that projects hidden states to vocabulary logits. The paper explicitly notes that "quantizing lm_head may introduce noticeable degradation in generation quality due to its direct impact on vocabulary logits." This is because even small quantization errors in the output projection directly shift the token probabilities that govern sampling, and the vocabulary size (typically 50K–250K tokens) means that small relative errors can change which tokens get sampled, propagating errors through the autoregressive generation.

This careful layer selection reflects a pragmatic engineering principle: quantize the bulk computation where the speedup is substantial (the large matrix multiplications that dominate FLOPs), but leave numerically sensitive bottleneck layers in full precision to preserve output quality.

3.4.2 Dynamic Weight Synchronization Pipeline

The Three-Phase Workflow

The paper's central engineering contribution is a synchronization pipeline that operates at the granularity of individual RL steps. Figure 1 illustrates the workflow, which proceeds through three distinct phases:

Phase 1: Initialization. Before the RL loop begins, the system configures the inference engine to use FP8 operations and applies "patches" (code modifications) to enable dynamic loading of quantized weights. In a standard FP8 inference deployment with vLLM or SGLang, the engine expects to load a pre-quantized model once at startup and serve requests indefinitely. The RL setting requires the engine to accept new quantized weights at arbitrary times without restarting. The initialization phase sets up this capability—it modifies the inference engine's weight-loading pathway to support repeated updates, and configures the FP8 recipe (which layers are quantized, the quantization scheme, block size, etc.).

Phase 2: Weight synchronization (every RL step). At the start of each RL step, after the training backend has updated the policy weights, the synchronization module performs these operations in order:

1. Retrieve BF16/FP16 weights from the training backend. If using FSDP (Fully Sharded Data Parallelism), this involves gathering the sharded weight tensors from across GPUs into a complete BF16 copy. If using Megatron-LM, weights are already in a format that can be directly accessed.
2. Apply blockwise FP8 quantization to the retrieved weights using the scheme described in Section 3.4.1. For each $128 \times 128$ block of each quantized linear layer, compute the scaling factor from the block's maximum absolute value, divide all weights in the block by that scale, and round to the nearest E4M3 FP8 value. The output is a set of FP8 weights (the quantized values) and FP8 scales (one per block) for each quantized layer.
3. Load quantized weights and scales into the inference engine. This step transfers the FP8 weight tensors and their associated scaling factors from the training process (or training GPU memory) to the inference engine's GPU memory. The exact mechanism depends on the backend integration: in veRL with vLLM, this uses shared memory or NCCL-based transfers depending on whether training and inference share GPUs or run on separate devices.

Phase 3: Inference (rollout). The inference engine now executes generation using the freshly loaded FP8 weights. For each forward pass during autoregressive decoding:

• Weights are already in FP8 format in GPU memory (loaded during Phase 2).
• Activations arrive in BF16/FP16 from the previous layer.
• The engine dynamically quantizes activations to FP8 (computing per-token or per-tile scaling factors on the fly).
• Matrix multiplications execute on FP8 tensor cores using the quantized weights and dynamically quantized activations.
• Results are accumulated in FP32 (to preserve precision during summation) and then converted back to BF16/FP16 for the next layer.

This three-phase cycle repeats for every RL step: train → synchronize → rollout → compute rewards → train → synchronize → rollout → ... For a typical RL run with thousands of steps, the synchronization phase executes thousands of times, making its efficiency critical.

Why Dynamic Synchronization Is Non-Trivial

In a standard static inference deployment, FP8 quantization is a one-time offline cost: the model is quantized ahead of time (possibly on a different machine), the quantized weights are saved to disk, and the inference engine loads them once at startup. The quantization itself can be slow (seconds to minutes) because it only happens once.

In RL, this approach breaks for two reasons:

1. The quantization must be fast enough to not dominate step time. If quantization takes 10 seconds per step but generation takes 2 seconds, the "speedup" from FP8 is meaningless because the quantization overhead outweighs any inference gains. The paper's blockwise scheme with $128 \times 128$ tiles is designed to be computationally efficient (each block's scale is just the max absolute value, which can be computed with a reduction operation), and the actual quantization is a simple division and rounding. For a 30B-parameter model, quantizing the relevant linear layers involves processing tens of billions of weights, but the operation is embarrassingly parallel across blocks and layers.

2. The decoupled training-inference architecture adds coordination complexity. In modern RL stacks, training and inference often run on different GPU sets (or at least different processes on the same GPUs) and communicate through explicit synchronization points. The training backend (FSDP or Megatron-LM) owns the "canonical" weights. The inference engine (vLLM or SGLang) is a separate process with its own GPU memory space. The synchronization module must bridge these two worlds: it must pull weights from the training process's memory (which may be sharded across GPUs in FSDP), quantize them, and push them into the inference engine's memory—all before the inference engine can start generating. The paper's implementation handles multiple backend combinations (FSDP→vLLM, FSDP→SGLang, Megatron-LM→vLLM, Megatron-LM→SGLang) through the veRL framework's abstraction layer, which standardizes the weight transfer interface.

Performance Considerations

The paper provides specific recommendations for maximizing the performance of the FP8 pipeline:

• CUDA version 12.9+ required: Newer CUDA versions include optimized FP8 DeepGEMM kernels that provide "significant speedups" over earlier implementations.
• DeepGEMM library: This open-source library (from DeepSeek-AI, hosted at github.com/deepseek-ai/DeepGEMM) provides highly optimized FP8 general matrix-multiply (GEMM) kernels. For vLLM 0.11+ and SGLang 0.55+, DeepGEMM is enabled by default. For older vLLM versions (0.10.x), the user must manually enable it by setting the environment variable VLLM_USE_DEEP_GEMM=True.
• Weight transfer cost: The paper does not report explicit numbers for synchronization overhead, but the fact that overall speedups of 10–50% are achieved implies that the quantization and transfer cost is small relative to the generation time saved. For the 30B MoE model with 20K-token responses, generation is so expensive (seconds to minutes per batch) that even a 1–2 second quantization step is negligible amortized over the rollout.

3.4.3 Train-Inference Mismatch and Importance-Sampling Correction

Why FP8 Rollout Creates Off-Policy Data

To understand why importance sampling is necessary, we need to examine what RL algorithms like PPO and DAPO actually compute during training. The policy gradient update (simplified) uses an estimator of the form:

$\mathbb{E}_{a \sim \pi_\theta(\cdot|s)} \left[ \frac{\pi_\theta(a|s)}{\pi_{\theta_{\text{old}}}(a|s)} \cdot A(s, a) \right]$

where $\pi_\theta$ is the current policy being optimized, $\pi_{\theta_{\text{old}}}$ is the behavior policy that generated the data (the "old" policy from the previous step), and $A(s, a)$ is the advantage estimate (how much better action $a$ is compared to the average action from state $s$). The expectation is taken over actions sampled from $\pi_{\theta_{\text{old}}}$, and the importance ratio $\pi_\theta / \pi_{\theta_{\text{old}}}$ corrects for the fact that we're using data from an older policy to update the current policy.

The critical assumption is that the data was actually generated by $\pi_{\theta_{\text{old}}}$—that is, the token probabilities recorded during rollout accurately reflect the behavior policy's distribution. When the rollout uses FP8 quantization, the actual behavior policy is $\pi_{\theta_{\text{old}}}^{\text{FP8}}$ (the quantized version), but the training algorithm computes importance ratios using $\pi_{\theta_{\text{old}}}$ (the full-precision version). This means the importance ratio is wrong: it corrects for the wrong baseline distribution. Effectively, the data is off-policy by an additional factor—the gap between $\pi_{\theta_{\text{old}}}^{\text{FP8}}$ and $\pi_{\theta_{\text{old}}}$—that is not accounted for in the standard PPO update.

This mismatch manifests in two ways in the training dynamics:

1. Biased gradient estimates: The advantage-weighted probability ratios no longer provide unbiased estimates of the policy gradient, potentially pushing the policy toward suboptimal regions.
2. The mismatch KL divergence $D_{\text{KL}}(\pi_{\theta}^{\text{FP8}} \parallel \pi_{\theta})$ grows over training: As the policy evolves, the quantization error pattern may change (because the weight distribution changes), and the accumulated divergence between what the trainer thinks happened and what actually happened during rollout can compound. The paper tracks this mismatch KL metric explicitly in all training plots (Figures 2, 4, 7, 9), showing that it is consistently higher for FP8 configurations than BF16 baselines.

The Importance Weight Formula

The correction mechanism is token-level importance sampling. For each action $a$ (where "action" means a generated token) taken from a state $s$ (the preceding context), the importance weight is:

$w(a|s) = \frac{\pi_\theta(a|s)}{\pi_\theta^{\text{FP8}}(a|s)}$

where $\pi_\theta(a|s)$ is the full-precision training policy's probability of generating token $a$ given context $s$, and $\pi_\theta^{\text{FP8}}(a|s)$ is the FP8 rollout policy's probability of generating that same token.

What it computes: For each token in each generated sequence, this ratio measures how much more (or less) likely the full-precision policy was to generate that token compared to the quantized policy that actually generated it. If the ratio is 1.0, the two policies agree perfectly on that token's probability. If the ratio is greater than 1, the full-precision policy considers the token more likely than the quantized policy did—the quantized policy was "lucky" to sample it, and the token should be upweighted in the training update. If the ratio is less than 1, the token was over-sampled by the quantized policy relative to the true policy, and it should be downweighted.

Why this form: This is the standard importance sampling correction for off-policy evaluation. Under the true policy $\pi_\theta$, the expected value of any function $f(a)$ is $\mathbb{E}_{a \sim \pi_\theta}[f(a)]$. If we only have samples from a different distribution $\pi_\theta^{\text{FP8}}$, we can still estimate the expectation by reweighting: $\mathbb{E}_{a \sim \pi_\theta^{\text{FP8}}}[w(a|s) \cdot f(a)] = \mathbb{E}_{a \sim \pi_\theta}[f(a)]$. This is an unbiased estimator (assuming the support of $\pi_\theta^{\text{FP8}}$ covers the support of $\pi_\theta$), meaning that on average, the reweighted gradient estimates point in the correct direction. The paper embeds this weight into the PPO-style objective, multiplying the per-token advantage by the importance weight before computing the policy loss.

Truncated Importance Sampling (TIS)

Naive importance weights can be problematic because they can become extremely large when the two distributions diverge significantly. If $\pi_\theta(a|s) = 0.01$ but $\pi_\theta^{\text{FP8}}(a|s) = 0.0001$, the weight is $100\times$. This means that a single token—potentially an outlier generated because of a quantization artifact—can dominate the entire batch's gradient update, causing high-variance, unstable training.

The paper adopts token-level truncated importance sampling (TIS), which clips the weights to a bounded range:

$w_{\text{TIS}}(a|s) = \text{clip}(w(a|s), C)$

where the clipping operation $\text{clip}(x, C)$ limits $x$ to the range $[\frac{1}{C}, C]$, and $C = 2$ is used in all experiments.

What it computes: For each token, compute the raw importance weight $w(a|s)$. If the weight is between $1/2$ and $2$, keep it unchanged. If it's below $1/2$, set it to $1/2$. If it's above $2$, set it to $2$. This bounds the influence of any single token on the gradient update, preventing individual outlier tokens from destabilizing training.

Why this form: Truncation is a standard variance-reduction technique in importance sampling. It introduces bias (the estimator is no longer exactly unbiased) but dramatically reduces variance, and in practice the bias-variance tradeoff strongly favors truncation when the two distributions differ non-trivially. The choice of $C=2$ is a conservative threshold: it allows the weights to vary by a factor of up to $4\times$ (from $0.5\times$ to $2\times$) before clipping, which provides meaningful correction for moderate distribution shift while capping the influence of extreme outliers. The paper does not report ablations over $C$, so the sensitivity of results to this threshold is unknown from the provided content.

Why Token-Level Rather Than Sequence-Level Correction

The paper applies importance sampling at the token level, not the sequence level. A sequence-level correction would compute a single weight for the entire generated response, multiplying the per-token probabilities across the sequence:

$w_{\text{seq}}(a_{1:T}|s_1) = \prod_{t=1}^T \frac{\pi_\theta(a_t|s_t)}{\pi_\theta^{\text{FP8}}(a_t|s_t)}$

The problem with sequence-level weights is that they multiply per-token ratios, causing the variance to explode exponentially with sequence length. For a 20K-token response (the paper's maximum), a sequence-level weight could easily be astronomically large or vanishingly small, making the correction useless in practice. Token-level correction applies the weight independently at each decoding step, keeping the variance manageable and enabling meaningful correction even for very long generations.

The Critical Empirical Finding: TIS Is Necessary

The paper's ablation study (Figure 2, green vs. blue curves for the 8B dense model) provides the key empirical evidence. FP8 W8A8 rollout without TIS (green) shows "noticeable accuracy degradation compared to both the BF16 baseline and the FP8+TIS configuration." With TIS enabled (blue), the FP8 rollout "closely tracks the BF16 baseline across all key training metrics." This establishes a clear operational prescription: FP8 rollout without correction is unsafe; always pair with importance-sampling correction.

The paper notes that the mismatch KL is "naturally higher for the FP8 run due to quantization-induced precision loss," but "with TIS enabled, this divergence remains stable and within an acceptable range." This suggests that TIS does not eliminate the mismatch—it still exists—but it prevents the mismatch from corrupting the gradient signal to the point of degrading final model quality.

MoE-Specific Mismatch Dynamics

An important additional observation comes from the MoE experiments (Section 2.2.3, Figure 4). Unlike the dense model where mismatch KL remains relatively stable over training, "MoE models inherently exhibit a trend of increasing mismatch KL during training in both the BF16 and FP8 runs." The paper attributes this to the dynamic nature of expert routing: "slight differences in precision or implementation between the inference engine and training backend can lead to inconsistent expert selection across the two systems. As training progresses and the policy evolves, these routing inconsistencies accumulate."

This means that for MoE models, even the BF16 baseline experiences growing mismatch (due to non-FP8 sources like kernel implementation differences), and FP8 exacerbates it. The paper mentions that while token-level TIS was sufficient in their experiments, "scenarios with more severe instability may require advanced correction techniques such as Masked Importance Sampling (MIS) or Rollout Router Replay (RRR)", the latter of which records which experts were selected during inference and forces the training forward pass to use the same expert assignments.

3.4.4 KV-Cache Quantization

Why KV-Cache Quantization Matters for Long-Context RL

The KV-cache is the memory structure that stores the key and value tensors for every previously generated token during autoregressive decoding. Without a cache, each new token would require recomputing attention over the entire prefix, turning an $O(n)$ per-token cost into $O(n^2)$. The cache trades memory for computation: by storing $2 \times \text{num\_layers} \times \text{num\_heads} \times \text{head\_dim}$ floating-point numbers per token, the model can compute attention for each new token with just a single query-key-value operation against the cached history.

In long-context generation (the paper tests up to 20K tokens), the KV-cache becomes the dominant memory consumer. For an 8B-parameter model with, say, 32 layers, 32 attention heads, and 128-dimensional head embeddings, the KV-cache stores $2 \times 32 \times 32 \times 128 = 262,144$ elements per token. At BF16 (2 bytes per element), this is 512 KB per token. For a batch of 1536 sequences (the paper's rollout batch size of $32 \times 3 \times 16$) at 20K tokens each, the total KV-cache requirement is $1536 \times 20,000 \times 0.5 \text{ MB} \approx 15 \text{ GB}$. On an 80GB H100 GPU, this competes directly with model weights (about 16 GB in BF16 for an 8B model) and activation memory, creating severe memory pressure.

The paper reports that under BF16 and linear W8A8 configurations (which don't quantize the KV-cache), "vLLM monitoring revealed frequent request preemptions caused by insufficient KV-cache space, which wasted computation and throttled throughput." Preemption means the inference engine must pause some in-progress generations, evict their KV-caches, generate other requests, and later resume the evicted ones from scratch (recomputing the entire prefix). This is catastrophically inefficient for long sequences.

By quantizing the KV-cache to FP8, the per-token memory footprint is halved (from 2 bytes to 1 byte per element), doubling the effective cache capacity. The paper reports that this "significantly mitigates memory pressure, resulting in a stable, high-throughput generation stream that maximizes GPU utilization."

The Dynamic QKV Scale Recalibration Challenge

KV-cache quantization requires computing scaling factors for the key and value tensors before they are stored in the cache. In static inference, these scales can be calibrated once (typically on a representative calibration dataset) and reused indefinitely. But in RL, the policy weights—including the query, key, and value projection matrices (q_proj, k_proj, v_proj)—change every step. This means the distribution of keys and values produced by the model changes every step, and the scaling factors calibrated for the previous step's weights may be inappropriate for the new weights, leading to suboptimal quantization (values falling outside the representable range or being quantized too coarsely).

The paper presents two design paradigms for handling this recalibration, illustrated in Figure 6:

Paradigm 1: Inference-Side Calibration (implemented in veRL). This approach leverages a built-in capability of modern inference engines like vLLM, which can dynamically compute QKV scaling factors during the first forward pass after model initialization. The paper adapts this for RL by forcing a recalibration at the start of every rollout phase. The mechanism is straightforward: before generation begins for a new RL step, the system resets the internal calculate_kv_scales flags in all attention layers of the inference engine. On the first forward pass of the new rollout batch, the engine observes the actual key and value tensors produced by the current weights, computes appropriate per-layer or per-head scaling factors, and uses these for the remainder of the rollout.

The advantage of this approach is simplicity: it requires no external coordination, uses the inference engine's native calibration logic, and automatically adapts to whatever weights are loaded. The calibration data is the actual rollout prompts, so the scales are computed on exactly the distribution that will be generated. The disadvantage is that it adds a small warm-up overhead to the first forward pass of each rollout (the calibration pass), but given that rollouts involve thousands of tokens, this is negligible.

Paradigm 2: Trainer-Side Calibration (implemented in NeMo-RL). In this approach, QKV scale recalibration happens on the training side, after the policy update. At the end of each training step, the training backend:

1. Takes the newly updated policy weights.
2. Runs a calibration forward pass using a subset of training data (prompts and generated responses from the previous or current step).
3. Records the key and value tensor statistics from this calibration pass.
4. Computes optimal FP8 scaling factors for each attention layer.
5. Synchronizes these scaling factors to the inference engine along with the quantized weights.

The advantage of this approach is "fine-grained control over the calibration data and process." The trainer can use a curated calibration dataset rather than relying on the actual rollout prompts, potentially providing more representative statistics. The disadvantage is tighter coupling between training and inference backends and a "minor calibration overhead" that the paper quantifies as "approximately 2–3% of total step time." This is a small but non-zero cost.

The paper validates both approaches and reports that "observations from the Trainer-Side calibration approach are largely consistent with the Inference-Side calibration results reported in the main text, indicating that both calibration paradigms achieve comparable effectiveness in maintaining training stability under FP8 KV cache quantization" (Appendix B.3.2). This gives practitioners flexibility: use the simpler inference-side calibration in veRL if you want minimal configuration; use trainer-side calibration in NeMo-RL if you need tighter control over calibration data.

KV-Cache Quantization Scope

The paper's full FP8 configuration quantizes three components: linear layers (W8A8), KV-cache storage, and attention computations. The attention computation quantization refers to performing the actual attention score computation (the softmax of query-key dot products, followed by the weighted sum of values) in FP8 rather than BF16. This is the most aggressive setting because attention involves numerically sensitive operations (exponentiation via softmax, normalization) that can amplify quantization errors.

The paper's ablation study (Figure 7) compares four configurations:

• BF16 baseline
• Linear W8A8 only (KV-cache and attention in BF16)
• KV-cache FP8 only (linear layers and attention in BF16, only the cache storage is FP8)
• Full FP8 (all three quantized)

The results show that even the most aggressive Full FP8 setting, when paired with TIS, "still follows the general accuracy trend, confirming that even aggressive full-stack quantization can be viable for RL training when paired with robust mismatch mitigation."

Performance Analysis: Why KV-Cache Quantization Provides Disproportionate Gains

Figure 8 shows that for the 8B dense model at 20K context, KV-cache FP8 alone provides a 38% speedup—nearly double the 20% speedup from linear W8A8 alone. The combined speedup is 44%. The paper explains this disproportionate gain (Section 2.3.2, Performance Analysis) through the preemption mechanism:

Under BF16 or linear W8A8 (which don't reduce KV-cache size), the memory pressure from 20K-token sequences in a batch of 1536 causes the inference engine to frequently preempt requests. Each preemption wastes all the computation already spent on the preempted tokens (since they must be recomputed from scratch when the request resumes) and introduces scheduling gaps while the engine juggles limited cache space. By halving the per-token cache footprint, KV-cache FP8 doubles the number of tokens that can coexist in GPU memory, which dramatically reduces preemption frequency. The resulting throughput improvement comes not just from faster individual operations but from eliminating the massive waste of recomputation and idle time caused by memory pressure.

For models and context lengths where the KV-cache fits comfortably in memory without preemption, the gains from KV-cache quantization would be smaller—limited to the modest speedup from reduced memory bandwidth during cache reads. The paper's results are therefore most applicable to memory-constrained long-context scenarios, which is precisely the regime where RL rollouts for reasoning tasks operate (chain-of-thought over thousands of tokens).

3.4.5 End-to-End FP8 RL (Training + Rollout)

Motivation for Extending FP8 to Training

The preceding sections focus on FP8 rollout only—training remains in BF16. Section 2.4 explores end-to-end FP8, where both training and rollout use FP8 precision. The paper identifies three benefits:

1. FP8 training is already validated at scale: Prior work (FP8-LM, DeepSeek-V3) has demonstrated that FP8 training converges comparably to BF16 when using appropriate quantization formats and scaling rules. This reduces the risk of adopting FP8 for the training side.

2. Reduced train-inference mismatch: When rollout uses FP8 but training uses BF16, there is a precision gap between the two paths: the trainer optimizes a BF16 policy using data from an FP8 rollout. By making both training and rollout use FP8, the policy being optimized is numerically closer to the policy that generates data, potentially reducing the mismatch KL and improving gradient signal quality. The paper confirms this empirically: "FP8 training + rollout exhibits smaller training–inference mismatch, reflected by a sampling importance ratio closer to 1 and a lower mismatch KL" (Section 2.4.2).

3. Training-side acceleration: FP8 training uses FP8 tensor cores for forward and backward passes, reducing training step time. The paper reports "approximately 20% lower training time compared to BF16 training + rollout" for end-to-end FP8.

Implementation and Experimental Design

The end-to-end FP8 experiments are conducted in the NeMo-RL framework (which has native FP8 training support) rather than veRL. The experimental setup mirrors the previous experiments: Qwen3-8B-Base, DAPO algorithm, AIME24 validation, 8×H100 GPUs, prompt batch size 32 with $n=16$ responses, max response length 20K tokens, token-level TIS with $C=2$.

Three configurations are compared (Figure 9):

• BF16 training + BF16 rollout: The full-precision baseline.
• BF16 training + FP8 rollout: Rollout-only FP8 (the configuration studied in Sections 2.2 and 2.3).
• FP8 training + FP8 rollout: End-to-end FP8.

The key finding is that "FP8 training + rollout closely tracks the BF16 training + rollout baseline across response length, reward, and validation accuracy." This is the cleanest demonstration that FP8 can be applied end-to-end without sacrificing model quality—at least for this model, dataset, and training configuration.

The Remaining Mismatch

Importantly, the paper notes that "FP8 training + rollout still shows slightly higher mismatch than BF16 training + rollout, suggesting that precision alignment helps but does not fully remove all sources of mismatch." This is a nuanced finding: some residual mismatch arises from sources other than the training-rollout precision gap—potentially kernel implementation differences, numerical differences in attention softmax or layer norm between the NeMo-RL training code and the vLLM inference code, or non-determinism in parallel operations. Aligning precision eliminates one source of mismatch but doesn't solve the broader problem of train-inference numerical consistency, which remains an active research area (see the discussion of bitwise consistency work in Section 3.3 of the paper).

3.4.6 Hyperparameter and Configuration Summary

For completeness and reproducibility, here are all concrete values and settings mentioned in the paper:

Training algorithm: DAPO [10] with online validation on AIME24.

Hardware:

• 8B dense model: 8×H100 GPUs
• 30B MoE model: 2×8×H100 GPUs (16 GPUs total)

Batch sizes:

• Prompt batch size: 32
• Responses per prompt ($n$): 16
• Total rollout batch size: $32 \times 3 \times 16 = 1536$ (the factor of 3 is from the multi-pass DAPO protocol)
• Training batch size: 32
• PPO mini-batch size: 32
• The training and mini-batch sizes equal to 32 ensures "the policy is updated using rollout outputs only once per iteration, which helps isolate the impact of quantization by removing additional off-policy noise."

Generation:

• Maximum response length: 20,000 tokens
• Temperature and top-p/top-k values are specified in the configuration but not reported in the paper (the reader is referred to the DAPO Qwen3-30B FP8 Rollout Recipe).

FP8 quantization:

• Format: E4M3 (4 exponent bits, 3 mantissa bits)
• Block size: $128 \times 128$ for weights
• Static quantization: weights
• Dynamic quantization: activations

Importance sampling:

• Type: Token-level truncated importance sampling (TIS)
• Clipping threshold: $C = 2$
• Applied to all FP8 configurations (except the ablation runs that explicitly omit it)

Software dependencies:

• CUDA 12.9+
• DeepGEMM library (enabled by default in vLLM 0.11+ and SGLang 0.55+; manually enabled via VLLM_USE_DEEP_GEMM=True for vLLM 0.10.x)
• veRL framework (for inference-side KV-cache calibration)
• NeMo-RL framework (for trainer-side KV-cache calibration and end-to-end FP8 experiments)

KV-cache quantization configuration (veRL):

actor_rollout_ref:
  rollout:
    quantization:
      kv_cache_dtype: fp8_e4m3
      calculate_kv_scales: True

KV-cache quantization configuration (NeMo-RL):

policy:
  generation:
    vllm_cfg:
      precision: fp8
      kv_cache_dtype: fp8

Basic FP8 linear rollout enablement (veRL):

actor_rollout_ref.rollout.quantization=fp8

This single flag automatically triggers weight conversion, blockwise quantization, and loading into the inference backend, encapsulating the entire engineering pipeline described in this section.

4. Key Insights and Innovations

Innovation 1: FP8 in RL Is an Integration Problem, Not a Quantization Problem

The paper's most distinctive intellectual move is reframing FP8 adoption in RL from a quantization research question to an integration engineering question. The dominant assumption in the low-precision literature—from FP8-LM [13] to DeepSeek-V3 [8]—is that the central challenge is designing quantization formats and schemes that preserve model accuracy: find the right block size, the right scaling strategy, the right mixed-precision layer selection. This paper argues that for RL specifically, the hard problems are elsewhere.

The quantization scheme itself—E4M3 format, $128 \times 128$ blockwise, static weights and dynamic activations, specific layer exclusions—is not novel. It is adopted directly from prior work (particularly DeepSeek-V3 [8]) with minimal modification. What is novel is the recognition that even a perfectly good quantization scheme will fail in RL unless two surrounding problems are solved: (1) the engineering problem of re-quantizing and re-synchronizing weights at step frequency across decoupled training and inference systems, and (2) the algorithmic problem of correcting the off-policy bias introduced when a full-precision trainer optimizes against data from a quantized rollout policy.

This is a fundamentally different kind of contribution than a new quantization technique. It is closer to a systems integration insight: the barrier to FP8 adoption in RL is not that we lack good FP8 formats, but that nobody had built the plumbing to make those formats work inside a dynamic weight-updating loop. The paper's evidence for this framing is the fact that the quantization scheme is off-the-shelf, yet the resulting system delivers up to 44% throughput gains while matching BF16 learning curves—something that was not available to practitioners before despite FP8 inference being mature technology. If the bottleneck were quantization quality, off-the-shelf FP8 would have worked without the synchronization pipeline and correction module; the paper's ablation showing that FP8 without TIS degrades accuracy (Figure 2, green curve) proves that the integration—not the quantization—is the critical failure mode.

This reframing has practical significance beyond this paper: it suggests that future work on low-precision RL should prioritize system-level concerns (synchronization latency, calibration frequency, mismatch monitoring) rather than pursuing marginally better quantization formats, because the integration challenges dominate the cost-benefit calculus.

Innovation 2: Train-Inference Mismatch as a Quantization-Induced Phenomenon with a Demonstrated Fix

Before this paper, the literature on train-inference mismatch in LLM RL had identified several causes: kernel implementation differences, parallelism-induced non-determinism, floating-point rounding in BF16 vs. FP16 [5, 6, 7, 21, 22]. Each of these is a software discrepancy—the training code and inference code compute slightly different outputs from the same weights. The correction strategies proposed for these discrepancies (bitwise consistency [21], FP16 alignment [22]) aim to make training and inference numerically identical.

This paper introduces a fundamentally different category of mismatch: quantization-induced mismatch, where the training and inference policies differ because the inference policy uses quantized weights, and no amount of kernel alignment can fix it because the models are literally computing different functions. This is a more severe form of mismatch in principle—the gap between $\pi_\theta$ and $\pi_\theta^{\text{FP8}}$ is systematic and weight-dependent—but the paper shows it is correctable using a standard tool (token-level TIS) that was developed for a different purpose (handling stale policies in off-policy RL).

What makes this an innovation rather than an obvious application is the empirical demonstration that the correction is both necessary and sufficient. The necessity is shown by the ablation in Figure 2: FP8 rollout without TIS (green) degrades validation accuracy relative to BF16, proving that quantization mismatch is not negligible and does not self-correct. The sufficiency is shown by the main result (blue vs. orange curves across Figures 2, 4, 7): with TIS enabled, learning curves are essentially indistinguishable from BF16 baselines. This closes a loop that prior work left open: prior mismatch papers either identified the problem without providing a validated fix specific to quantization, or proposed fixes (like bitwise consistency) that are fundamentally inapplicable when the model itself has been quantized.

A subtle but important finding is that TIS does not eliminate the mismatch—the mismatch KL remains elevated for all FP8 configurations—but it prevents it from corrupting the gradient signal. This is a diagnostic insight: mismatch KL divergence is a monitoring metric, not a direct measure of training harm, and elevated KL is acceptable as long as the importance-sampling correction keeps the gradient estimates well-behaved. This distinction between "mismatch exists" and "mismatch hurts" is not obvious a priori and is a valuable conceptual contribution for practitioners monitoring FP8 RL training runs.

Innovation 3: Diagnosing Why MoE Models Benefit Disproportionately from FP8 Rollout

The paper reports that FP8 W8A8 linear rollout delivers 10–20% speedup for the 8B dense model (Figure 3) but 30–50% for the 30B MoE model (Figure 5)—a 2–3× larger relative gain. This is not a marginal difference; it is a qualitative shift in the value proposition of FP8 between architectures. The paper unpacks this into three contributing mechanisms: higher arithmetic intensity in larger models (making FP8's compute acceleration more impactful), the 2× memory footprint reduction for the 30B parameter set (directly reducing weight-loading time), and—most interestingly—freed GPU memory expanding KV-cache capacity, which reduces preemption frequency in long-context generation.

The innovation here is not the speedup numbers themselves but the diagnostic decomposition of where the gains come from. Prior work on low-precision inference typically reports aggregate throughput improvements without analyzing why the improvement varies across models and workloads. This paper identifies the preemption-elimination mechanism as the dominant factor for KV-cache quantization (38% speedup from KV-cache FP8 vs. 20% from linear W8A8, Figure 8) and connects it to a measurable system behavior: "vLLM monitoring revealed frequent request preemptions caused by insufficient KV-cache space." This transforms the speedup from a black-box number into a predictable, workload-dependent effect: if your inference workload is KV-cache-memory-bound (long sequences, large batches), FP8 KV-cache quantization will provide disproportionately large gains by eliminating recomputation waste; if your workload is compute-bound (short sequences, small batches), the gains will be closer to the linear speedup from reduced-precision matrix multiplications.

This diagnostic framework is practically significant because it tells practitioners where to look when deciding whether to invest in FP8 rollout. It moves the conversation from "FP8 provides X% speedup on average" to "measure your preemption rate; if it's high, prioritize KV-cache quantization; if it's low, focus on linear layer quantization; if you have an MoE model, expect outsized gains from the combination." This is a conceptual advance in understanding the system-level dynamics of low-precision inference, not just a benchmarking result.

Innovation 4: The Negative Result That Precision Alignment Does Not Eliminate All Mismatch

Section 2.4 explores end-to-end FP8—quantizing both training and rollout—motivated by the hypothesis that aligning precision between the two phases would reduce train-inference mismatch. The results partially support this: end-to-end FP8 does exhibit a "sampling importance ratio closer to 1 and a lower mismatch KL" compared to rollout-only FP8 (Figure 9, green vs. orange curves). However, the paper explicitly notes that end-to-end FP8 "still shows slightly higher mismatch than BF16 training + rollout, suggesting that precision alignment helps but does not fully remove all sources of mismatch."

This is a negative result with constructive implications. It rules out the simple hypothesis that precision mismatch is the sole or dominant source of train-inference divergence—if it were, aligning both sides to FP8 would close the gap entirely. The residual mismatch must come from other sources: kernel implementation differences between the training and inference backends (even when both use FP8, NeMo-RL's training kernels and vLLM's inference kernels may compute slightly different results), numerical differences in non-quantized operations (layer norm, attention softmax, residual connections), or non-determinism in parallel reductions.

This finding is intellectually valuable because it constrains the space of solutions for the train-inference mismatch problem. It tells the field that precision alignment is a partial fix—worth doing, as it does reduce mismatch and improve training stability—but not a complete one. The remaining gap must be addressed by other means, such as the bitwise consistency approaches in [21] or the router-replay techniques for MoE models in [20]. This is a more nuanced position than either "precision is everything" [22] or "precision doesn't matter, just use importance sampling," and it provides a roadmap for which interventions address which components of the mismatch.

The significance of this finding is amplified by the fact that it comes from the paper's own extension beyond its primary contribution. Most systems papers would stop at demonstrating that rollout-only FP8 works and call it done. By pushing further to end-to-end FP8 and honestly reporting the residual mismatch, the paper provides a finding that shapes the research agenda beyond its own scope.

Innovation 5: FP8 as a First-Class Configuration Primitive in RL Systems

The paper's final contribution is more architectural than algorithmic: it establishes FP8 rollout as a first-class, configuration-level feature of RL training frameworks, rather than an ad-hoc, per-project optimization requiring bespoke integration. The veRL implementation enables FP8 W8A8 linear rollout via a single flag (actor_rollout_ref.rollout.quantization=fp8); KV-cache FP8 is similarly enabled through a few YAML configuration lines. The NeMo-RL implementation provides an analogous interface. This means that a practitioner who wants FP8 rollout does not need to understand blockwise quantization, dynamic weight synchronization, or QKV scale recalibration—they set a configuration flag, and the system handles the rest.

This may seem like a minor engineering convenience, but it represents a conceptual shift in how FP8 is positioned in the RL ecosystem. Prior to this work, FP8 in LLM training was something that framework developers supported (e.g., DeepSpeed's FP8 optimizer, Megatron-LM's FP8 training), but FP8 in LLM RL rollouts was not systematized—it was left to individual research teams to wire together FP8 inference engines with training backends, handle quantization on their own, and hope the mismatch didn't cause collapse. The paper's contribution is to productize this integration, making FP8 rollout as easy to adopt as any other hyperparameter.

This is significant because it lowers the barrier to entry for the entire field. When a capability moves from "possible with significant engineering effort" to "enabled by default in a widely-used framework," its adoption accelerates, and the community can shift from asking "can we use FP8?" to "how should we configure FP8 for our specific workload?" The paper's extensive ablation experiments—showing that FP8+TIS works across model architectures (dense and MoE), quantization scopes (linear only, KV-cache only, full stack), and calibration paradigms (inference-side and trainer-side)—provide the evidence base that justifies this productization. The framework developers are not just shipping a feature; they're shipping a feature whose safety has been systematically validated under the training conditions that matter (long-context RL, DAPO algorithm, AIME24 benchmark).

5. Experimental Analysis

Evaluation Methodology

• Dataset. All experiments use AIME24 (American Invitational Mathematics Examination 2024), a competition mathematics benchmark, for online validation during training. The paper does not specify the exact number of AIME24 problems, but the validation measurement is the percentage of correctly solved problems. Training prompts are not drawn from AIME24 itself—AIME24 serves purely as a held-out validation set to track policy quality across RL steps.

• Base model(s). The paper evaluates on two model architectures: Qwen3-8B-Base (an 8-billion-parameter dense transformer) and Qwen3-30B-A3B-Base (a 30-billion-parameter Mixture-of-Experts model with approximately 3 billion active parameters). The authors do not explicitly state why these specific models were chosen beyond their availability and representativeness of modern LLM architectures (dense and MoE). The 8B dense model serves as the primary testbed for most ablations (W8A8 linear, KV-cache quantization, TIS necessity, end-to-end FP8), while the 30B MoE model tests generalization to sparsely-activated architectures and larger scale. All models are base (pre-trained) models that undergo RL fine-tuning from scratch during the experiments, not instruction-tuned checkpoints.

• Metrics. The paper tracks five distinct metrics across all experiments:

  1. Validation accuracy on AIME24: The percentage of AIME24 problems solved correctly by the current policy checkpoint, used as the primary indicator of learning quality. This is not a training objective—it is an evaluation metric measured periodically during training.
  2. Reward: The average reward per response computed by the reward model during training, tracking optimization progress against the RL objective.
  3. Response length: The average token count per generated response. The paper notes this "typically increases as the policy learns more complex reasoning chains in long-context RL," serving as a proxy for reasoning complexity.
  4. Mismatch KL: The Kullback-Leibler divergence $D_{\text{KL}}(\pi_{\theta}^{\text{FP8}} \parallel \pi_{\theta})$ between the rollout policy $\pi_{\theta}^{\text{FP8}}$ (the quantized inference policy) and the training policy $\pi_{\theta}$ (the full-precision or FP8 training policy, depending on configuration). This quantifies distribution shift introduced by quantization. Lower values indicate better alignment between training and inference.
  5. Rollout performance: Measured via time-per-token (milliseconds per token) during generation, where lower values indicate faster rollout. This is the primary efficiency metric.

  For the end-to-end FP8 experiments (Section 2.4), an additional metric is reported: sampling importance ratio, measuring how close the importance weights $w(a|s) = \pi_\theta(a|s)/\pi_\theta^{\text{FP8}}(a|s)$ are to 1.0, with values closer to 1.0 indicating smaller train-inference mismatch.

• Baselines. The primary baseline across all experiments is BF16 rollout—full BF16 precision for both training and rollout generation, with the same training configuration (DAPO algorithm, same batch sizes, same maximum response length). Specific configurations vary by experiment:

  • For W8A8 linear experiments (Sections 2.2.2, 2.2.3): BF16 without token-level TIS serves as the baseline for the dense model (Figure 2, orange). For the MoE model (Figure 4), both BF16 and FP8 configurations use TIS, making BF16+TIS the baseline.
  • For KV-cache quantization experiments (Section 2.3.2): Four configurations are compared: BF16 baseline, Linear W8A8 only, KV-cache FP8 only, and Full FP8 (Linear + KV-cache + Attention)—all with TIS. The BF16 serves as the reference.
  • For end-to-end FP8 experiments (Section 2.4): Three configurations are compared: BF16 training + BF16 rollout (baseline), BF16 training + FP8 rollout, and FP8 training + FP8 rollout.
  • The paper also includes a critical ablation baseline: FP8 W8A8 without TIS (Figure 2, green curve) to demonstrate the necessity of importance-sampling correction.

• Generation budget / compute accounting. The paper uses a fixed generation budget per RL step rather than sweeping across budgets:

  • Prompt batch size: 32
  • Responses per prompt ($n$): 16
  • Total rollout batch size: $32 \times 3 \times 16 = 1536$ responses per step (the factor of 3 comes from the multi-pass DAPO protocol)
  • Maximum response length: 20,000 tokens
  • Training batch size and PPO mini-batch size are both set to 32, meaning the policy is updated exactly once per rollout batch—a deliberate choice "to help isolate the impact of quantization by removing additional off-policy noise" from multiple epochs.

  The compute budget is thus held constant across configurations; the independent variable is precision (BF16 vs. FP8 linear only vs. FP8 linear + KV-cache vs. Full FP8), and the dependent variables are training dynamics (accuracy, reward, length, mismatch KL) and generation throughput (time-per-token). No explicit FLOP counting or total compute budget analysis is performed beyond the throughput measurements.

• Cross-validation / statistical protocol. The paper reports no cross-validation, error bars, confidence intervals, or statistical significance tests for any result. All training curves (Figures 2, 4, 7, 9) show single-run trajectories over RL steps. The validation accuracy on AIME24 is measured on the full held-out set (size unspecified) without repeated sampling or variance estimation. For the compute-optimal strategy selection (there is none in this paper, unlike the reference example), no cross-validation is needed since the paper compares fixed configurations rather than selecting among them adaptively. The experimental protocol relies on visual alignment of training curves to establish equivalence, which is common in systems + RL papers where running multiple seeds is prohibitively expensive (8–16 H100 GPUs per run). The authors do not discuss seed sensitivity or run-to-run variance.

Main Quantitative Results

W8A8 Linear Rollout: Dense Model (Qwen3-8B-Base)

Training effectiveness. Figure 2 presents the training dynamics for three configurations on the 8B dense model over approximately 300 RL steps:

• BF16 baseline (orange): Serves as the reference for learning quality.
• FP8 W8A8 with token-level TIS (blue): The paper's proposed configuration.
• FP8 W8A8 without TIS (green): Ablation to isolate the effect of importance sampling.

The headline finding is that FP8 W8A8 with TIS closely tracks the BF16 baseline across all key training metrics. Specifically:

• Validation accuracy: The blue and orange curves "align well throughout training, with both configurations achieving comparable final accuracy on the AIME24 benchmark." The paper does not report the specific final accuracy values in the text, and the y-axis in Figure 2 is not numerically labeled in the provided content, so exact percentages cannot be determined. The visual alignment is the presented evidence.
• Reward: The curves "overlap significantly, confirming that policy optimization proceeds effectively under FP8 rollout." Again, specific reward values are not quoted.
• Response length: "Grows consistently in both configurations, indicating that the policy successfully learns to generate increasingly complex reasoning chains regardless of rollout precision."
• Mismatch KL: Is "naturally higher for the FP8 run due to quantization-induced precision loss. However, with TIS enabled, this divergence remains stable and within an acceptable range, ensuring training stability while maintaining accuracy comparable to the BF16 baseline." The absolute KL values are not reported.

The critical ablation—FP8 W8A8 without TIS (green)—"exhibits noticeable accuracy degradation compared to both the BF16 baseline and the FP8+TIS configuration. This performance drop demonstrates the critical role of importance sampling in mitigating the distribution mismatch introduced by quantization." This is the paper's most important empirical claim: TIS is necessary, not optional, for FP8 rollout in RL. The degradation is visually apparent in Figure 2 but not quantified as a specific accuracy gap.

Rollout performance. Figure 3 shows time-per-token measurements across varying response lengths for the 8B dense model:

• FP8 rollout consistently outperforms BF16 across all sequence lengths, with the gap widening at longer sequence lengths.
• The overall speedup is approximately 10–20% for the 8B dense model.
• The paper attributes the larger gains at longer lengths to memory bandwidth constraints becoming the dominant bottleneck, where FP8's reduced memory traffic provides greater benefit.

The exact time-per-token values are not quoted in the text; the speedup range is the only quantification provided.

W8A8 Linear Rollout: MoE Model (Qwen3-30B-A3B-Base)

Training effectiveness. Figure 4 presents training dynamics for the 30B MoE model, comparing two configurations over approximately 300 RL steps:

• BF16 with token-level TIS (orange): The baseline.
• FP8 W8A8 with token-level TIS (blue): The proposed configuration.

Unlike the dense model experiments, both configurations use TIS because "MoE models exhibit more pronounced train-inference mismatch even at full precision, making correction beneficial for optimal convergence."

The headline finding: "With token-level TIS enabled, the FP8 rollout (blue) aligns remarkably well with the BF16 baseline (orange) across all key training metrics." Specifically:

• Validation accuracy: "Curves track closely throughout training, demonstrating that FP8 quantization preserves learning quality for MoE models."
• Reward and response length: "Curves also overlap significantly, confirming that policy optimization proceeds effectively under FP8 rollout despite the additional complexity introduced by mixture-of-experts routing."

MoE-specific observations. The paper identifies a qualitative difference in the mismatch KL dynamics between dense and MoE models. Unlike the dense model where mismatch KL remains relatively stable, "MoE models inherently exhibit a trend of increasing mismatch KL during training in both the BF16 and FP8 runs." The explanation proposed is that "slight differences in precision or implementation between the inference engine and training backend can lead to inconsistent expert selection across the two systems. As training progresses and the policy evolves, these routing inconsistencies accumulate, causing the mismatch KL to gradually increase." This is a diagnostic observation with practical implications: MoE RL training may require more aggressive mismatch correction (MIS or router replay) as training progresses, even for the BF16 baseline.

Rollout performance. Figure 5 shows time-per-token for the MoE model, revealing substantially larger gains than the dense case:

• FP8 W8A8 rollout achieves an overall speedup of approximately 30–50% across varying response lengths.
• This represents a 2–3× larger improvement compared to the 10–20% speedup observed for the 8B dense model.

The paper decomposes this disproportionate gain into three mechanisms:

1. Higher arithmetic intensity: The 30B model's matrix multiplications are significantly larger, making FP8's compute acceleration more impactful in absolute terms.
2. Reduced memory traffic: Loading and transferring the 30B parameter set consumes substantial bandwidth in BF16; FP8's 2× memory footprint reduction directly translates to faster weight loading and reduced memory pressure.
3. Expanded KV-cache capacity: Quantizing the 30B weights frees significant GPU memory, which expands available KV-cache space. This "allows the system to host more concurrent requests and tokens, reducing preemption frequency and increasing overall throughput."

The paper emphasizes that these factors "compound in long-context generation scenarios where both compute intensity and memory pressure are high," explaining the outsized benefit for larger MoE models.

KV-Cache Quantization Results (Qwen3-8B-Base)

Training effectiveness. Figure 7 presents training dynamics for the 8B dense model comparing four configurations over approximately 300 RL steps:

• BF16 baseline (blue): Full BF16 precision.
• Linear W8A8 + TIS (yellow): W8A8 quantization for linear layers only, KV-cache and attention in BF16.
• KV-cache FP8 only + TIS (red): FP8 quantization applied exclusively to KV-cache storage; linear layers and attention computations remain in BF16.
• Full FP8 + TIS (green): FP8 quantization across all three components—linear layers (W8A8), KV-cache storage, and attention computations.

Key findings from the training curves:

• KV-cache FP8 only (red) "aligns closely with the BF16 baseline on validation accuracy," indicating that quantizing the KV-cache alone has minimal impact on model convergence. Reward curves and response length growth also track the baseline.
• The KV-cache FP8 only run exhibits a slightly higher mismatch KL compared to the Linear W8A8 run. The paper hypothesizes this stems from "error accumulation in long-context generation: unlike weight quantization which is static per layer, KV-cache quantization affects every token's attention computation dynamically over the entire long sequence." Despite this elevated KL, validation accuracy and reward remain stable, demonstrating that TIS successfully mitigates the additional mismatch.
• Full FP8 (green) "shows the largest mismatch KL as expected from compounded quantization errors." This is the most aggressive setting, combining quantization across three components. Nevertheless, "with token-level TIS enabled, it still follows the general accuracy trend, confirming that even aggressive full-stack quantization can be viable for RL training when paired with robust mismatch mitigation."

Rollout performance. Figure 8 presents the speedup numbers for each configuration relative to the BF16 baseline:

• Linear W8A8: approximately 20% speedup over BF16 baseline.
• KV-cache FP8 only: approximately 38% speedup over BF16 baseline—nearly double the linear-only gain.
• Full FP8 (Linear + KV + Attention): approximately 44% speedup over BF16 baseline, "demonstrating the complementary benefits of quantizing both components."

The paper's performance analysis attributes the disproportionate gain from KV-cache quantization (38% vs. 20% from linear) to the preemption-elimination mechanism: "In our experiments with the BF16 baseline and linear W8A8 configurations, vLLM monitoring revealed frequent request preemptions caused by insufficient KV-cache space, which wasted computation and throttled throughput. By enabling KV-cache FP8, we effectively double the KV-cache capacity, increasing maximum concurrency and reducing preemption frequency. This expanded headroom significantly mitigates memory pressure, resulting in a stable, high-throughput generation stream that maximizes GPU utilization."

The paper explicitly notes that this outcome is "highly dependent on model size and use case: for small dense models (like 8B) generating very long responses (maximum 20K tokens), the workload becomes heavily memory-bound and bottlenecked on KV-cache capacity rather than compute." For shorter contexts or larger GPU memory configurations where preemptions are rare, the gains from KV-cache quantization would be smaller.

End-to-End FP8 RL Results (Qwen3-8B-Base)

Training effectiveness and mismatch. Figure 9 presents training dynamics for the 8B dense model comparing three configurations over approximately 300 RL steps:

• BF16 training + BF16 rollout (blue): Full-precision baseline.
• BF16 training + FP8 rollout (orange): Rollout-only FP8 (the configuration from Sections 2.2–2.3).
• FP8 training + FP8 rollout (green): End-to-end FP8.

Key findings:

• Accuracy alignment: "FP8 training + rollout (green) closely tracks the BF16 training + rollout baseline (blue) across response length, reward, and validation accuracy. This demonstrates that end-to-end FP8 can preserve the learning dynamics and final quality of the BF16 run under this setup."

• Reduced mismatch vs. FP8 rollout-only: "Compared to FP8 rollout-only (orange), FP8 training + rollout (green) exhibits smaller training–inference mismatch, reflected by a sampling importance ratio closer to 1 and a lower mismatch KL." This confirms the hypothesis that aligning precision between training and rollout reduces distribution drift.

• Residual mismatch: Despite the improvement over rollout-only FP8, "FP8 training + rollout still shows slightly higher mismatch than BF16 training + rollout, suggesting that precision alignment helps but does not fully remove all sources of mismatch." This is the paper's most significant "negative result" diagnostic.

Training-side speedup. "Enabling FP8 training reduces the learner-side training time. In our runs, FP8 training + rollout achieves approximately 20% lower training time compared to BF16 training + rollout, indicating meaningful end-to-end efficiency gains beyond rollout-only acceleration." This speedup is additive to the rollout-side gains: end-to-end FP8 accelerates both the generation phase (via FP8 rollout) and the training phase (via FP8 forward/backward passes).

Trainer-Side KV-Cache Calibration Results (NeMo-RL, Appendix B.3)

Training effectiveness. Figure 10 presents training curves for the 8B dense model using the NeMo-RL implementation with trainer-side QKV scale calibration, comparing three configurations:

• BF16 baseline (blue)
• Linear W8A8 (orange)
• Full FP8 (Linear + KV Cache + Attention) (green)

All FP8 configurations use token-level TIS with C=2. Key finding: "After enabling token-level TIS, the validation accuracy of Full FP8 successfully aligns with both the BF16 baseline and Linear W8A8 configurations." The mismatch KL is "higher when both KV cache and attention are quantized to FP8, compared to the Linear W8A8 setting," which is "expected due to compounded quantization errors across multiple components."

The paper explicitly states that "these observations from the Trainer-Side calibration approach are largely consistent with the Inference-Side calibration results reported in the main text, indicating that both calibration paradigms achieve comparable effectiveness in maintaining training stability under FP8 KV cache quantization."

Rollout performance. Figure 11 shows the speedup breakdown for the NeMo-RL implementation:

• Full FP8 (Linear + KV Cache + Attention) yields an additional ~30% speedup on top of Linear W8A8, resulting in an overall ~48% speedup compared to the BF16 baseline.
• "Performance gains are particularly pronounced at longer response lengths, where attention computations constitute a larger fraction of the overall workload."
• The QKV scale recalibration process "consumes approximately 2–3% of the total step time, representing a minor cost relative to the substantial rollout acceleration achieved."

The 48% speedup in NeMo-RL (trainer-side calibration, Full FP8) is higher than the 44% reported in veRL (inference-side calibration, Full FP8). The paper does not explicitly discuss this discrepancy, but possible explanations include differences in the calibration approach, the inference engine configuration, or measurement conditions.

Ablation Studies and Robustness Checks

• Importance sampling necessity (TIS on vs. off for dense model): The FP8 W8A8 rollout without TIS configuration (Figure 2, green curve) demonstrates "noticeable accuracy degradation compared to both the BF16 baseline and the FP8+TIS configuration." This is the paper's central ablation: it establishes that the mismatch introduced by FP8 quantization is not self-correcting and that TIS is a necessary component of the solution, not an optional safeguard. The magnitude of degradation is not quantified numerically beyond the visual gap in Figure 2.

• Quantization scope (linear only vs. KV-cache only vs. Full FP8): Figure 7 systematically disambiguates the contributions of different quantization components. Linear W8A8 alone (yellow) provides a specific speedup/accuracy operating point; KV-cache FP8 alone (red) demonstrates that quantizing the cache is independently safe for training; Full FP8 (green) shows that combining all three components is viable but increases mismatch KL. This ablation provides a granular decomposition of where training stability margin comes from: KV-cache quantization adds slightly more mismatch than linear quantization alone, and attention quantization adds further mismatch on top of that, but TIS absorbs all three.

• KV-cache calibration paradigm (inference-side vs. trainer-side): Appendices B.1 and B.3 validate two different approaches to per-step QKV scale recalibration. The paper reports that training curves from trainer-side calibration (Figure 10) are "largely consistent" with inference-side calibration results (Figure 7), and both achieve comparable training effectiveness. This is a robustness check confirming that the choice of calibration paradigm does not fundamentally affect the viability of FP8 KV-cache quantization—practitioners can choose based on engineering convenience (inference-side for simplicity, trainer-side for calibration data control) without sacrificing training quality.

• End-to-end FP8 vs. rollout-only FP8: Section 2.4 compares BF16 training + FP8 rollout against FP8 training + FP8 rollout (Figure 9). This ablation tests whether extending FP8 to the training side provides benefits beyond rollout-only acceleration. The finding is nuanced: end-to-end FP8 reduces mismatch KL and brings the sampling importance ratio closer to 1.0, but does not fully close the gap to the BF16 baseline. This demonstrates that precision alignment is a partial, not complete, solution to train-inference mismatch. Additionally, end-to-end FP8 provides approximately 20% training-side speedup, establishing that the total efficiency gain from end-to-end FP8 exceeds rollout-only gains.

• MoE architecture as an implicit ablation: By testing on both dense (Qwen3-8B-Base) and MoE (Qwen3-30B-A3B-Base) models, the paper implicitly ablates across model architecture. The key differential finding is that MoE models exhibit growing mismatch KL during training even in BF16 (Figure 4) and benefit disproportionately from FP8 rollout speedup (30–50% vs. 10–20%, Figures 3 and 5). This establishes boundary conditions: the mismatch dynamics and performance gains of FP8 rollout are architecture-dependent, not universal.

• Negative result: ReST$^{EM}$ revision model instability (not applicable to this paper). Unlike the reference example which included a negative result from attempted optimization (ReST$^{EM}$ actually hurt revision performance), this paper does not report any attempted optimizations that failed. The closest analog is the finding that end-to-end FP8 does not fully eliminate mismatch, which is a constructive negative result but not a failed optimization attempt.

• Missing ablation: TIS clipping threshold C. The paper uses C=2 for all experiments but does not report any sweep over C values. The sensitivity of training quality to this hyperparameter is unknown. If C=1 (no correction except preventing extreme outliers) or C=∞ (unclipped importance sampling) were tested, they are not reported. This is a significant gap because the choice of clipping threshold directly controls the bias-variance tradeoff of the importance sampling correction, and practitioners need guidance on how to tune it.

• Missing ablation: Block size for quantization. The paper adopts $128 \times 128$ blockwise quantization following DeepSeek-V3 [8] but does not experiment with alternative block sizes (e.g., $64 \times 64$, $256 \times 256$, per-tensor, per-channel). The sensitivity of mismatch KL and training stability to block granularity is unexplored. Coarser blocks would be faster to quantize but introduce more error; finer blocks would preserve more accuracy but increase quantization overhead and scale storage.

• Missing ablation: Response length sensitivity. While the paper reports speedup across varying response lengths in the performance figures (Figures 3, 5), the training quality experiments fix the maximum response length at 20K tokens. Whether FP8 rollout with TIS maintains BF16-comparable accuracy at shorter contexts (e.g., 2K, 8K tokens) or even longer contexts (32K+ tokens) is not tested. Given that the preemption-elimination mechanism is specifically important at long contexts, the 20K-token setting may represent a worst case for BF16 and a best case for FP8 KV-cache quantization—results at shorter contexts might show smaller relative gains.

Critical Assessment

Claim 1: FP8 W8A8 linear rollout delivers 10–20% speedup for dense models and 30–50% for MoE models, with up to 44% combined improvement when KV-cache quantization is added.

Supported with specific quantification. Figures 3, 5, 8, and 11 provide time-per-token measurements across response lengths that directly support these ranges. The numbers are reported as ranges rather than single points, which appropriately captures variance across sequence lengths. The 44% combined speedup (Figure 8) and 48% in the NeMo-RL implementation (Figure 11) are clearly documented.

However, several important caveats apply:

1. Hardware and software specificity: All measurements are on H100 GPUs with CUDA 12.9+ and DeepGEMM enabled. The speedup numbers may not transfer to other hardware generations (A100, B200), other CUDA versions, or inference engines without DeepGEMM integration. The paper does not provide ablation across hardware configurations.

2. Context-length dependence of gains: The paper explicitly attributes a large fraction of the KV-cache quantization benefit to eliminating preemptions in memory-constrained scenarios. This means the 44% figure is specific to the tested regime (8B model, 20K-token responses, batch of 1536 concurrent sequences on 8 GPUs). Practitioners with shorter contexts or larger GPU memory pools (e.g., H200 with 141GB) may see substantially smaller gains, potentially making the investment in FP8 infrastructure less compelling. The paper acknowledges this implicitly (performance analysis in Section 2.3.2) but does not quantify the speedup at shorter context lengths.

3. No accounting for weight synchronization overhead: The speedup numbers measure time-per-token during generation, which captures the inference phase but not the weight quantization and synchronization phase (Phase 2 of the pipeline in Figure 1). If synchronization adds, say, 2 seconds per step and generation takes 10 seconds in BF16, a 44% generation speedup (to 5.6 seconds) plus 2 seconds of synchronization gives 7.6 seconds total—a 24% end-to-end speedup, not 44%. The paper does not report the absolute step times or the fraction of total step time consumed by synchronization, making it impossible to compute the true end-to-end speedup from the reported numbers.

Claim 2: FP8 rollout with TIS closely tracks BF16 baseline learning curves across all key training metrics (validation accuracy, reward, response length).

Qualitatively supported by the presented figures. The training curves in Figures 2, 4, 7, 9, and 10 show visual alignment between FP8+TIS and BF16 configurations. This is consistent across dense models, MoE models, linear-only quantization, KV-cache-only quantization, Full FP8, inference-side calibration, and trainer-side calibration—a broad robustness check.

However, the strength of evidence is weakened by several factors:

1. No quantitative accuracy comparisons: The paper never reports the final validation accuracy values for any configuration. Statements like "both configurations achieving comparable final accuracy" or "curves track closely throughout training" are qualitative judgments based on visual inspection of plots without axis labels in the provided figures. The reader cannot determine whether "comparable" means within 1%, 5%, or 10% of the BF16 baseline. A table reporting final accuracy, reward, and response length with confidence intervals would substantially strengthen this claim.

2. Single training run per configuration: All curves represent single runs with no error bars, seed replicates, or variance estimates. RL training is known to exhibit high run-to-run variance due to stochasticity in sampling, reward assignment, and optimization dynamics. Without multiple seeds, it is impossible to distinguish whether small deviations between FP8 and BF16 curves are due to quantization effects or random variation. If the FP8 run happened to have a favorable random seed while a different seed would have diverged, the conclusions would be misleading. This is a significant weakness given the paper's core claim of training equivalence.

3. Limited training duration: The training curves span approximately 300 RL steps. It is unclear whether FP8 and BF16 configurations would continue to track each other over longer training horizons (1000+ steps) or whether accumulated quantization error would eventually cause divergence. The MoE observation that mismatch KL increases during training (Figure 4) suggests that divergence could grow over time, potentially exceeding what TIS can correct at some threshold. The paper does not discuss this possibility or suggest monitoring strategies for long-running training jobs.

4. Single benchmark (AIME24): All validation accuracy measurements are on a single mathematics competition dataset. The paper does not test whether FP8 rollout preserves learning quality on other reasoning benchmarks, instruction-following tasks, or safety-related metrics. AIME24 is a specific distribution (competition math) that may not be representative of general RL fine-tuning objectives. If FP8 quantization subtly biases the policy toward certain reasoning patterns that happen to work for AIME24 but not for other tasks, the validation setup would not detect this.

5. Single RL algorithm (DAPO): All experiments use the DAPO algorithm. Whether FP8 rollout with TIS would work equally well with other RL algorithms (vanilla PPO, GRPO, RLOO, Reinforce) is untested. DAPO's specific objective and advantage estimation might be more or less sensitive to off-policy data than other algorithms. The paper does not discuss this potential interaction.

Claim 3: TIS is necessary—FP8 rollout without correction degrades accuracy.

Partially supported with limited quantification. The ablation in Figure 2 (green vs. blue) shows a visible accuracy gap, and the paper states that the no-TIS configuration exhibits "noticeable accuracy degradation." This establishes the existence of a problem requiring correction.

Weaknesses in this evidence:

1. Magnitude of degradation not reported: How much worse is the no-TIS configuration? 2% absolute accuracy? 10%? The paper does not say. Without quantification, practitioners cannot assess the risk of omitting TIS in their own settings, where the mismatch might be smaller (e.g., shorter sequences, larger models with more inherent noise tolerance) or larger (e.g., even lower precision like FP4).

2. No comparison to alternative corrections: The paper tests only token-level TIS with C=2. It does not compare against other correction strategies: unclipped importance sampling (which would test whether clipping is necessary), masked importance sampling (MIS, which masks tokens with extreme weights), sequence-level weighting, or no correction at all for the KV-cache-only configuration. The claim "TIS is necessary" could be refined to "some form of importance-sampling correction is necessary, and TIS with C=2 is sufficient for the tested configurations." Whether a simpler correction (e.g., just clipping importance ratios already present in PPO, or using a different clipping threshold) would work equally well is unknown.

3. No exploration of the mechanism: The paper states that no-TIS degrades accuracy but does not analyze how—does the policy overfit to reward hacking? Does response quality degrade? Does the optimization become unstable (diverging loss)? Understanding the failure mode would help practitioners diagnose TIS-related issues in their own runs.

Claim 4: KV-cache quantization is independently safe and provides disproportionate speedup in memory-bound regimes.

Strongly supported for the tested regime. Figure 7 (red curve) shows KV-cache FP8 only tracking BF16 closely on training metrics, confirming safety. Figure 8 quantifies the 38% standalone speedup and 44% combined speedup. The preemption-elimination mechanism is well-explained and plausible.

Caveats:

1. Regime specificity: The paper acknowledges that the 38% KV-cache speedup is "highly dependent on model size and use case" and is driven by eliminating preemptions in memory-constrained scenarios. The claim should be interpreted as: given a workload where KV-cache memory pressure causes frequent preemptions (8B model, 20K tokens, 1536 concurrent sequences, H100 GPUs), KV-cache FP8 provides large speedup by doubling effective cache capacity. Whether other workloads would see similar gains depends on their preemption frequency. The paper could have strengthened this claim by systematically varying batch size or context length to show the speedup as a function of memory pressure, but does not.

2. Attention computation quantization not isolated: The Full FP8 configuration (Figure 7, green) combines linear W8A8, KV-cache FP8, and attention computation FP8. The paper does not include a "Linear W8A8 + KV-cache FP8, attention BF16" configuration to isolate whether quantizing attention computations specifically adds value (speedup) or risk (mismatch). The contribution of attention computation quantization to both the speedup and the mismatch KL cannot be determined from the reported experiments.

Claim 5: End-to-end FP8 reduces train-inference mismatch compared to rollout-only FP8.

Supported with nuance. Figure 9 shows that FP8 training + rollout (green) has a "sampling importance ratio closer to 1 and a lower mismatch KL" than BF16 training + FP8 rollout (orange). The ~20% training-side speedup is also reported.

Important limitations:

1. Residual mismatch unexplained: The fact that end-to-end FP8 still shows higher mismatch than the BF16 baseline is a significant finding that the paper acknowledges but does not deeply investigate. What are the remaining sources of mismatch? Are they kernel-level differences between NeMo-RL's training code and vLLM's inference code? Differences in how attention softmax or layer norm are implemented? Non-determinism in parallel operations? Identifying these residual sources would guide future work, but the paper stops at observing their existence.

2. Single framework (NeMo-RL): The end-to-end FP8 experiments are conducted only in NeMo-RL, not in veRL. This limits the generality of the finding—it is possible that NeMo-RL's specific FP8 training implementation is particularly well (or poorly) matched to vLLM's FP8 inference, and results would differ in other frameworks. The paper does not discuss this scope limitation.

Experiments That Would Have Strengthened the Paper

1. Multiple random seeds (minimum 3) with error bars on all training curves. This is the single most impactful missing element. RL training is stochastic, and the paper's central claim of "training equivalence" between FP8 and BF16 requires demonstrating that observed differences are within run-to-run variance.

2. Quantitative final metrics table. A simple table reporting final validation accuracy, reward, response length, and mismatch KL for each configuration with standard deviations across seeds would transform the paper's evidence quality from qualitative visual inspection to quantitative comparison.

3. Ablation over TIS clipping threshold C. Testing C=1 (effectively no reweighting except clipping existing PPO ratios), C=2 (the paper's default), C=5, and C=∞ (unclipped) would map the sensitivity of training quality to this hyperparameter and provide guidance for practitioners.

4. Systematic context-length sweep. Measuring speedup at 2K, 5K, 10K, 20K, and 32K token maximum lengths would characterize how the speedup scales with memory pressure, directly testing the preemption-elimination hypothesis and providing predictive guidance for other workloads.

5. Additional benchmarks beyond AIME24. Testing on at least one additional validation benchmark (e.g., MATH, GSM8K, HumanEval for code) would test whether FP8-induced mismatch affects different reasoning domains differently.

6. Wall-clock step time measurement including synchronization. Reporting the absolute time breakdown per RL step (weight retrieval, quantization, weight loading, generation, training) for each configuration would enable practitioners to compute true end-to-end speedups for their specific workload parameters.

7. Longer training horizons. Extending training to 1000+ steps and observing whether mismatch KL continues to grow or stabilizes would address concerns about accumulated quantization error causing late-training divergence.

8. Alternative RL algorithm test. Running the same FP8 configuration with at least one other algorithm (e.g., standard PPO, GRPO) would test the interaction between quantization-induced off-policy data and algorithm-specific sensitivity to off-policy updates.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Unaccounted for in the Headline Throughput Numbers

The assumption or constraint. The paper estimates question difficulty by generating 2048 samples per prompt and using the process reward model's final-answer score as a proxy for correctness, then binning questions into five quintiles. The authors explicitly flag this in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The consequence. The reported 4× efficiency gains over best-of-N are computed after difficulty is known, without amortizing the cost of learning it. In practice, the total cost would be difficulty estimation plus strategy execution, and the former could dominate the latter. Generating 2048 samples per question is extraordinarily expensive—comparable to or greater than the largest test-time compute budgets studied (256–512 generations). A practitioner deploying this system would pay this upfront cost on every prompt, potentially negating or even reversing the reported speedup for problems that are ultimately easy (where the strategy itself requires only a few generations).

What evidence exists in the paper. The paper demonstrates that the PRM-based difficulty prediction works—predicted bins closely track oracle bins in Figures 4 and 8—but never measures the wall-clock time or FLOP cost of the estimation process itself. No comparison shows total cost (estimation + execution) versus the BF16 baseline. The paper's speedup numbers measure only the execution phase.

Mitigation status. The paper acknowledges this is a gap and flags it as future work:

"we leave the exploration of more computationally efficient difficulty estimation strategies (e.g., pretraining or finetuning models to directly predict difficulty of a question) to future work"

It does not propose or evaluate any cheaper estimation method. The current approach is a proof-of-concept that difficulty-adaptive allocation works, not a deployment-ready system. A practitioner today would need to develop their own lightweight difficulty estimator to realize the headline gains.

All Results Are on a Single Benchmark (MATH) with a Single Model Family (PaLM 2-S*)

The assumption or constraint. Every experiment in the paper uses the MATH benchmark (500 test questions) with PaLM 2-S* as the base model. The paper states in Section 4:

"We believe this model is representative of the capabilities of many contemporary LLMs."

No results are reported for other reasoning benchmarks (GSM8K, HumanEval, ARC), other model families (LLaMA, Qwen, DeepSeek), or non-math domains.

The consequence. Three aspects of the findings could be model-specific or domain-specific, and the paper provides no evidence to distinguish universal from contingent results:

• PRM quality and over-optimization behavior: The difficulty-dependent reversal—beam search hurts on easy problems due to verifier over-optimization (Figure 3, right), but helps on medium problems—depends on the PRM's calibration properties, which are a function of the base model's output distribution. A model with different error patterns or different calibration might exhibit different crossover points or no crossover at all.

• Revision model effectiveness: The revision model's ability to learn from edit-distance-paired incorrect-to-correct trajectories depends on the base model's in-context learning capabilities and the specific failure modes it exhibits. Models with different pretraining data or architectures might produce systematically different incorrect answers that are harder or easier to correct.

• MATH specificity: Competition math problems have clean ground-truth answers and well-defined reasoning steps. The PRM training relies on being able to verify intermediate steps via Monte Carlo rollout correctness. For tasks without such clean step-level verification (open-ended generation, dialogue, summarization), the entire PRM training pipeline would need to be redesigned.

What evidence exists in the paper. None. The paper does not include a single experiment outside the MATH benchmark or the PaLM 2 model family. This is a gap that the paper does not discuss as a limitation.

Mitigation status. Not addressed. The paper makes no claims about generalization beyond the tested setting, but also does not acknowledge this as a scope limitation. The reader is left to assume that the difficulty-dependent patterns generalize, without evidence.

The 14× Larger Model Baseline Is Weakened by Non-Compute-Optimal Pretraining and No Test-Time Compute

The assumption or constraint. The FLOPs-matched comparison in Section 7 scales model parameters by ~14× while holding training data fixed, following the LLaMA paradigm rather than Chinchilla-optimal scaling (where both parameters and data are scaled equally). The authors acknowledge this explicitly:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

Additionally, the larger model uses greedy decoding only—no majority voting, no best-of-N, no search of any kind.

The consequence. This makes the pretraining baseline weaker than it needs to be in two ways:

1. A Chinchilla-optimal model trained with 14× more total FLOPs (scaling both parameters and data) would likely outperform a parameter-only-scaled model, making the reported advantages of test-time compute (+27.8% on easy questions at R ≪ 1) potentially shrink or reverse.

2. Giving the larger model even a modest test-time compute budget—say, best-of-8 with majority voting—would create a much stronger baseline. The current comparison is asymmetric: the smaller model gets sophisticated, difficulty-adaptive test-time compute, while the larger model gets none. A fairer question is: "does a smaller model with optimized test-time compute beat a larger model with basic test-time compute?" The paper does not answer this.

What evidence exists in the paper. The paper is transparent about the parameter-only scaling choice (Section 7, quoted above). However, it does not discuss the asymmetry of giving test-time compute only to the smaller model, nor does it run the obvious ablation of the larger model with best-of-N or majority voting.

Mitigation status. The parameter-vs-Chinchilla comparison is explicitly deferred to future work. The asymmetric test-time compute allocation is not acknowledged as a limitation. Both issues mean the FLOPs-matched results should be interpreted as an upper bound on the advantage of test-time compute—the gap would likely narrow against stronger baselines.

The Revision Model Has a 38% Correct-to-Incorrect Reversion Rate with Only Patchwork Mitigations

The assumption or constraint. The revision model is trained exclusively on trajectories where all in-context answers are incorrect, followed by a correct answer. As a result, the model has never seen an example of what to do when the current answer is already correct. Section 6.1 reports:

"approximately 38% of correct answers get converted back to incorrect ones using a naive approach"

The consequence. This means that simply taking the last revision in a chain is unreliable—the model may overwrite a correct answer with an incorrect one. The paper mitigates this with majority voting or verifier-based selection across the whole chain (picking the best answer from any step), but these are patches that work around the problem rather than solving it. The underlying issue—the model has not learned to recognize and preserve correct answers—remains. For problems where the model produces a correct answer early but then revises it away, the revision approach can be strictly worse than generating a single answer. This also complicates latency: to benefit from revisions, the system must generate the full chain and then select, meaning the worst-case latency is the full chain length regardless of whether an early step was already correct.

What evidence exists in the paper. The 38% reversion rate is reported explicitly (Section 6.1). The paper does not break down how often early-correct answers get reverted at different chain positions or difficulty levels—for instance, is reversion more common on easy problems (where correct answers appear early and the model has more opportunities to break them) or hard problems? This breakdown would help practitioners decide whether revisions are net-beneficial for their problem distribution.

Mitigation status. The paper treats within-chain selection (majority voting or verifier) as the mitigation and does not explore more principled solutions, such as training the model with mixed trajectories that include correct-in-context examples, or using a classifier to detect likely-correct answers and halt revision. The ReST$^{EM}$ experiment (Appendix K, Figure 16) further demonstrates the fragility of revision training: attempting to optimize the revision model with RL caused performance to degrade substantially, suggesting that the positive revision results depend sensitively on the specific offline data construction procedure. This is a fundamental unresolved tradeoff: revisions add value on average, but introduce a failure mode (correct-to-incorrect reversion) that requires ad-hoc post-hoc correction.

No Latency or Wall-Clock Time Analysis Despite Serial vs. Parallel Tradeoffs

The assumption or constraint. The paper measures all compute in "generations"—the number of complete solutions sampled—and reports speedup in terms of generations-equivalent or time-per-token during rollout. This treats all generations as fungible units of compute. However, sequential revisions and parallel best-of-N sampling have fundamentally different wall-clock time profiles on real hardware.

The consequence. A strategy that allocates 128 generations as 64 sequential × 2 parallel takes roughly 64× longer in wall-clock time than one that runs 128 parallel samples simultaneously (assuming sufficient GPU memory and batch capacity). The compute-optimal policy in Section 6.3 selects strategies based on generation count, not wall-clock time, which means it may recommend sequential-heavy strategies for easy problems that are latency-disastrous in practice. For interactive or near-real-time applications, a strategy that achieves high accuracy but takes minutes to produce an answer is unacceptable, even if it is "compute-optimal" in terms of total FLOPs. The paper never measures or discusses this dimension.

What evidence exists in the paper. None. The paper reports no end-to-end wall-clock times for any strategy, no latency distributions, and no analysis of how hardware parallelism constraints affect the sequential-vs-parallel tradeoff. The speedup numbers (Figures 3, 5, 8) are generation-time measurements that do not include the revision chain's serial dependency cost. The paper's performance model assumes that reducing total FLOPs is sufficient, which is only true for throughput-bound batch workloads, not latency-bound interactive ones.

Mitigation status. Not addressed at all. The paper never mentions latency as a consideration. For practitioners deploying in latency-sensitive settings, this is a major gap: the compute-optimal allocation framework needs to be extended with a latency budget constraint (maximum wall-clock time per problem) in addition to a FLOP budget, and the current results provide no guidance on how to do that.

Hard Problems (Difficulty Bin 5) Remain Completely Unsolved Regardless of Compute Budget

The constraint. Across all methods studied—PRM search, iterative revisions, and their compute-optimal combinations—the hardest quintile of problems (bin 5, where the base model's pass@1 is near zero) shows negligible improvement. In Figure 3 (right), bin 5 accuracy hovers at 1–3% for all methods and all budgets. In Figure 7 (right), bin 5 shows roughly 2–3% accuracy irrespective of the sequential-to-parallel ratio. In the FLOPs-matched comparison (Figure 9), the bin 5 scaling line is essentially flat near 0–5% across all $R$ values.

The consequence. Test-time compute can amplify existing capability—it helps the model find and refine correct solutions that already exist in its output distribution at some non-trivial rate—but it cannot create capability from nothing. If the base model's pass@1 is approximately zero on a problem class, no amount of search or revision will help, because there are no correct solutions in the proposal distribution to find or refine. This establishes a hard ceiling: the approach offers no path forward for genuinely novel or out-of-distribution reasoning that exceeds the base model's pretraining-derived competence. For such problems, scaling pretraining (larger models, more data, or both) remains the only viable path. The paper is transparent about this (Section 7), but the implication is that practitioners who need their systems to handle a long tail of genuinely hard problems must either accept near-zero accuracy on those problems or invest in pretraining, not just inference-time optimization.

What evidence exists in the paper. The bin 5 results are consistent and unambiguous across all experiments. The near-flat lines at 1–5% accuracy in Figures 3 (right), 7 (right), and 9 are the most robust finding in the paper—no method, no budget, no strategy makes a meaningful dent in the hardest problems.

Mitigation status. The paper acknowledges this explicitly, framing it as a boundary condition: test-time compute amplifies existing capability but does not create it. The authors do not attempt to solve hard problems (e.g., by combining with retrieval, tool use, or multi-model collaboration) and do not suggest that any amount of additional test-time compute engineering would help. This is a fundamental limitation that the paper characterizes clearly rather than mitigates.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around low-precision inference for LLM reinforcement learning from a capability demonstration to an operational practice. Before FP8-RL, the field knew that FP8 inference existed (vLLM, TensorRT-LLM, SGLang all supported it) and that RL systems existed (veRL, NeMo-RL, OpenRLHF)—but nobody had stitched them together into a working pipeline that handles the unique demands of RL: per-step weight re-quantization, dynamic synchronization across decoupled training and inference backends, and algorithmic correction for the off-policy bias introduced by quantized rollouts. The paper's contribution is not a new quantization format or RL algorithm; it is the demonstration that existing components, when properly integrated and paired with a specific correction strategy (token-level TIS with C=2), can deliver FP8's throughput benefits without degrading training quality.

The magnitude of the shift is incremental but practically significant. This is not a paradigm-shifting theoretical insight like the Chinchilla scaling laws. It is closer to a systems integration milestone with validated recipes: the paper tells the field "here is exactly how to make FP8 work in RL, here is what happens if you skip the correction (accuracy degrades), and here is evidence that it works across architectures (dense and MoE), quantization scopes (linear only, KV-cache only, full stack), and calibration paradigms (inference-side and trainer-side)." This transforms FP8 rollout from a bespoke engineering effort requiring per-project expertise into a configuration-level feature—in veRL, a single flag (actor_rollout_ref.rollout.quantization=fp8). The economic implications are direct: rollout consumes up to 80% of RL iteration time (cited from Seer [1]), and FP8-RL delivers 10–50% speedup on the generation phase alone, with up to 44% when KV-cache quantization eliminates preemption waste in memory-constrained regimes. For organizations running thousand-GPU RL fine-tuning jobs, this translates to millions of dollars in saved compute and substantially shortened experimentation cycles—without sacrificing final model quality.

The paper resolves a specific tension in prior work: the apparent conflict between "FP8 inference is mature and safe for static deployment" and "RL introduces unique challenges that make naïve FP8 unsafe." Prior work on train-inference mismatch [5, 6, 7] had identified that off-policy data from rollout discrepancies can destabilize RL, and prior work on bitwise consistency [21] and FP16 alignment [22] had proposed fixes that focus on software-level numerical alignment. The FP8-RL paper demonstrates that quantization-induced mismatch is a distinct category—the inference policy $\pi_\theta^{\text{FP8}}$ computes a genuinely different function from the training policy $\pi_\theta$, and no amount of kernel alignment can fix that because the weights themselves are different. The critical finding is that this mismatch is correctable using a standard off-policy RL tool (token-level truncated importance sampling) that was developed for a different purpose (handling stale policies across training steps). The paper shows that TIS is both necessary (Figure 2, green vs. blue) and sufficient (blue tracks orange across Figures 2, 4, 7, 9) for making FP8 rollout safe. This provides a concrete operational prescription—"always pair FP8 rollout with importance-sampling correction"—that resolves the ambiguity left by prior mismatch work, which identified the problem without providing a validated, quantization-specific fix.

The paper also reframes which research directions become more or less attractive:

Directions that become MORE attractive:

• Cheap difficulty estimation for adaptive FP8. The paper's difficulty estimation (2048 samples per prompt) is far too expensive for deployment. But now that the compute-optimal allocation framework is validated (from the reference example's approach, which this paper echoes in spirit by adaptively allocating precision through its correction strategy rather than a one-size-fits-all approach), the natural next step is lightweight difficulty predictors—possibly distilled from the PRM or trained directly on question text—that make adaptive FP8 allocation practical. This paper lowers the barrier by providing a working end-to-end pipeline into which better difficulty estimators can be plugged.

• Verifier robustness under quantization. The finding that mismatch KL grows during training for MoE models (Figure 4) and that end-to-end FP8 does not fully eliminate mismatch (Figure 9) points to an open problem: how do we make RL training robust to the residual distribution shift that persists even after importance-sampling correction and precision alignment? This connects to the broader verifier over-optimization agenda from the reference example—improving verifier and value function robustness becomes a priority when precision-induced noise is added to the rollout distribution.

• FP8-specific monitoring and diagnostics. The paper introduces mismatch KL as a key monitoring metric and shows that elevated KL does not necessarily imply training harm (FP8+TIS has elevated KL but tracks BF16 accuracy; Figure 2). This suggests a research program around diagnostic tools for low-precision RL: what metrics predict actual training degradation vs. benign distribution shift? Can we detect incipient divergence before it affects validation accuracy?

Directions that become LESS attractive:

• Pursuing marginal improvements in FP8 quantization formats for RL specifically. The paper's quantization scheme (E4M3, 128×128 blocks, specific layer exclusions) is adopted directly from prior work and works without modification. The gains come from the integration and correction, not from a better format. This suggests that the quantization research community's effort is better spent on general-purpose improvements (which will percolate to RL) rather than RL-specific format optimization.

• Complex search or planning algorithms for test-time compute allocation in the low-precision regime. The reference example showed that sophisticated search (lookahead search, beam search with large width) can backfire due to verifier over-optimization. The FP8-RL paper adds a related caution: aggressive precision reduction without correction causes accuracy degradation (Figure 2, green). Together, these findings suggest that simple, well-corrected strategies outperform complex, uncorrected ones—a principle that should guide both precision allocation and test-time compute allocation. Researchers should prioritize robust correction mechanisms (importance sampling, verifier calibration) over sophisticated optimization algorithms that amplify uncorrected errors.

Follow-Up Research This Work Enables

Systematic characterization of the speedup-mismatch tradeoff as a function of context length, batch size, and GPU memory. The paper's headline 44% speedup comes from a specific regime where KV-cache memory pressure causes frequent preemptions (8B model, 20K-token responses, 1536-concurrent-sequence batch, H100 GPUs). The performance analysis in Section 2.3.2 explicitly states this is "highly dependent on model size and use case." A strong follow-up would sweep context length (2K, 5K, 10K, 20K, 32K tokens), batch size (varying concurrent sequences), and GPU memory configurations (H100 80GB vs. H200 141GB vs. B200) and measure both throughput speedup and training accuracy for FP8 rollout with TIS. The key output would be a phase diagram: under what (model size, context length, batch size) conditions does FP8 KV-cache quantization provide >30% speedup, 10–30%, or <10%? This would transform the paper's point observation into a predictive model that practitioners can use to decide whether FP8 rollout is worth the engineering investment for their specific workload. The experiment is straightforward—it uses the same DAPO + AIME24 recipe and software stack, just varying the generation parameters—and would substantially increase the paper's practical impact.

Is token-level TIS with C=2 always sufficient, or can it fail under more aggressive quantization? The paper demonstrates TIS's sufficiency for FP8 (E4M3) quantization but explicitly flags NVFP4 as future work (Section 4) and notes "reported instability from accumulated quantization error" for more aggressive formats. A natural stress-test is to repeat the 8B dense model experiments with FP4 weights or FP4 KV-cache (using blockwise quantization, same TIS correction with C=2) and measure whether the mismatch KL grows beyond what TIS can correct. The paper's MoE finding—that mismatch KL increases during training even in BF16 (Figure 4)—hints that accumulated error over training could eventually overwhelm a fixed clipping threshold. A strong follow-up would measure whether C=2 remains sufficient at FP4, or whether adaptive clipping (C decreasing over training steps) or alternative corrections (masked importance sampling, sequence-level variance reduction) are needed. The experimental cost is moderate (re-running the 8B dense model experiment with FP4 quantization, ~300 steps, 8 GPUs), and the outcome—either "TIS with C=2 works for FP4" or "TIS fails at FP4, requiring new correction strategies"—would directly inform the next generation of low-precision RL systems.

Diagnosing and mitigating the residual mismatch in end-to-end FP8 RL. Section 2.4 shows that end-to-end FP8 (training + rollout) reduces mismatch KL relative to rollout-only FP8, but still shows higher mismatch than the BF16 baseline. The paper attributes this to non-precision sources (kernel differences, attention softmax implementation, layer norm numerics) without identifying them. A high-value follow-up would systematically narrow down the residual mismatch source: run end-to-end FP8 with (a) identical attention kernels between NeMo-RL training and vLLM inference, (b) bitwise-deterministic layer norm implementations, (c) disabled non-deterministic optimizations (e.g., matmul autotuning), and (d) identical random seeds for dropout. Measure which intervention(s) reduce the residual mismatch KL. The goal is to answer: can end-to-end FP8 achieve BF16-equivalent mismatch KL if all non-precision sources are aligned? If yes, the path forward is engineering consistency. If no (residual mismatch persists even with aligned kernels), there is a deeper numerical effect from FP8 training that requires algorithmic mitigation beyond importance sampling—a finding that would redirect research toward robust training objectives rather than kernel alignment.

Cross-algorithm validation of FP8 rollout safety. All experiments use the DAPO algorithm. DAPO's specific advantage estimation and clipping scheme may be more or less tolerant of off-policy data than other popular RL algorithms (vanilla PPO, GRPO, RLOO, Reinforce). A systematic study would replicate the 8B dense model FP8 W8A8 + TIS experiment with at least two alternative algorithms (say, GRPO and standard PPO with the same batch size and response length), measuring validation accuracy and mismatch KL. The question is whether TIS with C=2 is a universal fix or algorithm-specific. If FP8+TIS works with PPO but degrades with GRPO, that reveals an interaction between quantization-induced off-policy data and the algorithm's sensitivity to importance weight variance—a finding that would guide algorithm selection in low-precision RL pipelines. The experiment is a direct replication with algorithm substitution and would take approximately the same compute as the original experiment (~300 steps, 8 GPUs per algorithm).

Lightweight difficulty estimation for adaptive precision allocation. The reference example's compute-optimal scaling framework and this paper's FP8-RL system share a common missing piece: cheap, reliable difficulty estimation that can inform per-prompt decisions about how much precision or compute to allocate. A concrete follow-up would train a small classifier (perhaps a 100M-parameter model) on top of frozen LLM embeddings to predict question difficulty from the prompt text alone, using the paper's PRM-based difficulty estimate (average final-answer score over 2048 samples) as training labels. Evaluate whether the classifier's predicted difficulty bins match the PRM-based bins closely enough to recover the 4× speedup from adaptive allocation (Figures 4, 8 in the reference example) without the 2048-sample estimation cost. This directly addresses the most significant deployment bottleneck identified in both papers and would make adaptive test-time strategies—whether for compute allocation or precision allocation—practical for production.

Practical Applications and Downstream Use Cases

Cost-efficient RL fine-tuning for reasoning models (math, code, science). The paper's experiments on AIME24 with DAPO directly model a production workflow: taking a base model (Qwen3-8B-Base, Qwen3-30B-A3B-Base), applying RL fine-tuning to improve complex reasoning, and validating on a competition benchmark. For organizations running such pipelines—whether fine-tuning open-weight models or developing proprietary reasoning models—the paper's 10–50% rollout speedup (Figures 3, 5, 8) translates to proportionally reduced training time and compute cost. For a representative training run: if rollout consumes 80% of iteration time (per Seer [1]), and FP8 W8A8 + KV-cache quantization provides a combined 44% speedup on the rollout phase, the end-to-end training time reduction is approximately 35% (0.8 × 0.44). On a 16-GPU H100 cluster running for weeks, this represents tens of thousands of GPU-hours saved—enough to run multiple additional experiments or train with more data within the same budget. The paper's validation that FP8 rollout with TIS preserves final model accuracy (Figures 2, 4, 7, 9) makes this a low-risk optimization: practitioners can enable FP8 with confidence that their model quality will not degrade. The primary requirement is ensuring their training framework supports the paper's synchronization pipeline (veRL 0.11+ or NeMo-RL with FP8 support) and CUDA 12.9+ with DeepGEMM.

Long-context RL training where KV-cache memory pressure causes catastrophic throughput degradation. The paper's diagnostic finding—that BF16 KV-cache at 20K tokens with 1536 concurrent sequences causes "frequent request preemptions" that "wasted computation and throttled throughput" (Section 2.3.2)—identifies a specific failure mode that FP8 KV-cache quantization eliminates. This is particularly relevant for RL training on reasoning tasks that require long chain-of-thought (mathematical proofs, multi-step code generation, scientific reasoning), where response lengths routinely exceed 10K tokens. Without KV-cache quantization, practitioners may find their H100 GPUs spending a significant fraction of time recomputing preempted sequences rather than making forward progress—a silent efficiency loss that is not obvious from GPU utilization metrics alone (GPU utilization can remain high even as effective throughput collapses). The paper's 38% KV-cache-only speedup (Figure 8) is effectively a "preemption tax refund"—it recovers computation that was being wasted on recomputation. For any team observing that their long-context RL throughput is lower than expected given their hardware's theoretical FLOPs, enabling KV-cache FP8 (via kv_cache_dtype: fp8_e4m3 in the configuration) is a high-priority diagnostic and fix. The paper provides configurations for both veRL (inference-side calibration) and NeMo-RL (trainer-side calibration with ~2-3% overhead), giving practitioners flexibility based on their framework.

MoE model training where FP8 provides outsized returns. The paper's finding that FP8 W8A8 rollout delivers 30–50% speedup for the 30B MoE model versus 10–20% for the 8B dense model (Figures 3 vs. 5) is actionable guidance for teams working with MoE architectures. The disproportionate benefit comes from three compounding factors—higher arithmetic intensity, reduced weight-loading bandwidth, and expanded KV-cache capacity reducing preemptions—all of which scale with model size. This suggests that the value proposition of FP8 rollout becomes stronger as models grow, making it an increasingly important optimization for frontier-scale RL fine-tuning. Teams training MoE models in the 100B+ parameter range (where the rollout cost is already the dominant bottleneck) should prioritize FP8 rollout integration, as the relative speedup is likely to exceed the 50% measured at 30B scale. The paper's MoE-specific finding about growing mismatch KL (Figure 4) also provides a warning: MoE models require importance-sampling correction even at BF16 due to routing inconsistencies, and FP8 exacerbates this growth. Teams should monitor mismatch KL during training and be prepared to escalate from token-level TIS to masked importance sampling (MIS) or Rollout Router Replay (R3) if the divergence becomes unstable.

FP8 adoption as a standard configuration in RL frameworks. The paper's implementation of FP8 rollout as a single-configuration-flag feature in veRL (actor_rollout_ref.rollout.quantization=fp8, Appendix A) and NeMo-RL has a downstream impact beyond the specific experiments reported: it lowers the barrier to FP8 adoption across the entire RL fine-tuning community. Before this work, enabling FP8 in RL required deep understanding of the quantization scheme, the inference engine's weight-loading internals, the synchronization protocol between training and inference, and the mismatch correction math—a combination of expertise that few teams possess. Now, a practitioner who would never read a quantization paper can set one flag and get FP8 rollout with TIS correction enabled. The paper's extensive validation across architectures, quantization components, and calibration paradigms provides the safety case for making FP8 a default rather than an experimental feature. Framework maintainers (veRL, NeMo-RL, and potentially OpenRLHF and others that adopt similar integrations) can confidently ship FP8 support knowing that the failure mode (no-TIS accuracy degradation) is documented and the mitigation (enable TIS) is validated. This has a multiplier effect: every team that adopts FP8 through these frameworks realizes the paper's 10–50% speedup without needing to understand the underlying mechanisms, collectively saving enormous compute across the field. The primary adoption requirement is CUDA 12.9+ and DeepGEMM—a dependency that will become standard as H100/B200 clusters upgrade their CUDA versions.