Defeating the Training-Inference Mismatch via FP16

🎯 Pitch

RL fine-tuning of LLMs collapses not because of algorithmic flaws, but because BF16's 7-bit mantissa silently corrupts the probability distributions between training and inference—a mismatch that no importance sampling correction can fully fix. Simply switching to FP16 eliminates this discrepancy, delivering stable optimization and higher final accuracy without any model or algorithm changes.

1. Executive Summary

This paper identifies and resolves a root cause of instability in reinforcement learning (RL) fine-tuning of large language models: the numerical mismatch between training and inference policies introduced by BF16's 7-bit mantissa, which accumulates rounding errors during autoregressive generation and produces divergent probability distributions. Through a systematic sanity test — a curated subset of MATH problems (1,460 questions) where the base model's initial accuracy falls between 20% and 80%, creating a perfectible dataset where a reliable algorithm should reach near-100% training accuracy — the authors demonstrate across DeepSeek-R1-Distill-Qwen-1.5B, Qwen3-30B-A3B-Base, Qwen3-14B-Base, OctoThinker-3B, and Qwen2.5-Math-1.5B that simply switching from BF16 to FP16 for both training and inference eliminates the mismatch, enabling even the vanilla unbiased policy gradient estimator (Equation 5) to outperform all BF16-based algorithmic corrections — including token-level truncated importance sampling and sequence-level masked importance sampling patches that each add ~25% computational overhead — achieving 99% training accuracy versus 95% for the best BF16 method and a 5-percentage-point improvement on AIME 2024 (39% vs. 34%), while reducing the sequence-level log-probability ratio mismatch by approximately 24× (KL divergence of 0.32 bits vs. 7.64 bits in BF16). The work establishes that FP16's 10 mantissa bits provide sufficient precision to absorb the implementation differences between training and inference engines across diverse frameworks (VeRL, Oat), architectures (dense, MoE), training regimes (full fine-tuning, LoRA), and RL algorithms, fundamentally resolving the bias-variance tradeoff that forces BF16 methods to choose between fast-but-collapsing biased estimators and stable-but-slow unbiased ones.

2. Context and Motivation

The Core Problem: RL Fine-Tuning of LLMs Is Brittle Despite Its Importance

The paper addresses a specific, practical obstacle that has made reinforcement learning fine-tuning of large language models notoriously unreliable. Over the past several years, RL has emerged as perhaps the most powerful paradigm for aligning LLMs — particularly for improving mathematical and logical reasoning. Landmark results like DeepSeek-R1 (Guo et al., 2025) demonstrated that RL alone, without supervised fine-tuning on reasoning traces, can induce sophisticated chain-of-thought behaviors. This has sparked a wave of follow-up work and production systems (Zeng et al., 2025; Liu et al., 2025c; Qi et al., 2025).

However, the paper documents a reality that practitioners routinely encounter but that the literature has struggled to systematically characterize: RL fine-tuning is fragile. The training process is, in the authors' words, "notoriously sensitive to hyperparameters and can suffer from training collapse" (Section 1). This isn't merely an inconvenience — it means that promising algorithmic ideas routinely fail to translate into reliable improvements, that reproducing published results requires painstaking hyperparameter tuning, and that scaling RL to larger models or longer training horizons often hits unexpected failure modes. The paper cites a substantial body of recent work that has grappled with this instability (Yao et al., 2025; Liu et al., 2025a; Team et al., 2025a; Zheng et al., 2025; Yu et al., 2025; Cui et al., 2025), underscoring that this is a widely recognized challenge, not a niche concern.

The Identified Mechanism: Training-Inference Mismatch

The paper focuses on one particular mechanism behind this instability: the training-inference mismatch. To understand why this arises, it's necessary to appreciate the system architecture of modern RL frameworks for LLMs.

When training an LLM with RL, the system must perform two computationally distinct operations per iteration:

Rollout (inference): Generate responses from the current model for a batch of prompts. This is inherently autoregressive — tokens are produced one at a time, with each new token conditioning on all previous ones. Modern frameworks use highly optimized inference engines for this, such as vLLM (Kwon et al., 2023), which employ techniques like PagedAttention for memory efficiency and specialized CUDA kernels for fast autoregressive decoding.
Training (gradient computation): Compute policy gradients and update model parameters. Unlike rollout, training processes entire sequences in parallel — the model sees all tokens simultaneously, and attention patterns are computed across the full sequence at once. Training frameworks typically use different distributed strategies (e.g., FSDP, DeepSpeed ZeRO) with their own optimized kernels and parallelization approaches.

These two operations are mathematically specified to use the identical model weights and an identical computation (a forward pass through the transformer). In a perfect world with infinite numerical precision, the probability distribution over next tokens produced during training would match exactly the distribution produced during inference — that is, the training policy $\pi(\cdot|\theta)$ and the inference policy $\mu(\cdot|\theta)$ would be identical.

In practice, they are not. The distinct computational paths — different engines, different kernels, different parallelization strategies — introduce small numerical discrepancies. A slight difference in the order of floating-point operations, a different accumulation strategy, or a CUDA kernel that fuses operations differently can all cause the same mathematical expression to produce slightly different numerical results. When these discrepancies occur at every token generation step, they accumulate across the autoregressive sequence. What starts as a tiny per-token rounding error compounds into a substantial divergence in the probability distribution over full responses.

This mismatch creates two distinct problems, which the paper formalizes in Section 2:

Biased gradients (Equation 3). The policy gradient computed using samples from $\mu$ but evaluated using log-probabilities from $\pi$ is no longer an unbiased estimate of the true gradient. Formally:

$\nabla_\theta J_{\text{biased}}(x, \theta) = \mathbb{E}_{y \sim \mu(\cdot|x,\theta)}\left[\nabla_\theta \log \pi(y|x,\theta) \cdot R(x,y)\right] \neq \nabla_\theta J(x,\theta)$

This bias means the optimization is moving toward a different objective than intended. The gradient signal becomes corrupted, potentially pushing parameters in directions that don't actually improve the policy under its own distribution.

The deployment gap (Equation 4). Even if the gradient bias were somehow corrected during training, a deeper problem persists: the parameters $\theta$ are being optimized with respect to $\pi$ (what the training engine computes), but at deployment time the model runs through $\mu$ (the inference engine). The parameter vector that maximizes performance under $\pi$ is not necessarily the same as the one that maximizes performance under $\mu$ :

$\arg\max_\theta \mathbb{E}_{x \sim p_X, y \sim \mu(\cdot|x,\theta)}[R(x,y)] \neq \arg\max_\theta \mathbb{E}_{x \sim p_X, y \sim \pi(\cdot|x,\theta)}[R(x,y)]$

This deployment gap is particularly insidious because it means that even apparently successful training — runs that achieve high training rewards — may not translate into correspondingly strong evaluation performance. The model has been optimized for a distribution that differs from the one it will face at test time.

Prior Approaches: Algorithmic Corrections and Their Limitations

Before this paper, the research community had pursued two broad strategies for addressing the training-inference mismatch: algorithmic corrections that modify the gradient estimator, and engineering approaches that try to reduce the mismatch directly. Neither has proven fully satisfactory.

Algorithmic Corrections via Importance Sampling

The theoretically principled approach is to use importance sampling (IS) to correct the gradient estimator (Section 2.1). Rather than treating samples from $\mu$ as if they were from $\pi$ (which introduces bias), one can re-weight them by the probability ratio:

$\nabla_\theta J_{\text{pg-is}}(x) = \mathbb{E}_{y \sim \mu(\cdot|x,\theta')}\left[\frac{\pi(y|x,\theta)}{\mu(y|x,\theta')} \nabla_\theta \log \pi(y|x,\theta) \cdot A(x,y)\right]$

where $\theta'$ are the parameters used during sampling (which may differ from $\theta$ in off-policy settings) and $A(x,y)$ is the advantage. This estimator is unbiased, but it comes with a severe practical problem: the probability ratio $\frac{\pi}{\mu}$ can be extreme for long sequences. In the context of LLMs where responses routinely span thousands of tokens, small per-token probability differences compound multiplicatively, producing importance weights that vary by orders of magnitude. This high variance makes the estimator impractical for stable optimization.

Two main strategies have been proposed to reduce this variance, trading some bias for stability:

Truncated Importance Sampling (TIS) (Equation 6), introduced to LLM RL by Yao et al. (2025): clip the importance weight to a maximum value $C$ , preventing any single sample from dominating the gradient.
Masked Importance Sampling (MIS) (Equation 7), proposed by Liu et al. (2025a): zero out samples whose importance weight exceeds the threshold $C$ entirely, rather than clipping them.

However, the paper identifies important limitations in how these ideas have been implemented in practice. The dominant open-source RL framework, VeRL (Sheng et al., 2024), is built around GRPO (Shao et al., 2024), which does not naturally support importance-weighted policy gradients. Consequently, the algorithmic corrections developed by Yao et al. (2025) and Liu et al. (2025a) are effectively implemented as patches on top of GRPO rather than as clean, standalone policy gradient estimators. The specific implementations, given in Equations 9 and 10, layer the importance sampling correction onto the existing GRPO gradient formula — which already includes its own clipping mechanism (the PPO-style $r_t A_t$ clipping).

This patching approach introduces two fundamental problems that the paper argues are inherent to the algorithmic strategy itself, regardless of implementation quality:

1. Computational overhead. Computing the importance sampling correction requires an additional forward pass. The implementations from Yao et al. (2025) and Liu et al. (2025a) need to evaluate $\pi(\cdot|\theta')$ — the training engine's probabilities for the responses that were generated by the inference engine. This is separate from the forward pass already required for the gradient computation. Assuming a backward pass costs roughly twice a forward pass (a standard approximation in distributed training literature, cited from Qi et al., 2023), this extra forward pass adds approximately 25% to the training cost. For large-scale RL training, this is a significant penalty.

2. The deployment gap persists by design. Importance sampling corrections fix the biased gradient — they ensure that the gradient computed during training is, in expectation, correct with respect to $\pi$ . But they do nothing to address the deployment gap. The optimization still targets $\pi$ , while deployment uses $\mu$ . The parameters that maximize performance under the training engine are not guaranteed to maximize performance under the inference engine. As the paper puts it (Section 2.1): "While algorithmic patches fix the biased gradient, by nature they cannot close the deployment gap, which calls for a fundamental solution to remove the mismatch altogether."

Engineering Approaches to Reduce the Mismatch

A separate line of work has attempted to address the mismatch from an engineering perspective — trying to make the training and inference computations produce numerically identical results. The paper surveys these efforts and finds them insufficient (Section 2.2):

FP32 language model head (Chen et al., 2025): Early attempts used higher precision only in the final projection layer (the LM head), under the intuition that this is where token probabilities are computed. However, Yao et al. (2025) and Liu et al. (2025a) showed this is insufficient to prevent training collapse — the mismatch accumulates throughout the entire transformer stack, not just in the final layer.
Manual alignment of training and inference implementations (Team et al., 2025a): Very recently, a team reported success by carefully aligning the specific CUDA kernels, parallelization strategies, and numerical operations between their training and inference stacks. However, the paper notes that this approach "requires deep domain knowledge and substantial engineering effort, and it is unclear whether such bespoke fixes can be generalized across different frameworks or models."
Enforced determinism (He, 2025): Work on making inference fully deterministic addresses a related but distinct issue. Deterministic inference ensures that the same input always produces the same output, which is valuable for reproducibility. However, the paper argues this "cannot directly address the training-inference mismatch" because the mismatch is between two different computational graphs (training and inference), not between two runs of the same graph.

The paper identifies fundamental reasons why engineering alignment is inherently difficult (Section 2.2): tokens are generated autoregressively in inference but processed in parallel during training; different parallelization strategies (e.g., tensor parallelism for inference vs. data parallelism for training) change the order of operations; and precision-sensitive operations like top-k expert selection in Mixture-of-Experts (MoE) models amplify small discrepancies in ways that are hard to predict or control. The sheer diversity of hardware, frameworks, and model architectures makes a universal engineering solution impractical.

Empirical Evidence of the Problem's Severity

The paper's offline analysis (Section 3.5, Figure 2) provides concrete evidence for the magnitude of the mismatch under BF16. When sampling responses with temperature 1.0 (so that the inference policy $\mu$ can be directly compared to the training policy $\pi$ using the same model weights evaluated through the DeepSpeed training engine), the authors find that:

At the token level, BF16 produces substantial scatter in the $\mu$ vs. $\pi$ probability scatter plot, with many points far from the diagonal (Figure 2, left). FP16 produces a much tighter concentration around the diagonal.
At the sequence level, the log-probability ratio $\log\frac{\pi(y|x)}{\mu(y|x)}$ — which directly determines the importance sampling weight — scales linearly with sequence length under BF16, with a slope of approximately −1.01. The total mismatch for a 25K-token sequence, measured by $\text{KL}[\mu \parallel \pi]$ , reaches 7.64 bits. Under FP16, the slope is essentially flat at −0.07, and the total mismatch is only 0.32 bits — roughly a 24× reduction.

This exponential growth of mismatch with sequence length is particularly damaging. It means that longer responses — precisely the kind that RL is intended to incentivize for complex reasoning problems — suffer from the worst mismatch. The very behavior the training objective is trying to encourage (long, detailed reasoning chains) is also the behavior for which the gradient signal is most corrupted. This creates a perverse dynamic where training may appear to make progress on short problems while silently failing on the long-form reasoning that matters most.

The Bias-Variance Tradeoff Under BF16

The paper's empirical results in Section 4.2 reveal a striking pattern in the behavior of different BF16 algorithms under the sanity test. This pattern, which the paper characterizes as a bias-variance tradeoff (Section 6), explains why prior work has produced conflicting recommendations about which algorithmic correction to use:

Low-bias, high-variance methods (PG-Seq-IS, GRPO-Seq-MIS): These methods use unbiased sequence-level importance sampling corrections. They are stable — they don't collapse — but converge slowly because the sequence-level ratio has enormous variance. Even at their peak, they exhibit a deployment gap: the best BF16 method (GRPO-Seq-MIS) reaches only 95% training accuracy versus 99% for our FP16 approach under the sanity test, and 34% vs. 39% on AIME 2024.
High-bias, low-variance methods (vanilla GRPO, GRPO-Token-TIS, GSPO): These methods either ignore the mismatch (vanilla GRPO) or apply token-level corrections that reduce variance but leave residual bias (token-level TIS). They converge quickly initially but eventually collapse — the bias accumulates over training steps, corrupting the optimization trajectory. The collapse is preceded by a growing $\pi - \mu$ divergence that reaches extreme values (one policy's probability approaching 1 while the other's approaches 0 for the same token, despite using identical weights), which the paper suggests may serve as an early-warning signal.

This tradeoff is fundamental to the BF16 regime: any method must choose between speed and stability, between low variance (fast convergence) and low bias (guaranteed correctness). The tension arises because BF16's low precision creates a large mismatch that forces the importance sampling correction to be aggressive, which in turn inflates variance. The paper's key insight — addressed in Section 3 — is that this tradeoff is not inherent to RL fine-tuning itself but is rather an artifact of the precision regime.

How This Paper Positions Itself

The paper explicitly positions itself as a precision-level solution to a problem that has previously been addressed at the algorithmic or engineering level (Section 3, Section 6). Rather than adding complexity — additional loss terms, extra forward passes, or bespoke kernel modifications — the proposed fix removes a source of error: the low precision of BF16. The change is a single configuration switch (or a few lines of code) to use FP16 for both training and inference, relying on mature loss scaling techniques (Micikevicius et al., 2017) that are already built into all major frameworks.

This positioning is important because it reframes the problem. Prior work implicitly accepted BF16 as a fixed constraint and asked: "Given that we're using BF16, how can we correct for the mismatch?" This paper asks a more fundamental question: "Is BF16 the right precision for RL fine-tuning at all?" The answer it provides — no — has implications that extend beyond any specific algorithmic correction. If the mismatch can be eliminated at the precision level, then:

The complex importance sampling patches become unnecessary, eliminating their 25% overhead.
The deployment gap closes, because $\pi \approx \mu$ .
The bias-variance tradeoff dissolves — all algorithms converge to similar performance under FP16 (Section 4.3, Figure 4).
Even the simplest, classic policy gradient estimator (Equation 5) becomes practical, dramatically outperforming all BF16 algorithmic corrections (Figure 3).

The paper is careful not to claim that FP16 is universally optimal. It acknowledges (Section 6) that BF16's wide dynamic range is genuinely valuable for pretraining, where gradient stability across diverse parameter scales matters enormously. The argument is specifically scoped to RL fine-tuning, where (1) the model's weight and activation ranges are already established from pretraining, making BF16's range less critical, and (2) the training-inference mismatch becomes a dominant source of error.

The paper also explicitly connects to the broader trend toward lower precision (FP8, INT8) and notes that FP16's tighter range may present engineering challenges for extremely large models. However, it frames these as solvable — citing recent successes in FP8 training — and argues that the community should reconsider whether BF16's default status in RL fine-tuning is justified, rather than accepting it as an immutable constraint.

3. Technical Approach

This is primarily an empirical diagnosis and remedy paper whose core idea is that the training-inference mismatch in RL fine-tuning is fundamentally a floating-point precision problem, not an algorithmic one — and that switching from BF16 to FP16 (a single configuration change) eliminates the mismatch at its source, rendering complex importance-sampling corrections unnecessary.

3.1 Reader orientation (approachable technical breakdown)

The "system" here is not a new piece of software but a precision configuration for existing RL fine-tuning frameworks. When you train an LLM with reinforcement learning, the system uses two different computational engines: a fast inference engine (like vLLM) to generate responses, and a training engine (like PyTorch FSDP or DeepSpeed) to compute gradients and update weights. These engines produce slightly different numerical outputs even when running the same model with the same weights, because they use different optimized CUDA kernels, different parallelization strategies, and different operation orders. The paper's solution is to switch the floating-point format used by both engines from BF16 (7 mantissa bits, low precision) to FP16 (10 mantissa bits, higher precision), so that the extra precision absorbs the implementation differences and the two engines produce nearly identical probability distributions. This solves both the biased gradient problem and the deployment gap in one stroke, without adding any algorithmic complexity or computational overhead.

3.2 Big-picture architecture (diagram in words)

The "architecture" of the proposed solution is a configuration stack with four layers:

Base Model (LLM): A pretrained language model (e.g., DeepSeek-R1-Distill-Qwen-1.5B, Qwen3-30B-A3B-Base, OctoThinker-3B) whose weights are already established from pretraining. The model's parameter ranges and activation scales are fixed — BF16's wide dynamic range is no longer critical.
Precision Configuration (FP16 for both paths): The single change that everything else depends on. Both the inference engine (used for rollout/generation) and the training engine (used for forward/backward passes) are configured to use FP16 precision. This is enabled by adding loss scaling (a mature technique from Micikevicius et al., 2017) to the training path, which prevents gradient underflow in FP16's limited exponent range. In modern frameworks, enabling FP16 + loss scaling typically requires changing a single configuration flag or a few lines of code.
RL Algorithm (any standard policy gradient method): Because FP16 reduces the mismatch to near-zero, complex algorithmic corrections become unnecessary. The paper shows that even the simplest unbiased policy gradient estimator with importance sampling (Equation 5) works well. The algorithm layer can be vanilla GRPO, PG-Seq-IS, or any standard method — the precision fix makes the choice of algorithm much less consequential.
RL Framework (VeRL or Oat): The distributed training infrastructure that orchestrates rollout, gradient computation, and parameter updates. The paper validates across two independent frameworks to rule out implementation-specific artifacts.

Information flows through these layers in a standard RL loop: a batch of prompts enters → the inference engine (FP16) generates responses autoregressively → the training engine (FP16) computes log-probabilities and policy gradients → the optimizer updates model weights → repeat. The critical property that FP16 provides is that the probability distributions computed by the inference engine ( $\mu$ ) and the training engine ( $\pi$ ) are numerically nearly identical at every step, so the gradient signal is unbiased and the optimized parameters are optimal for the deployment engine.

3.3 Roadmap for the deep dive

First, the formal problem statement and the mismatch equations (Section 2 recap with full mathematical detail): I'll walk through exactly how the biased gradient and deployment gap arise from $\mu \neq \pi$ , since understanding these equations is prerequisite to seeing why FP16 helps. This includes the REINFORCE estimator, the importance-sampling correction, and the GRPO-specific variants.
Second, the floating-point precision analysis (FP16 vs. BF16, Section 3): I'll explain the bit allocation, dynamic range, and precision of both formats, why BF16's 7-bit mantissa causes large rounding errors that compound autoregressively, and why FP16's 10-bit mantissa provides sufficient precision to absorb implementation differences. This includes the loss scaling mechanism that makes FP16 training stable despite its limited exponent range.
Third, the offline mismatch quantification (Section 3.5, Figure 2): I'll present the empirical evidence that FP16 reduces the sequence-level mismatch by ~24× (KL divergence of 0.32 vs. 7.64 bits), showing both the token-level probability scatter plots and the sequence-level log-ratio distributions that demonstrate the exponential growth of mismatch with sequence length under BF16.
Fourth, the sanity test construction (Section 4): I'll explain the filtering procedure that creates a perfectible dataset of 1,460 MATH problems (initial accuracy 20–80%), why this test is a sharp diagnostic for algorithm reliability, and what "passing" means (≥95% training accuracy).
Fifth, the algorithm comparisons under both precisions (Sections 4.2–4.3, Figures 1, 3, 4): I'll walk through the specific algorithms tested (vanilla GRPO, GRPO-Token-TIS, GRPO-Seq-MIS, GSPO, PG-Seq-IS, PG-Seq-MIS), their mathematical forms, their behavior under BF16 (bias-variance tradeoff, early collapse, deployment gap), and how all algorithms converge to similar performance under FP16.
Sixth, the ablation on precision combinations (Section 4.4, Figure 5): I'll explain the experiment that isolates training precision from inference precision, showing that both must be FP16 for optimal results, and that FP32 inference is stable but 3× slower.

3.4 Detailed, sentence-based technical breakdown

The Formal Mismatch: Why $\mu \neq \pi$ Matters

The paper's entire argument rests on a specific mathematical pathology that arises when the inference policy $\mu(\cdot|\theta)$ (the probability distribution over responses produced by the inference engine) diverges from the training policy $\pi(\cdot|\theta)$ (the distribution computed by the training engine during gradient calculation). In an ideal world with infinite numerical precision, $\mu = \pi$ identically, and there is no problem. In practice, with BF16 precision, they differ enough to corrupt optimization.

The RL objective. The standard RL fine-tuning objective is to maximize expected reward over prompts:

$J(\theta) = \mathbb{E}_{x \sim p_X}\left[ \mathbb{E}_{y \sim \pi(\cdot|x,\theta)}[R(x, y)] \right]$

where $x$ is a prompt sampled from a distribution $p_X$ , $y$ is a response sampled from the training policy $\pi(\cdot|x,\theta)$ , $R(x,y)$ is the scalar reward assigned to the response (e.g., 1 if the final answer is correct, 0 otherwise), and $\theta$ represents the model parameters being optimized.

What this represents: This is the expected reward if we could both train and deploy using the exact same probability distribution $\pi$ . The outer expectation is over prompts from the training distribution; the inner expectation is over responses the model would generate under its own training-time distribution.

Why this matters as a starting point: Everything that follows is about the gap between this ideal objective (which assumes $\mu = \pi$ ) and what actually gets optimized in practice (where $\mu \neq \pi$ ).

The REINFORCE gradient. The policy gradient of this objective, computed via the REINFORCE estimator (Williams, 1992), is:

$\nabla_\theta J(\theta) = \mathbb{E}_{x \sim p_X}\left[ \mathbb{E}_{y \sim \pi(\cdot|x,\theta)}\left[ \nabla_\theta \log \pi(y|x,\theta) \cdot R(x,y) \right] \right]$

where $\nabla_\theta \log \pi(y|x,\theta)$ is the score function — the gradient of the log-probability the training policy assigns to the generated response $y$ , and $R(x,y)$ is the reward acting as a scalar weight.

What this computes: The gradient that, when followed, increases the probability of high-reward responses and decreases the probability of low-reward ones. For each prompt $x$ , the estimator (1) samples a response $y$ from $\pi$ , (2) computes how much the log-probability of that response would change if we tweaked each parameter (the score), and (3) multiplies that direction by the reward — so parameters move more strongly toward responses that yielded higher reward.

Why this form: REINFORCE is the foundational policy gradient estimator. It is unbiased when samples come from the same policy whose gradient is being computed. The score function $\nabla_\theta \log \pi$ emerges from the identity $\nabla_\theta \pi = \pi \cdot \nabla_\theta \log \pi$ , which lets us move the gradient inside the expectation. The unbiasedness property is crucial — it guarantees that stochastic gradient descent converges to a local optimum of $J(\theta)$ .

The mismatch introduces bias. In practice, responses are not sampled from $\pi$ (the training engine) but from $\mu$ (the inference engine), because generating autoregressively through the training engine would be prohibitively slow. If we simply substitute $\mu$ for $\pi$ in the sampling step but continue using $\pi$ for the score function, we get a biased estimator:

$\nabla_\theta J_{\text{biased}}(x, \theta) = \mathbb{E}_{y \sim \mu(\cdot|x,\theta)}\left[ \nabla_\theta \log \pi(y|x,\theta) \cdot R(x,y) \right] \neq \nabla_\theta J(x, \theta)$

where $\mu(\cdot|x,\theta)$ is the inference policy — the distribution from which responses are actually drawn — and $\pi(\cdot|x,\theta)$ is the training policy used to evaluate log-probabilities.

What this computes: A gradient that uses the wrong distribution for sampling but the right distribution for scoring. The expectation is taken under $\mu$ (what the inference engine actually produces) but the score function evaluates $\pi$ (what the training engine would have produced). The result is not equal to the true gradient $\nabla_\theta J(x,\theta)$ — the estimator is biased, meaning it converges to a different objective than intended.

Why this inequality holds: The score function $\nabla_\theta \log \pi(y|x,\theta)$ depends on which responses are being scored. If $\mu$ produces a different distribution of responses than $\pi$ would, the average score function under $\mu$ differs from the average under $\pi$ , even though the score function formula is the same. This is the core mechanism: the mismatch in which responses get generated corrupts the gradient signal, because the gradient is an average over the responses that actually appear.

The deployment gap (separate from the gradient bias). Even if the gradient bias were corrected (which importance sampling can do in principle), a second problem persists. The parameters $\theta$ are optimized with respect to $\pi$ during training, but the model is deployed and evaluated using $\mu$ . The optimal parameters for $\pi$ are not necessarily optimal for $\mu$ :

$\arg\max_\theta \mathbb{E}_{x \sim p_X, y \sim \mu(\cdot|x,\theta)}[R(x,y)] \neq \arg\max_\theta \mathbb{E}_{x \sim p_X, y \sim \pi(\cdot|x,\theta)}[R(x,y)]$

What this states: The left-hand side is the deployment objective (maximize reward under the inference engine). The right-hand side is the training objective (maximize reward under the training engine). These two optimization problems have different solutions when $\mu \neq \pi$ , because the reward landscape — which responses get high reward and how likely they are — differs between the two distributions.

Why this cannot be fixed algorithmically: Importance sampling corrections (Section 2.1) re-weight the gradient to make it unbiased with respect to $\pi$ , but they do not change what $\pi$ is. The optimization still moves toward parameters that are good under $\pi$ . When the final model is deployed with $\mu$ , there is necessarily a gap between training performance and deployment performance. The only way to close this gap is to make $\mu \approx \pi$ — which is exactly what FP16 achieves.

Importance Sampling Corrections and Their GRPO-Specific Implementations

The unbiased importance-sampling estimator. A principled way to correct the biased gradient from Equation 3 is importance sampling. Instead of treating samples from $\mu$ as if they came from $\pi$ , we re-weight each sample by the probability ratio:

$\nabla_\theta J_{\text{pg-is}}(x) = \mathbb{E}_{y \sim \mu(\cdot|x,\theta')}\left[ \frac{\pi(y|x,\theta)}{\mu(y|x,\theta')} \nabla_\theta \log \pi(y|x,\theta) \cdot A(x,y) \right]$

where $\theta'$ denotes the parameters used during sampling (which may differ from $\theta$ in off-policy settings — $\theta'$ is from the previous iteration), $\frac{\pi(y|x,\theta)}{\mu(y|x,\theta')}$ is the per-sequence importance weight, and $A(x,y) = R(x,y) - B(x)$ is the advantage with baseline $B(x)$ for variance reduction.

What this computes: An unbiased estimate of the policy gradient under $\pi$ , using samples from $\mu$ . The importance weight $\frac{\pi}{\mu}$ compensates for the distributional mismatch: responses that are more likely under $\pi$ than $\mu$ get up-weighted, responses that are less likely get down-weighted. The baseline $B(x)$ typically is the average reward of other responses to the same prompt, which reduces variance without introducing bias (since its expectation is zero).

Why this is theoretically correct but practically problematic: The importance weight is a product of per-token probability ratios over the entire response sequence. For a response of length $|y|$ , this ratio compounds multiplicatively: $\frac{\pi(y|x,\theta)}{\mu(y|x,\theta')} = \prod_{t=1}^{|y|} \frac{\pi(y_t|x, y_{<t}, \theta)}{\mu(y_t|x, y_{<t}, \theta')}$ . Even small per-token discrepancies (which BF16 produces abundantly) multiply into enormous sequence-level ratios. The variance of this estimator scales with the variance of the importance weights, which can be extreme for long sequences — leading to unstable, high-variance gradient estimates.

Truncated Importance Sampling (TIS). To control the variance, TIS clips the importance weight to a maximum value $C$ :

$\nabla_\theta J_{\text{pg-tis}}(x) = \mathbb{E}_{y \sim \mu(\cdot|x,\theta')}\left[ \min\left(\frac{\pi(y|x,\theta)}{\mu(y|x,\theta')}, C\right) \cdot \nabla_\theta \log \pi(y|x,\theta) \cdot A(x,y) \right]$

where $C$ is the clipping threshold (set to 3 in the paper's experiments).

What this computes: The same re-weighted gradient, but any importance weight exceeding $C$ is capped at $C$ . This prevents a single high-weight sample from dominating the gradient estimate.

Why this trades bias for variance: Clipping introduces bias (the estimator is no longer exactly unbiased) but substantially reduces variance. The bias is typically small if $C$ is set reasonably, because extreme weights are rare and the clipping mostly affects outliers. However, when the mismatch $\mu \neq \pi$ is large (as under BF16), many samples hit the clipping threshold, and the accumulated bias can be substantial — this is one reason why TIS eventually collapses under BF16.

Masked Importance Sampling (MIS). An alternative to clipping is to zero out samples whose importance weight exceeds $C$ entirely:

$\nabla_\theta J_{\text{pg-mis}}(x) = \mathbb{E}_{y \sim \mu(\cdot|x,\theta')}\left[ \frac{\pi(y|x,\theta)}{\mu(y|x,\theta')} \cdot \mathbb{I}\left\{ \frac{\pi(y|x,\theta)}{\mu(y|x,\theta')} \leq C \right\} \cdot \nabla_\theta \log \pi(y|x,\theta) \cdot A(x,y) \right]$

where $\mathbb{I}\{\cdot\}$ is the indicator function — it is 1 if the importance weight is $\leq C$ and 0 otherwise.

What this computes: Only samples whose importance weight is within the acceptable range contribute to the gradient. Samples with extreme weights are dropped entirely.

Why this is more stable than TIS but slower: MIS eliminates the influence of extreme-weight samples completely, which is more conservative than clipping them. This makes training more stable (less likely to be derailed by a few outlier samples), but it effectively reduces the sample size — fewer responses contribute to each gradient estimate — which slows convergence.

GRPO and its variants. The dominant open-source RL framework (VeRL, Sheng et al., 2024) implements GRPO (Group Relative Policy Optimization, Shao et al., 2024) rather than a generic policy gradient. GRPO computes advantages within a group of $G$ responses to the same prompt and applies PPO-style clipping. The standard GRPO gradient (using the Dr.GRPO variant from Liu et al., 2025c, which removes length and difficulty biases) is:

$\nabla_\theta J_{\text{grpo}}(x) = \mathbb{E}_{y \sim \mu(\cdot|x,\theta')}\left[ \sum_{t=1}^{|y|} \nabla_\theta \min\left(r_t A_t, \text{clip}(r_t, 1 - \epsilon, 1 + \epsilon) A_t\right) \right]$

where $r_t = \frac{\pi(y_t|x, y_{<t}, \theta)}{\pi(y_t|x, y_{<t}, \theta')}$ is the per-token probability ratio between the current and sampling policies (both under $\pi$ , not $\mu$ — GRPO does not account for the training-inference mismatch), $A_t = R(x,y) - \frac{1}{G-1}\sum_{i=1}^{G-1} R(x,y_i)$ is the group-based advantage (reward minus the average reward of other responses in the group), and $\epsilon$ is the PPO clipping parameter.

What this computes: A gradient that, for each token, compares the probability the current policy assigns to that token against the probability the sampling policy assigned to it ( $r_t$ ). If $r_t > 1$ (the current policy likes this token more than the old policy did), and the advantage $A_t$ is positive, the gradient increases the token's probability — but the $\min$ and $\text{clip}$ operations prevent the update from being too aggressive.

Why GRPO is vulnerable to the mismatch: GRPO uses $\mu$ for sampling (because that's what the inference engine produces) but uses $\pi$ for both $r_t$ and $\pi(\cdot|\theta')$ . The ratio $r_t$ compares $\pi_{\text{current}}$ to $\pi_{\text{old}}$ , neither of which equals the actual sampling distribution $\mu$ . The mismatch between $\mu$ and $\pi$ is simply ignored, which introduces bias into both the sampling and the probability ratio computation.

GRPO with token-level TIS (Yao et al., 2025). This variant patches the GRPO gradient by multiplying a token-level importance weight:

$\nabla_\theta J_{\text{grpo-tok-tis}}(x) = \mathbb{E}_{y \sim \mu(\cdot|x,\theta')}\left[ \sum_{t=1}^{|y|} \min(\rho_t, C) \cdot \nabla_\theta \min\left(r_t A_t, \text{clip}(r_t, 1 - \epsilon, 1 + \epsilon) A_t\right) \right]$

where $\rho_t = \frac{\pi(y_t|x, y_{<t}, \theta')}{\mu(y_t|x, y_{<t}, \theta')}$ is the per-token importance ratio between the training and inference policies at the sampling parameters $\theta'$ .

What this computes: For each token, the GRPO gradient term is multiplied by a clipped importance weight that corrects for the difference between $\mu$ and $\pi$ at the sampling parameters. The idea is that tokens where $\mu$ and $\pi$ disagree are up-weighted or down-weighted to compensate.

Why this requires an extra forward pass: Computing $\rho_t$ requires evaluating $\pi(y_t|x, y_{<t}, \theta')$ — the training engine's probability for each token under the sampling parameters. This is not the same forward pass used for the gradient (which uses the current parameters $\theta$ ). The implementation must run the full response sequence through the training engine a second time, with the old parameters $\theta'$ frozen, to extract these token probabilities. This is the source of the ~25% computational overhead.

GRPO with sequence-level MIS (Liu et al., 2025a). This variant applies a single sequence-level mask to the entire GRPO gradient:

$\nabla_\theta J_{\text{grpo-seq-mis}}(x) = \mathbb{E}_{y \sim \mu(\cdot|x,\theta')}\left[ \rho \cdot \mathbb{I}\{\rho \leq C\} \cdot \sum_{t=1}^{|y|} \nabla_\theta \min\left(r_t A_t, \text{clip}(r_t, 1 - \epsilon, 1 + \epsilon) A_t\right) \right]$

where $\rho = \frac{\pi(y|x, \theta')}{\mu(y|x, \theta')}$ is the sequence-level importance ratio at the sampling parameters.

What this computes: The entire GRPO gradient term for a response is either included (if $\rho \leq C$ ) or zeroed out (if $\rho > C$ ). The inclusion/exclusion decision is based on the full-sequence probability ratio, not per-token ratios.

Why this is more stable than token-level TIS: The sequence-level ratio is unbiased in expectation (it's a proper importance weight), whereas the token-level corrections in the GRPO-Token-TIS implementation are applied as a multiplicative patch on top of an already-biased GRPO gradient, leading to a more complex bias structure. However, the sequence-level ratio has extremely high variance (as shown in Figure 2), which makes this estimator converge slowly — many batches have most or all responses masked out, effectively reducing the sample size.

Floating-Point Precision Analysis: FP16 vs. BF16

The paper's core contribution is identifying that the mismatch $\mu \neq \pi$ is fundamentally a floating-point precision problem. This section explains exactly why BF16 causes large mismatches and why FP16 avoids them.

How floating-point numbers work. A 16-bit floating-point number allocates its 16 bits between two components: exponent bits (which determine the range — how large or small values can be) and mantissa bits (which determine the precision — how finely values can be distinguished within that range). The value represented is approximately $\text{mantissa} \times 2^{\text{exponent}}$ . More exponent bits → larger dynamic range (can represent very large and very small numbers without overflow/underflow). More mantissa bits → higher precision (can distinguish nearby values more accurately).

BF16 (Brain Float 16). BF16 allocates 8 bits to the exponent and 7 bits to the mantissa. The 8-bit exponent matches the exponent range of FP32 (single precision, 32 bits), giving BF16 the same enormous dynamic range: the smallest positive normal value is approximately $1.2 \times 10^{-38}$ and the largest value is approximately $3.4 \times 10^{38}$ . However, the 7-bit mantissa provides relatively coarse precision: the next representable number above 1.0 is $1 + 2^{-7} \approx 1.007812$ , meaning values between 1.0 and 1.007812 are all rounded to one of these two numbers. The rounding error for a single operation can be up to ~0.4% of the value's magnitude.

FP16 (IEEE 754 half-precision). FP16 allocates 5 bits to the exponent and 10 bits to the mantissa. The 5-bit exponent provides a much smaller dynamic range: the smallest positive normal is approximately $6.1 \times 10^{-5}$ and the largest value is approximately $6.6 \times 10^{4}$ . This limited range makes FP16 susceptible to overflow (values exceeding $6.6 \times 10^4$ ) and underflow (values below $6.1 \times 10^{-5}$ rounding to zero). However, the 10-bit mantissa provides substantially higher precision: the next representable number above 1.0 is $1 + 2^{-10} \approx 1.000977$ . The rounding error for a single operation is at most ~0.05% of the value's magnitude — about 8× smaller than BF16's.

Why precision matters more than range for RL fine-tuning. During pretraining, the model's weights and activations span a wide range of magnitudes — some parameters are very small (e.g., in early layers) and some are very large (e.g., in later layers), and the optimizer's gradient updates can vary dramatically in scale. BF16's wide dynamic range is essential here because it prevents gradients from underflowing to zero or overflowing to infinity. However, during RL fine-tuning, the model's weight distribution is already established from pretraining. The parameters don't change radically — the learning rates are small (typically $1 \times 10^{-6}$ ), and the updates are modest relative to the existing weight magnitudes. The dynamic range of FP16 ( $6.1 \times 10^{-5}$ to $6.6 \times 10^4$ ) is sufficient for this regime. What does matter critically is precision, because the training-inference mismatch comes from small per-operation rounding errors that accumulate over thousands of operations in a forward pass.

How rounding errors cause policy divergence. A transformer forward pass involves hundreds of thousands of floating-point operations — matrix multiplications, attention softmax computations, layer normalizations, residual additions. Each operation rounds its result to the nearest representable value in the current precision. Under BF16, each rounding step can introduce an error of up to ~0.4%. Under FP16, the per-step error is ~0.05%. These errors compound as computation proceeds through the network. Crucially, the training engine and inference engine execute these operations in slightly different orders (due to different kernel implementations, parallelization strategies, and memory layouts). The rounding errors therefore accumulate differently in the two engines — even though both are using BF16, the pattern of rounding differs because the operation order differs. The result is that the final logit vectors (and hence the probability distributions after softmax) diverge.

Why autoregressive generation amplifies the mismatch. The divergence is not just per-token — it compounds across tokens. During autoregressive generation, each new token is sampled conditioned on all previous tokens. If the probability distribution for the first token differs slightly between $\mu$ and $\pi$ , the sampled first token may differ. Then the second token's distribution is conditioned on a different prefix in $\mu$ versus $\pi$ , causing an even larger divergence. This is the mechanism behind Figure 2's finding that the sequence-level log-probability ratio $\log \frac{\pi}{\mu}$ grows linearly with sequence length under BF16 (slope ≈ −1.01), while staying essentially flat under FP16 (slope ≈ −0.07). The key insight is that FP16's higher precision reduces per-token divergence enough that the autoregressive amplification is negligible — the mismatch stays small even for very long sequences.

Loss scaling: making FP16 training stable. FP16's limited exponent range means that small gradient values can underflow to zero — they fall below $6.1 \times 10^{-5}$ and become unrepresentable. This would cause the optimizer to miss important gradient signals, particularly for parameters with small-magnitude updates. Loss scaling (Micikevicius et al., 2017) is a simple technique that prevents this:

Multiply the loss by a large scaling factor $S$ before backpropagation. This scales up all gradients by $S$ , shifting small values into FP16's representable range.
Backpropagate normally — all intermediate gradients are now larger and stay above the underflow threshold.
Before the optimizer step, divide gradients by $S$ to restore the correct scale.

Modern frameworks implement dynamic loss scaling: the scaling factor $S$ starts at a large value (e.g., $2^{16}$ ) and is automatically adjusted — increased if no gradient overflows (NaN or infinity values) are detected for a number of consecutive steps, decreased immediately if an overflow occurs. This requires a global synchronization across all GPUs before each optimizer step to check for overflows and ensure the scaling factor is consistent, which adds a small communication overhead. However, this overhead is far smaller than the 25% computational cost of running an extra forward pass for importance sampling corrections.

The key design tradeoff. BF16 was adopted as the default for LLM training because it eliminates the need for loss scaling entirely — its 8-bit exponent makes overflow and underflow essentially impossible in practice. This was a genuine advance for pretraining, where gradient scales can vary enormously across training stages. However, the paper argues that for RL fine-tuning, this tradeoff is wrong: the dynamic range of BF16 is unnecessary (the model's weight distribution is already established), while the precision it sacrifices is exactly what's needed to prevent the training-inference mismatch. FP16 trades unnecessary range for critical precision, enabled by the mature and lightweight loss scaling infrastructure that all modern frameworks already support.

Offline Mismatch Quantification (Section 3.5, Figure 2)

Before running any RL training, the paper conducts an offline analysis to quantify the magnitude of the training-inference mismatch under BF16 versus FP16, using the DeepSeek-R1-Distill-Qwen-1.5B model.

Experimental procedure for mismatch measurement. The authors sample 32 responses per question from the AMC and AIME benchmarks (Li et al., 2024) using the inference engine (vLLM) configured with temperature 1.0 and no top-p filtering. Temperature 1.0 is critical here — with lower temperatures or top-p filtering, the sampling process is not directly comparable to the training engine's full probability distribution, because temperature scaling and top-p truncation modify the distribution. At temperature 1.0 and no top-p, the inference policy $\mu$ is the raw softmax distribution over the full vocabulary, which can be directly compared to the training policy $\pi$ computed by running the same model weights through the DeepSpeed training engine.

Token-level probability comparison (Figure 2, left two plots). For each token in each generated response, the authors record two probabilities: $\mu(y_t|x, y_{<t})$ (what the inference engine predicted) and $\pi(y_t|x, y_{<t})$ (what the training engine predicted for the same token given the same prefix). These are plotted as a scatter plot with $\mu$ on the x-axis and $\pi$ on the y-axis. If there were no mismatch, all points would lie exactly on the diagonal $\mu = \pi$ .

Under BF16, the scatter plot shows substantial dispersion away from the diagonal, particularly in the mid-range (probabilities between 0.2 and 0.8). Many tokens show deviations of 0.1 or more — a token that the inference engine assigns 30% probability might get 40% from the training engine. Under FP16, the scatter plot is much tighter, with points densely concentrated around the diagonal and only small deviations visible.

What this means operationally: When the training engine computes $\nabla_\theta \log \pi(y|x,\theta)$ for a response sampled from $\mu$ , it is computing the gradient of the log-probability under a distribution that differs from the one that actually generated the response. The gradient signal is being applied to the wrong probability mass — parameters are updated to increase the probability of tokens under $\pi$ , but those tokens may not be the ones $\mu$ would have assigned high probability to. The larger the scatter, the more the gradient signal is misaligned with the actual generation process.

Sequence-level log-ratio analysis (Figure 2, right two plots). The authors compute, for each full response $y$ , the log probability ratio $\log \frac{\pi(y|x)}{\mu(y|x)}$ , which is the log of the importance sampling weight that would be used in Equation 5. This is plotted as a function of the response length (in thousands of tokens), with each point representing one response.

Under BF16, the log-ratios exhibit a clear linear trend with negative slope ≈ −1.01. For short responses (a few thousand tokens), the log-ratio is near zero — the mismatch is small. But as length increases, the log-ratio becomes increasingly negative, reaching values of −40 to −50 for responses of 20K–25K tokens. A log-ratio of −40 means $\frac{\pi}{\mu} \approx e^{-40} \approx 4 \times 10^{-18}$ — the training engine assigns virtually zero probability to responses that the inference engine actually generated. The overall sequence-level mismatch, measured by $\text{KL}[\mu \parallel \pi] = \mathbb{E}_{y \sim \mu}[\log \frac{\mu(y)}{\pi(y)}]$ , is 7.64 bits under BF16.

Under FP16, the log-ratios are clustered near zero regardless of sequence length, with slope ≈ −0.07 (essentially flat). Even for the longest responses, log-ratios stay within ±10, corresponding to importance weights between $e^{-10} \approx 4.5 \times 10^{-5}$ and $e^{10} \approx 2.2 \times 10^4$ . The KL divergence is only 0.32 bits — roughly 24× smaller than BF16.

Why this exponential growth under BF16 matters for RL. The importance weight $\frac{\pi}{\mu}$ directly determines the variance of the importance-sampling-corrected gradient estimator. When $\frac{\pi}{\mu}$ is near 1 (log-ratio near 0), the estimator is well-behaved — all samples contribute roughly equally to the gradient. When $\frac{\pi}{\mu}$ spans many orders of magnitude (as under BF16 for long responses), a few samples with extreme weights dominate the gradient estimate, making it essentially a single-sample estimator with enormous variance. This is why PG-Seq-IS and GRPO-Seq-MIS converge slowly under BF16 — most of the effective sample size is lost to variance.

Moreover, the fact that the mismatch grows with sequence length means that RL training systematically under-signals on long responses. Long responses — which are precisely what we want to incentivize for complex reasoning — receive importance weights near zero (because $\pi$ assigns them much lower probability than $\mu$ did), so their contribution to the gradient is effectively masked out. The training process becomes blind to the quality of long reasoning chains, precisely when it most needs to distinguish good long chains from bad ones.

Baseline performance comparison (Table 2). As a sanity check, the authors also sample responses with the standard decoding settings (temperature 0.6, top-p 0.95) and evaluate accuracy on AMC23 and AIME24 under BF16, FP16, and FP32 precisions. At both 8K and 32K token budgets, the performance is "largely comparable" across precisions — for example, AIME24 at 32K: BF16 = 29.90%, FP16 = 30.94%, FP32 = 28.44%. This confirms that higher inference precision alone does not improve performance — the benefit of FP16 comes from reducing the mismatch during training, not from more accurate inference per se. The inference-only precision matters mainly insofar as it affects the accumulated mismatch.

The Sanity Test: A Perfectible Dataset for Sharp Diagnosis (Section 4)

Standard benchmarks like the full MATH dataset contain a mix of problems: some are trivially easy for the base model (initial accuracy near 100%), some are unsolvable (initial accuracy near 0%), and some are in the "perfectible" range where good training can make a difference. The paper argues that this mixture makes it hard to diagnose algorithmic failures. If an RL run collapses, is it because the algorithm is flawed, or because the model simply cannot solve the hard problems and the optimizer is thrashing on impossible objectives? If an RL run achieves high accuracy, is it because the algorithm is effective, or because the model memorized trivially easy problems?

Dataset construction procedure. The authors construct a "perfectible" subset of MATH by:

For each problem in the MATH training set, unroll 40 responses from the base model (DeepSeek-R1-Distill-Qwen-1.5B) with standard decoding settings.
Compute the initial accuracy (fraction of the 40 responses that are correct) for each problem.
Keep only problems where the initial accuracy is between 20% and 80%.

This filtering yields 1,460 questions from the original MATH dataset.

Why these thresholds? Problems with initial accuracy below 20% are likely too hard — the model rarely produces correct solutions even with 40 attempts, meaning the reward signal is extremely sparse and the optimization has almost nothing to work with. Problems with initial accuracy above 80% are trivial — the model already solves them most of the time, so RL training provides little signal (the reward is almost always 1). The 20–80% band selects problems that are "perfectible": the model sometimes solves them and sometimes fails, meaning there is a genuine learning signal (some responses get reward 1, some get 0) and the model has the latent capability to improve (it can produce correct solutions, just not consistently).

The sanity test criterion. An RL algorithm "passes" the sanity test if its training accuracy on this perfectible dataset converges above a high threshold — the paper uses 95% as a representative value. The reasoning: if the dataset consists entirely of problems the model can solve (as demonstrated by the initial accuracy being ≥20%), a properly functioning RL algorithm should be able to guide the model to solve them consistently. The 95% threshold is not an absolute requirement (the paper doesn't claim all good algorithms must hit exactly 95%), but serves as a sharp diagnostic: "An algorithm that fails this test can be considered unreliable or fundamentally flawed, as it is unable to guide the model to solve problems known to be within its reach."

Why this test is efficient. With only 1,460 questions, achieving near-100% accuracy is computationally feasible within reasonable training budgets (a few thousand steps). Unlike full-scale RL training on 50K+ problems where training collapse might take tens of thousands of steps to manifest and is difficult to attribute to specific causes, the sanity test provides rapid feedback on whether an algorithm is fundamentally sound. The smaller dataset also means that hyperparameter sweeps are practical — one can test multiple configurations and algorithm variants with controlled computational cost.

Experimental configuration for the sanity test. All sanity test experiments use:

Base model: DeepSeek-R1-Distill-Qwen-1.5B
Context length: 8,000 tokens
Hardware: 8 NVIDIA A100 80GB GPUs
Batch size: 64 questions per policy iteration
Rollouts per question: 8 (so 512 total responses per iteration)
Gradient steps per iteration: 4 (following the standard PPO-style multi-epoch update)
For GRPO-family algorithms: clip_higher = 0.28 (following Yu et al., 2025)
For importance sampling methods: clipping threshold C = 3 (Equations 7 and 10)
Frameworks: VeRL (Sheng et al., 2024) and Oat (Liu et al., 2025b), run independently to validate robustness

Algorithm Comparisons Under BF16 (Section 4.2, Figure 3)

The paper evaluates six algorithms under BF16 precision on the sanity test, with results shown in Figure 3 (training reward curves over steps, AIME 2024 evaluation scores, and mismatch metrics). The algorithms span a spectrum from high-bias/low-variance to low-bias/high-variance.

Vanilla GRPO (Dr.GRPO variant). This baseline uses Equation 8 — it ignores the training-inference mismatch entirely, computing advantages and probability ratios entirely within $\pi$ while sampling from $\mu$ . Under BF16, vanilla GRPO collapses early: it reaches a peak training accuracy of only 73% in VeRL and 84% in Oat before performance degrades. The collapse is preceded by a growing mismatch — the policy difference $\pi(\cdot|\theta') - \mu(\cdot|\theta')$ diverges to extreme values, with one policy's probability approaching 1 while the other approaches 0 for the same token. The AIME 2024 evaluation score peaks at around 28–30% before declining.

GRPO with token-level TIS (Yao et al., 2025). Using Equation 9, this method applies per-token importance weight clipping. It trains longer than vanilla GRPO — reaching 82% (VeRL) and 88% (Oat) accuracy — but still eventually collapses. The token-level correction reduces but does not eliminate the bias, and the residual bias accumulates over enough training steps to derail optimization. The AIME 2024 evaluation score peaks higher than vanilla GRPO (around 32–34% in VeRL) but also degrades after the collapse point.

GSPO (Zheng et al., 2025). Although originally designed for MoE models, GSPO is included in the comparison. It demonstrates more stable training than GRPO-Token-TIS, achieving higher rewards for a longer period. However, in the VeRL experiment, the GSPO gradient norm becomes NaN after approximately 1200 steps, halting further model updates. The paper notes this as evidence that even relatively stable BF16 methods eventually hit numerical failure modes.

GRPO with sequence-level MIS (Liu et al., 2025a). Using Equation 10, this is the only BF16 method that maintains stable training without collapsing. It passes the sanity test by not degrading, and achieves a maximum training accuracy of 95%. However, the cost of this stability is slow convergence — the sequence-level importance ratio has enormous variance under BF16 (as quantified in Figure 2), so many batches have most responses masked out (their $\rho$ exceeds $C=3$ ). The method takes many more steps to reach its peak compared to the faster-but-collapsing alternatives. Critically, even at its peak, GRPO-Seq-MIS exhibits a deployment gap: 95% training accuracy versus 99% for the FP16 approach, and 34% (vs. 39%) on AIME 2024.

PG-Seq-IS (standard unbiased policy gradient with importance sampling). Using Equation 5 without any clipping or masking. This is the theoretically cleanest estimator — fully unbiased under importance sampling. Under BF16, it converges slowly for the same variance reason as GRPO-Seq-MIS, and its final performance is similar.

The bias-variance tradeoff pattern. The paper characterizes these results as revealing a fundamental tradeoff under BF16. Methods that are aggressive about reducing variance (by ignoring the mismatch, as in vanilla GRPO, or by applying token-level corrections, as in GRPO-Token-TIS) converge quickly but eventually collapse because the residual bias accumulates. Methods that properly correct for the mismatch (PG-Seq-IS, GRPO-Seq-MIS) avoid collapse but suffer from high variance that makes them converge slowly and leaves a deployment gap. The tradeoff arises because BF16 creates a mismatch so large that correcting for it requires importance weights spanning many orders of magnitude, which in turn creates enormous variance. Any method must choose between bias (fast but wrong) and variance (correct but slow).

Mismatch as an early-warning signal. A striking empirical regularity across all BF16 algorithms: those that eventually collapse consistently show a growing training-inference mismatch beforehand (visible in the third row of metrics in Figure 3). The policy difference $\pi(\cdot|\theta') - \mu(\cdot|\theta')$ converges to extreme values (approaching +1 or −1) — meaning that for some tokens, one policy assigns probability near 1 while the other assigns probability near 0, despite using what is supposed to be the same copy of model weights. The paper hypothesizes this is driven by a particular optimization bias but leaves full validation to future work. Importantly, this pattern provides a diagnostic: monitoring the mismatch during training can warn of impending collapse before performance degrades.

The Efficacy of FP16 (Section 4.2 continued, Figures 1, 3, 6)

The paper's central empirical claim is that simply switching both training and inference precision from BF16 to FP16 eliminates these problems. The evidence is presented in multiple forms.

Training stability and convergence speed. All algorithms trained under FP16 (shown in Figure 1, which presents training reward curves for 12 different settings spanning algorithms, model families, and frameworks) exhibit stable training without collapse. The reward curves increase smoothly and saturate at high values, without the sudden drops that characterize BF16 runs. Convergence is substantially faster — FP16 runs reach high rewards in fewer steps because the gradient estimates have lower variance (the importance weights are well-behaved, as shown in Figure 2).

Sanity test pass rates. Under FP16, even the simplest algorithm — PG-Seq-IS with standard importance sampling and no clipping — achieves 99% training accuracy on the perfectible dataset (Figure 3). This is 4 percentage points higher than the best BF16 method (GRPO-Seq-MIS at 95%), and achieved with faster convergence (the FP16 curve rises more steeply).

AIME 2024 evaluation. The FP16-trained PG-Seq-IS model achieves 39% on AIME 2024, compared to 34% for the best BF16 method (GRPO-Seq-MIS). This 5-percentage-point improvement on a challenging math benchmark demonstrates that the benefits of eliminating the mismatch translate to genuine deployment performance, not just training metrics.

The deployment gap closes. Because FP16 makes $\mu \approx \pi$ , the parameters optimized for the training engine are also near-optimal for the inference engine. The paper shows (Figure 6) that FP16-trained models consistently achieve higher evaluation scores than their BF16 counterparts across all tested settings, with the gap being particularly pronounced for algorithms that were most affected by the mismatch under BF16.

Why the unbiased estimator works under FP16. The key mechanism is visible in Figure 2: under FP16, the sequence-level importance weight $\frac{\pi}{\mu}$ stays close to 1 for responses of all lengths. This means the importance-sampling estimator in Equation 5 is genuinely practical — the weights are well-behaved, variance is low, and no clipping or masking is needed. The classical, theoretically clean estimator that was too unstable to use under BF16 becomes the best-performing method under FP16. This is a striking result: FP16 doesn't just improve existing algorithms, it makes the simplest possible algorithm work well, which in turn makes complex algorithmic patches unnecessary.

Framework-specific differences. While the core findings hold across both VeRL and Oat, the paper notes subtle differences (Section 4.2, "Framework-Specific Differences"). The initial training-inference mismatch is slightly smaller in Oat than in VeRL (the policy difference $\pi - \mu$ has a minimum near −0.9 in Oat versus −1.0 in VeRL under BF16). Even under FP16, where both frameworks exhibit a small mismatch, VeRL is more prone to occasional numerical spikes. The authors attribute these differences to the different distributed backends (DeepSpeed ZeRO in VeRL vs. PyTorch FSDP in Oat), which may have slightly different numerical properties in their communication and aggregation operations. Oat yields slightly higher training rewards, particularly for the algorithms that eventually collapse under BF16. These differences are minor and do not affect the main conclusion, but they highlight that the mismatch is sensitive to implementation details at a level below the algorithm design.

Algorithm Convergence Under FP16 (Section 4.3, Figure 4)

Having established that FP16 dramatically improves stability, the paper asks a natural follow-up: under FP16, does the choice of algorithm still matter? The answer, shown in Figure 4, is "barely."

All algorithms perform similarly. Figure 4 compares five algorithms (GRPO, GRPO-TIS, GRPO-Seq-MIS, GSPO, PG-Seq-IS) all trained with FP16. The training reward curves are nearly indistinguishable — they all converge to approximately the same high reward value. The AIME 2024 scores cluster closely: roughly 37–39% for most algorithms, with GRPO scoring slightly lower on AIME 2024 but slightly higher on AIME 2025. The response length dynamics are also similar across algorithms.

Why this happens. The paper attributes this convergence to FP16 fundamentally changing the nature of the optimization problem. Under BF16, the mismatch $\mu \neq \pi$ is large, which creates an effective off-policy setting — the samples come from a different distribution than the one being optimized. In this regime, different algorithms make different tradeoffs in how they handle the distributional shift (bias vs. variance), leading to divergent behaviors. Under FP16, the mismatch is so small that the optimization is effectively on-policy — $\mu \approx \pi$ . In the on-policy regime, all reasonable policy gradient estimators converge to similar solutions, because there is no distributional shift to correct for. The complex machinery of importance sampling corrections becomes unnecessary — the simplest estimator works fine.

Practical implication. This finding has an important practical consequence: when using FP16, practitioners do not need to carefully choose among RL algorithms based on their mismatch-handling properties. The choice of algorithm can be driven by other considerations (implementation simplicity, computational efficiency, compatibility with existing infrastructure) rather than by concerns about stability or bias correction. The paper does not claim that algorithm choice is entirely irrelevant — there may be differences in sample efficiency or final performance on specific benchmarks — but the dramatic algorithm-dependent instability that characterizes BF16 training is eliminated.

Precision Ablation Study (Section 4.4, Figure 5)

To isolate the effects of training precision from inference precision, the paper conducts an ablation study on the VeRL framework, systematically varying the precision used by the inference engine (vLLM) and the training engine (PyTorch FSDP).

Configurations tested:

fp32vllm-bf16fsdp: FP32 inference, BF16 training
fp16vllm-bf16fsdp: FP16 inference, BF16 training
fp16vllm-fp16fsdp: FP16 inference, FP16 training (the proposed configuration)
bf16vllm-bf16fsdp: BF16 inference, BF16 training (the baseline)

Results (Figure 5).

When training with BF16, increasing inference precision helps. The configuration with FP32 inference and BF16 training (fp32vllm-bf16fsdp) is fully stable with no signs of collapse. The mismatch (shown in the bottom panel of Figure 5) stays small throughout training, and the AIME 2024 evaluation score reaches competitive levels. The reason is that FP32 inference has 23 mantissa bits — enormously more precision than BF16's 7 — so the inference policy $\mu$ is computed with very high accuracy, and the mismatch between $\mu$ and the BF16 training policy $\pi$ is dominated by the training engine's lower precision rather than by divergences in both engines. FP16 inference with BF16 training (fp16vllm-bf16fsdp) also shows improved stability compared to the all-BF16 baseline, but not as much as FP32 inference.

However, FP32 inference is impractically slow. The rollout time (middle panel of Figure 5) for FP32 inference is approximately 300 seconds per iteration, compared to roughly 100–120 seconds for FP16 or BF16 inference — nearly 3× slower. For large-scale RL training where rollout time is often the bottleneck, this overhead is prohibitive.

FP16 for both training and inference is the optimal configuration. The fp16vllm-fp16fsdp configuration achieves the lowest mismatch, the most stable training dynamics, and nearly 100% training accuracy on the perfectible dataset, all without any inference slowdown. The rollout time is essentially identical to the BF16 baseline (since both are 16-bit formats with the same memory footprint and compute throughput on modern GPUs). This configuration delivers the stability benefits of FP32 inference without its speed penalty.

Key insight from the ablation. The ablation reveals that the mismatch is driven by the lower of the two precisions. BF16's 7-bit mantissa is the bottleneck — as long as either engine uses BF16, the mismatch is non-negligible. Switching both to FP16 (10-bit mantissa) eliminates the bottleneck, while switching only one side to an even higher precision (FP32, 23-bit mantissa) helps but leaves the BF16 side as the limiting factor. The practical optimum is therefore to use FP16 uniformly — it provides sufficient precision to make the mismatch negligible while maintaining the speed of 16-bit computation.

Generalization Experiments (Section 5, Figure 1 panels g–l, Figure 6)

Beyond the sanity test, the paper validates that the FP16 advantage generalizes across diverse settings.

MoE RL (Section 5.1). Mixture-of-Experts models pose additional challenges for training-inference mismatch because their routing decisions (top-k expert selection) are highly sensitive to small numerical differences — a tiny change in a router logit can change which experts are activated, leading to completely different computation paths. The authors train Qwen3-30B-A3B-Base (a 30B-parameter MoE with 3B active parameters) on DAPO-Math-17k using three algorithms: GRPO-Seq-MIS, GRPO-Token-TIS, and PG-Seq-TIS. Hyperparameters include: 8 nodes × 8 GPUs, batch size 512, 16 rollouts per prompt, max response length 20,480, learning rate $1 \times 10^{-6}$ , AdamW with betas $[0.9, 0.95]$ and epsilon $1 \times 10^{-15}$ , clip ratio high 0.28, clip ratio low 0.2, $C=3$ for importance sampling. As shown in Figure 1(i–k), all three algorithms achieve higher and more stable training rewards under FP16 than BF16, and the validation rewards on AIME 2024 (Figure 6(i–k)) are consistently higher for FP16-trained checkpoints.

LoRA RL (Section 5.2). Low-Rank Adaptation (Hu et al., 2022) has regained popularity for RL fine-tuning due to its parameter efficiency. The authors train Qwen2.5-Math-1.5B on the standard MATH dataset using GRPO-Token-TIS with LoRA applied to all layers (rank 32, scaling factor $\alpha = 64$ , learning rate $4 \times 10^{-5}$ , following Schulman and Lab, 2025). Under BF16 (Figure 1h), the LoRA training collapses after approximately 600 steps. Under FP16, it remains stable throughout training. This demonstrates that the precision effect is not specific to full fine-tuning — LoRA, which operates in a lower-dimensional parameter subspace, experiences the same mismatch pathology under BF16.

Large dense models (Section 5.3). To test whether the FP16 advantage scales to larger models, the authors train Qwen3-14B-Base using DAPO (Yu et al., 2025) on a curated math dataset of 54.4K problems (aggregated from OR1, DAPO, and DeepScaler, deduplicated and filtered). Hyperparameters: 8 nodes × 8 GPUs, batch size 512 (training) / 1536 (generation), max prompt length 2,048, max response length 20,480, 16 rollouts per prompt, learning rate $1 \times 10^{-6}$ with 10 warmup steps, weight decay 0.1, AdamW with betas $[0.9, 0.999]$ and epsilon $1 \times 10^{-8}$ , clip ratio high 0.28, clip ratio low 0.2, clip ratio C 10.0, overlong buffer enabled with length 4,096 and penalty factor 1.0, filter groups enabled with accuracy metric and max 10 gen batches. As shown in Figure 1(l), FP16 training rewards increase much faster than BF16, and the validation accuracy on AIME 2024 (Figure 6(l)) is higher for FP16-trained checkpoints throughout training.

Alternative model families (Section 5.4). To ensure the results are not specific to the Qwen model family, the authors train OctoThinker-3B (Wang et al., 2025b), a model mid-trained from Llama3.2-3B on reasoning-intensive data, using GRPO. Under BF16 (Figure 1g), training destabilizes after approximately 150 steps due to numerical mismatch. Under FP16, training continues smoothly without collapse. This demonstrates that the precision effect generalizes across model architectures and pretraining recipes.

Design Choices and Their Justifications

Why FP16 rather than FP32? FP32 (23 mantissa bits) would provide even higher precision and would certainly eliminate the mismatch. However, FP32 inference is approximately 3× slower than FP16/BF16 inference (Figure 5, middle panel), making it impractical for the rollout phase, which is often the throughput bottleneck in RL training. FP32 training is also slower and uses twice the memory of FP16. FP16 hits the sweet spot: sufficient precision to make the mismatch negligible (KL divergence 0.32 vs. 7.64 under BF16), while maintaining the speed and memory efficiency of 16-bit computation.

Why not mixed precision (FP16 training, BF16 inference, or vice versa)? The ablation study (Figure 5) shows that mismatched precisions leave a residual gap. If training uses FP16 but inference uses BF16 (or vice versa), the lower precision (BF16) becomes the bottleneck, and the mismatch persists. Uniform FP16 is necessary because both engines need sufficient mantissa bits to produce numerically similar results. The mismatch arises from differences in how the two engines round — if both use the same precision with sufficient mantissa, the rounding differences are small enough to be inconsequential.

Why does FP16 work for RL fine-tuning when BF16 was adopted for good reasons? The paper does not dispute that BF16 was a genuine advance for pretraining. During pretraining, (1) the model's weights and activations span a wide dynamic range that changes throughout training (from random initialization to converged values spanning many orders of magnitude), making BF16's 8-bit exponent genuinely necessary, and (2) there is no training-inference mismatch problem because pretraining uses the same engine for both forward and backward passes (there's no separate inference engine). RL fine-tuning differs on both counts: the weight range is already established (FP16's range is sufficient), and the training-inference mismatch becomes a dominant source of error (BF16's low precision is harmful).

Why loss scaling is acceptable despite its global synchronization overhead. The paper acknowledges that loss scaling requires a global synchronization before each optimizer step to check for gradient overflows and align the scaling factor across workers. This adds a small communication overhead. However, this overhead is dwarfed by the 25% computational cost of the extra forward pass required by the algorithmic corrections (Section 2.1.1). Moreover, loss scaling is a mature, well-optimized component in all major frameworks — it is not a new technique that requires bespoke implementation.

Why the sanity test is a better diagnostic than standard benchmarks. The sanity test's 20–80% accuracy filtering creates a dataset where every problem is solvable but not trivial. This eliminates two sources of noise: (1) unsolvable problems, where even a perfect algorithm cannot improve performance, muddying the comparison between algorithms, and (2) trivially easy problems, where any algorithm (even a bad one) achieves high accuracy, masking differences. The test provides a sharp signal: if an algorithm cannot achieve near-100% accuracy on a dataset of problems the model demonstrably can solve, the algorithm is fundamentally flawed. The 1,460-question size makes it computationally practical to run many experiments, enabling the comprehensive algorithm and precision comparisons in the paper.

Why multiple frameworks (VeRL and Oat) were used. The training-inference mismatch is sensitive to implementation details — different distributed backends (DeepSpeed ZeRO vs. PyTorch FSDP), different kernel libraries, and different communication patterns can all affect the magnitude and behavior of the mismatch. By replicating the core findings across two independent frameworks, the paper rules out the possibility that the results are artifacts of a particular implementation. The framework-specific differences that were observed (slightly smaller initial mismatch in Oat, occasional numerical spikes in VeRL under FP16) are noted transparently and do not affect the main conclusions.

4. Key Insights and Innovations

Innovation 1: Reframing the Training-Inference Mismatch as a Floating-Point Precision Problem, Not an Algorithmic One

The paper's most intellectually distinctive move is a reframing of what kind of problem the training-inference mismatch actually is. Prior work — exemplified by Yao et al. (2025), Liu et al. (2025a), and the broader RLHF/RLAIF community — treated the mismatch as an algorithmic problem: the training and inference engines produce different probability distributions, so the solution must be to correct the gradient estimator to account for this distributional shift. This framing led to increasingly sophisticated importance-sampling patches layered on top of GRPO (token-level TIS, sequence-level MIS), each trading bias for variance in a zero-sum game specific to the BF16 regime.

This paper shifts the frame entirely. It argues that the mismatch is fundamentally a numerical precision problem, not an algorithmic one. The root cause is not that we lack the right importance-sampling formula — it is that BF16's 7-bit mantissa produces per-operation rounding errors large enough that the training and inference engines' different operation orders cause their probability distributions to diverge, and this divergence compounds exponentially with sequence length (Figure 2: slope ≈ −1.01 in log-ratio vs. length, KL divergence of 7.64 bits). The algorithmic patches are, in this view, symptoms-management — they try to compensate for a pathology that should not exist in the first place. The paper's counterproposal is to eliminate the pathology at its source by switching to a precision format (FP16, 10-bit mantissa) where the per-operation rounding errors are small enough (~8× smaller) that the two engines' outputs stay numerically nearly identical regardless of implementation differences.

This reframing is significant for several reasons beyond the specific FP16 recommendation:

It changes what counts as a solution. Under the algorithmic framing, success means designing a gradient estimator that remains unbiased or stable despite $\mu \neq \pi$ . Under the precision framing, success means making $\mu \approx \pi$ in the first place. The latter is strictly stronger — it solves both the biased gradient and the deployment gap (Equation 4), whereas algorithmic corrections solve only the former by design. The paper's results bear this out: even the best BF16 algorithmic correction (GRPO-Seq-MIS) shows a deployment gap of 95% training accuracy vs. 99% under FP16, and 34% vs. 39% on AIME 2024 (Figure 3, Section 4.2).

It explains why prior engineering approaches failed. Attempts to manually align training and inference implementations (Team et al., 2025a) or to use FP32 for the LM head (Chen et al., 2025) were working within the BF16 paradigm — they accepted BF16 as a constraint and tried to compensate. The precision framing explains why these were insufficient: the mismatch accumulates throughout the entire transformer stack, not just in the final layer or in any single operation that can be aligned. The only way to solve it globally is to raise the precision floor everywhere, which is what uniform FP16 does.

It has predictive power for future work. If the precision framing is correct, then any future low-precision format (FP8, INT8, or custom formats) for RL fine-tuning must be evaluated not only on training stability (the usual criterion) but on whether it keeps the training-inference mismatch below some acceptable threshold. The paper's offline mismatch quantification methodology (Section 3.5, Figure 2) provides a diagnostic for this: measure the sequence-level log-probability ratio distribution between $\mu$ and $\pi$ as a function of sequence length. If the slope is far from zero, the precision is insufficient regardless of how stable individual training runs appear.

This is a fundamental reframing, not an incremental improvement. It changes the default question from "how do we correct for the BF16 mismatch?" to "why are we using BF16 for RL fine-tuning at all?" The paper is careful to scope this reframing — it does not claim BF16 is bad for pretraining or for inference-only deployment — but within the specific context of RL fine-tuning where both training and inference engines run simultaneously, the precision framing is a genuinely new lens on a problem the community had been attacking from the wrong angle.

Innovation 2: The Sanity Test as a Sharp Diagnostic for RL Algorithm Reliability

The paper introduces a novel evaluation methodology — the sanity test — that serves as a diagnostic instrument for RL algorithm design. This is not a benchmark in the traditional sense (it doesn't measure peak performance on held-out problems) but rather a capability probe: a specially constructed dataset where a properly functioning RL algorithm should, in principle, be able to achieve near-perfect training accuracy.

The construction is simple but surgically precise. From the MATH training set, the authors filter for problems where the base model's initial accuracy (over 40 rollouts) falls between 20% and 80%, yielding 1,460 questions. The 20% lower bound excludes problems the model fundamentally cannot solve (where no amount of optimization can create capability that isn't latent). The 80% upper bound excludes problems the model already solves trivially (where training signal is weak because almost all responses get reward 1). The remaining problems are perfectible: the model sometimes succeeds and sometimes fails, so there is genuine learning signal, and the model demonstrably possesses the latent capability to solve them.

The diagnostic logic is: if an RL algorithm cannot guide the model to solve these problems consistently (the paper uses 95% as a representative threshold), the algorithm is fundamentally unreliable — the failure is not due to problem difficulty or model capability limits, but to a flaw in the algorithm itself. Conversely, passing the sanity test is not a guarantee of universal success, but failing it is a strong indicator that something is wrong.

What makes this methodology novel relative to standard practice. The dominant evaluation paradigm in LLM RL research is to train on a large dataset (e.g., full MATH, 50K+ problems) and evaluate on held-out benchmarks (AIME, AMC, GPQA). This conflates multiple sources of variation: algorithmic quality, problem difficulty distribution, model capability boundaries, and training stochasticity. When a training run collapses or underperforms, it is unclear whether the algorithm failed, the problems were too hard, or the hyperparameters were wrong. The sanity test disentangles these by controlling problem difficulty — every problem is known to be solvable — so that algorithmic failures become sharply visible.

This is analogous to unit testing in software engineering: before deploying an algorithm to a complex, heterogeneous benchmark, test it on a controlled set of inputs where the expected behavior is well-defined. The paper is essentially arguing that the field has been debugging RL algorithms in production (on full benchmarks) rather than in a controlled test environment.

Practical value beyond this paper. The sanity test methodology is reusable. Any research group developing a new RL algorithm for LLMs can construct a perfectible dataset for their specific base model by following the same 20–80% filtering procedure. This provides a rapid, computationally tractable (1,460 questions, manageable within a few thousand training steps) diagnostic that can catch fundamental flaws early, before scaling to expensive large-dataset experiments. The paper's own use of the sanity test to reveal the bias-variance tradeoff under BF16 (Section 4.2) — where different algorithms exhibit qualitatively different failure modes (early collapse vs. slow convergence) that would be partially obscured on a full benchmark — demonstrates the methodology's diagnostic power.

Why this is more than just a filtered dataset. The innovation is not the filtering procedure itself (which is straightforward) but the interpretive framework around it: the claim that an algorithm's ability to achieve near-100% accuracy on a perfectible dataset constitutes a necessary condition for algorithmic soundness. This reframes evaluation from "how high can this algorithm push the benchmark score?" to "does this algorithm have a fundamental flaw that prevents it from solving problems the model demonstrably can solve?" The latter question is more actionable for algorithm designers, because a negative answer directly motivates debugging, while a low benchmark score could have many explanations.

This is a methodological innovation rather than a theoretical or empirical one. It provides the field with a sharper tool for distinguishing algorithmic flaws from capability limitations — a distinction that is often blurred in standard benchmarks. Its significance lies in its potential to accelerate algorithmic research by enabling faster, more diagnostic feedback cycles.

Innovation 3: The Dissolution of the Bias-Variance Tradeoff Under FP16

The paper's empirical results reveal a phenomenon that is more surprising than a simple performance improvement: under FP16, the bias-variance tradeoff that characterizes BF16 RL training effectively disappears. This is not a claim that FP16 "works better" in a continuous sense — it is a claim that FP16 qualitatively changes the optimization landscape, collapsing the performance differences between algorithms that exhibit starkly divergent behaviors under BF16.

The evidence for this dissolution is Figure 4. Under BF16 (Figure 3), the six tested algorithms span a wide spectrum: vanilla GRPO collapses early (73–84% peak accuracy), GRPO-Token-TIS trains longer but still collapses (82–88%), GSPO is moderately stable until NaN, GRPO-Seq-MIS is stable but slow (95% peak), and PG-Seq-IS is stable but high-variance. The paper characterizes this spectrum as a fundamental bias-variance tradeoff (Section 6): methods that aggressively reduce variance (by ignoring or only partially correcting the mismatch) converge quickly but accumulate bias that eventually causes collapse; methods that properly correct the mismatch avoid collapse but suffer from high variance that slows convergence. Under BF16, the practitioner must choose between fast-but-fragile and slow-but-stable.

Under FP16, this tradeoff vanishes. All five algorithms shown in Figure 4 (GRPO, GRPO-TIS, GRPO-Seq-MIS, GSPO, PG-Seq-IS) produce training reward curves that are nearly indistinguishable. AIME 2024 scores cluster in the 37–39% range across algorithms. The response length dynamics are similar. The paper explicitly states: "the performance differences between algorithms become almost indistinguishable."

Why this is significant beyond "FP16 is better." This result demonstrates that the bias-variance tradeoff is not an inherent property of RL fine-tuning for LLMs — it is an artifact of BF16's low precision. Under BF16, the mismatch $\mu \neq \pi$ is large enough that the importance-sampling correction must be aggressive, which inflates variance to problematic levels. The tradeoff emerges because no algorithm can simultaneously achieve low bias (which requires accurate correction) and low variance (which requires small importance weights) when the underlying mismatch forces importance weights to span many orders of magnitude (as shown in Figure 2: log-ratios reaching −40 to −50 under BF16).

FP16 eliminates the mismatch at its source (KL divergence 0.32 vs. 7.64 bits), which means the importance weights stay near 1 even for long sequences (log-ratios within ±10, Figure 2). In this regime, there is no tension between bias and variance — the unbiased estimator (Equation 5) already has low variance because the importance weights are well-behaved. The optimization becomes effectively on-policy, and in an on-policy setting, all reasonable policy gradient estimators converge to similar solutions.

The practical implication is a simplification of the RL practitioner's decision space. Under BF16, choosing an RL algorithm for LLM fine-tuning required navigating a complex tradeoff space: which importance-sampling variant to use, what clipping threshold to set, whether to use token-level or sequence-level corrections, and how to balance convergence speed against collapse risk. Under FP16, the paper's results suggest these choices matter much less — practitioners can use the simplest algorithm (vanilla policy gradient with importance sampling, Equation 5) and expect performance comparable to or better than the most sophisticated BF16 methods, with none of the collapse risk. The complexity cost of RL fine-tuning drops substantially.

This finding also has implications for algorithm research. If algorithm choice matters primarily in regimes where the training-inference mismatch is large, then algorithm development should be evaluated under the precision regime where that mismatch is controlled. Otherwise, improvements that appear significant under BF16 may simply be better at compensating for BF16's artifacts, rather than being genuinely better optimization algorithms. The sanity test under FP16 provides a cleaner evaluation environment where algorithmic improvements must come from better optimization, not better mismatch compensation.

This is an empirical discovery with theoretical implications. The paper does not prove formally that the bias-variance tradeoff must dissolve under FP16 — it demonstrates it empirically and provides a mechanistic explanation (reduced mismatch → well-behaved importance weights → low variance for unbiased estimators). The result is intellectually distinctive because it shows that a single engineering change (precision format) can resolve a tradeoff that the field had been treating as fundamental and had invested substantial algorithmic effort in navigating.

Innovation 4: Identifying the Mismatch Growth Rate as a Diagnostic Signal for Training Health

The paper uncovers an empirical regularity that has practical value beyond the FP16 recommendation: algorithms that eventually collapse consistently exhibit a growing training-inference mismatch beforehand, making the mismatch magnitude a potential early-warning signal for training instability. This is not presented as a fully validated diagnostic tool — the paper explicitly states "further validation is required" — but the pattern is sufficiently consistent across algorithms and frameworks to constitute a meaningful empirical finding.

The evidence appears in Figure 3 (third row of metrics per framework). For vanilla GRPO and GRPO-Token-TIS under BF16 — both of which eventually collapse — the policy difference $\pi(\cdot|\theta') - \mu(\cdot|\theta')$ grows over training, eventually reaching extreme values where one policy's probability approaches 1 while the other's approaches 0 for the same token, despite both using what should be identical model weights. In contrast, the stable methods (GRPO-Seq-MIS under BF16, all methods under FP16) maintain a bounded mismatch throughout training. Under FP16, the mismatch stays small for all algorithms, and no collapse occurs.

Why this matters as a diagnostic concept. In standard RL training, collapse is often detected after the fact — the reward curve drops, evaluation scores degrade, and the damage is done. If the mismatch magnitude can serve as a leading indicator, practitioners could monitor it during training and intervene (e.g., reduce learning rate, switch precision, or roll back to an earlier checkpoint) before performance degrades. This is analogous to monitoring gradient norm or weight norm as early-warning signals for training instability in supervised learning.

The paper hypothesizes a mechanism: the growing mismatch may be driven by an optimization bias — the biased gradients (Equation 3) push parameters in directions that improve the objective under $\pi$ but increase the divergence between $\pi$ and $\mu$ . This creates a feedback loop: more bias → larger mismatch → more bias in the next gradient step, until the policies diverge completely and collapse occurs. The authors label this as requiring further validation, but the empirical regularity is clear enough to be reported.

Connection to the FP16 story. This finding strengthens the paper's central argument by showing that the mismatch is not just a static nuisance (a fixed level of noise in the gradient) but a dynamically evolving pathology under BF16. Algorithms that ignore or partially correct the mismatch are not just operating with a biased gradient — they are actively making the mismatch worse over time, digging themselves into a hole that eventually causes collapse. FP16 prevents this dynamic entirely by keeping the mismatch small from the start, so the feedback loop never initiates.

Limitations as a diagnostic. The paper does not establish a quantitative threshold for "dangerous" mismatch levels, nor does it demonstrate that monitoring mismatch enables successful early intervention. These are left as future work. The contribution at this stage is the identification of the pattern — the observation that mismatch growth precedes collapse — which provides a concrete direction for developing practical training monitoring tools.

This is a modest but genuine empirical discovery that emerges from the paper's systematic mismatch measurement methodology. It is not the paper's primary contribution (which is the precision-level solution), but it adds to the conceptual toolkit for understanding and diagnosing RL training instability, independent of whether practitioners adopt FP16. Even for those who continue using BF16, monitoring the mismatch could help catch impending collapse early.

5. Experimental Analysis

Evaluation Methodology

Dataset. The primary diagnostic tool is the paper's novel sanity test dataset: 1,460 questions filtered from the MATH training set (Hendrycks et al., 2021) where the base model DeepSeek-R1-Distill-Qwen-1.5B achieves initial accuracy between 20% and 80% (measured over 40 rollouts per question). For generalization experiments, the authors use DAPO-Math-17k (Yu et al., 2025) for MoE RL, the standard MATH dataset for LoRA RL, and a curated math dataset of 54.4K problems (aggregated from OR1, DAPO, and DeepScaler, deduplicated and filtered; Cheng et al., 2025) for large dense model experiments. Evaluation benchmarks include AIME 2024, AIME 2025, and AMC23 (Li et al., 2024), with the specific benchmark per experiment noted in each result section.
Base model(s). The primary model for the sanity test and offline analysis is DeepSeek-R1-Distill-Qwen-1.5B (Guo et al., 2025), chosen because it represents a widely-used open-source reasoning model with non-trivial but imperfect MATH performance (~10–20% pass@1 depending on configuration), leaving room for RL fine-tuning to demonstrate improvement. Generalization experiments use Qwen3-30B-A3B-Base (MoE architecture), Qwen2.5-Math-1.5B (for LoRA experiments), Qwen3-14B-Base (large dense model), and OctoThinker-3B (Wang et al., 2025b, mid-trained from Llama3.2-3B on reasoning data) to test across model families, scales, and architectures. For the offline analysis (Table 2), the same DeepSeek-R1-Distill-Qwen-1.5B model is used with recommended decoding settings (temperature 0.6, top-p 0.95).
Metrics. Training reward: the fraction of generated responses that receive a positive reward (typically correctness-based, i.e., 1 if the final answer matches ground truth, 0 otherwise), reported per training step. Training accuracy: the fraction of prompts in the sanity test dataset for which the model's selected response is correct, used to determine whether an algorithm "passes" the sanity test (threshold: ≥95%). Evaluation accuracy: for held-out benchmarks (AIME 2024, AIME 2025, AMC23), computed as the fraction of questions answered correctly, reported using either avg@32 (MoE experiments), avg@8 (large dense model experiment), or the best-of-N metric specified in each experiment. Training-inference mismatch: measured via three related metrics — the mean absolute difference $\mathbb{E}[|\pi - \mu|]$ , the policy difference range $\max(\pi - \mu)$ and $\min(\pi - \mu)$ , and the KL divergence $\text{KL}[\mu \parallel \pi]$ (Figure 2 offline, Figure 3, Figure 5). Response length: average number of tokens in generated responses, tracked to monitor whether training induces length hacking.
Baselines. The paper evaluates a suite of algorithms that represent the state of prior work on addressing the training-inference mismatch:
- Vanilla GRPO (specifically the Dr.GRPO variant from Liu et al., 2025c, Equation 8): standard GRPO with no mismatch correction.
- GRPO with token-level TIS (Yao et al., 2025, Equation 9): GRPO augmented with per-token truncated importance sampling weights.
- GRPO with sequence-level MIS (Liu et al., 2025a, Equation 10): GRPO augmented with a sequence-level masked importance sampling gate.
- GSPO (Zheng et al., 2025): group sequence policy optimization, originally designed for MoE models.
- PG-Seq-IS (Equation 5): standard unbiased policy gradient with sequence-level importance sampling.
- PG-Seq-MIS (Equation 7): policy gradient with sequence-level masked importance sampling.
All baselines are run under BF16 precision (the default in modern frameworks) to establish the performance floor. The proposed approach is these same algorithms run under FP16 precision, with no algorithmic modifications.
Generation budget / compute accounting. For the sanity test, each policy iteration uses a batch of 64 questions with 8 rollouts per question (512 total responses), followed by 4 gradient steps. For MoE RL: batch size 512, 16 rollouts per prompt, max response length 20,480 tokens, 8 nodes × 8 GPUs. For large dense model RL: training batch size 512, generation batch size 1,536, max prompt length 2,048, max response length 20,480, 16 rollouts per prompt, 8 nodes × 8 GPUs. The paper argues that the proposed FP16 approach eliminates the ~25% computational overhead incurred by BF16 algorithmic corrections (which require an extra forward pass to compute importance sampling ratios, Section 2.1.1), but does not report wall-clock training times or FLOP counts for the main RL experiments. An exception is Figure 5, which reports rollout time (seconds per iteration) to quantify the speed penalty of FP32 inference (~3× slower than FP16/BF16). All experiments use a single configuration change (training and inference precision to FP16) with no modifications to batch sizes, learning rates, or other hyperparameters.
Cross-validation / statistical protocol. No formal cross-validation or statistical significance testing is reported. The sanity test serves as the primary reliability diagnostic: an algorithm "passes" if it achieves ≥95% training accuracy on the perfectible dataset. To rule out framework-specific artifacts, core experiments are replicated across two independent frameworks: VeRL (Sheng et al., 2024, using DeepSpeed ZeRO backend) and Oat (Liu et al., 2025b, using PyTorch FSDP backend). The paper notes and transparently reports framework-specific differences (e.g., slightly smaller initial mismatch in Oat, occasional numerical spikes in VeRL under FP16) but does not run multiple seeds or report variance across runs for individual configurations. The perfectible dataset construction itself uses 40 rollouts per question to estimate initial accuracy, which provides a reasonably stable difficulty estimate but is not cross-validated across different random seeds or model checkpoints.

Main Quantitative Results

Offline Mismatch Quantification (Section 3.5, Figure 2, Table 2)

Headline result: Under BF16, the sequence-level training-inference mismatch grows exponentially with response length (log-ratio slope ≈ −1.01 per 1K tokens, KL divergence = 7.64 bits), while under FP16 the mismatch is approximately 24× smaller and nearly independent of sequence length (slope ≈ −0.07, KL divergence = 0.32 bits).

The paper first confirms that inference-only precision does not meaningfully affect performance: when sampling with standard decoding settings (temperature 0.6, top-p 0.95), AIME24 accuracy at 32K token budget is 29.90% (BF16), 30.94% (FP16), and 28.44% (FP32) (Table 2). These differences are "largely comparable," establishing that higher precision during inference alone does not improve the model's raw capability.

The mismatch analysis uses temperature 1.0 with no top-p filtering (so that $\mu$ reflects the raw softmax distribution, directly comparable to $\pi$ computed through the training engine). For each of 32 responses per question from AMC and AIME benchmarks, the authors record token-level probabilities from both the inference engine (vLLM) and the training engine (DeepSpeed) using the same model weights.

Token-level scatter (Figure 2, left two plots): Under BF16, the scatter plot of $\mu$ vs. $\pi$ shows substantial dispersion away from the diagonal, with many tokens in the mid-probability range (0.2–0.8) deviating by 0.1 or more. Under FP16, points are tightly concentrated around the diagonal. This directly visualizes the per-token probability discrepancies that cause biased gradients.

Sequence-level log-ratio (Figure 2, right two plots): For each full response $y$ , the log importance weight $\log \frac{\pi(y|x)}{\mu(y|x)}$ is plotted against response length. Under BF16, a clear linear trend emerges with negative slope ≈ −1.01: responses of 20–25K tokens have log-ratios reaching −40 to −50, corresponding to importance weights of $e^{-40} \approx 4 \times 10^{-18}$ — the training engine assigns essentially zero probability to responses the inference engine generated. The overall KL divergence $\text{KL}[\mu \parallel \pi] = 7.64$ bits. Under FP16, the points cluster near zero across all sequence lengths, with slope ≈ −0.07 (essentially flat). Even the longest responses stay within log-ratios of ±10 (importance weights between ~ $4.5 \times 10^{-5}$ and ~ $2.2 \times 10^4$ ). The KL divergence is 0.32 bits — roughly 24× smaller.

Why this matters for RL: The sequence-level importance weight $\frac{\pi}{\mu}$ directly determines the variance of the importance-sampling-corrected gradient. Under BF16, weights spanning ~18 orders of magnitude make the estimator essentially single-sample (dominated by a few extreme-weight responses). Under FP16, weights stay well-behaved, making the unbiased estimator practical.

Sanity Test: Algorithm Comparisons Under BF16 (Section 4.2, Figure 3)

Headline result: Under BF16, all tested algorithms either collapse (vanilla GRPO, GRPO-Token-TIS, GSPO) or converge slowly with a deployment gap (GRPO-Seq-MIS at 95% training accuracy vs. 34% on AIME 2024), revealing a fundamental bias-variance tradeoff. Under FP16, the simplest unbiased estimator (PG-Seq-IS) achieves 99% training accuracy and 39% on AIME 2024, outperforming all BF16 methods.

The sanity test experiments use DeepSeek-R1-Distill-Qwen-1.5B on the 1,460-question perfectible dataset with 8,000-token context, 8 A100 GPUs, batch size 64 questions × 8 rollouts × 4 gradient steps. The GRPO-family algorithms use clip_higher = 0.28; importance sampling methods use C = 3. All methods are evaluated under BF16 precision first, then under FP16.

Vanilla GRPO (BF16) collapses early. In VeRL, it reaches a peak training accuracy of 73% before degrading; in Oat, 84%. The AIME 2024 evaluation score peaks around 28–30% (VeRL) before declining. The collapse is preceded by a growing policy difference $\pi - \mu$ that reaches extreme values (approaching ±1), visible in Figure 3 (third row, VeRL: Max&Min of $\pi - \mu$ ; Oat: KL[ $\mu \parallel \pi$ ]).

GRPO-Token-TIS (BF16) trains longer but still collapses. It reaches 82% (VeRL) and 88% (Oat) training accuracy before degrading. AIME 2024 peaks higher than vanilla GRPO at roughly 32–34% (VeRL) but also degrades after collapse. The token-level correction reduces but does not eliminate the bias; residual bias accumulates over training steps.

GSPO (BF16) demonstrates more stable training than GRPO-Token-TIS for a longer period. However, in VeRL, the gradient norm becomes NaN after approximately 1,200 steps, halting further updates. The paper notes this as evidence that even relatively stable BF16 methods eventually hit numerical failure modes.

GRPO-Seq-MIS (BF16) is the only BF16 method that maintains stable training without collapsing. It achieves a maximum training accuracy of 95% and an AIME 2024 score of 34%. However, convergence is slow — the sequence-level importance ratio has enormous variance under BF16, so many batches have most responses masked out. Critically, even at its peak, it shows a deployment gap: 95% training accuracy vs. 99% under FP16, and a 5-percentage-point deficit on AIME 2024 (34% vs. 39%).

PG-Seq-IS (BF16) behaves similarly to GRPO-Seq-MIS: stable but slow convergence, with similar final performance.

The bias-variance tradeoff pattern (explicitly discussed in Section 6). Methods with lower variance but higher bias (vanilla GRPO, GRPO-Token-TIS, GSPO) converge quickly initially but eventually collapse because residual bias accumulates over training steps. Methods with lower bias but higher variance (PG-Seq-IS, GRPO-Seq-MIS) are stable but converge slowly because the sequence-level importance weight variance under BF16 (Figure 2) reduces effective sample size. The tradeoff is forced by BF16's large mismatch: correcting for it requires importance weights spanning many orders of magnitude, which inflates variance; not correcting for it leaves bias that eventually destroys optimization.

Mismatch as an early-warning signal. A consistent empirical regularity: algorithms that eventually collapse show a growing training-inference mismatch beforehand (Figure 3, third row of metrics). The policy difference $\pi(\cdot|\theta') - \mu(\cdot|\theta')$ diverges to values near ±1 — one policy's probability approaching 1 while the other approaches 0 for the same token, despite identical weights. The paper hypothesizes an optimization bias feedback loop but leaves full validation to future work. Stable methods maintain a bounded mismatch throughout training.

Sanity Test: FP16 Performance (Section 4.2 continued, Figures 1, 3, 6)

Headline result: Switching to FP16 for both training and inference eliminates collapse for all algorithms, accelerates convergence, and enables the simple PG-Seq-IS estimator to achieve 99% training accuracy (vs. 95% for the best BF16 method) and 39% on AIME 2024 (vs. 34% for the best BF16 method).

Training stability (Figure 1, panels a–f; Figure 3, FP16 curves). All algorithms trained under FP16 exhibit smooth reward curves that increase and saturate without the sudden drops characteristic of BF16 runs. Convergence is faster — the FP16 curves rise more steeply than their BF16 counterparts for the same algorithm. None of the FP16-trained algorithms collapse within the tested training horizon (2,000–2,500 steps for sanity test experiments).

Sanity test pass rates (Figure 3). Under FP16, the simplest algorithm — PG-Seq-IS with standard importance sampling and no clipping (Equation 5) — achieves 99% training accuracy on the perfectible dataset, exceeding the 95% threshold by a wide margin. This is 4 percentage points higher than the best BF16 method (GRPO-Seq-MIS at 95%), and achieved with substantially faster convergence (the FP16 PG-Seq-IS curve rises more steeply than the BF16 GRPO-Seq-MIS curve).

AIME 2024 evaluation (Figure 3, middle row; Figure 6, panels a–f). The FP16-trained PG-Seq-IS model achieves 39% on AIME 2024, compared to 34% for the best BF16 method (GRPO-Seq-MIS). This 5-percentage-point gap demonstrates that eliminating the mismatch translates to genuine deployment performance improvement. Figure 6 confirms that FP16-trained checkpoints consistently achieve higher evaluation scores than their BF16 counterparts across all algorithms, with the gap particularly pronounced for algorithms that were most affected by mismatch under BF16.

Why PG-Seq-IS works under FP16. The mechanism is directly visible in Figure 2: under FP16, $\frac{\pi}{\mu} \approx 1$ for responses of all lengths. The importance-sampling estimator in Equation 5 becomes genuinely practical — weights are well-behaved, variance is low, and no clipping or masking is needed. The classical unbiased estimator that was too unstable to use under BF16 becomes the best-performing method.

Framework-specific differences (Section 4.2, "Framework-Specific Differences"). The initial training-inference mismatch is slightly smaller in Oat than in VeRL (policy difference $\pi - \mu$ minimum near −0.9 in Oat vs. −1.0 in VeRL under BF16). Even under FP16, where both frameworks exhibit a small mismatch, VeRL shows occasional numerical spikes that Oat does not. The authors attribute these to different distributed backends (DeepSpeed ZeRO vs. PyTorch FSDP) having slightly different numerical properties in their communication and aggregation operations. Oat yields slightly higher training rewards, particularly for algorithms that eventually collapse under BF16. These differences do not affect the main conclusion — FP16 uniformly outperforms BF16 in both frameworks.

Algorithm Convergence Under FP16 (Section 4.3, Figure 4)

Headline result: Under FP16, the performance differences between algorithms become "almost indistinguishable," with all tested methods (GRPO, GRPO-TIS, GRPO-Seq-MIS, GSPO, PG-Seq-IS) producing nearly identical training reward curves and clustering within ~2 percentage points on AIME benchmarks.

Figure 4 shows five algorithms all trained with FP16. Training reward curves are nearly superimposed — they rise at the same rate and saturate at the same value. AIME 2024 scores range from approximately 37% to 39% across algorithms, with GRPO scoring slightly lower on AIME 2024 but slightly higher on AIME 2025 (the paper notes this makes it "difficult to draw a definitive conclusion about its relative performance"). Response length dynamics are similar across all algorithms.

Why this convergence happens. Under BF16, the large mismatch $\mu \neq \pi$ creates an effective off-policy setting where different algorithms make different bias-variance tradeoffs, leading to divergent behaviors. Under FP16, $\mu \approx \pi$ (KL divergence 0.32 bits), so the optimization is effectively on-policy. In the on-policy regime, all reasonable policy gradient estimators converge to similar solutions because there is no distributional shift to correct for. The complex importance-sampling machinery becomes unnecessary — the simplest estimator works fine, and adding complexity (clipping, masking, token-level corrections) provides no additional benefit because there is nothing to correct.

Practical implication. When using FP16, practitioners can choose algorithms based on implementation simplicity and computational efficiency rather than mismatch-handling properties. The dramatic algorithm-dependent instability that characterizes BF16 training is eliminated.

Precision Ablation Study (Section 4.4, Figure 5)

Headline result: FP32 inference with BF16 training is stable but 3× slower (rollout time ~300 seconds vs. ~100–120 seconds for FP16/BF16). FP16 for both training and inference is the optimal configuration: lowest mismatch, fastest convergence, no speed penalty.

The ablation compares four precision configurations using the VeRL framework with PG-Seq-IS on the sanity test:

fp32vllm-bf16fsdp (FP32 inference, BF16 training): Fully stable with no signs of collapse. The mismatch stays small throughout training, and AIME 2024 reaches competitive levels. However, rollout time is approximately 300 seconds per iteration — nearly 3× slower than FP16 or BF16 inference (~100–120 seconds). The paper declares this combination "impractical for large-scale experiments."
fp16vllm-bf16fsdp (FP16 inference, BF16 training): Shows improved stability compared to the all-BF16 baseline (bf16vllm-bf16fsdp) but not as much as FP32 inference. The mismatch is intermediate.
fp16vllm-fp16fsdp (FP16 inference, FP16 training): The optimal configuration. Achieves the lowest mismatch (bottom panel of Figure 5: Max&Min of $\pi - \mu$ stays near zero), the most stable training dynamics, and nearly 100% training accuracy on the perfectible dataset, all with rollout time essentially identical to the BF16 baseline (~100–120 seconds).
bf16vllm-bf16fsdp (BF16 inference, BF16 training): The standard baseline. Exhibits growing mismatch and eventual performance degradation.

Key insight. The mismatch is driven by the lower of the two precisions. As long as either engine uses BF16 (7-bit mantissa), the mismatch remains non-negligible. Switching both to FP16 (10-bit mantissa) eliminates the bottleneck. Uniform FP16 provides sufficient precision to make the mismatch negligible while maintaining 16-bit computation speed.

Generalization Experiments (Section 5, Figure 1 panels g–l, Figure 6 panels g–l)

Headline result: The FP16 advantage generalizes across model architectures (dense, MoE), training regimes (full fine-tuning, LoRA), model families (Qwen, Llama-derived OctoThinker), model scales (1.5B to 30B parameters), and RL algorithms (GRPO variants, PG variants, DAPO), with FP16-trained models consistently achieving higher training rewards and evaluation scores than their BF16 counterparts across all tested settings.

MoE RL (Section 5.1, Figure 1i–k, Figure 6i–k). Training Qwen3-30B-A3B-Base on DAPO-Math-17k using GRPO-Seq-MIS, GRPO-Token-TIS, and PG-Seq-TIS, all three algorithms under FP16 achieve higher and more stable training rewards (Figure 1i–k) and consistently higher validation rewards on AIME 2024 (Figure 6i–k) compared to BF16. The improvement is consistent across algorithms, indicating FP16 mitigates the training-inference mismatch even in MoE architectures where routing decisions amplify numerical sensitivity. Detailed hyperparameters (Table 3): 8 nodes × 8 GPUs, batch size 512, 16 rollouts per prompt, max response length 20,480, learning rate $1 \times 10^{-6}$ , AdamW with betas $[0.9, 0.95]$ and epsilon $1 \times 10^{-15}$ , clip ratio high 0.28, clip ratio low 0.2, $C = 3$ for importance sampling, loss aggregation mode "seq-mean-token-sum-norm" (a corrected Dr.GRPO implementation).

LoRA RL (Section 5.2, Figure 1h, Figure 6h). Training Qwen2.5-Math-1.5B on the standard MATH dataset using GRPO-Token-TIS with LoRA (rank 32, $\alpha = 64$ , learning rate $4 \times 10^{-5}$ , applied to all layers), BF16 training collapses after approximately 600 steps, while FP16 remains stable throughout training. This demonstrates that the precision effect is not specific to full fine-tuning — LoRA, which operates in a lower-dimensional parameter subspace, experiences the same mismatch pathology under BF16. Figure 6h shows FP16 evaluation scores consistently above BF16 across the training horizon.

Large dense model RL (Section 5.3, Figure 1l, Figure 6l). Training Qwen3-14B-Base using DAPO (Yu et al., 2025) on a 54.4K-problem curated math dataset (aggregated from OR1, DAPO, DeepScaler; Cheng et al., 2025), FP16 training rewards increase substantially faster than BF16 (Figure 1l). FP16 achieves higher validation accuracy on AIME 2024 throughout training (Figure 6l). Detailed hyperparameters (Table 3): 8 nodes × 8 GPUs, training batch size 512, generation batch size 1,536, max prompt length 2,048, max response length 20,480, 16 rollouts per prompt, learning rate $1 \times 10^{-6}$ with 10 warmup steps, weight decay 0.1, AdamW with betas $[0.9, 0.999]$ and epsilon $1 \times 10^{-8}$ , clip ratio high 0.28, clip ratio low 0.2, clip ratio C 10.0, overlong buffer enabled (length 4,096, penalty factor 1.0), filter groups enabled (accuracy metric, max 10 gen batches).

Alternative model family (Section 5.4, Figure 1g, Figure 6g). Training OctoThinker-3B (Wang et al., 2025b, mid-trained from Llama3.2-3B on reasoning-intensive data) using GRPO, BF16 training destabilizes after approximately 150 steps due to numerical mismatch, while FP16 continues to train smoothly without collapse. Figure 6g shows FP16 evaluation scores rising throughout training while BF16 scores plateau or degrade after the destabilization point. This demonstrates the precision effect generalizes beyond the Qwen model family to Llama-derived architectures.

Ablation Studies and Robustness Checks

Precision combination ablation (Section 4.4, Figure 5): As detailed above, varying inference precision (FP32, FP16, BF16) against training precision (BF16, FP16) reveals that the mismatch is dominated by the lower of the two precisions. FP32 inference with BF16 training is stable but 3× slower (rollout time ~300 seconds vs. ~100–120 seconds). FP16 for both is the Pareto-optimal configuration: lowest mismatch and no speed penalty. The all-BF16 baseline (bf16vllm-bf16fsdp) shows growing mismatch and eventual degradation.
Framework replication (Section 4.2, Figure 3): Core sanity test experiments are replicated across VeRL (DeepSpeed ZeRO backend) and Oat (PyTorch FSDP backend). The main conclusions — FP16 eliminates collapse, outperforms all BF16 methods — hold in both frameworks. Subtle differences: initial mismatch is slightly smaller in Oat (policy difference min ≈ −0.9 vs. −1.0 in VeRL under BF16), and VeRL shows occasional numerical spikes under FP16 that Oat does not. The authors attribute these to backend-specific numerical properties and note they do not affect the main conclusions. The transparency about these differences strengthens the paper's credibility.
Algorithm convergence under FP16 (Section 4.3, Figure 4): This serves as an implicit ablation of algorithmic complexity: under FP16, adding importance-sampling corrections (token-level TIS, sequence-level MIS) to GRPO provides no meaningful benefit over vanilla GRPO, and the simple PG-Seq-IS estimator matches or exceeds all GRPO variants. This demonstrates that the algorithmic complexity introduced by prior work was compensating for BF16's precision artifacts, not providing fundamental optimization improvements.
Inference-only precision (Table 2): Evaluating DeepSeek-R1-Distill-Qwen-1.5B on AMC23 and AIME24 under BF16, FP16, and FP32 using standard decoding (temperature 0.6, top-p 0.95) shows "largely comparable" performance: AIME24 at 32K tokens is 29.90% (BF16), 30.94% (FP16), 28.44% (FP32). This confirms that higher inference precision alone does not improve model capability — the benefit of FP16 comes from reducing the training-inference mismatch during RL optimization, not from more accurate inference per se.
LoRA as a training regime variant (Section 5.2, Figure 1h): The collapse of LoRA-based RL under BF16 (~600 steps) versus stability under FP16 demonstrates that the mismatch is not specific to full fine-tuning. LoRA's reduced parameter count does not immunize it against precision-induced mismatch, likely because the mismatch accumulates in the forward pass (which involves the full model) regardless of which parameters are being updated.
Model family generalization (Section 5.4, Figure 1g): The OctoThinker-3B experiment (Llama-derived architecture) serves as an ablation of model family. BF16 destabilization at ~150 steps versus FP16 stability confirms the effect is not Qwen-specific.
Model scale generalization (Sections 5.1, 5.3): MoE experiments (30B total, 3B active) and dense experiments (14B) serve as ablations of model scale. The FP16 advantage persists at larger scales, suggesting the precision effect is not limited to small models where numerical issues might be expected to be more pronounced.
Implicit algorithmic ablation (Figure 1 all panels, Figures 3 and 4): Across 12 different experimental configurations (panels a–l in Figure 1) spanning 7 algorithm variants (GRPO, GRPO-Token-TIS, GRPO-Seq-MIS, GSPO, PG-Seq-IS, PG-Seq-MIS, DAPO), the FP16 advantage is uniform. This serves as a robustness check that the precision effect is not an artifact of a particular algorithm's interaction with precision — it is a property of the training-inference interface itself.

Critical Assessment

Does FP16 genuinely eliminate the mismatch, or just reduce it to a manageable level?

The paper's title and abstract claim that FP16 "effectively eliminates" and can "virtually eliminate" the training-inference mismatch. The evidence supports substantial reduction but not necessarily elimination. Figure 2 shows that under FP16, the sequence-level log-ratio slope is approximately −0.07 (vs. −1.01 under BF16) and KL divergence is 0.32 bits (vs. 7.64 bits). This is a ~24× reduction, which is dramatic, but the mismatch is not zero — the KL divergence of 0.32 bits represents a non-zero distributional difference, and Figure 2 (right) shows that some long-response log-ratios still deviate from zero by ±5–10 units under FP16. Whether this residual mismatch is small enough to be practically irrelevant is supported by the RL results (99% training accuracy, stable training), but the paper should be more precise: FP16 dramatically reduces the mismatch to a level where its effects on optimization become negligible, rather than "eliminating" it in an absolute sense. This distinction matters for future work on even lower-precision formats (FP8), where the residual mismatch might become relevant again.

Does the 24× mismatch reduction fully explain the performance improvement, or are there confounding factors?

The paper's causal chain is: BF16 causes large mismatch → mismatch causes biased gradients and deployment gap → biased gradients cause collapse/slow convergence. Switching to FP16 reduces mismatch → gradients become unbiased → training stabilizes. The evidence for each link is:

BF16 causes large mismatch: Strongly supported by Figure 2 (token scatter, sequence-level log-ratio, KL divergence).
Mismatch causes biased gradients: Supported theoretically (Equation 3) and by the observation that mismatch grows before collapse (Figure 3, third row).
Biased gradients cause collapse: Plausible but not directly tested. The paper does not run an experiment where mismatch is artificially varied while holding precision constant to isolate this causal link. For example: using BF16 but with a modified training engine that artificially inflates or reduces the mismatch would test whether mismatch magnitude directly predicts collapse probability.
FP16 reduces mismatch: Strongly supported by Figure 2.
Reduced mismatch causes stable training: Supported by the correlation between small mismatch (Figure 5, bottom panel) and stable training (Figure 5, top panel) across precision configurations, but again, no experiment cleanly isolates mismatch as the only varying factor.

An alternative partial explanation: FP16 may have other numerical properties (different rounding behavior, different accumulation patterns) that affect optimization dynamics independently of the mismatch. The paper's ablation (Figure 5) partially addresses this by showing that FP16 training with BF16 inference (fp16vllm-bf16fsdp) performs intermediately — better than all-BF16 but worse than all-FP16 — which is consistent with the mismatch explanation. But a cleaner test would be to use FP16 for both engines but with an artificial mismatch injected (e.g., by adding controlled noise to the inference engine's logits) to see whether stability degrades proportionally.

The sanity test is a strong diagnostic, but what does it actually measure?

The sanity test's filtering criterion (initial accuracy 20–80%) creates a dataset where the model can solve every problem (since accuracy ≥ 20% means correct solutions exist in the 40-rollout sample) but does not do so consistently. The test therefore measures: can the RL algorithm guide the model to produce correct solutions more consistently on problems within its capability range? This is a necessary condition for algorithmic soundness — if an algorithm cannot do this, it is fundamentally unreliable — but it is not sufficient. An algorithm that passes the sanity test might still fail on harder problems (where the model rarely produces correct solutions, making the reward signal extremely sparse) or exhibit other pathologies (catastrophic forgetting, reward hacking) that the perfectible dataset's controlled difficulty range does not expose.

The paper uses 95% as a representative passing threshold but does not justify this number theoretically or empirically. Is 90% insufficient? Is 99% meaningfully different from 95%? The threshold is somewhat arbitrary, and the paper would be stronger if it discussed what performance on the sanity test predicts about performance on standard benchmarks, or if it varied the threshold to show robustness.

The generalization experiments are broad but thin.

The paper shows FP16 outperforming BF16 across 12 different experimental configurations (Figure 1, all panels). This breadth is impressive and convincing — the effect is clearly not a fluke of one model, one algorithm, or one framework. However, each configuration is typically run once (no multiple seeds reported), and the experiments are relatively short — many BF16 runs collapse within a few hundred to a few thousand steps, so the comparison is between "FP16 training smoothly" and "BF16 collapsing." The paper does not demonstrate that FP16-trained models continue to improve beyond the point where BF16 would have collapsed, or that the final FP16 models are substantially better than the best BF16 checkpoint (which could be saved before collapse). The AIME evaluation curves (Figure 6) partially address this by showing FP16 evaluation scores continuing to rise while BF16 scores plateau or decline, but longer training runs would strengthen the case that FP16's benefits compound over time rather than just preventing early collapse.

The computational cost comparison is incomplete.

The paper argues that FP16 eliminates the ~25% overhead of algorithmic corrections (the extra forward pass for importance sampling ratios). This is a valid point, but the paper does not provide a complete cost accounting:

Loss scaling overhead: FP16 training requires dynamic loss scaling, which adds a global synchronization before each optimizer step to check for gradient overflows and align the scaling factor across workers. The paper acknowledges this (Section 3.3) but does not quantify the overhead. In large distributed settings (8 nodes × 8 GPUs as used in the paper's MoE and large-model experiments), this synchronization could be non-trivial.
Throughput comparison: The paper does not report tokens-per-second or wall-clock time for FP16 vs. BF16 training. Since both are 16-bit formats, the compute throughput should be identical on modern GPUs (both use the same tensor core instructions), but memory bandwidth and communication patterns might differ slightly. A throughput comparison would confirm that FP16 provides its stability benefits without hidden speed penalties.
Memory usage: FP16 and BF16 use the same 2 bytes per value, so memory footprint should be identical. The paper could state this explicitly.

The missing FP8 discussion.

The paper mentions FP8 in Section 6 as a direction for even lower precision but does not run FP8 experiments. Given that modern hardware (H100, H200) has native FP8 support and several large-scale training efforts are moving to FP8, a natural question is whether FP8's even lower mantissa precision would reintroduce the mismatch problem. An offline analysis of FP8 mismatch (analogous to Figure 2) would have been a valuable addition, even if full RL training experiments were beyond scope.

The "early-warning signal" claim is suggestive but not validated.

The observation that mismatch grows before collapse (Figure 3, third row) is intriguing and potentially practically useful, but the paper explicitly states "further validation is required" and does not demonstrate that monitoring mismatch enables successful intervention (e.g., reducing learning rate, rolling back to an earlier checkpoint). Without such a demonstration, the claim remains an empirical regularity rather than a validated diagnostic tool. The paper would be stronger if it either (a) demonstrated a successful intervention triggered by mismatch monitoring, or (b) scoped the claim more modestly as an observation that motivates future work.

The paper does not disentangle mantissa bits from other FP16 properties.

FP16 and BF16 differ in more than mantissa precision — they have different exponent ranges, different rounding behavior (FP16 uses round-to-nearest-even by default; BF16 uses round-to-nearest-even as well, but the larger rounding unit changes the practical behavior), and potentially different hardware kernel implementations. Could some of FP16's benefit come from its smaller dynamic range acting as implicit gradient clipping, rather than from higher precision per se? This is unlikely (loss scaling explicitly prevents underflow, so the dynamic range difference shouldn't matter for gradients), but the paper does not directly test it. A hypothetical experiment using a custom 16-bit format with BF16's exponent range and FP16's mantissa would isolate the mantissa effect, but this hardware does not exist. The paper's precision ablation (Figure 5) comes closest by varying precision independently for training and inference, and the results are consistent with the mantissa-precision explanation.

Claims vs. evidence summary.

Claim: "FP16 effectively eliminates the training-inference mismatch." Supported with qualifications. The mismatch is reduced ~24× (KL: 0.32 vs. 7.64 bits) but not zero. The reduction is sufficient to stabilize RL training across all tested settings.

Claim: "FP16 enables stable training without algorithmic corrections." Strongly supported. All FP16-trained algorithms (Figures 1, 3, 6) train stably; none collapse. The simplest unbiased estimator (PG-Seq-IS) works well.

Claim: "FP16 outperforms all BF16 algorithmic corrections." Supported on the tested benchmarks (sanity test: 99% vs. 95%; AIME 2024: 39% vs. 34%). The comparison is between FP16 with no algorithmic corrections and BF16 with the best available corrections. The gap is clear and consistent.

Claim: "The bias-variance tradeoff dissolves under FP16." Supported by Figure 4 (all algorithms converge to similar performance) and the mechanistic explanation (well-behaved importance weights under FP16 eliminate the tension between bias correction and variance). This is an empirical finding, not a theoretical proof.

Claim: "The deployment gap closes under FP16." Supported indirectly: FP16 makes $\mu \approx \pi$ , so the parameters optimal for $\pi$ should also be near-optimal for $\mu$ . The AIME evaluation improvements (Figure 6) are consistent with this, but the paper does not run an experiment that directly measures the deployment gap (e.g., comparing training reward vs. evaluation reward at matched checkpoints) across precisions.

Overall. The paper's central empirical claims are well-supported by the presented experiments. The evidence is consistent, replicated across frameworks, and generalizes across models and algorithms. The main limitation is that the causal chain (precision → mismatch → gradient bias → collapse) is inferred from correlations rather than tested through direct experimental manipulation, and the paper does not fully characterize the residual mismatch under FP16 or its implications for even lower-precision formats. Despite these limitations, the empirical case for preferring FP16 over BF16 for RL fine-tuning is compelling and actionable.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Unaccounted For and Prohibitively Expensive

The assumption or constraint. The sanity test dataset is constructed by unrolling 40 responses per question from the base model and computing the initial accuracy — problems with 20–80% accuracy are retained as "perfectible." The paper acknowledges that this filtering is essential to the diagnostic framework: "On this perfectible dataset, a reliable RL algorithm should theoretically be able to achieve 100% training accuracy" (Section 4). However, the paper does not account for the cost of constructing this dataset in any budget calculation, nor does it provide a method for identifying perfectible problems without this expensive pre-computation. The sanity test is treated as a fixed evaluation instrument whose construction cost is externalized.

The consequence. In a deployment or research setting where a new base model is being evaluated, constructing a perfectible dataset requires generating and scoring 40 × (full dataset size) responses. For the MATH training set (12,000 questions per the Lightman et al., 2022 split), this means generating 480,000 responses and evaluating correctness — a computation that may rival or exceed the cost of an RL training run itself. The 1,460-question dataset used in the paper represents only those problems that passed the 20–80% filter; the remaining ~88% of MATH problems were generated and discarded. More critically, the sanity test's diagnostic value depends on the 20–80% band accurately capturing "perfectibility" — if the base model changes (e.g., a different pretrained checkpoint, a different model family), the filtering must be re-run. The paper provides no guidance on whether cheaper alternatives exist (e.g., using a smaller number of rollouts, using the PRM's confidence as a proxy, or reusing difficulty estimates across related models).

What evidence exists in the paper. The sanity test construction is described in Section 4: "we unroll 40 responses for each problem in the MATH dataset, and only keep problems where the initial accuracy is between 20% and 80%. This process yielded a targeted dataset of 1,460 questions." The computational cost of this unrolling is never quantified. The paper explicitly states that the smaller dataset size "makes achieving near-100% accuracy computationally feasible, allowing for efficient and conclusive testing" — but this refers to the RL training cost on the filtered dataset, not the dataset construction cost that precedes it.

Mitigation status. The paper does not address this limitation. The sanity test is presented as a methodological contribution (a better diagnostic instrument), but the cost of deploying that instrument is not discussed. A practitioner wanting to adopt the sanity test for their own model would need to independently absorb the dataset construction cost. The paper does not suggest approximations (e.g., using fewer rollouts, leveraging difficulty estimates from related models, or using lightweight confidence proxies). This limitation is particularly significant because the sanity test is the paper's primary evaluation framework — all algorithm comparisons in Section 4 depend on it — yet its construction cost is invisibly amortized.

The Residual Mismatch Under FP16 Is Non-Zero and Not Fully Characterized

The assumption or constraint. The paper's title and abstract claim that FP16 "effectively eliminates" and can "virtually eliminate" the training-inference mismatch. However, the empirical evidence in Figure 2 shows that the mismatch is substantially reduced but not zero. Under FP16, the KL divergence between µ and π is 0.32 bits (vs. 7.64 bits under BF16), and the sequence-level log-ratio slope is −0.07 (vs. −1.01 under BF16). A KL divergence of 0.32 bits represents a genuine, nonzero distributional difference — it is ~24× smaller than BF16's, but not zero. The paper does not investigate whether this residual mismatch can grow under certain conditions (longer sequences than the 25K tokens tested, different model architectures, different decoding strategies) or whether it imposes a ceiling on how far FP16-based RL training can scale.

The consequence. The residual mismatch might become significant in regimes not tested in the paper: (1) extremely long responses beyond 25K tokens, since even a flat slope of −0.07 accumulates over sufficient length; (2) models with different numerical properties (e.g., deeper networks, different normalization schemes, different activation functions) that might amplify per-operation rounding errors; (3) very large training runs where the residual bias, however small, accumulates over many more gradient steps than tested (the sanity test runs are 1,200–2,500 steps); (4) future lower-precision formats like FP8, where the mismatch would likely be larger than FP16's but smaller than BF16's, and the paper provides no framework for predicting where the acceptable threshold lies. The paper's framing of FP16 as "eliminating" rather than "dramatically reducing" the mismatch may lead practitioners to assume the problem is fully solved rather than substantially mitigated.

What evidence exists in the paper. Figure 2 (right, "Seq mismatch vs. Len (FP16)") shows that individual responses under FP16 can still have log-ratios deviating by ±5–10 units, and the reported KL divergence of 0.32 bits is explicitly non-zero. The paper does not run an ablation where sequence length is systematically extended beyond 25K to test whether the flat slope holds, nor does it measure mismatch after prolonged training to check whether the mismatch grows over time (as it does under BF16 for collapsing algorithms). Figure 5 (bottom panel) shows that even under the optimal FP16-FP16 configuration, the policy difference π − µ is not identically zero — it fluctuates within a small band but occasionally shows spikes (particularly in VeRL, as noted in Section 4.2).

Mitigation status. The paper partially acknowledges this in its discussion (Section 6) by noting that "using FP16 for extremely large models might present engineering challenges related to its limited range, such as managing potential overflows" — but this addresses FP16's dynamic range limitation, not the residual precision limitation. The residual KL divergence of 0.32 bits is not discussed as a potential concern. The paper does not propose methods to further reduce the residual mismatch (e.g., using FP32 for specific numerically sensitive operations while keeping FP16 for the bulk of computation), nor does it establish a threshold for what KL divergence is "small enough" for stable RL training.

Single Task Domain and Narrow Evaluation Scope

The assumption or constraint. All experiments in the paper — the sanity test, the generalization experiments, and the offline mismatch quantification — use mathematical reasoning datasets: MATH (Hendrycks et al., 2021) for the sanity test and LoRA experiments, DAPO-Math-17k for MoE experiments, a curated math dataset of 54.4K problems (aggregated from OR1, DAPO, and DeepScaler) for large dense model experiments, and AMC/AIME for evaluation. The paper provides no evidence about whether FP16's mismatch-reduction benefits extend to other RL fine-tuning domains: code generation, instruction following, safety alignment, multi-turn dialogue, or tasks where rewards are learned (via a reward model) rather than rule-based (string matching against ground-truth answers).

The consequence. Mathematical reasoning has several properties that may make it a best-case scenario for the FP16 fix: (1) rewards are binary and deterministic (correct final answer or not), which produces clean gradient signals — in domains with noisy or learned rewards, other sources of variance might dominate, making the precision-induced mismatch relatively less important; (2) responses are typically on the order of thousands of tokens — in domains with very short responses (single-turn QA, classification), the autoregressive accumulation of mismatch is less severe, potentially reducing the benefit of FP16; (3) the base models used (DeepSeek-R1-Distill-Qwen, Qwen-Math) were explicitly pretrained or fine-tuned for math, meaning their weight distributions may be more stable and predictable than models fine-tuned for more open-ended tasks. A practitioner working on RLHF for dialogue or safety alignment cannot confidently extrapolate from the paper's math-only results — the mismatch might be less severe (short responses), equally severe (long multi-turn dialogues with similar autoregressive accumulation), or dominated by other noise sources (learned reward model variance).

What evidence exists in the paper. The generalization experiments (Section 5) test across model families (Qwen, OctoThinker/Llama-derived), architectures (dense, MoE), training regimes (full fine-tuning, LoRA), and model scales (1.5B to 30B), but all within the math reasoning domain. The evaluation benchmarks are AIME 2024, AIME 2025, and AMC23 — all math competition datasets. There are no experiments on code generation (HumanEval, MBPP), general reasoning (GPQA, MMLU), instruction following (AlpacaEval, MT-Bench), or safety alignment. The paper does not claim domain-generality but also does not discuss this as a limitation.

Mitigation status. None. The paper does not acknowledge the domain restriction as a limitation, nor does it suggest that future work should validate FP16 in non-math RL fine-tuning settings. Section 8 (implications and future directions in the full paper) is not included in the provided excerpts, but the main text (Sections 1–6) contains no discussion of domain generalization. This is a significant gap because the paper's core claim — that FP16 eliminates the training-inference mismatch — is mechanistic (precision causes mismatch regardless of domain), yet the empirical validation is confined to a single task family where the signal is unusually clean.

No Demonstration of Sustained Benefit Beyond the Collapse Horizon

The assumption or constraint. The paper's primary empirical demonstration is that FP16 prevents training collapse and achieves higher rewards than BF16 within the training horizons tested: 1,000–2,500 steps for sanity test experiments (Figures 1a–f, 3, 4), and shorter horizons for generalization experiments (150–1,400 steps; Figures 1g–l). The BF16 baselines collapse or degrade within these horizons. However, the paper does not demonstrate what happens when FP16 training is extended significantly beyond the point where BF16 would have collapsed — does FP16 training continue to improve, plateau, or eventually develop its own pathologies? The AIME evaluation curves (Figure 6) show FP16 scores rising throughout the tested horizon, but this covers at most ~2,000 steps for the sanity test and fewer for generalization experiments.

The consequence. A practitioner adopting FP16 for a production RL fine-tuning run that may last tens of thousands of steps needs to know whether FP16's stability is sustained or merely delays the same underlying collapse mechanism. The residual mismatch (KL 0.32 bits, as discussed above) could theoretically accumulate over many training iterations in the same way that BF16's larger mismatch accumulates over fewer iterations — if the optimization bias feedback loop hypothesized in Section 4.2 operates at any non-zero mismatch level, then FP16 might eventually collapse, just much later than BF16. The paper's evidence cannot distinguish between "FP16 eliminates the collapse mechanism entirely" and "FP16 pushes the collapse horizon beyond the tested training budget." For large-scale training runs where cost per step is high, this distinction matters enormously.

What evidence exists in the paper. The longest sanity test runs extend to approximately 2,000–2,500 steps (Figures 1a–f, 3). The generalization experiments are shorter: OctoThinker GRPO runs for ~1,000 steps (Figure 1g), LoRA GRPO-Token-TIS for ~1,400 steps (Figure 1h), MoE experiments for 150–200 steps (Figures 1i–k), and Dense-14B DAPO for ~80 steps (Figure 1l). The paper shows no experiments where FP16 training is continued to, say, 10,000 steps to verify sustained stability. The training reward curves for FP16 (Figure 1) typically saturate rather than continuing to rise, but this saturation appears to be at a high reward value (near 1.0 for sanity test, near 0.7–0.9 for other experiments) rather than followed by degradation. The AIME evaluation curves (Figure 6) show FP16 scores generally rising or plateauing at a high level, not declining.

Mitigation status. The paper does not acknowledge this as a limitation. The discussion (Section 6) frames FP16 as resolving the stability problem: "This enhanced stability allows even the most naive policy gradient estimator to converge efficiently." There is no caveat about the tested training horizon being potentially insufficient to observe late-stage FP16 degradation, nor a suggestion that future work should run extended training durations to verify asymptotic stability. The mismatch metrics in Figure 3 (third row) show that under FP16, the mismatch remains bounded and small throughout the tested horizon, which is encouraging but not conclusive for longer runs.

The Deployment Gap Closure Is Demonstrated Implicitly, Not Measured Directly

The assumption or constraint. The paper argues that FP16 closes the deployment gap (Equation 4) because µ ≈ π — the parameters optimized under the training engine are near-optimal for the inference engine. The evidence for this claim is indirect: (1) FP16-trained models achieve higher AIME 2024 evaluation scores than BF16-trained models (39% vs. 34% for PG-Seq-IS, Figure 3; consistently higher across Figure 6); (2) the mismatch under FP16 is small (KL 0.32 bits, Figure 2). However, the paper never directly measures the deployment gap — defined as the difference between training reward and evaluation reward at matched checkpoints — and compares it across precisions.

The consequence. The higher evaluation scores under FP16 could arise from multiple mechanisms, not all of which are attributable to deployment gap closure: (1) FP16 training may simply optimize better (less biased gradients → better parameter values), which would improve both training and evaluation performance regardless of the deployment gap; (2) FP16 inference may produce slightly different (better) responses at evaluation time than BF16 inference, independent of training (though Table 2 argues against this by showing comparable inference-only performance across precisions); (3) the stability of FP16 training may allow more effective exploration that discovers better policies, again independent of the deployment gap. Without a direct measurement — e.g., plotting training accuracy vs. evaluation accuracy for matched FP16 and BF16 checkpoints, or computing the gap |J_train(θ) − J_eval(θ)| — the paper cannot attribute the evaluation improvement specifically to deployment gap closure rather than to generally better optimization.

What evidence exists in the paper. The deployment gap is defined mathematically in Equation 4 and discussed conceptually in Sections 2 and 3. Empirical evidence for its closure comes from: (1) Figure 3, which shows that the best BF16 method (GRPO-Seq-MIS) reaches 95% training accuracy but only 34% on AIME 2024, while FP16 PG-Seq-IS reaches 99% training accuracy and 39% on AIME 2024 — the ratio of evaluation-to-training performance is higher for FP16; (2) Figure 6, which shows FP16 evaluation scores consistently above BF16 across all experiments; (3) Figure 2, which shows the small mismatch under FP16. However, none of these directly isolate and measure the deployment gap as a quantity.

Mitigation status. The paper does not acknowledge the indirect nature of this evidence. The claim that "FP16 closes the deployment gap" appears in the abstract and is treated as established. A direct deployment gap measurement — perhaps the simplest being a scatter plot of training reward vs. AIME evaluation for checkpoints throughout training, colored by precision, with a diagonal reference line representing zero gap — would substantially strengthen this claim. The paper's existing data (training and evaluation curves in Figures 1, 3, 6) could support such an analysis but does not present it.

The Comparison Against Algorithmic Baselines Uses a Weak BF16 Configuration

The assumption or constraint. The paper compares FP16-trained algorithms (which use no algorithmic corrections beyond the base algorithm) against BF16-trained algorithms that use the best available importance-sampling corrections (GRPO-Seq-MIS, GRPO-Token-TIS). However, the BF16 baselines all use a single clipping threshold (C = 3 for importance sampling methods; clip_higher = 0.28 for GRPO-family algorithms) identified in Section 4.1. There is no evidence that these hyperparameters are optimal for BF16 training. Since the paper's central claim is that FP16 with no algorithmic corrections outperforms BF16 with the best available corrections, the strength of this claim depends on whether the BF16 baselines are genuinely well-tuned.

The consequence. If the BF16 algorithms could achieve better performance with different hyperparameters — e.g., a different clipping threshold, a different learning rate, a different number of gradient steps per iteration, or a different ratio of rollouts to gradient steps — then the gap between FP16 and BF16 might be smaller than reported. The bias-variance tradeoff the paper identifies under BF16 (Section 6) might be partly navigable through hyperparameter tuning: for instance, a smaller clipping threshold (C = 1.5 instead of C = 3) might make GRPO-Seq-MIS converge faster (by accepting more bias in exchange for lower variance), potentially approaching FP16's convergence speed. The paper's conclusion that "the performance differences between algorithms become almost indistinguishable" under FP16 (Section 4.3) could partially reflect that FP16's stability makes hyperparameter tuning less critical, which is itself a practical advantage, but the headline comparison of FP16 vs. BF16 peak performance would benefit from a demonstration that the BF16 baselines are near their performance ceiling.

What evidence exists in the paper. Section 4.1 specifies the hyperparameters: clip_higher = 0.28 for GRPO-family algorithms, C = 3 for importance sampling methods, batch size 64 with 8 rollouts per question, 4 gradient steps per iteration. The paper cites Yu et al. (2025) for the clip_higher choice, suggesting it is a community standard rather than a per-experiment optimization. There is no evidence of hyperparameter sweeps for the BF16 baselines — no ablation over C, learning rate, batch size, or gradient accumulation steps. The paper's ablation studies (Section 4.4, Figure 5) vary precision but keep all other hyperparameters fixed. The algorithm comparisons under FP16 (Section 4.3, Figure 4) implicitly show that algorithm choice matters little under FP16, but this does not address whether better-tuned BF16 baselines would close the gap.

Mitigation status. The paper does not discuss hyperparameter sensitivity for the BF16 baselines. This is partially defensible — the paper's contribution is the precision-level solution, not a hyperparameter optimization study — and the breadth of experiments (12 configurations across models, algorithms, and frameworks) makes it unlikely that all BF16 baselines are simultaneously poorly tuned in the same direction. However, for the specific claim that FP16 PG-Seq-IS (99% training accuracy, 39% AIME) outperforms the best BF16 method (GRPO-Seq-MIS: 95%, 34%), even a modest hyperparameter improvement in the BF16 baseline could narrow the gap. The paper would be stronger if it included at least a limited sensitivity analysis for the key BF16 hyperparameters, or acknowledged this as a caveat.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper should reframe how the field thinks about numerical precision in RL fine-tuning — a reframing, not a paradigm shift. The theoretical foundations of RL for LLMs (REINFORCE, importance sampling, PPO-style clipping) remain unchanged. What changes is the default assumption about which precision format is appropriate for the RL phase. Before this work, BF16 was the unquestioned standard for all LLM training stages, inherited from pretraining where its wide dynamic range is genuinely valuable. This paper provides the first systematic evidence that BF16 is actively harmful during RL fine-tuning, because its 7-bit mantissa creates a training-inference mismatch that corrupts gradient signals and causes training collapse — problems that prior work had been addressing through increasingly complex algorithmic patches (Yao et al., 2025; Liu et al., 2025a; Zheng et al., 2025) rather than by questioning the precision choice itself.

The magnitude of this reframing is substantial but bounded. It does not change what RL algorithms do — it changes under what numerical conditions they do it, and shows that getting those conditions right makes the algorithmic complexity of the past two years largely unnecessary. The paper's most striking result — that the simplest unbiased policy gradient estimator (Equation 5, PG-Seq-IS) under FP16 outperforms all BF16 algorithmic corrections, achieving 99% training accuracy vs. 95% for the best BF16 method and 39% vs. 34% on AIME 2024 (Figure 3) — resolves an apparent contradiction in the prior literature. Yao et al. (2025) found that token-level importance sampling corrections helped stabilize GRPO; Liu et al. (2025a) found those corrections insufficient and proposed sequence-level masking instead. Both were right about their specific observations but wrong about the root cause: the corrections were compensating for BF16's precision artifacts, and the disagreement reflected different points on the bias-variance tradeoff curve that BF16 forces (Section 6). Under FP16, that tradeoff dissolves (Figure 4), and all algorithms converge to similar performance — neither token-level TIS nor sequence-level MIS provides meaningful benefit. The field had been optimizing the wrong thing: algorithm design to compensate for precision loss, rather than precision itself.

This reframing makes several research directions more attractive. Verifier and reward model quality becomes the primary bottleneck for RL fine-tuning, since algorithmic instability from numerical mismatch is now controlled. Longer training horizons become practical — if FP16 prevents the collapse that BF16 algorithms experience within hundreds to low thousands of steps (Figure 1), researchers can investigate whether RL fine-tuning benefits from substantially more training, rather than having to stop before collapse occurs. Scaling RL to larger models and more complex tasks becomes less risky, since the precision fix is architectural (a configuration flag) rather than algorithmic (requiring hyperparameter tuning per model/task combination). Conversely, research on ever-more-elaborate importance sampling corrections for BF16 becomes less urgent — the paper shows these corrections are compensating for a problem that can be eliminated at the precision level with zero algorithmic overhead and a ~25% computational saving (the extra forward pass for importance ratio computation, Section 2.1.1).

The paper also introduces a methodological contribution that may outlast the specific FP16 recommendation: the sanity test (Section 4) as a diagnostic instrument. By filtering for problems where the base model's accuracy is 20–80%, it creates a "perfectible" dataset where a reliable RL algorithm should achieve near-perfect training accuracy. This provides a sharper signal than standard benchmarks for algorithmic debugging — if an algorithm cannot reach 95%+ accuracy on problems the model demonstrably can solve, something is fundamentally wrong. As the field moves to even lower precisions (FP8) or new hardware with different numerical properties, the sanity test provides a reusable framework for detecting whether precision-induced mismatch is corrupting optimization, before scaling to expensive large-dataset experiments.

Follow-Up Research This Work Enables

Quantifying the acceptable mismatch threshold for stable RL training. The paper shows that a KL divergence of 7.64 bits (BF16) causes collapse while 0.32 bits (FP16) enables stable training, but it does not establish where the boundary lies. A natural follow-up would systematically vary the mismatch magnitude — for instance, by injecting controlled noise into the inference engine's logits at varying scales, or by using custom mixed-precision configurations with intermediate mantissa widths — and measure at what KL divergence training begins to destabilize. This would produce a mismatch budget that could guide the design of future low-precision formats: if FP8 produces KL divergence of, say, 1.5 bits, is that safe? The paper's offline mismatch measurement methodology (Figure 2: sample with temperature 1.0, no top-p, compare µ vs. π log-ratios) provides the measurement tool; the missing piece is the RL training stability curve as a function of mismatch. A strong follow-up would sweep mismatch levels and report the maximum training horizon before collapse, the final training accuracy, and whether the bias-variance tradeoff (Section 6) re-emerges at intermediate mismatch values.

Extending the analysis to FP8 and other emerging formats. Modern hardware (H100, H200, B200) supports FP8 natively, and several large-scale training efforts are adopting it for throughput. FP8 typically uses either 4 or 5 exponent bits and 3 or 2 mantissa bits (E4M3 or E5M2 formats) — substantially less precision than FP16's 10 mantissa bits. The paper's framework predicts that FP8 will produce a larger training-inference mismatch than FP16, potentially large enough to reintroduce the instability FP16 eliminated. A direct experiment would replicate the offline analysis (Figure 2) and sanity test (Section 4) using FP8 for training and/or inference, measuring the sequence-level log-ratio slope and KL divergence, and then running RL training to see whether collapse re-emerges. If FP8 does cause collapse, the natural next question is whether the algorithmic corrections that were unnecessary under FP16 (token-level TIS, sequence-level MIS) become necessary again under FP8 — closing the loop on the paper's central argument that precision choice determines algorithmic requirements. If FP8 is stable, then the community can confidently adopt it for RL fine-tuning with the throughput benefits it offers.

Validating the FP16 benefit in non-math RL domains. All of the paper's experiments use mathematical reasoning tasks with binary, rule-based rewards (final answer matches ground truth). This domain has properties that may make the mismatch particularly consequential: responses are long (thousands of tokens, amplifying autoregressive accumulation), and the reward signal is clean (zero noise). In other RL fine-tuning domains — RLHF for dialogue with a learned reward model, safety alignment with multi-objective rewards, code generation with unit-test-based rewards — the mismatch may interact differently with the reward signal. A learned reward model introduces its own variance, potentially dominating the precision-induced gradient noise and making the FP16 benefit smaller. Conversely, code generation often produces very long responses (full programs with comments), potentially making the mismatch more severe than in math. A strong follow-up would replicate the sanity test construction (filter for perfectible problems) in code generation (using HumanEval or MBPP, filtering by pass@1 rate) and dialogue (using a learned reward model's confidence as a proxy for perfectibility), then compare FP16 vs. BF16 training stability and final performance. This would establish whether the FP16 recommendation is domain-specific or universal.

Characterizing the residual mismatch under FP16 at extended training horizons and sequence lengths. The paper's training runs extend to 1,000–2,500 steps for sanity test experiments, and the offline mismatch analysis covers sequences up to ~25K tokens. The residual mismatch under FP16 (KL 0.32 bits, log-ratio slope −0.07) is small but non-zero. Two stress tests would clarify whether this residual matters in practice: (1) extended training to 10,000+ steps, monitoring whether the mismatch grows over time (as it does under BF16 for collapsing algorithms, Figure 3 third row) and whether late-stage collapse eventually occurs; (2) offline mismatch measurement for sequences of 50K–100K tokens (achievable with long-context models), to determine whether the near-flat slope holds or eventually bends. If FP16 eventually collapses at, say, 8,000 steps, or if the mismatch grows substantially for 50K+ token sequences, then FP16 is a mitigation rather than a solution, and practitioners training very large models on very long horizons would need additional strategies (e.g., FP32 for specific sensitive operations, periodic resynchronization of µ and π). If FP16 remains stable indefinitely, the residual mismatch is practically irrelevant, and the field can treat FP16 as a solved parameter.

Developing lightweight, online difficulty estimation to replace the sanity test's offline filtering. The sanity test's construction — generating 40 rollouts per problem and filtering by initial accuracy — is expensive (480,000 responses to filter MATH's 12,000 training questions) and must be re-run for each new base model. A practical deployment of the sanity test methodology would benefit from a cheaper proxy: can a small number of rollouts (e.g., 4–8) combined with a confidence signal (e.g., the model's own sequence-level probability, or the variance of reward across rollouts) accurately identify perfectible problems? A strong follow-up would compare the perfectible subsets identified by cheap proxies against the full 40-rollout gold standard, reporting precision/recall of problem inclusion and, critically, whether RL training on the proxy-identified subset produces similar algorithmic diagnoses (i.e., do algorithms that pass the gold-standard sanity test also pass the proxy version, and vice versa?). This would make the sanity test methodology practical for routine use in RL algorithm development.

Investigating the optimization bias feedback loop hypothesized in Section 4.2. The paper observes that under BF16, algorithms that eventually collapse show a growing training-inference mismatch beforehand (Figure 3, third row), and hypothesizes that biased gradients create a feedback loop: bias → larger mismatch → more bias in subsequent gradient steps → eventual divergence. This hypothesis is plausible but untested. A direct experiment would intervene during training when the mismatch crosses a threshold: (1) pause training, (2) resynchronize µ and π by copying weights and re-running the inference engine to produce fresh samples from the current (post-bias) policy, and (3) resume training. If the feedback loop hypothesis is correct, resynchronization should reset the mismatch and delay or prevent collapse. If collapse continues despite resynchronization, the bias has already corrupted the parameter values themselves (not just the gradient estimates), which would indicate a deeper pathology. This experiment would clarify whether mismatch monitoring (Figure 3) can serve as an actionable early-warning signal with a known intervention, or whether it is merely a correlated symptom of an irreversible process.

Practical Applications and Downstream Use Cases

Open-source RL fine-tuning pipelines (VeRL, Oat, TRL) adopting FP16 as the default precision. The paper's most immediate practical impact is a configuration change in the major open-source RL frameworks. Both VeRL (Sheng et al., 2024) and Oat (Liu et al., 2025b) currently default to BF16 for RL training, following the pretraining convention. The paper shows across both frameworks (Figures 1, 3) that switching to FP16 prevents collapse, accelerates convergence (FP16 curves rise more steeply than BF16 in Figure 1a–f), and eliminates the need for algorithmic corrections that add ~25% computational overhead (Section 2.1.1). For a research group running RL fine-tuning on a 1.5B–14B model (the scale tested in the paper), this translates to: (1) no more debugging mysterious training collapses; (2) ~25% more training throughput per GPU-hour (by dropping the extra forward pass for importance ratio computation); (3) reduced need for hyperparameter tuning across algorithms (since Figure 4 shows algorithm choice matters little under FP16). The change is a single configuration flag in most frameworks (e.g., --dtype fp16 in VeRL's launch script, or setting torch_dtype=torch.float16 in the model loader). The paper's evidence that FP16 works across model families (Qwen, Llama-derived OctoThinker), architectures (dense, MoE), training regimes (full fine-tuning, LoRA), and scales (1.5B to 30B parameters) gives framework maintainers confidence that this default change will not break existing workflows.

Cost-efficient RL fine-tuning for small labs and individual researchers. The ~25% throughput improvement from dropping the extra forward pass, combined with faster convergence (fewer steps to reach the same reward), means that FP16-trained models require fewer total GPU-hours than BF16-trained models with algorithmic corrections. Using the sanity test numbers: FP16 PG-Seq-IS achieves 99% training accuracy vs. 95% for BF16 GRPO-Seq-MIS (Figure 3), with convergence that is visibly faster (steeper reward curves in Figure 3, top row). For a small lab running on 8× A100 GPUs (the hardware used in the paper's sanity test), this could mean the difference between a weekend experiment and a multi-day run. More importantly, the reduced collapse risk means fewer wasted runs — under BF16, a run that collapses at step 600 (as in the LoRA experiment, Figure 1h) represents a total loss of the compute invested up to that point. FP16 eliminates this failure mode across all tested configurations (Figure 1, all panels). For researchers iterating on reward functions, prompt datasets, or hyperparameters, the reliability improvement from FP16 enables faster experimentation cycles.

Scaling RL fine-tuning to larger models where BF16 instability is most costly. The paper's MoE experiment (Qwen3-30B-A3B-Base, Figure 1i–k) and large dense model experiment (Qwen3-14B-Base, Figure 1l) demonstrate that the FP16 benefit persists at scale. For organizations training models at the 70B–400B parameter range, training collapse is catastrophically expensive — a single failed run can cost tens of thousands of GPU-hours. The paper's finding that FP16 prevents collapse across MoE architectures (where top-k routing amplifies numerical sensitivity) is particularly relevant here, since many frontier models (Mixtral, DeepSeek-V2/V3, GPT-4-class models) use MoE. The practical recommendation is straightforward: before scaling an RL fine-tuning recipe to a large production run, validate on a smaller model using the sanity test methodology. If BF16 training collapses on the sanity test (as it does for all tested configurations in the paper) while FP16 passes, the large-scale run should use FP16. The paper provides the evidence base for making this decision without expensive large-scale trial-and-error.

The sanity test as a lightweight diagnostic for RL algorithm development. Independent of the precision recommendation, the sanity test methodology (Section 4) provides a practical tool for RL algorithm researchers. Constructing a perfectible dataset for a new base model requires generating 40 rollouts per problem and filtering for 20–80% initial accuracy — expensive in absolute terms (480K responses to filter MATH) but cheap relative to the cost of debugging algorithm failures on full benchmarks where problem difficulty confounds the diagnosis. A researcher developing a new RL algorithm can first test it on the sanity test; if it fails to reach 95%+ training accuracy, the algorithm has a fundamental flaw that must be fixed before scaling to larger datasets. This accelerates the development cycle by providing rapid, unambiguous feedback. The paper's own use of the sanity test to diagnose the bias-variance tradeoff under BF16 (Section 4.2) — where different algorithms exhibited qualitatively different failure modes that would be partially obscured on the full MATH benchmark — demonstrates the diagnostic value. Framework developers could ship sanity test datasets alongside their RL implementations, giving users a standard "unit test" for their training setup.

🎯 Pitch

1. Executive Summary

2. Context and Motivation

The Core Problem: RL Fine-Tuning of LLMs Is Brittle Despite Its Importance

The Identified Mechanism: Training-Inference Mismatch

Prior Approaches: Algorithmic Corrections and Their Limitations

Algorithmic Corrections via Importance Sampling

Engineering Approaches to Reduce the Mismatch

Empirical Evidence of the Problem's Severity

The Bias-Variance Tradeoff Under BF16

How This Paper Positions Itself

3. Technical Approach

3.1 Reader orientation (approachable technical breakdown)

3.2 Big-picture architecture (diagram in words)

3.3 Roadmap for the deep dive

3.4 Detailed, sentence-based technical breakdown

The Formal Mismatch: Why $\mu \neq \pi$ Matters

Importance Sampling Corrections and Their GRPO-Specific Implementations

Floating-Point Precision Analysis: FP16 vs. BF16

Offline Mismatch Quantification (Section 3.5, Figure 2)

The Sanity Test: A Perfectible Dataset for Sharp Diagnosis (Section 4)

Algorithm Comparisons Under BF16 (Section 4.2, Figure 3)

The Efficacy of FP16 (Section 4.2 continued, Figures 1, 3, 6)

Algorithm Convergence Under FP16 (Section 4.3, Figure 4)

Precision Ablation Study (Section 4.4, Figure 5)

Generalization Experiments (Section 5, Figure 1 panels g–l, Figure 6)

Design Choices and Their Justifications

4. Key Insights and Innovations

Innovation 1: Reframing the Training-Inference Mismatch as a Floating-Point Precision Problem, Not an Algorithmic One

Innovation 2: The Sanity Test as a Sharp Diagnostic for RL Algorithm Reliability

Innovation 3: The Dissolution of the Bias-Variance Tradeoff Under FP16

Innovation 4: Identifying the Mismatch Growth Rate as a Diagnostic Signal for Training Health

5. Experimental Analysis

Evaluation Methodology

Main Quantitative Results

Offline Mismatch Quantification (Section 3.5, Figure 2, Table 2)

Sanity Test: Algorithm Comparisons Under BF16 (Section 4.2, Figure 3)

Sanity Test: FP16 Performance (Section 4.2 continued, Figures 1, 3, 6)

Algorithm Convergence Under FP16 (Section 4.3, Figure 4)

Precision Ablation Study (Section 4.4, Figure 5)

Generalization Experiments (Section 5, Figure 1 panels g–l, Figure 6 panels g–l)

Ablation Studies and Robustness Checks

Critical Assessment

Does FP16 genuinely eliminate the mismatch, or just reduce it to a manageable level?

Does the 24× mismatch reduction fully explain the performance improvement, or are there confounding factors?

The sanity test is a strong diagnostic, but what does it actually measure?

The generalization experiments are broad but thin.

The computational cost comparison is incomplete.

The missing FP8 discussion.

The "early-warning signal" claim is suggestive but not validated.

The paper does not disentangle mantissa bits from other FP16 properties.

Claims vs. evidence summary.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Unaccounted For and Prohibitively Expensive

The Residual Mismatch Under FP16 Is Non-Zero and Not Fully Characterized

Single Task Domain and Narrow Evaluation Scope

No Demonstration of Sustained Benefit Beyond the Collapse Horizon

The Deployment Gap Closure Is Demonstrated Implicitly, Not Measured Directly

The Comparison Against Algorithmic Baselines Uses a Weak BF16 Configuration

7. Implications and Future Directions

How This Work Changes the Landscape

Follow-Up Research This Work Enables

Practical Applications and Downstream Use Cases

Adjacent in domain

PairUni: Pairwise Training for Unified Multimodal Language Models

Memory Retrieval and Consolidation in Large Language Models through Function Tokens