Stabilizing Reinforcement Learning with LLMs: Formulation and Practices

ArXiv: 2512.01374

🎯 Pitch

Optimizing a sequence-level reward with token-level policy gradients is only valid when two hidden mismatches are controlled—the gap between training and inference engines, and the gap between rollout and target policies. Once training is stabilized by managing these factors, prolonged RL consistently matches the best performance regardless of how poorly the model was initialized, overturning the assumption that starting quality dictates final results.

---

1. Executive Summary

This paper proposes a novel formulation for reinforcement learning with large language models, explaining why and under what conditions a surrogate token-level objective can serve as a valid first-order approximation to the true sequence-level reward. Through extensive experiments with a 30B Mixture-of-Experts model on competition-level math benchmarks (HMMT25, AIME25, AIME24), the authors demonstrate that minimizing both the training–inference discrepancy (numerical mismatches between deployment engines) and policy staleness (divergence between the rollout policy and the target policy) is essential for stable RL training, with importance sampling correction, clipping, and Routing Replay (fixing routed experts during gradient updates) each playing a critical role in preserving this approximation. The paper establishes that on-policy training with basic policy gradient and importance sampling correction achieves the highest stability, while off-policy training requires both clipping and Routing Replay to prevent collapse — with Rollout Routing Replay (R3) becoming necessary over Vanilla Routing Replay (R2) under larger off-policiness — and further shows that prolonged stable training consistently yields comparable final performance regardless of cold-start initialization, establishing that training stability, not initialization specifics, is the decisive factor for successfully scaling RL.

2. Context and Motivation

The Core Problem: A Mismatch Between Reward and Optimization Granularity in LLM Reinforcement Learning

The fundamental problem this paper addresses is a structural misalignment in how reinforcement learning is applied to large language models. In RL for LLMs, the reward signal — a scalar score like "correct" or "incorrect" — is assigned to an entire response sequence. You can't meaningfully say whether the third token of a math solution was "correct"; correctness only makes sense when the full answer is available. Yet the optimization algorithms that dominate LLM training — REINFORCE, GRPO, PPO variants — operate at the token level, computing per-token gradients that drive parameter updates. The paper puts this tension in plain terms (Section 1):

"this mismatch between the reward (assigned at the sequence level) and the optimization unit (typically at the token level) raises concerns about the soundness and training stability of such approaches"

This isn't merely a philosophical concern. The mismatch creates genuine instability: token-level objectives can drift away from what the sequence-level reward actually incentivizes, leading to training collapses where the model's behavior abruptly degrades, reward signals become unreliable, and the policy enters a failure mode that is difficult or impossible to recover from. The authors define stable training explicitly in a footnote that is worth quoting because it captures what practitioners actually experience:

"By stable training, we refer to a training process in which model performance steadily improves over training steps — reflected in both the training reward and benchmark scores — and, crucially, the model's internal state evolves smoothly and without abrupt shifts."

This definition matters because it distinguishes between training that merely sometimes produces a good final checkpoint (unstable but lucky) and training that reliably improves. For organizations investing hundreds of thousands of GPU hours into a single RL run — the scale at which this paper operates — the difference between these two regimes is the difference between a successful project and wasted compute.

Why This Problem Is Important: Theoretical and Practical Stakes

Theoretical stakes. The paper's concern is foundational. If you're optimizing a sequence-level objective using token-level gradients, you are implicitly making an approximation. Whether that approximation is valid, and under what conditions, determines whether the entire RL-for-LLMs paradigm rests on solid mathematical ground. Prior to this paper, the field largely operated on empirical intuition: certain algorithms (GRPO, PPO) seemed to work, so they were adopted. But there was no clear theoretical account of why token-level optimization should approximate sequence-level reward optimization, when that approximation might break, or what observable signals would indicate that the approximation is failing. The paper aims to provide exactly this missing theoretical foundation, which matters not just for intellectual clarity but for diagnosing and preventing training failures before they happen.

Practical stakes — MoE models amplify the instability. The problem is especially acute for Mixture-of-Experts (MoE) models, which have become the dominant architecture for frontier LLMs (as evidenced by DeepSeek-R1, Qwen3, and others). In MoE models, each token dynamically routes through a subset of the model's parameters (the "experts"). This routing introduces a new layer of non-determinism that compounds the sequence-token mismatch. Specifically, as the authors detail in Section 3.1, the training engine and inference engine can route tokens to different experts even given identical model parameters and inputs, simply because the computational kernels differ between engines. This means the token-level importance sampling weight — the correction factor that links the rollout policy's probabilities to the training policy's probabilities — becomes entangled with which experts were activated. If the experts differ between rollout and optimization, the importance weight is computed relative to the wrong policy, and the approximation collapses. Without a principled understanding of this failure mode, MoE RL training is a minefield.

Practical stakes — the cost of instability. RL training at frontier scale is extraordinarily expensive. The paper reports using a 30B MoE model with FP8 inference and BF16 training, consuming "hundreds of thousands of GPU hours" across experiments. Each gradient step costs approximately 5–6 GPU hours. A training collapse that occurs after 1,500 gradient steps represents roughly 7,500–9,000 GPU hours of wasted compute — not including the cost of diagnosis, hyperparameter tuning, and re-running experiments. Understanding and preventing these collapses is therefore an engineering problem with direct financial consequences.

Practical stakes — the cold-start question. There is an ongoing debate in the RL-for-LLMs community about the importance of cold-start initialization: how much does the quality of the supervised fine-tuned model you start with determine the success of subsequent RL training? Some prior work has emphasized careful data curation for the initial SFT phase, suggesting that RL can only polish what is already well-formed. This paper's stability-first perspective offers a different view: if the RL training process itself is stable, the model can be trusted to improve steadily from a wide range of starting points, and the specifics of initialization matter less than whether the RL recipe is sound. This shifts where practitioners should invest effort — from obsessing over cold-start data toward engineering reliable RL pipelines.

Prior Approaches and Where They Fall Short

The paper does not name a single prior approach as the "standard" against which it compares, but rather identifies a landscape of practices that share common weaknesses. Understanding these requires tracing several threads in the literature.

Thread 1: Standard policy gradient algorithms (REINFORCE, GRPO, PPO) applied to LLMs. Since at least DeepSeekMath (Shao et al., 2024), the dominant paradigm for RL with LLMs has been to take a policy gradient algorithm originally developed for continuous control or game-playing domains — where per-timestep rewards are natural — and apply it to language generation, where rewards are sequence-level. GRPO (Group Relative Policy Optimization), for instance, computes advantages by comparing a group of sampled responses to each other and normalizing, then applies a per-token ratio-based objective with clipping. The objective in GRPO is:

$J_{\text{GRPO}}(\theta) = \mathbb{E}_{x \sim \mathcal{D}, \{y_i\}_{i=1}^G \sim \mu_{\theta_{\text{old}}}(\cdot|x)} \left[ \frac{1}{G} \sum_{i=1}^G \frac{1}{|y_i|} \sum_{t=1}^{|y_i|} \min\left( r_{i,t}(\theta) \hat{A}_{i,t}, \text{clip}(r_{i,t}(\theta), 1-\varepsilon_{\text{low}}, 1+\varepsilon_{\text{high}}) \hat{A}_{i,t} \right) \right]$

where $r_{i,t}(\theta) = \pi_\theta(y_{i,t}|x, y_{i,<t}) / \pi_{\theta_{\text{old}}}(y_{i,t}|x, y_{i,<t})$.

The problem: GRPO's objective is not derived from any principled approximation to a sequence-level reward. It incorporates length normalization (dividing by $|y_i|$) and per-token clipping without theoretical justification for why these operations should preserve alignment with the sequence-level reward the practitioner actually cares about. The authors show in Section 4.3 (Figure 1) that length normalization — a seemingly innocuous design choice — produces suboptimal benchmark performance compared to the un-normalized objective. This is not because length normalization is inherently bad, but because it invalidates the first-order approximation that connects token-level optimization to sequence-level reward. Without this approximation, the gradients computed at each token are no longer coherently pointing toward improvement of the true objective.

Thread 2: Sequence-level optimization as an alternative. Recognizing the token-sequence mismatch, some recent work has proposed abandoning token-level objectives entirely in favor of sequence-level optimization. Zheng et al. (2025) introduced Group Sequence Policy Optimization (GSPO), which optimizes over complete response sequences using sequence-level importance ratios. Liu et al. (2025a) similarly advocated for sequence-level objectives. These approaches are theoretically cleaner: they optimize exactly what they aim to maximize, with no approximation needed.

The authors acknowledge this line of work but position it as facing a different problem: the sequence-level gradient (Equation 2) involves the ratio of full-sequence likelihoods $\pi_\theta(y|x) / \mu_{\theta_{\text{old}}}(y|x)$, which has "large numerical range and high variance" because sequence likelihoods are products of hundreds or thousands of per-token probabilities. This makes direct sequence-level optimization "usually intractable to utilize." The token-level objective, by contrast, decomposes the ratio into per-token terms that are individually well-behaved — but only if the first-order approximation holds. So the field faces a tradeoff: sequence-level objectives (pure but intractable) versus token-level objectives (tractable but potentially unsound). This paper's contribution is to characterize when the token-level approach is sound, effectively bridging these two positions.

Thread 3: Training-inference discrepancy as an unrecognized source of instability. Prior work on RL training stability for LLMs largely focused on policy staleness — the fact that when you do multiple gradient updates on a batch of responses, the policy changes and the responses become "off-policy" relative to the current model. This is a well-understood phenomenon from standard RL theory. But the training–inference discrepancy — the numerical mismatch between what the training engine computes and what the inference engine computes — has been largely ignored or treated as a nuisance rather than a first-class problem. Yao et al. (2025) and Liu et al. (2025a) recently identified that RL training collapses are often accompanied by sharp increases in the KL divergence between the inference and training engine outputs, but they did not connect this to the validity of the underlying optimization objective. This paper elevates the training–inference discrepancy to equal status with policy staleness, showing through the decomposition in Equation (5) that both terms multiply together to determine whether the token-level approximation is valid. If either term is large, the approximation breaks.

Thread 4: MoE training instability and prior Routing Replay work. Two prior papers proposed techniques for stabilizing MoE RL training. Zheng et al. (2025) introduced Vanilla Routing Replay (R2), which fixes the routed experts during gradient updates to those determined by the rollout policy in the training engine. Ma et al. (2025) proposed Rollout Routing Replay (R3), which fixes the experts to those determined by the rollout policy in the inference engine. Both papers demonstrated empirical improvements, but neither provided a clear theoretical account of why fixing experts helps, when one variant should be preferred over the other, or how Routing Replay interacts with the broader token-sequence approximation issue. Ma et al. (2025) validated R3 only on small-scale experiments (a maximum of 180 global steps) with BF16 inference, which the authors of this paper argue is an insufficient stress test — the training–inference discrepancy is much smaller under BF16 than under the FP8 setting used here, so the benefits of R3 may not have been fully tested. This paper provides both the missing theoretical grounding (Routing Replay works because it restores the first-order approximation by reducing the training–inference discrepancy and policy staleness; see Section 3.2) and a controlled comparison of R2 vs. R3 at scale across multiple off-policy regimes.

The cumulative gap. The field's situation before this paper can be summarized as follows: practitioners were using token-level RL algorithms that worked most of the time but collapsed unpredictably, especially with MoE models and under off-policy training. There was no clear diagnostic framework for understanding why collapses occurred, no theoretical criterion for evaluating whether a given algorithmic variation (length normalization, IS correction, clipping) would help or hurt, and no principled guidance on when to use R2 versus R3 for MoE models. The gap was not that token-level RL didn't work — it clearly does, as evidenced by DeepSeek-R1 and Qwen3 — but that nobody could explain why it works, when it stops working, or how to fix it when it breaks.

How This Paper Positions Itself

The paper positions itself as providing the theoretical formulation that was missing from prior empirical work, and then validating that formulation through controlled, large-scale experimentation. The key intellectual move is reframing the token-level objective not as an independent algorithm design but as a first-order Taylor approximation to the sequence-level objective we actually want to optimize.

The derivation in Section 2.3 is the conceptual core. The sequence-level gradient (Equation 2) involves the product $\prod_{t=1}^{|y|} \pi_\theta(y_t|x, y_{<t}) / \mu_{\theta_{\text{old}}}(y_t|x, y_{<t})$. If each per-token ratio is close to 1 — meaning the target policy and rollout policy assign similar probabilities to each token — then we can write each ratio as $1 + \delta_t$ where $\delta_t$ is small, and the product expands as:

$\prod_{t=1}^{|y|} (1 + \delta_t) = 1 + \sum_{t=1}^{|y|} \delta_t + \mathcal{O}(\delta^2)$

Dropping the second-order and higher terms (products like $\delta_i \delta_j$) yields the token-level objective, where the importance weight is a sum of per-token ratios rather than a product. This approximation is only valid when the $\delta_t$ are indeed small — which requires the target policy $\pi_\theta$ and the rollout policy $\mu_{\theta_{\text{old}}}$ to be close.

The critical decomposition in Equation (5) then shows that the gap between these two policies comes from two multiplicative factors: the training–inference discrepancy ($\pi_{\theta_{\text{old}}} / \mu_{\theta_{\text{old}}}$) and policy staleness ($\pi_\theta / \pi_{\theta_{\text{old}}}$). This is the paper's central theoretical innovation: it provides a unified framework that explains why seemingly unrelated techniques — importance sampling correction (addresses training–inference discrepancy), clipping (restrains policy staleness by preventing aggressive updates), and Routing Replay (addresses both, but in different ways for R2 vs. R3) — all contribute to stable training. They are not arbitrary tricks; they are mechanisms for preserving the validity of the first-order approximation.

For MoE models specifically, the paper extends this decomposition to Equation (6), showing that expert routing introduces additional terms into both the training–inference discrepancy and policy staleness. Routing Replay works by constraining these routing terms, but at the cost of biasing the target policy (the policy being optimized is no longer $\pi_\theta$ with naturally-routed experts, but $\pi_\theta^{\text{R2}}$ or $\pi_\theta^{\text{R3}}$ with fixed experts). The paper is explicit that this introduces a tradeoff — bias versus approximation validity — and that the optimal choice between R2 and R3 depends on the degree of off-policiness because R2 does not alter the target policy in the first mini-batch while R3 alters it in all mini-batches (Table 1). This tradeoff-aware analysis is a substantial advance over prior work that simply reported empirical results for one variant.

The empirical strategy follows directly from the theory. Rather than proposing a new algorithm, the paper constructs MiniRL — a minimalist baseline that strips down REINFORCE to exactly the components justified by the first-order approximation framework (IS correction, clipping, no length normalization) — and then systematically ablates those components to show that deviating from the approximation-valid regime causes training instability. The on-policy experiments (Section 4.3) demonstrate this cleanly: MiniRL with IS correction trains stably and achieves the best performance; removing IS correction causes rapid collapse; adding length normalization degrades performance but doesn't collapse (it biases the objective without destroying the approximation); adding R3 in on-policy training doesn't help and can hurt (because the bias from altered experts outweighs any benefit when policy staleness is already zero). The off-policy experiments (Section 4.4) then show that as policy staleness increases with off-policiness, the tradeoff shifts: clipping and Routing Replay become essential because policy staleness threatens the approximation, and R3 overtakes R2 at higher off-policiness because the bias from altering experts becomes less costly than the instability from an invalid approximation.

The paper thus positions itself not as an algorithm paper — MiniRL is intentionally unoriginal, derived directly from REINFORCE with standard PPO clipping — but as a formulation and analysis paper that explains why existing techniques work and provides a framework for evaluating future techniques. A new proposed method can be assessed by asking: does it preserve the validity of the first-order approximation, or does it introduce a bias that may be justified in certain regimes but not others? This reframing of the RL-for-LLMs problem from empirical trial-and-error to approximation-theory-guided design is the paper's primary intellectual contribution.

3. Technical Approach

3.1 Reader Orientation (Approachable Technical Breakdown)

This paper builds a theoretical framework for understanding why token-level reinforcement learning objectives work for training large language models, and what conditions must be satisfied for them to be valid. It solves the problem of unpredictable training instability in RL-for-LLMs by showing that stability depends on maintaining the validity of a first-order Taylor approximation that connects token-level optimization to the true sequence-level reward objective, and by identifying which engineering techniques preserve that approximation under different training regimes.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major conceptual components:

1. Sequence-Level Reward Objective ($J^{\text{seq}}$): The true quantity we want to maximize — the expected reward over responses sampled from the target policy. This is what the practitioner actually cares about but cannot directly optimize.

2. Token-Level Surrogate Objective ($J^{\text{token}}$): A tractable proxy derived by first-order Taylor expansion of the sequence-level objective. This is what gradient-based optimizers actually update. The surrogate decomposes the problematic product-of-ratios into a sum of per-token ratios, massively reducing variance.

3. Approximation Validity Conditions: Two multiplicative factors that determine whether the surrogate is valid — the training–inference discrepancy (numerical mismatch between what the rollout engine computes and what the training engine computes) and policy staleness (how far the current policy has drifted from the policy that generated the training responses). When either factor is large, the approximation breaks and training can collapse.

4. Stabilization Mechanisms: Engineering techniques that preserve the approximation's validity — importance sampling correction (cancels training–inference discrepancy in the gradient), clipping (prevents policy staleness from growing too large), and Routing Replay (constrains expert routing in MoE models to reduce both discrepancy and staleness).

Information flows as follows: prompts are sampled → the inference engine generates responses using the rollout policy $\mu_{\theta_{\text{old}}}$ → rewards are assigned to complete responses → the training engine computes per-token ratios between the target policy $\pi_\theta$ and the rollout policy → these ratios serve as importance weights in the token-level gradient → the gradient updates the model parameters → the updated policy becomes the new rollout policy for the next iteration. The stability of this loop depends crucially on whether the per-token ratios remain close to 1 throughout training.

3.3 Roadmap for the Deep Dive

• First, the sequence-level objective and its intractable gradient: why direct sequence-level optimization fails due to the product of per-token probabilities causing extreme variance. This establishes what we want to do and why we can't do it directly.

• Second, the token-level surrogate as a first-order approximation: the key mathematical insight — how expanding the sequence-level importance weight as a product of $(1 + \delta_t)$ terms, dropping second-order terms, yields the token-level objective. This is the paper's central theoretical contribution.

• Third, the decomposition of the approximation error: breaking the gap between target and rollout policies into the training–inference discrepancy and policy staleness (Equation 5). This provides the diagnostic framework for understanding when and why training becomes unstable.

• Fourth, the MoE challenge: how expert routing compounds both sources of error and why vanilla RL algorithms fail on MoE models (Equation 6). This motivates the need for Routing Replay.

• Fifth, Routing Replay mechanisms (R2 and R3): the two concrete implementations, how each addresses different components of the approximation error, and the bias-vs-validity tradeoff that determines when each variant should be preferred (Table 1).

• Sixth, MiniRL — the minimalist baseline algorithm: how the theoretical framework translates into a concrete training objective (Equation 7), including group-normalized advantages, per-token clipping, and the specific hyperparameters used in experiments. This grounds the theory in an implementable algorithm.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a theoretical formulation paper backed by extensive empirical validation. Its core idea is that token-level RL objectives for LLMs can be understood as first-order Taylor approximations to the true sequence-level reward objective, and that training stability depends on keeping this approximation valid by controlling two specific sources of error: the numerical discrepancy between training and inference engines, and the staleness of the policy that generated the training data relative to the current policy being optimized.

---

The Intractable Sequence-Level Objective

The paper begins from the quantity that practitioners actually want to maximize: the expected reward under the model's own policy. This is the natural objective for any RL problem — you want your policy to produce high-reward outputs.

The true objective. The sequence-level reward objective is defined as:

$J^{\text{seq}}(\theta) = \mathbb{E}_{x \sim \mathcal{D},\ y \sim \pi_\theta(\cdot|x)} \left[ R(x, y) \right]$

where $\pi_\theta$ is the target policy (the LLM being trained, parameterized by $\theta$), $\mathcal{D}$ is the prompt distribution, $y$ is a complete response sampled from the current policy, and $R(x, y) \in \{0, 1\}$ is the scalar reward assigned to the full response (binary in the paper's math experiments — correct or incorrect — but the formulation is general).

What it computes: the average reward we would get if we deployed the current model $\pi_\theta$ to generate responses to prompts from $\mathcal{D}$ and scored each response. This is exactly what we care about: higher $J^{\text{seq}}$ means the model produces more correct answers.

Why we cannot directly optimize it: the expectation is taken over $y \sim \pi_\theta(\cdot|x)$, meaning the responses come from the current policy. But during RL training, responses are generated by the inference engine (e.g., SGLang, vLLM) — not the training engine (e.g., Megatron, FSDP) where gradients are computed. The training engine holds the current parameters $\theta$ but does not itself generate text. So the responses available for computing gradients were generated under a different version of the policy — the rollout policy — not the current target policy.

The importance sampling transformation. To express the gradient in terms of responses we actually have (those sampled under the rollout policy), we apply importance sampling:

$J^{\text{seq}}(\theta) = \mathbb{E}_{x \sim \mathcal{D},\ y \sim \mu_{\theta_{\text{old}}}(\cdot|x)} \left[ \underbrace{\frac{\pi_\theta(y|x)}{\mu_{\theta_{\text{old}}}(y|x)}}_{\text{sequence-level IS weight}} R(x, y) \right]$

where $\mu_{\theta_{\text{old}}}$ denotes the rollout policy — the policy as computed by the inference engine at the time responses were sampled. The notation $\mu$ (rather than $\pi$) is deliberate: it acknowledges that the inference engine and training engine produce numerically different outputs even given identical parameters. $\theta_{\text{old}}$ denotes that the parameters used during rollout may differ from the current parameters $\theta$ (this is policy staleness — the rollout policy is "old" relative to the target policy).

What it computes: the same expected reward, but now the expectation is taken over responses generated under $\mu_{\theta_{\text{old}}}$, which is the distribution we actually have samples from. The importance weight $\pi_\theta(y|x) / \mu_{\theta_{\text{old}}}(y|x)$ corrects for the distribution shift — if a response is more likely under the target policy than under the rollout policy, it gets weighted more heavily, and vice versa.

The gradient of the sequence-level objective. Taking the gradient with respect to $\theta$ yields:

$\nabla_\theta J^{\text{seq}}(\theta) = \mathbb{E}_{x \sim \mathcal{D},\ y \sim \mu_{\theta_{\text{old}}}(\cdot|x)} \left[ \frac{\pi_\theta(y|x)}{\mu_{\theta_{\text{old}}}(y|x)} R(x, y) \nabla_\theta \log \pi_\theta(y|x) \right]$

Expanding the log-probability of the full sequence into its per-token decomposition:

$\nabla_\theta J^{\text{seq}}(\theta) = \mathbb{E}_{x \sim \mathcal{D},\ y \sim \mu_{\theta_{\text{old}}}(\cdot|x)} \left[ \frac{\pi_\theta(y|x)}{\mu_{\theta_{\text{old}}}(y|x)} R(x, y) \sum_{t=1}^{|y|} \nabla_\theta \log \pi_\theta(y_t | x, y_{<t}) \right]$

where $|y|$ is the number of tokens in the response, and $\nabla_\theta \log \pi_\theta(y_t | x, y_{<t})$ is the score function gradient for token $t$.

What it computes: for each response $y$ sampled under the rollout policy, we compute (1) the sequence-level importance weight — a product of per-token probability ratios across the entire response — (2) multiply by the reward, and (3) multiply by the sum of per-token score function gradients. The expectation of this quantity over responses gives us the policy gradient.

Why this gradient is intractable: the sequence-level importance weight is a product of $|y|$ terms (one per token), where $|y|$ can be thousands (the paper's experiments use a maximum generation length of 32,768 tokens). Each term is the ratio of two probabilities, both between 0 and 1. The product of thousands of such ratios has enormous numerical range — if the target policy is even slightly more or less likely to generate the response than the rollout policy, the product can be astronomically large or vanishingly small. This makes the gradient estimator extremely high-variance, to the point of being unusable for optimization. The authors state:

"this gradient is usually intractable to utilize due to the large numerical range and high variance of sequence likelihood"

This is not merely a computational inconvenience — it is a fundamental obstacle. The variance of the gradient estimator scales with the product of per-token ratios, which grows exponentially with sequence length.

---

The Token-Level Surrogate Objective as a First-Order Approximation

The paper's central technical insight is that we can construct a tractable surrogate objective by approximating the sequence-level importance weight using a first-order Taylor expansion.

The surrogate token-level objective:

$J^{\text{token}}(\theta) = \mathbb{E}_{x \sim \mathcal{D},\ y \sim \mu_{\theta_{\text{old}}}(\cdot|x)} \left[ \sum_{t=1}^{|y|} \underbrace{\frac{\pi_\theta(y_t | x, y_{<t})}{\mu_{\theta_{\text{old}}}(y_t | x, y_{<t})}}_{\text{token-level IS weight}} R(x, y) \right]$

where the key difference from $J^{\text{seq}}$ is that the importance weight is now a sum of per-token ratios rather than a product of per-token ratios.

What it computes: for each response, we compute the per-token importance weight at each position, sum them across the sequence, multiply by the reward, and take the expectation over rollout-sampled responses. This is a scalar objective that can be differentiated to obtain gradients.

The gradient of the token-level objective:

$\nabla_\theta J^{\text{token}}(\theta) = \mathbb{E}_{x \sim \mathcal{D},\ y \sim \mu_{\theta_{\text{old}}}(\cdot|x)} \left[ \sum_{t=1}^{|y|} \frac{\pi_\theta(y_t | x, y_{<t})}{\mu_{\theta_{\text{old}}}(y_t | x, y_{<t})} R(x, y) \nabla_\theta \log \pi_\theta(y_t | x, y_{<t}) \right]$

What this gradient computes: for each response, at each token position $t$, we compute a per-token advantage — the per-token importance weight multiplied by the reward — and use it to scale the score function gradient for that token. The total gradient is the sum across all tokens in all responses.

How this relates to the sequence-level gradient — the first-order approximation. The critical derivation proceeds as follows. Suppose the target policy $\pi_\theta$ and the rollout policy $\mu_{\theta_{\text{old}}}$ are close, so that at each token their probability ratio is close to 1. Write the per-token ratio as:

$\frac{\pi_\theta(y_t | x, y_{<t})}{\mu_{\theta_{\text{old}}}(y_t | x, y_{<t})} = 1 + \delta_t$

where each $\delta_t$ is a small quantity (close to 0). Now examine the sequence-level importance weight:

$\frac{\pi_\theta(y|x)}{\mu_{\theta_{\text{old}}}(y|x)} = \prod_{t=1}^{|y|} \frac{\pi_\theta(y_t | x, y_{<t})}{\mu_{\theta_{\text{old}}}(y_t | x, y_{<t})} = \prod_{t=1}^{|y|} (1 + \delta_t)$

Expanding this product:

$\prod_{t=1}^{|y|} (1 + \delta_t) = 1 + \sum_{t=1}^{|y|} \delta_t + \underbrace{\sum_{i \neq j} \delta_i \delta_j + \sum_{i \neq j \neq k} \delta_i \delta_j \delta_k + \cdots}_{\mathcal{O}(\delta^2)}$

The higher-order terms involve products of two or more $\delta$ values. Since each $\delta_t$ is small, products like $\delta_i \delta_j$ are second-order small and become negligible. Dropping all second-order and higher terms yields:

$\frac{\pi_\theta(y|x)}{\mu_{\theta_{\text{old}}}(y|x)} \approx 1 + \sum_{t=1}^{|y|} \delta_t$

Substituting back $\delta_t = \frac{\pi_\theta(y_t | x, y_{<t})}{\mu_{\theta_{\text{old}}}(y_t | x, y_{<t})} - 1$:

$\frac{\pi_\theta(y|x)}{\mu_{\theta_{\text{old}}}(y|x)} \approx 1 + \sum_{t=1}^{|y|} \left( \frac{\pi_\theta(y_t | x, y_{<t})}{\mu_{\theta_{\text{old}}}(y_t | x, y_{<t})} - 1 \right)$

Now consider the gradient of $J^{\text{seq}}$ with respect to $\theta$. The gradient involves $\nabla_\theta$ of the importance weight (the reward $R(x,y)$ does not depend on $\theta$ since it is a function of the rollout-sampled response, not the policy parameters):

$\nabla_\theta J^{\text{seq}}(\theta) = \mathbb{E}_{x, y \sim \mu_{\theta_{\text{old}}}} \left[ R(x, y) \ \nabla_\theta \left( \frac{\pi_\theta(y|x)}{\mu_{\theta_{\text{old}}}(y|x)} \right) \right]$

Using the first-order approximation for the importance weight, and noting that the constant 1 and the -1 terms vanish under $\nabla_\theta$:

$\nabla_\theta \left( \frac{\pi_\theta(y|x)}{\mu_{\theta_{\text{old}}}(y|x)} \right) \approx \nabla_\theta \left( 1 + \sum_{t=1}^{|y|} \left( \frac{\pi_\theta(y_t | x, y_{<t})}{\mu_{\theta_{\text{old}}}(y_t | x, y_{<t})} - 1 \right) \right) = \nabla_\theta \left( \sum_{t=1}^{|y|} \frac{\pi_\theta(y_t | x, y_{<t})}{\mu_{\theta_{\text{old}}}(y_t | x, y_{<t})} \right)$

Therefore:

$\nabla_\theta J^{\text{seq}}(\theta) \approx \mathbb{E}_{x, y \sim \mu_{\theta_{\text{old}}}} \left[ R(x, y) \ \nabla_\theta \left( \sum_{t=1}^{|y|} \frac{\pi_\theta(y_t | x, y_{<t})}{\mu_{\theta_{\text{old}}}(y_t | x, y_{<t})} \right) \right] = \nabla_\theta J^{\text{token}}(\theta)$

In plain operational English: when the per-token probability ratios between the target policy and the rollout policy are all close to 1 (meaning the two policies assign similar probabilities to each token), we can replace the high-variance product of ratios (sequence-level) with a low-variance sum of ratios (token-level) and get approximately the same gradient. The approximation error comes from the second-order and higher terms we dropped — products of two or more per-token deviations. When those deviations are small, the error is negligible.

Why this form is preferred over alternatives: the token-level gradient involves a sum of per-token ratios rather than a product, which is the difference between variance that grows linearly with sequence length versus variance that grows exponentially. Token-level importance weights are individually well-behaved numbers near 1; their sum across hundreds or thousands of tokens remains numerically stable. The sequence-level importance weight, by contrast, is a product that can easily overflow or underflow floating-point representations. The token-level surrogate is thus the only tractable form — the price we pay for tractability is that the approximation is only valid when the per-token ratios stay close to 1.

---

The Two Sources of Approximation Error

For the first-order approximation to hold, we need $\pi_\theta(y_t | x, y_{<t}) \approx \mu_{\theta_{\text{old}}}(y_t | x, y_{<t})$ at every token. The paper decomposes the gap between these two policies into two multiplicative factors:

$\frac{\pi_\theta(y_t | x, y_{<t})}{\mu_{\theta_{\text{old}}}(y_t | x, y_{<t})} = \underbrace{\frac{\pi_{\theta_{\text{old}}}(y_t | x, y_{<t})}{\mu_{\theta_{\text{old}}}(y_t | x, y_{<t})}}_{\text{training–inference discrepancy}} \times \underbrace{\frac{\pi_\theta(y_t | x, y_{<t})}{\pi_{\theta_{\text{old}}}(y_t | x, y_{<t})}}_{\text{policy staleness}}$

where $\pi_{\theta_{\text{old}}}$ is the rollout policy as computed by the training engine — crucially, this is different from $\mu_{\theta_{\text{old}}}$, which is the rollout policy as computed by the inference engine. The same parameter values $\theta_{\text{old}}$ produce different probability distributions depending on whether they're evaluated in the training stack (PyTorch/FSDP with BF16) or the inference stack (vLLM/SGLang with FP8).

What each factor represents:

• Training–inference discrepancy ($\pi_{\theta_{\text{old}}} / \mu_{\theta_{\text{old}}}$): Even when the parameters are identical ($\theta = \theta_{\text{old}}$), the inference engine and training engine compute slightly different probability distributions for the same input. This arises from multiple sources: different computational kernels optimized for different hardware characteristics, different floating-point precision (FP8 inference vs. BF16 training in the paper's setup), and, in the inference engine, the common practice of disabling batch-invariant kernels to maximize throughput — meaning the same input can get different outputs depending on batch composition. The paper notes:

"training and inference engines typically employ different computational kernels for peak performance, which would yield inconsistent outputs given the same model input. Even within a single engine, particularly the inference side, batch-invariant kernels are often disabled for maximizing throughput, so the same model input can still receive variant outputs."

• Policy staleness ($\pi_\theta / \pi_{\theta_{\text{old}}}$): After gradient updates, the current policy $\pi_\theta$ differs from the rollout policy $\pi_{\theta_{\text{old}}}$. This is the standard off-policy problem in RL — the data was generated under an older policy, and we're trying to use it to improve the current policy. In synchronous RL with a global batch split into $N$ mini-batches, the first mini-batch is exactly on-policy ($\theta = \theta_{\text{old}}$ exactly), but subsequent mini-batches see increasingly stale data as gradient updates accumulate. The paper notes:

"since the rollout stage in RL is typically bounded in time by the generation length, to accelerate convergence through increased computational resources, we often split a large batch of sampled responses into mini-batches for multiple gradient updates. Consequently, mini-batches consumed later may exhibit greater policy staleness."

Why this decomposition matters: it provides a diagnostic framework. If training is unstable, you can examine whether the training–inference discrepancy is large (monitored via $\text{D}_{\text{KL}}(\mu_{\theta_{\text{old}}} \| \pi_{\theta_{\text{old}}})$ — the KL divergence between inference and training engine outputs) or whether policy staleness is large (monitored via the per-token ratio $\pi_\theta / \pi_{\theta_{\text{old}}}$ and the entropy of the current policy). Different interventions target different sources: importance sampling correction addresses the training–inference discrepancy in the gradient computation; clipping addresses policy staleness by preventing aggressive updates; Routing Replay (in MoE models) addresses both, but through different mechanisms for R2 and R3.

The key implication: if either factor deviates substantially from 1, the per-token ratios $\pi_\theta / \mu_{\theta_{\text{old}}}$ are no longer close to 1, the $\delta_t$ in the Taylor expansion are no longer small, the higher-order terms we dropped become non-negligible, and the token-level gradient $\nabla_\theta J^{\text{token}}$ no longer approximates the true sequence-level gradient $\nabla_\theta J^{\text{seq}}$. At that point, gradient steps do not reliably improve the expected sequence-level reward, and training can enter unstable regimes where the policy degrades abruptly.

---

The MoE Challenge: Expert Routing Compounds Both Error Sources

Mixture-of-Experts models add a critical complication: at each token generation step, only a subset of the model's parameters (the "experts") are activated via a dynamic routing mechanism. This routing depends on the token context and the model parameters. Since the routing is determined by the forward pass, and the forward passes differ between training and inference engines, the same token can be routed to different experts in the two environments even with identical parameters.

Extending the decomposition to MoE models. Incorporating expert routing, the token-level importance weight becomes:

$\frac{\pi_\theta(y_t | x, y_{<t})}{\mu_{\theta_{\text{old}}}(y_t | x, y_{<t})} = \frac{\pi_\theta(y_t | x, y_{<t}, e^\pi_t)}{\mu_{\theta_{\text{old}}}(y_t | x, y_{<t}, e^\mu_{\text{old}, t})}$

where $e^\pi$ denotes the experts routed in the training engine and $e^\mu$ denotes the experts routed in the inference engine. The subscript "old" indicates experts as determined at rollout time.

Decomposing as before:

$\frac{\pi_\theta(y_t | x, y_{<t})}{\mu_{\theta_{\text{old}}}(y_t | x, y_{<t})} = \underbrace{\frac{\pi_{\theta_{\text{old}}}(y_t | x, y_{<t}, e^\pi_{\text{old}, t})}{\mu_{\theta_{\text{old}}}(y_t | x, y_{<t}, e^\mu_{\text{old}, t})}}_{\text{training–inference discrepancy}} \times \underbrace{\frac{\pi_\theta(y_t | x, y_{<t}, e^\pi_t)}{\pi_{\theta_{\text{old}}}(y_t | x, y_{<t}, e^\pi_{\text{old}, t})}}_{\text{policy staleness}}$

What's new for MoE models. There are two distinct failure modes introduced by expert routing:

1. Inconsistent routing amplifies training–inference discrepancy: Even when $\theta = \theta_{\text{old}}$, the inference engine routes tokens through experts $e^\mu_{\text{old}, t}$ while the training engine routes through $e^\pi_{\text{old}, t}$. If these expert assignments differ, the probability distributions computed by the two engines differ not just from numerical precision but from using different subsets of parameters. This makes the training–inference discrepancy much larger for MoE models than for dense models.

2. Routing shifts amplify policy staleness: When the policy changes ($\theta \neq \theta_{\text{old}}$), two things happen simultaneously: the token-level probabilities change (as in dense models), AND the expert routing changes ($e^\pi_t \neq e^\pi_{\text{old}, t}$). A parameter update that changes which experts are activated for a given token can produce a discontinuous jump in the policy, because completely different parameters now process that token. This means policy staleness in MoE models is not just a smooth drift in probabilities but can involve wholesale changes in which computation path processes each token.

The paper summarizes this crisply:

"expert routing is entangled with the training–inference discrepancy and policy staleness, increasing the likelihood that the first-order approximation underlying the surrogate token-level optimization objective breaks down"

Why this is practically devastating for MoE RL training: the first-order approximation requires $\pi_\theta$ and $\mu_{\theta_{\text{old}}}$ to be close at every token. But in MoE models, even a small parameter update can cause a token that was previously routed through Expert A to now route through Expert B, producing a completely different output distribution. The per-token ratio can jump from near 1 to very far from 1 in a single gradient step, immediately invalidating the approximation. This explains the observation (common in practice, and motivating prior work on Routing Replay) that MoE models are substantially more brittle under RL training than equivalently-sized dense models.

---

Routing Replay: Restoring the Approximation by Fixing Experts

The paper formalizes two variants of Routing Replay as mechanisms for controlling the expert-routing component of the approximation error. The core idea is straightforward: during policy optimization (gradient computation), instead of letting the training engine dynamically route each token to experts based on the current parameters, we fix the routed experts to those determined during rollout. This prevents the routing shifts that amplify both the training–inference discrepancy and policy staleness.

Vanilla Routing Replay (R2). In R2 (originally from Zheng et al., 2025), during gradient updates, the training engine replays the experts that were routed by the rollout policy as computed in the training engine ($e^\pi_{\text{old}, t}$). The modified token-level IS weight becomes:

$\frac{\pi^{\text{R2}}_\theta(y_t | x, y_{<t})}{\mu_{\theta_{\text{old}}}(y_t | x, y_{<t})} = \frac{\pi_\theta(y_t | x, y_{<t}, e^\pi_{\text{old}, t})}{\mu_{\theta_{\text{old}}}(y_t | x, y_{<t}, e^\mu_{\text{old}, t})}$

The superscript $\pi^{\text{R2}}_\theta$ indicates that this is no longer the original target policy $\pi_\theta$ with naturally-routed experts, but a modified target policy where expert routing is constrained to $e^\pi_{\text{old}, t}$ regardless of what the current parameters would naturally route.

Decomposing:

$\frac{\pi^{\text{R2}}_\theta(y_t | x, y_{<t})}{\mu_{\theta_{\text{old}}}(y_t | x, y_{<t})} = \frac{\pi_{\theta_{\text{old}}}(y_t | x, y_{<t}, e^\pi_{\text{old}, t})}{\mu_{\theta_{\text{old}}}(y_t | x, y_{<t}, e^\mu_{\text{old}, t})} \times \underbrace{\frac{\pi_\theta(y_t | x, y_{<t}, e^\pi_{\text{old}, t})}{\pi_{\theta_{\text{old}}}(y_t | x, y_{<t}, e^\pi_{\text{old}, t})}}_{\text{policy staleness} \downarrow}$

What R2 accomplishes: it eliminates the routing component of policy staleness because $e^\pi_{\text{old}, t}$ is fixed, so the policy staleness term now only reflects changes in the token probabilities given the same experts — it no longer includes the discontinuous effect of routing changes. The training–inference discrepancy term is unaffected: the numerator still uses $e^\pi_{\text{old}, t}$ and the denominator still uses $e^\mu_{\text{old}, t}$, so the routing mismatch between engines persists.

Rollout Routing Replay (R3). In R3 (originally from Ma et al., 2025), during gradient updates, the training engine replays the experts that were routed by the rollout policy as computed in the inference engine ($e^\mu_{\text{old}, t}$). The modified token-level IS weight becomes:

$\frac{\pi^{\text{R3}}_\theta(y_t | x, y_{<t})}{\mu_{\theta_{\text{old}}}(y_t | x, y_{<t})} = \frac{\pi_\theta(y_t | x, y_{<t}, e^\mu_{\text{old}, t})}{\mu_{\theta_{\text{old}}}(y_t | x, y_{<t}, e^\mu_{\text{old}, t})}$

Decomposing:

$\frac{\pi^{\text{R3}}_\theta(y_t | x, y_{<t})}{\mu_{\theta_{\text{old}}}(y_t | x, y_{<t})} = \underbrace{\frac{\pi_{\theta_{\text{old}}}(y_t | x, y_{<t}, e^\mu_{\text{old}, t})}{\mu_{\theta_{\text{old}}}(y_t | x, y_{<t}, e^\mu_{\text{old}, t})}}_{\text{training–inference discrepancy} \downarrow} \times \underbrace{\frac{\pi_\theta(y_t | x, y_{<t}, e^\mu_{\text{old}, t})}{\pi_{\theta_{\text{old}}}(y_t | x, y_{<t}, e^\mu_{\text{old}, t})}}_{\text{policy staleness} \downarrow}$

What R3 accomplishes: it eliminates the routing component of the training–inference discrepancy because both numerator and denominator now use the same experts ($e^\mu_{\text{old}, t}$) — the only remaining discrepancy comes from numerical precision differences between engines when computing probabilities given the same experts. It also eliminates the routing component of policy staleness for the same reason as R2: experts are fixed, so policy changes are limited to probability shifts rather than routing changes. R3 is strictly stronger than R2 in terms of reducing approximation error sources, because it addresses both factors rather than just one.

The bias introduced by Routing Replay. The cost of fixing experts is that the policy being optimized is no longer the true target policy $\pi_\theta$ (which would use naturally-routed experts $e^\pi_t$). Instead, R2 optimizes $\pi^{\text{R2}}_\theta$ (experts fixed to $e^\pi_{\text{old}, t}$) and R3 optimizes $\pi^{\text{R3}}_\theta$ (experts fixed to $e^\mu_{\text{old}, t}$). These are biased versions of the true policy. The bias can be understood as: we are improving the model's ability to generate tokens given specific expert assignments, but those expert assignments may not be the ones the model would naturally use at deployment. The paper explicitly flags this tradeoff:

"Routing Replay intuitively restores the validity of the first-order approximation in MoE models by reducing the training–inference discrepancy (in R3) and alleviating policy staleness (in R2 and R3). However, it also implicitly biases the target policy... Routing Replay constrains the routed experts... leading to another target policy $\pi^{\text{R2}}_\theta$ or $\pi^{\text{R3}}_\theta$ that deviates from the original $\pi_\theta$"

When does the bias matter? The paper identifies a crucial asymmetry in how R2 and R3 affect different mini-batches during off-policy training. As shown in Table 1:

• First mini-batch: When $N = 1$ (on-policy) or for the first mini-batch in an off-policy setting, $\theta = \theta_{\text{old}}$ exactly, so the training engine's naturally-routed experts equal the rollout policy's training-engine experts: $e^\pi_{\text{old}, t} = e^\pi_t$. In this case, R2 does not alter the target policy — the fixed experts are exactly the ones the current policy would have chosen anyway. R3, however, still alters the target policy because $e^\mu_{\text{old}, t} \neq e^\pi_t$ (the inference and training engines route differently). So in the first mini-batch, R2 is unbiased while R3 introduces bias without providing any staleness benefit (since staleness is zero).

• Non-first mini-batches: After gradient updates, $\theta \neq \theta_{\text{old}}$, so the training engine would naturally route tokens to different experts than during rollout. Both R2 and R3 alter the target policy ($e^\pi_{\text{old}, t} \neq e^\pi_t$ and $e^\mu_{\text{old}, t} \neq e^\pi_t$), but R3's alteration may be less severe because $e^\mu_{\text{old}, t}$ was determined by the inference engine that actually generated the response, making it more consistent with the data distribution.

The key hypothesis about R2 vs. R3. The paper conjectures that R2 should outperform R3 when off-policiness is small (few mini-batches per global batch) because R2 is unbiased in the first mini-batch and policy staleness is already modest. When off-policiness is large, R3 should outperform R2 because the bias from fixing to inference-engine experts becomes less costly than the instability from accumulating routing shifts across many stale mini-batches. This is exactly what the experiments in Section 4.4 confirm (Figures 2–4).

---

MiniRL: The Minimalist Baseline Algorithm

To empirically validate the theoretical framework, the paper constructs MiniRL — a minimal policy gradient algorithm that includes only those modifications justified by the first-order approximation analysis. MiniRL is intentionally not novel; it is REINFORCE with importance sampling correction and PPO-style clipping, stripped of common but theoretically unjustified additions like length normalization.

The MiniRL objective:

$J^{\text{MiniRL}}(\theta) = \mathbb{E}_{x \sim \mathcal{D},\ y \sim \mu_{\theta_{\text{old}}}(\cdot|x)} \left[ \sum_{t=1}^{|y|} M_t \cdot \text{sg}\left( \frac{\pi_\theta(y_t | x, y_{<t})}{\mu_{\theta_{\text{old}}}(y_t | x, y_{<t})} \right) \cdot \hat{A}(x, y) \cdot \log \pi_\theta(y_t | x, y_{<t}) \right]$

where:

• $\mu_{\theta_{\text{old}}}$ is the rollout policy in the inference engine,
• $\hat{A}(x, y)$ is the group-normalized advantage estimate for response $y$,
• $M_t \in \{0, 1\}$ is a per-token clipping mask (explained below),
• $\text{sg}(\cdot)$ is the stop-gradient operator — the importance weight is treated as a constant multiplier, not differentiated through,
• $\log \pi_\theta(y_t | x, y_{<t})$ is the log-probability of token $y_t$, which is the only term that receives gradient.

What this objective computes: for each response $y$ sampled under the rollout policy, we compute (1) a per-token importance weight $\pi_\theta / \mu_{\theta_{\text{old}}}$ at each position to correct for training–inference discrepancy and policy staleness, (2) stop the gradient through this weight so that we are optimizing the policy toward tokens that had high importance weights rather than trying to increase the importance weight itself, (3) multiply by the advantage $\hat{A}(x, y)$ — a positive number means the response was better than average and we should increase its probability, a negative number means it was worse and we should decrease it, and (4) optionally zero out the gradient for certain tokens via the clipping mask $M_t$ to prevent overly aggressive updates. The gradient of this objective with respect to $\theta$ produces the per-token REINFORCE update with importance sampling correction.

The group-normalized advantage estimate. The paper applies group-normalization (from Shao et al., 2024) to convert raw rewards into advantages:

$\hat{A}(x, y) = R(x, y) - \mathbb{E}_{y' \sim \mu_{\theta_{\text{old}}}(\cdot|x)} [R(x, y')]$

where the expectation is computed over the $G$ responses sampled for the same prompt $x$ (the "group"). In practice, with $G = 16$ responses per prompt, the advantage for response $i$ is:

$\hat{A}(x, y_i) = R(x, y_i) - \frac{1}{G} \sum_{j=1}^{G} R(x, y_j)$

Why group normalization: subtracting the group mean centers the rewards so that above-average responses get positive advantages (encouraged) and below-average responses get negative advantages (discouraged). This is a variance reduction technique — it removes the effect of prompt difficulty from the reward signal (hard prompts will have low average reward and easy prompts high average reward, but the advantage captures only relative quality within the group). Note that MiniRL does NOT divide by the standard deviation (unlike GRPO, which uses $z$-score normalization). The paper keeps the advantage in raw reward units rather than standard deviation units. It also does not apply length normalization — the advantage is uniform across all tokens in a response rather than being divided by $|y|$.

Why no length normalization: this is a deliberate design choice justified by the theory. Length normalization would modify the gradient to be:

$\nabla_\theta J^{\text{w/ length-norm}}(\theta) = \nabla_\theta \mathbb{E} \left[ \frac{1}{|y|} \sum_{t=1}^{|y|} \frac{\pi_\theta(y_t | x, y_{<t})}{\mu_{\theta_{\text{old}}}(y_t | x, y_{<t})} R(x, y) \right] = \mathbb{E} \left[ \frac{1}{|y|} \sum_{t=1}^{|y|} \frac{\pi_\theta(y_t | x, y_{<t})}{\mu_{\theta_{\text{old}}}(y_t | x, y_{<t})} R(x, y) \nabla_\theta \log \pi_\theta(y_t | x, y_{<t}) \right]$

This gradient is NOT equal to $\nabla_\theta J^{\text{token}}$ scaled by $1/|y|$ because the expectation involves $\nabla_\theta$ of the importance weights and the $1/|y|$ factor cannot be pulled through the gradient operator cleanly. Consequently, length normalization destroys the first-order approximation to $\nabla_\theta J^{\text{seq}}$. The gradient of the length-normalized objective does not, even approximately, point in the direction of improving expected sequence-level reward. The paper's experiments confirm this: MiniRL with length normalization achieves stable but suboptimal performance (Figure 1) because the objective is biased — the gradients are coherent enough to avoid collapse but are optimizing the wrong thing.

The clipping mechanism. Clipping is implemented as a per-token binary mask $M_t$ that selectively zeros out gradients for certain tokens based on how far the token-level probability ratio has moved:

$M_t = \begin{cases} 0 & \text{if } \hat{A}(x, y) > 0 \text{ and } r_t > 1 + \varepsilon_{\text{high}} \\ 0 & \text{if } \hat{A}(x, y) < 0 \text{ and } r_t < 1 - \varepsilon_{\text{low}} \\ 1 & \text{otherwise} \end{cases}$

where:

$r_t = \frac{\pi_\theta(y_t | x, y_{<t})}{\pi_{\theta_{\text{old}}}(y_t | x, y_{<t})}$

is the policy staleness ratio (comparing the current policy to the rollout policy, both computed in the training engine), and $\varepsilon_{\text{high}} = 0.27$, $\varepsilon_{\text{low}} = 0.2$ are clipping thresholds.

What clipping does operationally:

• Case 1: If a token's probability has INCREASED substantially under the current policy ($r_t > 1.27$) AND the response had positive advantage (it was better than average), we clip the gradient for that token to zero. Without clipping, the gradient would further increase the token's probability, driving $r_t$ even higher and making the token-level ratio deviate further from 1 — this would invalidate the first-order approximation and cause the gradient to become decoupled from the true sequence-level objective.

• Case 2: If a token's probability has DECREASED substantially ($r_t < 0.8$) AND the response had negative advantage (it was worse than average), we clip the gradient to zero. Without clipping, the gradient would further decrease the probability, again driving the ratio away from 1 and breaking the approximation.

• Default: In all other cases, the gradient flows normally. This includes all tokens where $r_t$ is near 1 (the approximation is valid) and tokens where the direction of change is beneficial for the approximation (increasing probability when advantage is negative, or decreasing when advantage is positive — both move $r_t$ toward 1).

Why this form of clipping: the paper follows the "decoupled PPO" approach from Hilton et al. (2022), using $\pi_{\theta_{\text{old}}}$ (the training-engine rollout policy) as the proximal policy — the reference point against which to measure how far the current policy has moved. This is a deliberate choice: clipping against $\pi_{\theta_{\text{old}}}$ rather than $\mu_{\theta_{\text{old}}}$ means clipping only addresses policy staleness, not training–inference discrepancy. The training–inference discrepancy is handled separately by the IS correction term $\pi_\theta / \mu_{\theta_{\text{old}}}$, which is stopped-gradient (not clipped). This separation allows each mechanism to target its intended error source: IS correction handles engine mismatch, clipping handles staleness.

The stop-gradient on the IS weight. The term $\text{sg}(\pi_\theta / \mu_{\theta_{\text{old}}})$ uses the stop-gradient operator. This means that during backpropagation, the IS weight is treated as a constant — gradients flow through $\log \pi_\theta(y_t | x, y_{<t})$ but not through the IS weight. Mechanically, this produces the gradient:

$\nabla_\theta J^{\text{MiniRL}} = \mathbb{E} \left[ \sum_t M_t \cdot \frac{\pi_\theta(y_t | x, y_{<t})}{\mu_{\theta_{\text{old}}}(y_t | x, y_{<t})} \cdot \hat{A}(x, y) \cdot \nabla_\theta \log \pi_\theta(y_t | x, y_{<t}) \right]$

which is exactly the token-level REINFORCE gradient of Equation (4) with clipping. Without the stop-gradient, there would be an additional gradient term through the IS weight that would attempt to optimize the IS weight itself rather than optimizing the policy.

Truncated Importance Sampling (TIS). The paper additionally applies truncation to the token-level IS weight, with a threshold of 5. This means that if $\pi_\theta(y_t | x, y_{<t}) / \mu_{\theta_{\text{old}}}(y_t | x, y_{<t}) > 5$, the weight is capped at 5. This is a standard variance reduction technique — extreme IS weights, even at the token level, can occasionally occur and dominate the gradient, and truncation prevents individual tokens from having disproportionate influence. The threshold of 5 is a practical choice; the paper does not ablate it directly but references Yao et al. (2025) for the technique.

Comparison with GRPO and CISPO. The paper explicitly contrasts MiniRL with two popular alternatives (Appendix A). GRPO uses:

$J^{\text{GRPO}}(\theta) = \mathbb{E}_{x, \{y_i\}_{i=1}^G \sim \mu_{\theta_{\text{old}}}} \left[ \frac{1}{G} \sum_{i=1}^G \frac{1}{|y_i|} \sum_{t=1}^{|y_i|} \min\left( r_{i,t}(\theta) \hat{A}_{i,t}, \text{clip}(r_{i,t}(\theta), 1-\varepsilon_{\text{low}}, 1+\varepsilon_{\text{high}}) \hat{A}_{i,t} \right) \right]$

where $r_{i,t}(\theta) = \pi_\theta(y_{i,t}|x, y_{i,<t}) / \pi_{\theta_{\text{old}}}(y_{i,t}|x, y_{i,<t})$ and $\hat{A}_{i,t}$ is a $z$-score normalized advantage. GRPO differs from MiniRL in three ways: (1) no training–inference IS correction ($\pi_{\theta_{\text{old}}}$ in the denominator rather than $\mu_{\theta_{\text{old}}}$), (2) length normalization (dividing by $|y_i|$), and (3) min-clipping rather than stop-gradient masking. CISPO uses:

$J^{\text{CISPO}}(\theta) = \mathbb{E}_{x, y \sim \mu_{\theta_{\text{old}}}} \left[ \frac{1}{\sum_{i=1}^G |y_i|} \sum_{i=1}^G \sum_{t=1}^{|y_i|} \text{sg}\left[ \text{clip}(r_{i,t}(\theta), 1-\varepsilon_{\text{low}}, 1+\varepsilon_{\text{high}}) \right] \hat{A}_{i,t} \log \pi_\theta(y_t | x, y_{<t}) \right]$

CISPO includes stop-gradient clipping but retains length normalization and lacks training–inference IS correction. MiniRL is thus the unique algorithm that (1) corrects for training–inference discrepancy via $\mu_{\theta_{\text{old}}}$ in the IS denominator, (2) avoids length normalization, and (3) uses stop-gradient IS weights rather than gradient-through clipping — all justified by the requirement that the gradient must approximate $\nabla_\theta J^{\text{seq}}$ for the optimization to be sound.

Experimental setup and hyperparameters. The paper's experiments use the following configuration, which is important for understanding the scale and stress-testing nature of the evaluation:

• Model: Qwen3-30B-A3B-Base, fine-tuned as a cold-start, then RL-trained. This is a 30B-parameter MoE model with 3B active parameters per token.
• Precision: FP8 inference, BF16 training. This deliberate precision mismatch creates a large training–inference discrepancy — exactly the stress test needed to evaluate whether the theoretical framework holds under challenging conditions. The authors note this explicitly: "providing a stress test for algorithmic correctness where the inference precision is lower than the training and the training–inference discrepancy is large."
• Task: Mathematical reasoning with binary rewards ($R(x,y) \in \{0, 1\}$) based on comparing the model's answer to a ground-truth answer.
• Prompt set: 4,096 math problems with verified answers, curated for RL training.
• Evaluation benchmarks: HMMT25, AIME25, and AIME24, each with 30 competition-level math problems (90 total). Reported as average accuracy over 32 sampled responses per problem.
• Synchronous RL framework: Each global step: sample $B$ prompts → generate $G$ responses per prompt in the inference engine → split the $B \times G$ responses into $N$ mini-batches → apply $N$ gradient updates in the training engine → the updated policy becomes the new rollout policy.
• Fixed mini-batch size: 1,024 responses ($B = 64$ prompts $\times G = 16$ responses per prompt) for every gradient update across all experiments.
• Maximum generation length: 32,768 tokens.
• Clipping thresholds: $\varepsilon_{\text{high}} = 0.27$, $\varepsilon_{\text{low}} = 0.2$ in MiniRL.
• TIS truncation threshold: 5.
• Compute cost: approximately 5–6 GPU hours per gradient step, with total experiments spanning hundreds of thousands of GPU hours.

Three variants of on-policy MiniRL ablated in Section 4.3. Under on-policy training ($N = 1$, so global batch size = mini-batch size = 1,024), $\theta = \theta_{\text{old}}$ exactly, so MiniRL simplifies to:

$J^{\text{MiniRL}}(\theta) = \mathbb{E}_{x \sim \mathcal{D},\ y \sim \mu_{\theta_{\text{old}}}(\cdot|x)} \left[ \sum_{t=1}^{|y|} \frac{\pi_{\theta_{\text{old}}}(y_t | x, y_{<t})}{\mu_{\theta_{\text{old}}}(y_t | x, y_{<t})} \hat{A}(x, y) \log \pi_\theta(y_t | x, y_{<t}) \right]$

where the IS weight now only corrects for training–inference discrepancy ($\pi_{\theta_{\text{old}}} / \mu_{\theta_{\text{old}}}$) since policy staleness is zero. The paper compares this against two ablated variants:

1. MiniRL + length-norm: adds $1/|y|$ length normalization — this invalidates the first-order approximation but is commonly used in practice (as in GRPO, CISPO).
2. MiniRL − train-infer-IS: removes the training–inference IS correction entirely, using $\hat{A}(x,y) \log \pi_\theta$ directly — this completely breaks the first-order approximation because the gradient no longer relates to $\nabla_\theta J^{\text{seq}}$.

Each variant is also tested with and without R3 (R2 is not applicable on-policy since $e^\pi_{\text{old}, t} = e^\pi_t$ in the first/only mini-batch — see Table 1).

Off-policy MiniRL variants in Section 4.4. For off-policy experiments, the global batch size is set to $N \times$ the mini-batch size, where $N = 2, 4, 8$ (global batch sizes of 2,048, 4,096, and 8,192 respectively). MiniRL is the full objective in Equation (7), with both IS correction and clipping active. The compared methods are:

• MiniRL (no clipping): removes the clipping mask $M_t$ — all tokens receive gradients regardless of ratio. This tests whether clipping is essential for controlling policy staleness.
• MiniRL + R2 (no clipping): uses Vanilla Routing Replay but without clipping.
• MiniRL + R2: both Routing Replay (R2) and clipping.
• MiniRL + R3: both Routing Replay (R3) and clipping.

The experiments are designed to isolate each component's contribution: the (no clipping) variants test whether Routing Replay alone suffices to prevent collapse, and the R2 vs. R3 comparison tests the tradeoff between bias and approximation validity under different off-policiness levels.

Training stability metrics. Beyond benchmark scores and training reward, the paper monitors two diagnostic metrics that operationalize the two sources of approximation error:

1. Token-level entropy of the target policy:

$H[\pi_\theta] \approx \mathbb{E}_{x \sim \mathcal{D},\ y_{<t} \sim \mu_{\theta_{\text{old}}}(\cdot|x)} \left[ -\sum_{w \in \mathcal{V}} \pi_\theta(w | x, y_{<t}) \log \pi_\theta(w | x, y_{<t}) \right]$

where $\mathcal{V}$ is the vocabulary. Entropy measures the policy's uncertainty — high entropy means the policy is exploring broadly; collapsing entropy (sharp drop) indicates the policy is becoming deterministic and potentially overfitting to spurious patterns. A healthy training run shows entropy decreasing gradually as the policy becomes more confident in correct reasoning patterns; an unhealthy run shows entropy crashing as the policy collapses to a degenerate mode.

2. KL divergence between inference and training engine rollout policies:

$D_{\text{KL}}(\mu_{\theta_{\text{old}}} \| \pi_{\theta_{\text{old}}}) = \mathbb{E}_{x \sim \mathcal{D},\ y_t \sim \mu_{\theta_{\text{old}}}(\cdot|x, y_{<t})} \left[ \log \frac{\mu_{\theta_{\text{old}}}(y_t | x, y_{<t})}{\pi_{\theta_{\text{old}}}(y_t | x, y_{<t})} \right]$

This directly measures the training–inference discrepancy: how different are the probability distributions computed by the two engines given identical parameters and inputs? A healthy run maintains low and stable KL divergence; a sharp increase in this metric is a leading indicator of training collapse, as identified by Yao et al. (2025) and Liu et al. (2025a). The paper plots this on a log scale because divergence can spike by orders of magnitude during collapse events.

Why these particular metrics: together they provide a real-time diagnostic for whether the first-order approximation is valid. Low and stable $D_{\text{KL}}(\mu_{\theta_{\text{old}}} \| \pi_{\theta_{\text{old}}})$ means the training–inference discrepancy is under control. Moderate and gradually decreasing entropy means the policy is learning without collapsing — policy staleness is accumulating gradually rather than catastrophically. A sudden divergence increase accompanied by an entropy crash is the signature of the first-order approximation breaking: the gradient steps are no longer coherently improving the sequence-level objective, and the policy enters a vicious cycle where bad updates drive larger discrepancies, which make subsequent updates even worse.

4. Key Insights and Innovations

Innovation 1: Token-Level Objectives as a First-Order Approximation — A Principled Criterion for Algorithmic Soundness

Before this paper, the field of RL-for-LLMs operated on empirical precedent rather than theoretical justification. Algorithms like GRPO (Shao et al., 2024) and PPO variants were adopted because they seemed to work in practice — not because anyone could explain why optimizing a token-level objective should reliably improve a sequence-level reward signal. This left practitioners in an uncomfortable position: training was fragile, collapses were unpredictable, and when things went wrong there was no diagnostic framework for identifying the root cause. Was a training collapse due to bad hyperparameters? To the verifier? To something about the model architecture? Nobody could say with confidence.

The paper's central conceptual move is to provide exactly this missing theoretical criterion. By deriving the token-level objective as a first-order Taylor approximation to the true sequence-level objective, the authors give the field a principled test for whether any given RL algorithm is doing something coherent: is its gradient approximately equal to the gradient of the expected sequence-level reward? If yes, the algorithm is directionally sound — gradient steps point (roughly) toward better expected reward. If no, the algorithm is optimizing something other than what the practitioner cares about, and the fact that it sometimes produces good checkpoints is fortuitous rather than reliable.

This is a fundamental conceptual advance, not an incremental refinement. It transforms the question "does this RL recipe work?" from an empirical one (requiring expensive trial-and-error at scale) to a partly analytical one (does the objective preserve the first-order approximation?). Consider the paper's length-normalization result (Section 4.3, Figure 1): MiniRL with length normalization trains stably — it doesn't collapse — but achieves suboptimal benchmark scores. Without the approximation framework, this would be a puzzling empirical finding: why does a seemingly innocuous design choice (dividing by response length) hurt performance without causing obvious instability? With the framework, the explanation is immediate: length normalization destroys the equality between $\nabla_\theta J^{\text{token}}$ and $\nabla_\theta J^{\text{seq}}$, creating a biased objective that converges to a different optimum. The training is stable because the gradients remain coherent — they just point in the wrong direction.

This diagnostic power extends beyond the paper's own experiments. Any future proposed RL algorithm for LLMs can, in principle, be evaluated against this criterion: does each component preserve, weaken, or break the first-order approximation? Clipping preserves it (by preventing policy staleness from growing); IS correction preserves it (by canceling the training–inference discrepancy in the gradient); length normalization breaks it; omitting IS correction breaks it. The framework provides a unified language for reasoning about algorithmic design choices that were previously justified only by folklore or small-scale ablation studies.

The significance goes beyond explanation into prediction. The paper's decomposition of approximation error into two multiplicative factors — training–inference discrepancy and policy staleness (Equation 5) — predicts observable signatures of impending failure: a spike in $D_{\text{KL}}(\mu_{\theta_{\text{old}}} \| \pi_{\theta_{\text{old}}})$ indicates the discrepancy term is growing, and a collapsing entropy indicates policy staleness is being driven by degenerate updates. These are precisely the patterns observed in the paper's collapse experiments (Figures 1–4). Prior work (Yao et al., 2025; Liu et al., 2025a) had noted that RL collapses correlate with increasing training–inference divergence, but had not connected this observation to the validity of the optimization objective itself. This paper elevates that correlation to a causal mechanism: divergence doesn't just correlate with collapse, it causes collapse by invalidating the approximation that makes token-level optimization meaningful.

In short: before this paper, the field had a collection of heuristics for making RL training work; after it, the field has a criterion for evaluating whether any given technique should work, and a diagnostic dashboard for detecting when it's failing. That's a qualitative shift in how the problem is understood.

---

Innovation 2: The Training–Inference Discrepancy as a First-Class Source of Instability (Not a Nuisance)

The dominant assumption in RL-for-LLMs, inherited from classical RL where the "environment" is an MDP with well-defined transition probabilities, was that the policy that generates data and the policy that computes gradients are the same mathematical object — or at least close enough that the difference can be ignored. Prior work on training–inference mismatch (Yao et al., 2025) treated it as an implementation artifact to be mitigated, not a fundamental determinant of algorithmic validity.

This paper makes a bolder claim: the training–inference discrepancy is structurally equivalent to policy staleness in determining whether token-level RL is sound. The decomposition in Equation (5) — $\pi_\theta / \mu_{\theta_{\text{old}}} = (\pi_{\theta_{\text{old}}} / \mu_{\theta_{\text{old}}}) \times (\pi_\theta / \pi_{\theta_{\text{old}}})$ — shows that these two factors multiply together. If either one deviates substantially from 1, the product does too, and the first-order approximation breaks. This is not two separate problems that can be addressed independently; it's a single product that must remain near 1. A system with perfect policy freshness (on-policy training) can still collapse if the training–inference discrepancy is large. Conversely, a system with a well-matched training–inference stack can still collapse if policy staleness accumulates too much.

The practical implication of this reframing is substantial. It means that importance sampling correction for the training–inference discrepancy is not optional — it's as essential as clipping for controlling policy staleness. The on-policy experiments (Figure 1) demonstrate this starkly: removing the IS correction (MiniRL − train-infer-IS) causes rapid entropy collapse and training failure, even though policy staleness is identically zero (since $\theta = \theta_{\text{old}}$ in on-policy training). This is a clean experiment: with staleness eliminated as a confounding factor, the training–inference discrepancy alone is sufficient to cause collapse when uncorrected. The IS weight is not a "nice-to-have" variance reduction tool; it is an inherent component of the first-order approximation, and omitting it makes the gradient of the token-level objective no longer equal to (or even proportional to) the gradient of the sequence-level objective.

This is a conceptual upgrade from prior work. Yao et al. (2025) identified the training–inference discrepancy as a source of off-policy training and proposed Truncated Importance Sampling (TIS) as a mitigation, but did not articulate why TIS helps beyond variance reduction. This paper provides the "why": TIS helps because it keeps the token-level IS weight bounded, which keeps the $\delta_t$ terms small in the Taylor expansion, which preserves the validity of the approximation. The truncation threshold of 5 is not an arbitrary hyperparameter; it's the point beyond which the $\delta_t$ for that token is large enough that the second-order terms we dropped in the expansion become non-negligible, and the gradient update on that token no longer reliably corresponds to improving expected sequence-level reward.

This insight also explains a puzzling pattern in Figure 1 that might otherwise be misinterpreted. MiniRL + R3 (Rollout Routing Replay) with IS correction shows lower benchmark scores than MiniRL without R3 in the on-policy setting, despite R3 reducing the training–inference KL divergence. If the training–inference discrepancy were simply "bad" and reducing it were simply "good," R3 should help. But the theory explains why it doesn't: in on-policy training, policy staleness is zero, and R3 introduces a bias (altering the target policy by fixing to inference-engine experts) without providing any staleness-reduction benefit. The IS correction alone is sufficient to handle the discrepancy in the gradient computation; adding R3 trades a solved problem for a new bias, and performance suffers. This nuanced prediction — that the same technique (Routing Replay) can be harmful in one regime and essential in another — would be impossible to derive without the approximation framework.

---

Innovation 3: A Unified Account of Why Standard Stabilization Techniques Work (and When They Don't)

The field of RL-for-LLMs has accumulated a toolkit of stabilization techniques — importance sampling correction, clipping, length normalization, Routing Replay — through a process of empirical trial-and-error. Practitioners knew these techniques often helped, but the understanding was fragmented: each technique was justified by its own local rationale, with no overarching framework for understanding how they interact or when one should be preferred over another.

This paper provides that unifying account. By showing that all effective stabilization techniques work through a single mechanism — preserving the validity of the first-order approximation — it transforms a collection of ad-hoc tricks into a coherent engineering discipline. Each technique targets a specific term in the decomposition of Equation (5):

• Importance sampling correction ($\pi_{\theta_{\text{old}}} / \mu_{\theta_{\text{old}}}$ in the weight) addresses the training–inference discrepancy factor directly. Without it, this factor is unaccounted for in the gradient, and the approximation breaks immediately (as shown by the collapse in Figure 1 for MiniRL − train-infer-IS).

• Clipping ($M_t$ masking based on $r_t = \pi_\theta / \pi_{\theta_{\text{old}}}$) addresses the policy staleness factor by preventing individual token ratios from growing beyond the small-$\delta_t$ regime. The clipping thresholds ($\varepsilon_{\text{high}} = 0.27$, $\varepsilon_{\text{low}} = 0.2$) are not arbitrary — they define the boundary of "close to 1" within which the first-order expansion is reliable. The paper's choice of $\varepsilon_{\text{high}} > \varepsilon_{\text{low}}$ (0.27 vs 0.2) is an interesting asymmetry: the policy is allowed to increase token probabilities slightly more aggressively than it decreases them, perhaps reflecting that encouraging correct tokens is safer than suppressing incorrect ones when the approximation is imperfect.

• Routing Replay addresses the expert-routing component that amplifies both factors in MoE models. R2 (replaying training-engine experts) reduces the routing contribution to policy staleness; R3 (replaying inference-engine experts) reduces the routing contribution to both staleness and the training–inference discrepancy. The prediction that R2 should be preferred when policy staleness is small (few mini-batches) and R3 when staleness is large (many mini-batches) — derived from the bias-vs-validity tradeoff in Table 1 — is confirmed experimentally (Figures 2–4).

What makes this account powerful is that it explains negative results as clearly as positive ones. Length normalization degrades performance not because of some mysterious interaction with the optimizer, but because it breaks the equality $\nabla_\theta J^{\text{token}} \approx \nabla_\theta J^{\text{seq}}$ — the objective being optimized is no longer the one the practitioner cares about. R3 hurts on-policy training because the bias it introduces (altered target policy) is not compensated by any staleness reduction when staleness is already zero. The ReST$^{EM}$ experiment mentioned in the prior sections (from the reference example) similarly fails because on-policy data collection amplifies spurious correlations — but in this paper's framework, the failure can be diagnosed as the revision model's training producing a large policy staleness that the stabilization mechanisms weren't configured to handle.

This is an advance in explanatory power rather than algorithmic novelty. The paper does not propose a new stabilization technique; it explains why the existing ones work and provides a decision procedure for choosing among them. For a practitioner setting up an RL training run with an MoE model, the framework says: (1) always include IS correction — it's not optional, (2) if you're doing off-policy updates, include clipping — it's necessary once staleness is non-zero, (3) if you're using an MoE model with off-policy updates, include Routing Replay — the routing shifts will otherwise amplify staleness beyond what clipping can handle, (4) choose R2 if off-policiness is modest ($N \leq 2$) and R3 if it's large ($N \geq 4$). This recipe, validated at scale across hundreds of thousands of GPU hours, is a practical distillation of the theory that would have been difficult to derive through pure empiricism.

---

Innovation 4: Training Stability, Not Initialization, Is the Binding Constraint — A Reframing of Research Priorities

There is an implicit assumption in much LLM fine-tuning work that the quality of the cold-start model — the supervised fine-tuned checkpoint you begin RL from — is the dominant determinant of final performance. The reasoning is intuitive: RL can only polish what already exists; if the starting policy doesn't produce any correct solutions, RL has no signal to amplify. This assumption motivates substantial investment in cold-start data curation, multi-stage SFT pipelines, and careful initialization strategies.

This paper challenges that assumption with a simple but powerful empirical finding: given stable RL training and sufficient optimization, different cold-start initializations converge to comparable final performance. Three cold-start models distilled from different frontier models — Qwen3-Max-Thinking-Preview, DeepSeek-R1-0528, and gpt-oss-120b (high mode) — all reach similar benchmark scores on AIME25 and AIME24 after extended RL training (Figure 5). The initial differences in quality are washed out by the optimization process.

The significance of this finding is not that cold-start quality is irrelevant — the models all start from reasonably capable checkpoints — but that the binding constraint on RL success is training stability, not initialization quality. If your RL recipe is unstable, it doesn't matter how good your cold-start is; training will collapse before reaching the performance ceiling. If your recipe is stable, the model will improve steadily from a wide range of starting points, and differences in initialization will matter less and less as training proceeds. The practical implication is a shift in where practitioners should invest effort: from obsessing over cold-start data toward engineering reliable, stable RL training pipelines that can sustain prolonged optimization.

This is a conceptual reframing with direct resource-allocation consequences. Organizations investing in RL-for-LLMs face a choice: spend engineering effort on better cold-start data, or spend it on better stabilization techniques (IS correction, clipping, Routing Replay, monitoring infrastructure). The paper's evidence suggests the latter has higher leverage — improvements to training stability compound across the entire RL run, while improvements to initialization only affect the starting point and their effects decay as optimization proceeds. This is reinforced by the finding that both on-policy and off-policy training, once stabilized, achieve similar peak performance (comparing Figure 1 with Figures 2–4). The specific training configuration matters less than whether the configuration is stable enough to sustain optimization.

The finding also has implications for the broader research agenda. If stable training consistently converges to similar performance regardless of initialization, then benchmarking RL algorithms requires controlling for training duration — a method that achieves higher scores after 500 steps but collapses at 1,000 steps may be worse than one that plateaus lower but never collapses. The paper's emphasis on monitoring training dynamics (entropy, training–inference KL) rather than just final benchmark scores provides a template for more rigorous evaluation. A "good" RL algorithm should not just achieve high performance; it should maintain stable internal dynamics throughout training, with smoothly decreasing entropy and bounded KL divergence, indicating that the first-order approximation remains valid and the model's internal state is evolving coherently.

Note that this finding does NOT mean cold-start is irrelevant in an absolute sense. The paper's experiments start from fine-tuned checkpoints that already achieve non-trivial math performance; the claim is about which differences in initialization matter given a reasonable starting point. A randomly initialized model with near-zero pass@1 on math problems would presumably not converge regardless of stability, because there would be no reward signal to learn from. The paper's framework actually predicts this: if the base model never produces correct solutions, the advantage $\hat{A}(x,y)$ is always zero or negative, and there is no gradient signal to shape the policy toward correctness. The cold-start model needs to be good enough to generate some correct answers so that the RL signal exists; beyond that threshold, stability matters more than further improvements to initialization.

This reframing connects back to the paper's theoretical framework in a satisfying way. Stable training means the first-order approximation remains valid throughout optimization, which means each gradient step reliably improves expected sequence-level reward. If the approximation holds, the optimization process is well-behaved and will converge toward a local optimum of $J^{\text{seq}}$ regardless of where it starts (within the basin of attraction). If the approximation breaks, the optimization process becomes a random walk that may or may not stumble into good regions of parameter space, and initialization matters enormously because it determines which random path is taken. The paper's cold-start experiment is thus a validation of the theory: stable training makes the optimization outcome more deterministic (less dependent on initialization) because the gradient signal is reliable.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The RL training uses a curated set of 4,096 math problems with verified answers as the prompt set. Evaluation is conducted on three competition-level math benchmarks: HMMT25, AIME25, and AIME24, each containing 30 problems (90 total). These are standard benchmarks for mathematical reasoning in LLMs, with problems requiring multi-step derivations and producing a single verifiable answer.

• Base model(s). The primary model is a cold-start checkpoint fine-tuned from Qwen3-30B-A3B-Base — a 30B-parameter Mixture-of-Experts model with 3B active parameters per token. The authors argue this architecture is "representative of the capabilities of many contemporary LLMs" (Section 4.2) and note that MoE models pose unique challenges for RL training stability, making it a meaningful test case. For the cold-start initialization experiment (Section 4.5), three different cold-start models are compared, each distilled from a different frontier model: Qwen3-Max-Thinking-Preview, DeepSeek-R1-0528, and gpt-oss-120b (high mode), fine-tuned onto an early-experimental Qwen3Next MoE model.

• Metrics. The primary evaluation metric is average accuracy over 32 sampled responses per problem on each benchmark (HMMT25, AIME25, AIME24). For training monitoring, the paper reports training reward (the fraction of sampled responses receiving a reward of 1, i.e., correct answers) and two diagnostic metrics designed to track the validity of the first-order approximation: (1) token-level entropy of the target policy $H[\pi_\theta]$, computed as $-\mathbb{E}[\sum_{w \in \mathcal{V}} \pi_\theta(w|x, y_{<t}) \log \pi_\theta(w|x, y_{<t})]$, measuring policy uncertainty — a sharp drop indicates collapse to a degenerate mode; and (2) KL divergence between inference and training engine rollout policies, $D_{\text{KL}}(\mu_{\theta_{\text{old}}} \| \pi_{\theta_{\text{old}}}) = \mathbb{E}[\log(\mu_{\theta_{\text{old}}}(y_t|x, y_{<t}) / \pi_{\theta_{\text{old}}}(y_t|x, y_{<t}))]$, directly measuring the training–inference discrepancy. The reward function is binary: $R(x,y) \in \{0, 1\}$ based on comparing the model's extracted final answer against the ground-truth answer.

• Baselines. The primary baseline is MiniRL — a minimalist policy gradient algorithm constructed by the authors (Equation 7) — which serves as the default configuration against which all ablations are compared. The paper also explicitly compares MiniRL to GRPO (Shao et al., 2024) and CISPO (Chen et al., 2025) in Appendix A, noting three key differences: (1) both GRPO and CISPO omit training–inference IS correction (using $\pi_{\theta_{\text{old}}}$ in the denominator rather than $\mu_{\theta_{\text{old}}}$), (2) both employ length normalization, and (3) CISPO uses stop-gradient clipping rather than gradient-through clipping. The paper's ablated variants — MiniRL + length-norm, MiniRL − train-infer-IS, MiniRL (no clipping), MiniRL + R2, MiniRL + R3 — serve as intra-method baselines for isolating specific mechanisms.

• Generation budget / compute accounting. The unit of compute is the gradient step, with each step processing a mini-batch of 1,024 responses (64 prompts × 16 responses each). The global batch size varies across experiments: 1,024 (on-policy, $N=1$), 2,048 ($N=2$ mini-batches), 4,096 ($N=4$), and 8,192 ($N=8$). The authors estimate approximately 5–6 GPU hours per gradient step, with total experiments spanning "hundreds of thousands of GPU hours" (Section 4.2). The maximum generation length is 32,768 tokens. Precision is FP8 for inference and BF16 for training — a deliberate mismatch that creates a stress test for algorithmic correctness. Comparisons between methods are made at equivalent total gradient steps, which controls for total compute since the per-step cost is identical across variants.

• Cross-validation / statistical protocol. The paper does not employ formal cross-validation or statistical significance testing. The evaluation protocol uses fixed benchmarks (HMMT25, AIME25, AIME24) with 32 sampled responses per problem, and results are reported as point estimates (no confidence intervals or error bars). Training dynamics are monitored continuously via training reward, entropy, and training–inference KL divergence across gradient steps. The synchronous RL framework ensures that all compared methods within a given off-policy regime ($N = 2, 4, 8$) operate under identical data conditions — same batch composition, same mini-batch size, same rollout policy — so that differences in stability and performance can be attributed to algorithmic choices rather than data sampling variation.

---

Main Quantitative Results

On-Policy Training: MiniRL with IS Correction Achieves Highest Stability and Performance

The on-policy experiments (Figure 1, Appendix B Figure 6) set global batch size equal to mini-batch size (1,024), meaning $\theta = \theta_{\text{old}}$ exactly — policy staleness is zero, and the IS weight reduces to $\pi_{\theta_{\text{old}}}(y_t|x, y_{<t}) / \mu_{\theta_{\text{old}}}(y_t|x, y_{<t})$, correcting only for the training–inference discrepancy.

Headline result. MiniRL — the basic policy gradient with IS correction and no length normalization — achieves the highest benchmark scores and the most stable training dynamics across all on-policy configurations. After approximately 1,200 gradient steps, MiniRL reaches a benchmark score (averaged across HMMT25, AIME25, AIME24) of roughly 0.73–0.75, with training reward approaching 0.85–0.90 and entropy decreasing gradually from roughly 0.20 to 0.10 without abrupt drops.

MiniRL + length-norm: stable but suboptimal. Adding length normalization (dividing the per-token objective by response length $|y|$) produces stable training — no collapse — but the benchmark score plateaus lower, at roughly 0.68–0.70 (Figure 1, green vs. orange curves). Training–inference KL divergence remains similarly low and controlled (roughly $10^{-2}$ range), indicating that length normalization does not destabilize the approximation in a way that causes collapse, but rather introduces a persistent bias in the optimization objective itself. The paper states this confirms that "length normalization invalidates the first-order approximation to the true expected sequence-level reward, resulting in a biased token-level optimization objective" (Section 4.3).

MiniRL − train-infer-IS: rapid collapse. Removing the training–inference IS correction entirely — using $\hat{A}(x,y) \log \pi_\theta(y_t|x, y_{<t})$ without the $\pi_{\theta_{\text{old}}} / \mu_{\theta_{\text{old}}}$ weight — causes training to collapse within approximately 200–400 gradient steps (Figure 1, red curve): entropy drops sharply from roughly 0.20 to below 0.05, training–inference KL divergence spikes by more than an order of magnitude, and benchmark scores plummet. This is the cleanest demonstration that the IS weight is not optional — even when policy staleness is identically zero, the training–inference discrepancy alone is sufficient to invalidate the first-order approximation and cause catastrophic failure when uncorrected.

On-policy Routing Replay (R3): no benefit, can harm. Adding R3 (Rollout Routing Replay) to MiniRL in the on-policy setting yields benchmark scores comparable to or slightly below MiniRL without R3 (Figure 1, purple vs. orange). Training–inference KL divergence is substantially lower with R3 (as expected, since R3 eliminates the routing component of the discrepancy), but this reduction does not translate to improved performance. Combining R3 with length normalization degrades scores further (below 0.65), and R3 without IS correction still collapses. The paper interprets this as evidence that "in on-policy training, the detrimental impact of R3's alteration to the target policy outweighs its benefit in preserving the validity of the first-order approximation" (Section 4.3).

Per-benchmark breakdown (Appendix B, Figure 6). The patterns are consistent across HMMT25, AIME25, and AIME24 individually. MiniRL achieves the highest scores on all three benchmarks. AIME24 (the easiest of the three, with MiniRL reaching roughly 0.85–0.90) shows the clearest separation between MiniRL and the length-normalized variant. HMMT25 (the hardest, with MiniRL plateauing around 0.60–0.65) shows more compressed differences between methods, but the rank ordering is preserved.

---

Off-Policy Training: Routing Replay and Clipping Become Essential

The off-policy experiments (Figures 2–4, Appendix B Figures 7–9) introduce policy staleness by splitting a larger global batch into $N$ mini-batches for multiple gradient updates, with $N = 2, 4, 8$ corresponding to global batch sizes of 2,048, 4,096, and 8,192 respectively. Mini-batch size is fixed at 1,024 responses throughout.

Headline result (across all off-policy regimes). Both clipping and Routing Replay are necessary for stable off-policy training. Omitting either causes premature training collapse and degraded peak performance. The optimal Routing Replay variant shifts from R2 to R3 as off-policiness increases.

Off-policiness $N=2$ (Figure 2, Appendix B Figure 7). At the lowest off-policy level (global batch = 2,048, 2 mini-batches):

• MiniRL (no clipping): Training collapses after roughly 1,000 gradient steps. Entropy drops from roughly 0.42 to 0.30, and training–inference KL divergence begins increasing, diverging from the stable methods after about 750 steps. Benchmark score peaks around 0.68–0.70 before collapsing.
• MiniRL + R2 (no clipping): Also unstable — training collapses after roughly 1,200 steps, though the collapse occurs later than without R2, suggesting Routing Replay alone provides some but insufficient stabilization.
• MiniRL + R2 (with clipping): Training remains stable throughout the full 2,500 gradient steps. Benchmark scores rise steadily to approximately 0.75, training reward approaches 0.85, entropy decreases gradually from 0.42 to 0.30 without crashing, and training–inference KL divergence stays in the $10^{-2}$ range.
• MiniRL + R3 (with clipping): Also stable, but benchmark scores are consistently lower than MiniRL + R2, plateauing around 0.72. This is the first piece of evidence for the R2 > R3 preference at low off-policiness: R3's alteration of the target policy in the first mini-batch (where R2 would be unbiased) hurts more than any additional staleness reduction helps.

Off-policiness $N=4$ (Figure 3, Appendix B Figure 8). At moderate off-policy level (global batch = 4,096, 4 mini-batches):

• MiniRL (no clipping): Collapses even earlier — after roughly 500–750 gradient steps — with entropy crashing from 0.65 to below 0.35 and training–inference KL spiking. Peak benchmark score is roughly 0.70 before collapse.
• MiniRL + R2 (no clipping): Collapse is delayed but still occurs after roughly 1,000–1,200 steps. The later collapse relative to MiniRL (no clipping) suggests R2 provides partial stabilization, but without clipping, policy staleness eventually overwhelms the approximation.
• MiniRL + R2 (with clipping): Training is more stable than without clipping, but the paper notes it "fails to sustain stable training" (Section 4.4), with benchmark scores beginning to decline after roughly 2,000–2,500 steps. The peak score of roughly 0.76 is slightly lower than the R3 variant's eventual plateau.
• MiniRL + R3 (with clipping): This is the strongest configuration at $N=4$. Training remains stable across 4,000 gradient steps, benchmark scores reach approximately 0.78–0.80, training reward approaches 0.90, entropy decreases gradually from 0.65 to 0.35 without crashing, and training–inference KL divergence remains in the $10^{-1}$ to $10^{-2}$ range. This marks the crossover point where R3 overtakes R2.

Off-policiness $N=8$ (Figure 4, Appendix B Figure 9). At high off-policy level (global batch = 8,192, 8 mini-batches):

• MiniRL + R2 (no clipping): Collapses early. The paper notes that MiniRL without any Routing Replay was not run at this off-policy level (presumably because it would collapse almost immediately based on the $N=4$ results), so R2 without clipping serves as the closest point of comparison.
• MiniRL + R2 (with clipping): Becomes unstable at this off-policy level. Benchmark scores reach roughly 0.74–0.75 but begin declining after roughly 2,000 steps, and training–inference KL divergence rises. The paper states that "under high off-policiness, R2 fails to sustain stable training, and its peak performance achieved before training collapse is also slightly lower than that of R3" (Section 4.4).
• MiniRL + R3 (with clipping): Remains stable through 5,000 gradient steps, with benchmark scores reaching roughly 0.76–0.77, training reward around 0.88, entropy decreasing from 0.55 to 0.30, and KL divergence controlled in the $10^{-2}$ range. This confirms R3 as the necessary choice under high off-policiness.

Per-benchmark breakdown (Appendix B, Figures 7–9). The patterns hold consistently across HMMT25, AIME25, and AIME24. AIME24 (the easiest) shows the clearest separation between stable and unstable methods, with stable configurations reaching roughly 0.88–0.90 and unstable ones collapsing below 0.80. HMMT25 shows more variance, but the rank ordering of methods is preserved.

Summary of off-policy findings:

The key pattern: R2 dominates when off-policiness is small because it does not alter the target policy in the first mini-batch (Table 1); R3 dominates when off-policiness is large because the bias from altering experts is offset by the greater need to control the training–inference discrepancy and routing-induced staleness across many stale mini-batches.

---

Cold-Start Initialization: Stable Training Yields Comparable Final Performance

The cold-start experiments (Figure 5, Appendix B Figure 10) use an early-experimental Qwen3Next MoE model with a global batch size of 4,096, mini-batch size of 2,048 ($B = 128$, $G = 16$, $N = 2$), and maximum generation length of 65,536 tokens. The training recipe is MiniRL + R2.

Headline result. Three cold-start models initialized from different teacher models — Qwen3-Max-Thinking-Preview, DeepSeek-R1-0528, and gpt-oss-120b (high mode) — converge to essentially identical benchmark scores on AIME25 and AIME24 after approximately 600 gradient steps of stable RL training. All three models reach roughly 0.86–0.88 on the combined AIME25 + AIME24 benchmark, with initial differences in the first 100–200 steps narrowing to near-zero by step 600.

Response length dynamics. All three models show increasing average response lengths throughout training, rising from roughly 24,000–26,000 tokens at step 0 to 30,000–32,000 tokens by step 600 (Figure 5, right panel). The trend is smooth and monotonic, with no sharp changes that would indicate instability. Response lengths are broadly comparable across cold-start variants, with the gpt-oss-120b initialization producing slightly longer responses early in training but converging to similar lengths as the others by step 600.

Interpretation. The paper frames this as evidence that "once training is stabilized, models with different cold-start initializations consistently achieve comparable final performance" (Section 4.5). This finding is not presented as an ablation in the traditional sense (there is no "no cold-start" control — all models start from capable SFT checkpoints), but rather as a demonstration that the effects of initialization differences diminish under prolonged stable optimization.

---

Ablation Studies and Robustness Checks

Importance sampling correction for training–inference discrepancy (on-policy, Section 4.3, Figure 1): Removing the $\pi_{\theta_{\text{old}}} / \mu_{\theta_{\text{old}}}$ correction term while keeping all other MiniRL components causes rapid training collapse within 200–400 gradient steps, even though policy staleness is identically zero. This is the strongest evidence that IS correction is an inherent requirement of the first-order approximation, not a variance-reduction heuristic. The collapse manifests as a sharp entropy drop (from ~0.20 to below 0.05) and a spike in training–inference KL divergence. On the HMMT25 benchmark (Appendix B, Figure 6), the uncorrected variant achieves a peak score of roughly 0.52 before collapsing versus MiniRL's stable 0.60–0.65.

Length normalization (on-policy, Section 4.3, Figure 1): Adding $1/|y|$ length normalization to MiniRL produces stable but suboptimal training. The length-normalized variant reaches benchmark scores of roughly 0.68–0.70 versus MiniRL's 0.73–0.75, despite comparable training–inference KL divergence. Training reward is also lower (~0.78 vs. ~0.87). The finding is significant because length normalization is a common design choice in GRPO and CISPO — the paper's theory predicts (and the experiment confirms) that it introduces a persistent bias by breaking the equality $\nabla_\theta J^{\text{token}} \approx \nabla_\theta J^{\text{seq}}$.

Clipping in off-policy training (Section 4.4, Figures 2–4): Removing the clipping mask $M_t$ from MiniRL causes training collapse at all off-policy levels tested ($N = 2, 4$). The collapse is characterized by sharply decreasing entropy, rising training–inference KL divergence, and declining benchmark scores. Even when combined with Routing Replay (R2 or R3), the absence of clipping eventually leads to instability, though Routing Replay delays the collapse relative to no-clipping-no-RoutingReplay. At $N=2$ (Figure 2), MiniRL + R2 (no clipping) collapses after roughly 1,200 steps versus roughly 1,000 steps for MiniRL (no clipping); at $N=4$ (Figure 3), the collapse points are roughly 1,000–1,200 steps versus 500–750 steps. The paper concludes that "once off-policy updates are introduced, both Routing Replay and clipping become essential for stable training" (Section 4.4).

R2 vs. R3 across off-policy regimes (Section 4.4, Figures 2–4, Table 1): The choice of which Routing Replay variant to use interacts with the degree of off-policiness. At $N=2$ (Figure 2), R2 + clipping achieves benchmark scores approximately 3 percentage points higher than R3 + clipping (~0.75 vs. ~0.72). At $N=4$ (Figure 3), the ordering reverses: R3 + clipping reaches roughly 0.78–0.80 versus R2 + clipping's peak of roughly 0.76 before declining. At $N=8$ (Figure 4), R3 + clipping is the only configuration that remains stable through 5,000 steps, with R2 + clipping becoming unstable after roughly 2,000 steps. The paper attributes this crossover to the bias introduced by R3 in the first mini-batch (Table 1): when off-policiness is small, the first mini-batch constitutes a large fraction of the total data (50% at $N=2$), so R3's bias is costly; when off-policiness is large, the first mini-batch is proportionally smaller (12.5% at $N=8$), and R3's stronger stabilization dominates.

On-policy R3 (Section 4.3, Figure 1): Applying Rollout Routing Replay in the on-policy setting ($N=1$) does not improve and can degrade performance. MiniRL + R3 achieves benchmark scores comparable to or slightly below MiniRL without R3, despite reducing training–inference KL divergence. When R3 is combined with length normalization, benchmark scores degrade further. When R3 is combined with removal of IS correction, training still collapses. The paper interprets this as evidence that Routing Replay introduces a bias in the target policy that is not compensated by any staleness reduction when staleness is already zero. This is a negative result with practical implications: R3 should not be used in on-policy training, even though it reduces the training–inference discrepancy.

TIS truncation threshold (Section 4.2): The paper applies Truncated Importance Sampling with a threshold of 5 to the token-level IS weight but does not ablate this threshold. The choice of 5 is noted but not experimentally justified within this paper; it is attributed to Yao et al. (2025). This is a missing ablation — the sensitivity of results to the TIS threshold is unknown, and it is possible that the reported benefits of IS correction are partly due to the truncation interacting with the specific numerical properties of the FP8/BF16 precision mismatch.

Per-benchmark consistency (Appendix B, Figures 6–10): All ablation findings are reported individually for HMMT25, AIME25, and AIME24. The patterns — MiniRL outperforming length-normalized variants in on-policy training, R2 > R3 at $N=2$ and R3 > R2 at $N=4,8$, collapse dynamics for uncorrected and unclipped variants — are qualitatively consistent across all three benchmarks. The relative gaps between methods are largest on AIME24 (the easiest, where stable methods approach 0.90 and unstable methods collapse below 0.80) and smallest on HMMT25 (where all methods cluster more tightly around 0.55–0.65). This consistency strengthens the claim that the findings are not artifacts of a particular benchmark.

---

Critical Assessment

Claim 1: The token-level objective is a valid first-order approximation to the sequence-level objective, and its validity depends on minimizing the training–inference discrepancy and policy staleness.

What the experiments demonstrate. The on-policy results (Figure 1) provide strong evidence for one direction of this claim: when the approximation is invalidated — by removing IS correction or (to a lesser degree) adding length normalization — performance degrades or training collapses. The IS-correction removal is the cleanest test: with policy staleness held at zero, removing the training–inference discrepancy correction causes rapid collapse, confirming that this factor is indeed critical to the approximation's validity. The length-normalization result confirms that modifying the gradient to break the $\nabla_\theta J^{\text{token}} \approx \nabla_\theta J^{\text{seq}}$ relationship produces a biased objective, even when training remains superficially stable.

The off-policy results (Figures 2–4) show that clipping and Routing Replay — mechanisms the theory identifies as preserving the approximation by controlling policy staleness and expert routing shifts — are indeed essential for stable training when staleness is non-zero. The monotonic relationship between off-policiness and the required strength of stabilization (R2 suffices at $N=2$, R3 required at $N=8$) aligns with the theory's prediction.

What the experiments do NOT demonstrate. The paper does not provide direct evidence that $\nabla_\theta J^{\text{token}}$ is numerically close to $\nabla_\theta J^{\text{seq}}$ in any of the experimental configurations. The validity of the approximation is inferred from training stability — if training is stable and performance improves, the approximation is presumed to hold; if training collapses, it is presumed to have broken. This is a reasonable inference given the theory, but it is correlational rather than causal. A direct test would compare the sequence-level and token-level gradients on the same batch of data and measure their cosine similarity or relative error as a function of training–inference discrepancy and policy staleness. Such a measurement is computationally expensive (requiring computation of the intractable sequence-level gradient for comparison) but would definitively validate the approximation theory. Without it, alternative explanations for the observed stability patterns — e.g., that IS correction serves as a variance reduction technique whose benefits are unrelated to the Taylor approximation — cannot be ruled out.

Conditional nature. The claim about the approximation's validity conditions is supported specifically for the regime tested: 30B MoE model, FP8 inference, BF16 training, binary-reward mathematical reasoning, synchronous RL framework. Whether the same thresholds for "small $\delta_t$" apply to dense models, different precision configurations, or different reward structures (continuous rewards, multi-turn tasks) is untested.

Claim 2: Several widely-adopted stabilization techniques — importance sampling correction, clipping, and Routing Replay — are effective because they preserve the validity of the first-order approximation.

What the experiments demonstrate. The ablation structure is explicitly designed to test this claim: each technique is removed in isolation, and training stability is observed. On-policy: removing IS correction → collapse. Off-policy: removing clipping → collapse (even with Routing Replay). Off-policy with MoE: removing Routing Replay → collapse (at higher off-policiness). The experiments convincingly show that these techniques are necessary for stable training in their respective regimes.

What the experiments do NOT fully establish. The claim has a causal component — the techniques work because they preserve the first-order approximation — but the experiments only establish that the techniques are necessary and that removing them causes instability. The paper monitors training–inference KL divergence and entropy as proxies for the two error sources, and these metrics do behave as predicted (KL divergence spikes during collapses, entropy crashes), which supports the causal interpretation. However, the monitoring metrics are correlated with instability, not proven to mediate the relationship between the techniques and stability. It is possible, for example, that clipping prevents collapse through some mechanism unrelated to the Taylor approximation (e.g., by preventing gradient spikes that would occur even in a correctly-specified sequence-level objective), and the observed correlation with approximation validity is incidental. The paper's theoretical framework makes this alternative less plausible — it explains why clipping at the specific threshold values should matter — but does not experimentally rule it out.

Strength of the Routing Replay evidence. The R2 vs. R3 comparison is the strongest element of this claim, because it involves a specific, non-obvious prediction from the theory: R2 should dominate at low off-policiness and R3 at high off-policiness, due to the bias introduced in the first mini-batch (Table 1). The experiments confirm this crossover pattern cleanly across $N=2, 4, 8$. A purely empirical hypothesis — "Routing Replay helps" — would not predict the crossover; any Routing Replay variant would be expected to help equally at all off-policy levels. The fact that the crossover occurs as predicted provides stronger evidence that the theoretical mechanism (bias-vs-approximation-validity tradeoff) is genuinely operating.

Claim 3: On-policy training with basic policy gradient and IS correction achieves the highest training stability.

Supported with qualifications. The on-policy experiments (Figure 1) show MiniRL outperforming all ablated variants (length-normalized, without IS correction, with R3) in both stability and final performance. However, "highest training stability" is demonstrated only relative to the other on-policy variants tested. The paper does not compare on-policy MiniRL against off-policy MiniRL + R3 in terms of stability — the off-policy runs (Figures 2–4) with appropriate stabilization also train stably for thousands of steps. The claim appears to be that on-policy training is the most stable regime (which is theoretically expected, since policy staleness is zero), not that on-policy training achieves higher final performance than off-policy — indeed, comparing Figure 1 (on-policy, ~0.73–0.75) with Figure 3 (off-policy $N=4$ with R3, ~0.78–0.80), the off-policy run actually achieves higher final scores, likely because the larger global batch size provides more data per gradient step and enables faster convergence in wall-clock-equivalent compute.

Claim 4: Once training is stabilized, prolonged optimization consistently yields comparable final performance regardless of cold-start initialization.

Supported, with caveats about scope. The cold-start experiment (Figure 5) shows three different initializations converging to nearly identical AIME25 + AIME24 scores after ~600 gradient steps. This is compelling but limited in several ways: (1) the three cold-start models are all distilled from frontier models and fine-tuned on the same base architecture — they differ in their teacher but not in their fundamental capability range; (2) only two benchmarks (AIME25, AIME24) are reported; (3) training proceeds for only 600 gradient steps — it is possible that differences would re-emerge with longer training; (4) the experiment uses a different base model (Qwen3Next) than the main experiments (Qwen3-30B-A3B), and the training recipe (MiniRL + R2, $N=2$) is only one of several stable configurations identified. The finding is best interpreted as evidence that within a reasonable range of initialization quality, stability matters more than the specific choice of cold-start — but the boundaries of that "reasonable range" are not explored. A cold-start model with dramatically lower pass@1 (e.g., near-zero) would presumably not converge regardless of stability, since there would be no reward signal. The paper acknowledges this implicitly by starting all experiments from capable SFT checkpoints.

Genuine weaknesses in the experimental design

Single architecture, single task domain. All experiments use MoE models (Qwen3-30B-A3B or Qwen3Next) on mathematical reasoning. The paper's theoretical framework makes no claims specific to math — the first-order approximation should apply to any sequence-level reward setting — but the empirical validation is entirely within one domain. Mathematical reasoning has several properties that may affect generalizability: binary rewards with a well-defined ground truth, long response sequences (tens of thousands of tokens) that stress the product-of-ratios issue the approximation addresses, and a relatively clean signal where reward is a monotonic function of solution quality. Domains with noisy rewards (e.g., human preference judgments), multi-turn interactions, or tasks where reward depends on specific subsequences rather than the full response may present different stability challenges.

No confidence intervals or statistical testing. All results are reported as point estimates over 90 evaluation problems (30 per benchmark × 3 benchmarks). With 32 responses sampled per problem, the effective sample size is modest, and the difference between methods — particularly in the on-policy experiments where scores are tightly clustered (0.73 vs. 0.70 for MiniRL vs. MiniRL + length-norm) — may not be statistically significant. The paper does not report standard errors, bootstrap confidence intervals, or any formal test of whether observed differences exceed sampling variation. The qualitative patterns (collapse vs. no-collapse) are robust to this concern, but the precise quantitative rankings among stable methods should be interpreted cautiously.

Missing ablation: IS correction only on the training–inference component. The IS weight in MiniRL corrects for both training–inference discrepancy AND policy staleness ($\pi_\theta / \mu_{\theta_{\text{old}}} = (\pi_{\theta_{\text{old}}} / \mu_{\theta_{\text{old}}}) \times (\pi_\theta / \pi_{\theta_{\text{old}}})$). The on-policy experiments isolate the training–inference component (since $\pi_\theta / \pi_{\theta_{\text{old}}} = 1$), but there is no experiment that ablates only the staleness component while keeping the training–inference correction — for example, using $\pi_\theta / \pi_{\theta_{\text{old}}}$ as the IS weight while omitting $\pi_{\theta_{\text{old}}} / \mu_{\theta_{\text{old}}}$. Such an ablation would test whether the training–inference discrepancy is the primary source of instability (as the on-policy collapse suggests) or whether policy staleness alone (without training–inference discrepancy) can also cause collapse in an otherwise-corrected objective.

Missing ablation: the interaction between TIS truncation and stability. The TIS truncation threshold of 5 is not ablated. Given that the theory's central condition is that per-token ratios remain close to 1, the truncation threshold determines the maximum allowable deviation before a token's weight is capped. A threshold of 5 means $\delta_t$ can be as large as 4 ($1 + \delta_t = 5 \implies \delta_t = 4$), which is arguably too large for the first-order approximation ($\delta_i \delta_j$ terms with $\delta = 4$ would be 16, not negligible). A lower truncation threshold might improve stability further by keeping more tokens in the small-$\delta_t$ regime; a higher threshold might be necessary if aggressive truncation introduces bias. The paper's stability is partly attributable to TIS preventing extreme weights, but the sensitivity to the specific threshold is unknown.

Missing comparison: MiniRL with GRPO and CISPO at scale. Appendix A provides an analytical comparison of MiniRL against GRPO and CISPO, identifying three differences (no IS correction, length normalization, clipping strategy). However, the paper does not experimentally compare MiniRL against GRPO or CISPO at the scale of the main experiments. The ablations (removing IS correction, adding length normalization) partially replicate GRPO/CISPO-like configurations, and those configurations perform worse — but a direct head-to-head comparison with hyperparameter tuning for the baseline algorithms would strengthen the claim that MiniRL's theoretically-justified design choices translate to practical improvements over widely-used alternatives.

Stability definition is qualitative. The paper defines stable training as a process where "model performance steadily improves... and the model's internal state evolves smoothly and without abrupt shifts" (Section 1, footnote). This is a reasonable operational definition, but the paper does not provide quantitative criteria for what constitutes a "collapse" versus an acceptable fluctuation. In the off-policy experiments, MiniRL + R2 at $N=4$ (Figure 3) shows a gradual decline after ~2,000 steps — is this a "collapse" or a mild degradation? The entropy decreases from 0.65 to 0.35 (a drop but not a crash), and training–inference KL rises but does not spike dramatically. The paper's interpretation of this as "failing to sustain stable training" is reasonable but subjective. Quantitative thresholds for entropy decline rate or KL divergence increase that predict imminent collapse would make the diagnostic framework more actionable.

6. Limitations and Trade-offs

The Single-Domain, Single-Architecture Validation

The constraint. All experiments in this paper use one model family (Qwen3-30B-A3B-Base and Qwen3Next, both MoE architectures), one task domain (competition-level mathematical reasoning), and one reward structure (binary correctness based on exact answer matching). The authors do not claim generality beyond this scope — they describe the 30B MoE model as "representative of the capabilities of many contemporary LLMs" (Section 4.2) but do not test this representativeness claim. The theoretical framework (the first-order approximation in Section 2.3) is formulated in domain-agnostic terms — the Taylor expansion argument does not reference math, binary rewards, or MoE architectures — but the empirical validation is entirely within one setting.

The consequence. A practitioner cannot determine from this paper alone whether the MiniRL recipe and the R2/R3 selection guidelines transfer to other important deployment scenarios. Several properties of the experimental setup may be load-bearing in ways that are not tested:

• Binary rewards with exact answer matching provide a clean, noise-free signal. In domains where rewards are continuous (e.g., human preference scores), learned (e.g., a reward model), or stochastic, the variance properties of the token-level gradient may differ, and the clipping thresholds ($\varepsilon_{\text{high}} = 0.27$, $\varepsilon_{\text{low}} = 0.2$) calibrated for binary rewards may be inappropriate. A noisy reward would inject additional variance into the advantage estimate $\hat{A}(x,y)$, potentially requiring different stabilization mechanisms.

• Mathematical reasoning produces very long responses (the paper reports response lengths of 24,000–32,000 tokens; Figures 5 and 10). The first-order approximation's validity depends on products of many small $\delta_t$ being negligible — this is the setting where the product-of-ratios problem is most severe, and the token-level surrogate provides the largest variance reduction. For short-response tasks (e.g., multiple-choice QA with responses of 50–200 tokens), the sequence-level importance weight is numerically less problematic, and the benefit of the token-level approximation may be smaller or the $4\times$ importance of IS correction may be reduced.

• MoE architectures are the explicit focus of the Routing Replay analysis, and the paper's strongest findings (the R2 vs. R3 crossover, the necessity of Routing Replay under off-policy training) are architecture-specific. A practitioner training a dense model would learn from this paper that IS correction and clipping are essential, but cannot determine whether Routing Replay (which has no analogue for dense models) is being replaced by some other necessary technique or whether dense models are simply easier to stabilize.

What evidence exists in the paper. The paper provides no cross-domain or cross-architecture experiments. The cold-start experiment (Section 4.5) tests a different MoE model (Qwen3Next) but still within mathematical reasoning. The FP8/BF16 precision mismatch is a deliberate stress test for training–inference discrepancy, which increases confidence that the IS correction finding generalizes to other precision configurations — but the stress test is about magnitude of discrepancy, not type of task or structure of reward.

Mitigation status. The paper partially acknowledges this limitation implicitly by scoping all claims to the experimental setting — it does not assert that MiniRL is universally optimal. However, it does not explicitly flag the single-domain limitation as a threat to generalization, and the theoretical sections (Sections 2 and 3) are presented in a task-agnostic manner that could be read as claiming universality. No future work is suggested on cross-domain validation.

---

The Cost of Difficulty Estimation and Monitoring Is Not Accounted For

The assumption. The paper's diagnostic framework for detecting and preventing training instability relies on continuous monitoring of two metrics: (1) the KL divergence between inference and training engine rollout policies, $D_{\text{KL}}(\mu_{\theta_{\text{old}}} \| \pi_{\theta_{\text{old}}})$, and (2) the token-level entropy of the target policy, $H[\pi_\theta]$. Computing the KL divergence requires running the same input through the training engine (which is not normally done during RL — the training engine computes gradients, not probabilities for rollout-generated tokens) to obtain $\pi_{\theta_{\text{old}}}(y_t|x, y_{<t})$ for tokens that were generated under $\mu_{\theta_{\text{old}}}$. This is an additional forward pass on every response, or at least on a representative subset, which the paper does not factor into its compute accounting. The paper reports training–inference KL divergence in all figures but does not discuss the cost of computing it.

The consequence. A practitioner attempting to replicate the paper's stability monitoring would face a non-trivial computational overhead. Each gradient step processes 1,024 responses. Computing the training–inference KL for all these responses requires re-running the training engine forward pass on each response's tokens to obtain $\pi_{\theta_{\text{old}}}$ probabilities, then comparing them against the stored $\mu_{\theta_{\text{old}}}$ probabilities from the inference engine. For responses of 24,000–32,000 tokens (typical in the paper's experiments), this is a substantial cost — the forward pass through the training engine on 1,024 long responses could rival the cost of the gradient computation itself. Without this monitoring, however, practitioners lose the early-warning signal that the paper's framework identifies as critical for detecting imminent collapse — the sharp increase in KL divergence that precedes entropy crashes in Figures 1–4.

More fundamentally, the paper's central methodological prescription — "monitor whether the first-order approximation is holding, and intervene when it breaks" — assumes that the monitoring is affordable. If computing the KL divergence doubles the per-step cost, the net efficiency of the approach degrades, and the headline stability gains must be weighed against the monitoring overhead.

What evidence exists in the paper. The paper does not measure or report the cost of computing training–inference KL divergence. The 5–6 GPU hours per gradient step estimate (Section 4.2) appears to exclude monitoring costs — it describes the cost of "each gradient step" (the forward and backward pass for parameter updates on the mini-batch), not the additional forward passes for diagnostic metrics. The entropy computation is likely cheaper (it uses the training-engine probabilities directly, which are computed during the gradient step anyway), but the KL divergence requires redundant computation.

Mitigation status. The paper does not acknowledge this as a limitation. The monitoring metrics are presented as informative diagnostics without discussion of their computational cost. A partial mitigation — subsampling (computing KL on a small fraction of tokens rather than all) — would reduce the cost but is not proposed or evaluated. The paper does not discuss whether the diagnostic value of monitoring justifies its overhead.

---

No Demonstration That the Theoretical Framework Improves Over Strong Tuned Baselines

The constraint. The paper's experimental program is structured as an internal ablation study: MiniRL is the baseline, and variants are constructed by adding or removing components (length normalization, IS correction, clipping, Routing Replay) to test the theory's predictions. This design cleanly isolates the effect of each component relative to the minimalist baseline, which is appropriate for validating the theoretical framework. However, the paper does not compare MiniRL — even in its best configuration — against independently tuned implementations of widely-used algorithms like GRPO (Shao et al., 2024) or PPO (Schulman et al., 2017) at comparable scale. GRPO and CISPO are analyzed analytically in Appendix A, and the ablations that remove IS correction or add length normalization partially replicate GRPO-like configurations (which perform worse than MiniRL), but these are degraded versions of MiniRL, not independently optimized implementations of the competing algorithms.

The consequence. The paper establishes that MiniRL's components are well-motivated by the theory and that removing them degrades performance. But it does not establish that MiniRL outperforms what a practitioner would actually use if they were not following this paper's framework. A well-tuned GRPO implementation might use different clipping thresholds, different advantage normalization, different learning rate schedules, or different batch-size ratios — any of which could close or reverse the performance gap observed against MiniRL's length-normalized ablated variant. The paper's claim is that its theoretical framework explains why certain design choices matter, not that MiniRL is a superior algorithm to all alternatives. However, the empirical results are presented in a way that could be read as establishing MiniRL's superiority (e.g., "MiniRL... achieves the best performance and training stability" in Section 4.3), which conflates "best among our tested variants" with "best among possible algorithms."

This is a meaningful distinction because GRPO, in particular, has been used successfully at scale (DeepSeekMath, DeepSeek-R1) to train models that achieve strong reasoning performance. If GRPO with appropriate hyperparameter tuning achieves comparable or better stability and final performance than MiniRL, the paper's practical contribution shifts from "use MiniRL" to "understand why your chosen algorithm works through the lens of the first-order approximation." The latter is intellectually valuable but has different implications for practitioners choosing an RL recipe.

What evidence exists in the paper. Appendix A provides an analytical comparison identifying three differences between MiniRL and GRPO/CISPO: (1) no training–inference IS correction in GRPO/CISPO, (2) length normalization in GRPO/CISPO, (3) different clipping strategies. The on-policy experiments (Figure 1) show that adding length normalization and removing IS correction from MiniRL — which moves it toward a GRPO-like configuration — degrades performance. But this is not a head-to-head comparison with GRPO, which also differs in its advantage normalization (z-score vs. group-mean), its min-clipping vs. stop-gradient masking, and potentially other implementation details. The ablation tells us that these specific differences matter, but not whether GRPO's full configuration compensates for them through its other design choices.

Mitigation status. The paper is transparent about what it is comparing — the ablated variants are explicitly described as modifications of MiniRL, not as implementations of GRPO or PPO. The claim in Appendix A is that the three identified differences are the "key differences" between MiniRL and the competing algorithms, which is an analytical claim rather than an empirical one. The paper does not claim to have benchmarked against tuned GRPO, but it also does not flag the absence of such a comparison as a limitation. A practitioner evaluating whether to adopt MiniRL over their existing GRPO pipeline would need this comparison and does not get it from this paper.

---

The Training–Inference IS Correction Depends on Access to Inference-Engine Probabilities

The assumption. MiniRL's IS correction requires computing the per-token probability ratio $\pi_\theta(y_t|x, y_{<t}) / \mu_{\theta_{\text{old}}}(y_t|x, y_{<t})$ during gradient computation. The numerator $\pi_\theta(y_t|x, y_{<t})$ is available from the training engine's forward pass. The denominator $\mu_{\theta_{\text{old}}}(y_t|x, y_{<t})$ is the probability the inference engine assigned to token $y_t$ when it generated the response during rollout — this must be recorded at generation time and stored alongside the response tokens. Storing per-token log-probabilities for responses of 24,000–32,000 tokens across global batch sizes of up to 8,192 responses represents a substantial memory and I/O burden: 8,192 responses × 30,000 tokens × 4 bytes (float32) ≈ 1 GB of additional data to transfer from inference to training infrastructure per global step.

Moreover, this requirement couples the inference and training stacks in a way that complicates deployment. The inference engine (SGLang, vLLM) must be configured to output per-token log-probabilities, which may not be the default behavior and can impose throughput penalties. Some inference engines discard or approximate log-probabilities for efficiency when they are not explicitly requested. The paper's experimental infrastructure (not described in detail) presumably supports this, but a practitioner integrating MiniRL into a production RL pipeline may face non-trivial engineering challenges.

The consequence. The paper's strongest finding — that IS correction is necessary for stable training, not optional — implies that any practitioner deploying RL for LLMs must solve the engineering problem of extracting and storing per-token inference-engine probabilities. This is not a theoretical limitation of the framework, but a practical barrier to adoption. If the inference engine cannot efficiently provide log-probabilities (e.g., because batch-invariant kernels are disabled for throughput, which the paper itself notes causes nondeterminism that makes the recorded probabilities inconsistent with the actual sampling distribution), the IS correction becomes noisy or biased, potentially undermining the stability it is supposed to provide.

What evidence exists in the paper. The paper does not discuss the engineering requirements or overhead of storing per-token probabilities. The synchronous RL framework is described at a high level (Section 4.2) — "we first sample a batch of B prompts and sample G responses for each prompt using the rollout policy in the inference engine. Then, we split the responses into N mini-batches and apply N gradient updates in the training engine" — without detailing how token-level probabilities are communicated between the two engines. The FP8/BF16 precision configuration is described as a "stress test" (Section 4.2), which implicitly acknowledges that the inference engine produces lower-precision probabilities, but the paper does not discuss whether FP8 log-probabilities are sufficiently accurate for the IS correction to be reliable — this is the central premise of the IS correction that the paper does not interrogate.

Mitigation status. The paper does not acknowledge this as a limitation. It treats the availability of $\mu_{\theta_{\text{old}}}$ probabilities as a given. In practice, RL frameworks that do not record per-token inference probabilities (which includes many production systems optimized for throughput) cannot implement MiniRL as described without infrastructure changes. The Truncated Importance Sampling (TIS) with threshold 5 partially mitigates the impact of noisy IS weights (since extreme values are capped), but TIS does not address the fundamental requirement of having $\mu_{\theta_{\text{old}}}$ available in the first place.

---

The Cold-Start Finding Has Narrow Scope and Limited Generality

The constraint and what the paper establishes. Section 4.5 shows that three cold-start models — all distilled from frontier LLMs (Qwen3-Max-Thinking-Preview, DeepSeek-R1-0528, gpt-oss-120b high mode) and fine-tuned onto the same base architecture (Qwen3Next MoE) — converge to comparable performance after ~600 gradient steps of stable RL training on mathematical reasoning benchmarks (AIME25, AIME24). The paper interprets this as evidence that "once training is stabilized, models with different cold-start initializations consistently achieve comparable final performance" (Section 4.5) and suggests that this "motivates future work to focus more on RL itself rather than overly on the specifics of cold-start initialization."

The consequence — what the finding does NOT establish. The three cold-start models differ only in their distillation teacher, not in their fundamental capability range. All three are fine-tuned from frontier reasoning models that already achieve strong math performance. The experiment does not test initialization diversity along several dimensions that matter to practitioners:

• Capability gap: What if one cold-start model achieves pass@1 of 30% and another achieves 5% on the training prompts? The paper's theory would predict that the 5% model may struggle because there are too few correct samples to provide a useful advantage signal ($\hat{A}(x,y)$ would be zero or negative for nearly all responses). The experiment does not explore this threshold — all three models presumably start from comparable pass@1 rates (the paper does not report them).

• Model scale: The cold-start experiment uses a single model size. It is unknown whether convergence to comparable performance would hold for smaller models (which may benefit more from better initialization because their capacity ceiling is lower) or larger models (which may be more robust to initialization because they have more capacity to overcome early mistakes).

• Training duration: The experiment runs for 600 gradient steps. The paper's main experiments run for 1,200–5,000 steps. The claim that "differences arising from the latter [cold-start specifics] are expected to vanish given prolonged RL training" (Section 1) is extrapolated from 600 steps of convergence. Whether the three cold-start models would remain indistinguishable after 2,000 or 5,000 steps — or whether they would diverge again — is not tested. The response-length trajectories in Figure 5 (right panel) show differences narrowing but not fully closing, suggesting some residual initialization effects.

• Benchmark coverage: Only AIME25 and AIME24 (60 problems total) are reported. HMMT25, which is part of the main evaluation suite, is omitted from the cold-start experiment without explanation.

• Domain: Mathematical reasoning only. In domains where the reward signal is weaker or where different cold-start strategies produce qualitatively different reasoning styles (e.g., creative writing, dialogue), initialization differences may persist under RL in ways that the paper's framework does not predict.

These limitations matter because the paper's recommendation — focus on RL stability rather than cold-start data — could lead practitioners to underinvest in cold-start quality, only to discover that their specific initialization regime (e.g., a much weaker base model, a different task, a different training duration) does not exhibit the same convergence property.

What evidence exists in the paper. The cold-start experiment is a single figure (Figure 5) with two panels (benchmark score and response length). The paper does not report training reward, entropy, or training–inference KL for these runs, making it impossible to verify that all three converged stably as opposed to coincidentally reaching similar scores through different dynamic paths. The absence of HMMT25 from this experiment is unexplained. The paper does not discuss the scope of the convergence claim or its boundary conditions.

Mitigation status. The paper presents the finding as an encouraging empirical observation rather than a theorem. The claim is hedged: "This motivates future work to focus more on RL itself rather than overly on the specifics of cold-start initialization, as differences arising from the latter are expected to vanish given prolonged RL training" (Section 4.5, emphasis on "motivates" and "expected"). The hedging is appropriate but does not substitute for testing the boundaries of the claim. The paper does not suggest specific follow-up experiments to characterize when initialization differences do versus do not vanish.

---

The First-Order Approximation Framework Is Validated Indirectly Through Training Stability, Not Through Direct Gradient Comparison

The constraint. The paper's central theoretical claim is that the token-level gradient $\nabla_\theta J^{\text{token}}$ approximates the sequence-level gradient $\nabla_\theta J^{\text{seq}}$ when the training–inference discrepancy and policy staleness are small (Section 2.3). The entire experimental program tests this claim indirectly: configurations predicted by the theory to preserve the approximation (IS correction, clipping, Routing Replay) produce stable training, while configurations predicted to break it produce training collapse. Stability is treated as evidence that the approximation holds; collapse is treated as evidence that it has broken.

The consequence. The paper provides no direct measurement of approximation quality. Neither the cosine similarity between $\nabla_\theta J^{\text{token}}$ and $\nabla_\theta J^{\text{seq}}$, nor the relative error in their magnitudes, nor the variance ratio between the sequence-level and token-level gradient estimators, is computed or reported for any experimental configuration. This leaves open alternative explanations for the observed stability patterns:

• IS correction as variance reduction (not approximation validity): The IS weight $\pi_{\theta_{\text{old}}} / \mu_{\theta_{\text{old}}}$ could be stabilizing because it reduces the variance of the gradient estimator — tokens where the inference engine assigned very low probability to the actually-generated token get downweighted, preventing noisy updates — rather than because it makes the token-level gradient approximate the sequence-level gradient. The Taylor expansion argument explains why the IS weight has this form, but the experimental evidence (training collapses without IS correction) is equally consistent with a pure variance-reduction story, since removing variance reduction from a high-variance estimator can cause optimization to diverge even if the estimator is unbiased.

• Clipping as gradient-norm control: Clipping prevents aggressive updates by zeroing out gradients for tokens with large probability ratios. This could stabilize training by keeping the effective learning rate bounded, independent of any first-order approximation considerations. The entropy crash and KL spike observed during collapses could be symptoms of uncontrolled gradient norms rather than consequences of the approximation breaking — a large gradient step could simultaneously drive entropy down, increase the training–inference discrepancy (if the policy shift causes routing changes in MoE), and degrade performance, without the causal mechanism being approximation invalidity.

These alternative explanations are not mutually exclusive with the paper's framework — the first-order approximation may be the reason why IS correction and clipping are the right forms of variance reduction and gradient-norm control. But without direct gradient measurements, the paper cannot rule out that simpler explanations account for the observed phenomena, which would weaken the claim that the approximation framework is necessary for understanding stability (as opposed to being a useful but non-exclusive interpretive lens).

What evidence exists in the paper. The strongest evidence for the approximation-theoretic interpretation — as opposed to a generic variance-reduction interpretation — is the length-normalization result. Length normalization produces stable training but lower final performance than MiniRL, even though it likely does not increase gradient variance (dividing by $|y|$ reduces variance). The theory explains this as a bias introduced by breaking $\nabla_\theta J^{\text{token}} \approx \nabla_\theta J^{\text{seq}}$ — the length-normalized gradient points in a different direction, optimizing the wrong objective. This pattern is harder to explain through pure variance reduction, since a variance-reduced but biased estimator could produce exactly this outcome (stable convergence to a suboptimal point). The R2 vs. R3 crossover (Section 4.4) also provides theory-specific evidence: the bias-vs-validity tradeoff in Table 1 makes a directional prediction that is confirmed, and a generic "Routing Replay helps" hypothesis would not predict the crossover. These results strengthen the case for the approximation framework, but they still do not directly measure approximation quality.

Mitigation status. The paper does not acknowledge this as a limitation, and does not discuss the computational or methodological challenges that would be involved in directly measuring $\nabla_\theta J^{\text{seq}}$ for comparison. Computing the sequence-level gradient on even a single response of 30,000 tokens faces the numerical-range problem the paper itself identifies as making $J^{\text{seq}}$ "intractable" (Section 2.2) — the sequence-level importance weight is the product of 30,000 per-token ratios, which may overflow or underflow floating-point representations. This makes direct gradient comparison genuinely difficult, and the paper's indirect validation strategy is a reasonable response to that difficulty. However, flagging this as an open validation gap — "we have not directly measured approximation error because doing so is numerically challenging, and our claims rest on the consistent alignment between theoretical predictions and observed stability patterns" — would strengthen the paper's transparency without weakening its contributions.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper provides the first principled theoretical criterion for evaluating whether a token-level RL objective for LLMs is sound, and in doing so transforms RL-for-LLMs from an empirical trial-and-error discipline into one guided by approximation theory. The magnitude of this shift is not an incremental refinement of existing algorithms — it is a diagnostic reframing that changes how we think about the problem, even though the algorithmic components (IS correction, clipping, Routing Replay) were known before. Prior to this work, the community understood that certain techniques "helped" — IS correction reduced variance, clipping prevented overly large updates, Routing Replay fixed experts during MoE optimization — but each was justified by its own local rationale, and there was no unified account of why these disparate mechanisms all contributed to stability. This paper provides that unified account: all effective stabilization techniques work by preserving the validity of the first-order Taylor approximation that connects the token-level surrogate objective to the true sequence-level reward. Whether a given design choice helps or hurts can now be evaluated by asking a single question: does it keep the per-token ratios between the target policy and the rollout policy close to 1, thereby keeping the dropped second-order terms in the expansion negligible?

This reframing resolves a cluster of prior contradictions and ambiguities. Before this paper, it was unclear why GRPO and PPO variants — which dominate production RL training for LLMs — sometimes worked brilliantly and sometimes collapsed catastrophically. The paper's decomposition of approximation error into the multiplicative product of training–inference discrepancy and policy staleness (Equation 5) explains this: a configuration that is stable under BF16 inference (small discrepancy) may collapse under FP8 inference (large discrepancy), not because the algorithm is wrong, but because the approximation it implicitly relies on has been invalidated by the precision change. Similarly, a configuration that is stable with on-policy updates may collapse when off-policy mini-batches are introduced, because policy staleness accumulates to the point where the $\delta_t$ terms in the Taylor expansion are no longer small. The framework also explains why length normalization degrades performance without causing collapse (it biases the objective without destroying the gradient's coherence), and why naive removal of IS correction causes immediate failure even on-policy (the approximation breaks at the first gradient step). These are not separate mysteries requiring separate explanations — they are all manifestations of a single underlying condition being violated.

The paper also reconciles the tension between two camps in the prior literature: those arguing for sequence-level optimization objectives (Zheng et al., 2025; Liu et al., 2025a) because token-level objectives are theoretically unsound, and those using token-level objectives (GRPO, PPO) because they are computationally tractable. The paper's framework shows that both camps are partially right: sequence-level objectives are indeed the "correct" thing to optimize, and token-level objectives introduce an approximation. But the approximation is not inherently broken — it is valid under specific, monitorable conditions. The practical implication is that the field can keep using tractable token-level objectives while having a principled understanding of when and why they work, rather than feeling compelled to switch to numerically challenging sequence-level methods.

Several research directions become newly attractive or newly tractable as a result:

• Diagnostic-driven RL training becomes possible: rather than running RL and hoping stability emerges, practitioners can now monitor training–inference KL divergence and policy entropy as direct indicators of approximation validity, and intervene (adjust clipping thresholds, reduce off-policiness, switch Routing Replay variants) before a collapse occurs. The paper's training curves (Figures 1–4) show that KL spikes and entropy crashes are leading indicators — they precede performance degradation, not merely accompany it — meaning the diagnostics are genuinely predictive.

• Algorithm design by approximation preservation replaces algorithm design by trial-and-error: any proposed modification to an RL objective (new normalization schemes, alternative advantage estimators, adaptive clipping) can be evaluated by asking whether it maintains $\nabla_\theta J^{\text{token}} \approx \nabla_\theta J^{\text{seq}}$. The paper's length-normalization analysis is a template for this kind of evaluation.

• The training–inference discrepancy is elevated to a first-class research problem, not an implementation nuisance. The paper shows that this discrepancy is structurally equivalent to policy staleness in determining whether RL optimization is coherent. This motivates investment in inference-engine determinism, precision-matched training–inference stacks, and IS correction schemes that do not require storing per-token inference probabilities. The finding that the discrepancy alone (without any staleness) is sufficient to cause collapse under FP8 (Figure 1, MiniRL − train-infer-IS) means that precision mismatch is not just an efficiency concern — it is a correctness concern for RL.

Conversely, some directions become less attractive:

• Obsessing over cold-start data quality beyond a reasonable baseline is shown to have diminishing returns when RL training is stable (Section 4.5, Figure 5). Three cold-start models distilled from different frontier teachers converge to nearly identical benchmark scores after ~600 gradient steps. This does not mean cold-start is irrelevant — a model that never produces correct answers will have no RL signal to amplify — but it means that, above a capability threshold, investment in RL stability has higher marginal returns than further investment in initialization quality.

• Pursuing ever-more-complex search or exploration mechanisms within RL may be premature if the underlying optimization is unstable. The paper shows that a minimalist algorithm (MiniRL — essentially REINFORCE with IS correction and PPO clipping) achieves stable, high-performing training when the approximation conditions are maintained. Adding complexity on top of an unstable foundation risks introducing new failure modes without addressing the root cause.

Follow-Up Research This Work Enables

Direct measurement of the first-order approximation error across training regimes. The paper validates its theory indirectly — stable training is taken as evidence that the approximation holds, collapse as evidence that it has broken. A direct measurement would compare $\nabla_\theta J^{\text{token}}$ and $\nabla_\theta J^{\text{seq}}$ on the same batch of data, computing their cosine similarity and relative magnitude error as a function of the training–inference discrepancy (varied by precision: FP8, BF16, FP32) and policy staleness (varied by mini-batch index within a global batch). The challenge is that $\nabla_\theta J^{\text{seq}}$ involves product-of-ratios that can overflow — but for moderate sequence lengths (e.g., 500–2000 tokens) and small per-token deviations, the product may remain computationally tractable. A follow-up could run a smaller-scale experiment (e.g., 7B dense model, 1000-token responses) where both gradients can be computed stably, and produce a plot of approximation error versus the two decomposed factors (training–inference KL and policy staleness ratio). This would convert the paper's correlational evidence into a direct causal validation, and would identify the specific numerical thresholds at which the approximation "breaks" — enabling practitioners to set monitoring alerts at those thresholds.

Cross-domain and cross-architecture stress-testing of MiniRL and the approximation framework. The paper's empirical validation is confined to mathematical reasoning with binary rewards on MoE architectures. Critical stress tests would include: (1) Dense models (e.g., Qwen3-30B dense, or LLaMA-family models), to determine whether Routing Replay's benefits are MoE-specific or whether dense models face analogous instability from other sources (e.g., activation outliers) that require different mitigations. If dense models achieve stable training with MiniRL alone (no Routing Replay) across all off-policy regimes, that would confirm Routing Replay is a targeted fix for expert routing, not a general-purpose stabilizer. (2) Continuous or learned rewards (e.g., reward model scores, human preference labels), where the advantage estimate has additional variance and may be miscalibrated relative to the binary-reward setting. The paper's clipping thresholds ($\varepsilon_{\text{high}} = 0.27$, $\varepsilon_{\text{low}} = 0.2$) were likely tuned for binary rewards; continuous rewards with larger variance may require different thresholds or adaptive clipping. (3) Short-response tasks (e.g., multiple-choice QA, classification) where sequences are 50–200 tokens rather than 24,000–32,000. The first-order approximation's variance reduction benefit is largest for long sequences (where the product-of-ratios problem is most severe); for short sequences, sequence-level optimization may be tractable, and the advantage of the token-level surrogate may be smaller. A systematic comparison of MiniRL against direct sequence-level optimization across response-length regimes would map out the Pareto frontier of when the approximation is worth incurring. (4) Multi-turn or interactive tasks (e.g., tool use, dialogue), where the reward may depend on intermediate steps rather than only the final response. The paper's framework assumes a single sequence-level reward; multi-turn settings introduce partial observability and per-turn reward signals that may require extending the approximation to handle reward decomposition.

Adaptive off-policiness and dynamic stabilization. The paper treats the global-batch-to-mini-batch ratio $N$ as a fixed hyperparameter and selects R2 versus R3 based on it. A more sophisticated approach would dynamically adjust the number of mini-batches and the choice of Routing Replay variant based on real-time monitoring of approximation validity. For example: start each global step with $N=1$ (on-policy), compute the training–inference KL after the first mini-batch, and if the KL is below a threshold (indicating the approximation is healthy), increase $N$ for the next global step to accelerate convergence; if the KL spikes above a warning threshold, reduce $N$ or switch from R2 to R3 mid-training. This would be an exploration–exploitation tradeoff in the space of training configurations rather than model outputs, and would directly operationalize the paper's diagnostic framework as a control signal. A strong follow-up would implement this adaptive controller on the same 30B MoE setup and compare total compute to reach a target benchmark score against the fixed-$N$ baselines from the paper (Figures 2–4). The hypothesis is that adaptive off-policiness could achieve the faster convergence of high-$N$ training while avoiding the collapse risk, yielding better compute efficiency than any fixed configuration.

IS correction without storing per-token inference-engine probabilities. The paper identifies IS correction ($\pi_{\theta_{\text{old}}} / \mu_{\theta_{\text{old}}}$) as essential for stability, but the denominator requires storing per-token log-probabilities from the inference engine — an engineering burden the paper does not address. A practical follow-up would investigate whether the IS weight can be approximated from training-engine quantities alone. For instance: train a lightweight correction model that predicts $\mu_{\theta_{\text{old}}}(y_t|x, y_{<t}) / \pi_{\theta_{\text{old}}}(y_t|x, y_{<t})$ from features available during training (the token embedding, the training-engine logit, the layer index), without requiring a full inference-engine forward pass. Or: use a fixed calibration dataset to measure the average per-token discrepancy between engines as a function of token position, response length, and model layer, then apply this average correction as a static IS weight rather than a per-token dynamic one. The paper's finding that the training–inference KL remains relatively stable during healthy training (Figures 1–4, KL curves in the $10^{-2}$ range) suggests that a static correction might be sufficient — the discrepancy is systematic rather than wildly fluctuating. A negative result (showing that per-token dynamic IS weights are indeed necessary) would also be valuable, as it would establish the minimum engineering requirements for stable RL and motivate investment in inference-engine infrastructure.

Scaling laws for RL training stability: precision, model size, and batch composition. The paper uses FP8/BF16 as a deliberate stress test and shows that this precision gap is large enough to cause collapse without IS correction. This opens a broader question: how does the required strength of stabilization scale with model size, precision gap, and training duration? A scaling study could sweep model sizes (1B, 7B, 30B, 70B), precision configurations (FP32/FP32, BF16/BF16, FP8/BF16, FP8/FP8), and off-policy ratios ($N = 1, 2, 4, 8, 16$), measuring for each combination: (1) the minimum IS correction strength (TIS threshold) needed to prevent collapse, (2) the minimum clipping tightness ($\varepsilon_{\text{high}}, \varepsilon_{\text{low}}$) needed for stability, and (3) whether Routing Replay becomes necessary at a predictable model size or off-policy threshold. This would produce RL stability scaling laws analogous to pretraining scaling laws (Hoffmann et al., 2022) — equations that tell a practitioner, given their model size, precision budget, and desired convergence speed, exactly which stabilization mechanisms are necessary and how to configure them. The paper's existing data points (30B MoE, FP8/BF16, $N = 1,2,4,8$) provide the anchor for such a study, and the theory provides the functional forms to fit (stability threshold as a function of the product of discrepancy and staleness).

Practical Applications and Downstream Use Cases

Production RL training pipelines for MoE-based reasoning models. The most direct application is for teams training reasoning models with RL at scale — the scenario the paper itself exemplifies. The decision procedure derived from the paper's experiments is concrete: (1) always include IS correction with the inference-engine probabilities in the denominator — this is non-optional and the paper shows collapse within 200–400 steps without it (Figure 1); (2) for on-policy training ($N=1$), MiniRL without Routing Replay is sufficient and optimal, and length normalization should be avoided; (3) for off-policy training to accelerate convergence, use clipping ($\varepsilon_{\text{high}}=0.27$, $\varepsilon_{\text{low}}=0.2$) and Routing Replay, with the variant chosen by off-policiness: R2 for $N \leq 2$, R3 for $N \geq 4$. The paper's experiments with $N=2$ and R2 reach benchmark scores of ~0.75 with stable training through 2,500 gradient steps (Figure 2); scaling to $N=4$ with R3 reaches ~0.78–0.80 through 4,000 steps (Figure 3). For a team with a fixed compute budget, the choice between these configurations translates to different convergence speed vs. stability tradeoffs, and the paper provides the data to make that choice quantitatively.

Diagnostic monitoring as an early-warning system for training collapse. The paper's training dynamics figures show that $D_{\text{KL}}(\mu_{\theta_{\text{old}}} \| \pi_{\theta_{\text{old}}})$ spikes and entropy crashes precede benchmark score degradation — not merely accompany it. For example, in Figure 2 (MiniRL with no clipping), the KL divergence begins rising after ~500 gradient steps, entropy begins dropping around ~750 steps, and benchmark scores peak around ~1,000 steps before collapsing. This means a practitioner monitoring these metrics can detect an impending collapse hundreds of gradient steps before it affects final performance — at the paper's rate of 5–6 GPU hours per step, this represents roughly 1,200–1,800 GPU hours of advance warning. The practical workflow: instrument the training loop to compute training–inference KL on a subset of tokens (e.g., 10% to reduce overhead), set an alert threshold (e.g., KL > 5× the running average), and when the alert fires, either reduce the off-policy ratio $N$, tighten clipping thresholds, switch from R2 to R3, or checkpoint and roll back to a pre-spike model state. The paper does not provide the exact trigger thresholds — they would need to be calibrated per-model — but it establishes that the metrics are predictive and that the interventions (clipping, Routing Replay) are effective.

Cost-efficient RL for smaller organizations with limited cold-start budgets. The cold-start convergence finding (Section 4.5, Figure 5) has direct implications for resource allocation. Three cold-start models — distilled from different frontier teachers (Qwen3-Max-Thinking-Preview, DeepSeek-R1-0528, gpt-oss-120b) — converge to within ~2 percentage points of each other on AIME25 + AIME24 after 600 gradient steps of stable RL. The practical implication: an organization with a limited budget for cold-start data curation need not invest in distilling from the single best frontier model or in exhaustive SFT data filtering to achieve competitive final performance. A reasonable cold-start (the paper's models are all distilled from frontier models, but the fact that three different frontier models produce interchangeable results implies the specific teacher is not critical) plus stable RL training will reach comparable performance to a more expensive cold-start plus the same RL. The savings can be redirected toward the RL infrastructure itself — ensuring IS correction is implemented, clipping is configured, and if using MoE, Routing Replay is available — which the paper shows is the binding constraint on success.

When to Prefer This Method

The paper articulates an explicit tradeoff between on-policy training stability and off-policy training convergence speed, mediated by the choice of Routing Replay variant. These tradeoffs are grounded in the paper's specific experimental comparisons and theoretical framework.

• Prefer on-policy MiniRL without Routing Replay ($N=1$, global batch = mini-batch = 1,024) when training stability is the paramount concern and wall-clock training time is less constrained. This configuration achieves the smoothest training dynamics (Figure 1: no entropy crashes, no KL spikes) and the paper's highest claimed stability. However, it converges more slowly in terms of total compute than off-policy configurations — Figure 1 shows benchmark scores plateauing around 0.73–0.75 after ~1,200 gradient steps, while Figure 3 ($N=4$ with R3) reaches ~0.78–0.80 after ~2,000 steps. Note that the off-policy run uses more total data per global step (4× the batch size), so the per-step comparison understates the data efficiency advantage of off-policy training.

• Prefer off-policy MiniRL with R2 + clipping ($N=2$, global batch = 2,048) when moderate acceleration is desired and the model architecture is MoE. This configuration achieves the best performance at low off-policiness (~0.75 benchmark vs. ~0.72 for R3 at $N=2$; Figure 2), because R2 does not alter the target policy in the first mini-batch (Table 1), avoiding the bias that R3 introduces. The convergence is roughly 2× faster in wall-clock terms than on-policy (twice as many gradient updates per rollout batch), with stability maintained through at least 2,500 gradient steps.

• Prefer off-policy MiniRL with R3 + clipping ($N \geq 4$, global batch $\geq$ 4,096) when maximum convergence speed is required and the infrastructure supports larger batches. R3 becomes necessary at these off-policy levels because policy staleness accumulates across 4–8 mini-batches, and R3's reduction of both training–inference discrepancy and routing-induced staleness outweighs the bias it introduces (Figure 3: R3 stable at ~0.78–0.80 through 4,000 steps vs. R2 declining after ~2,000 steps; Figure 4: R3 is the only configuration stable through 5,000 steps at $N=8$). The convergence speed benefit is substantial — $N=4$ trains on 4× more data per rollout batch than on-policy — but requires the engineering infrastructure to handle batch sizes of 4,096–8,192 responses.

• Avoid Routing Replay entirely in on-policy training, even for MoE models. The paper shows that R3 provides no benefit and can degrade performance when $N=1$ (Figure 1: MiniRL + R3 achieves scores comparable to or slightly below MiniRL alone, and combining R3 with length normalization degrades scores further). The bias introduced by fixing experts is not compensated by any staleness reduction when staleness is zero. This is a strong, specific recommendation that contradicts the intuition that "Routing Replay is good for MoE models and should always be used" — the paper's theory explains exactly when it helps and when it hurts.

• Do NOT use length normalization in the RL objective, regardless of on-policy or off-policy configuration. The paper shows that length-normalized variants consistently underperform their unnormalized counterparts (Figure 1: ~0.68–0.70 vs. ~0.73–0.75 for MiniRL; Figures 2–4: length-normalized variants are not tested but the on-policy finding likely extends). The theoretical reason — length normalization breaks the equality $\nabla_\theta J^{\text{token}} \approx \nabla_\theta J^{\text{seq}}$ — is fundamental to the approximation structure, not specific to a particular batch size or model architecture. This recommendation runs counter to common practice (GRPO and CISPO both use length normalization) and is one of the paper's most actionable prescriptions.