On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitation

ArXiv: 2603.22117

🎯 Pitch

RLVR doesn't just change how much a model deviates from its base—it shifts token probabilities in a highly specific direction, and ignoring that directionality makes you blind to the actual reasoning improvements. By simply amplifying the learned signed log-probability difference Δlog p at test time, models gain up to 3 accuracy points without any extra training.

1. Executive Summary

This paper analyzes how reinforcement learning with verifiable rewards (RLVR) transforms a base language model into a reasoning-specialized counterpart, arguing that the signed, token-level log-probability difference Δlog p—rather than magnitude-based metrics like entropy or KL divergence—is the critical lens for understanding RLVR's effects. Through statistical analysis and token-replacement interventions on the AIME-24 benchmark using Qwen2.5 and Qwen3 model pairs trained via DAPO, ORZ, and UniReason, the authors demonstrate that Δlog p identifies sparse yet reasoning-critical token updates (recovering RLVR accuracy with only ~10% token replacements, versus 15–30% for divergence or entropy). Building on this insight, the paper proposes two practical methods: a test-time extrapolation that amplifies the policy along the learned Δlog p direction (raising AIME-24 Avg@32 by 1–3 points over the RLVR model without additional training) and a training-time advantage reweighting that upweights low-probability tokens implicated by high Δlog p (improving average reasoning accuracy across AIME24, AIME25, and AMC by ~1.6 points over DAPO baselines). The analysis establishes that RLVR's performance gains originate from sparse, directional updates concentrated on low-probability tokens, with magnitude-only metrics proving insufficient to characterize this transformation.

2. Context and Motivation

The Core Problem: We Don't Understand How RLVR Changes a Model

The fundamental question this paper tackles is deceptively simple: when RLVR fine-tunes a base language model to become a reasoning specialist, what actually changes inside the model? This question matters because RLVR has become the dominant paradigm for building state-of-the-art reasoning models. Systems like OpenAI o1, DeepSeek R1, Gemini 2.5, and Qwen3 all rely on RLVR as a core algorithmic component. The approach works remarkably well in practice—it substantially improves mathematical reasoning, code generation, and other verifiable tasks—but the mechanism of improvement remains poorly understood.

This gap has real consequences. Without understanding how RLVR transforms a model's output distribution, practitioners are flying blind when making crucial design decisions: what training hyperparameters matter most, which tokens are actually being modified, and whether the learned changes generalize beyond the training distribution. More fundamentally, if we could precisely characterize what RLVR does, we might be able to replicate its benefits more efficiently, avoid its failure modes, or even surpass its performance through targeted interventions.

The Prevailing Narrative: RLVR Induces Sparse Updates

Prior work has converged on one consistent finding: RLVR produces sparse changes. The final RL-trained model $\pi_{\text{RL}}$ does not uniformly shift its entire output distribution away from the base model $\pi_{\text{Base}}$; instead, it modifies only a small subset of tokens. This sparsity has been observed through multiple independent analytical lenses:

• Entropy-based analyses (Wang et al., 2025b) find that RLVR updates concentrate on high-entropy tokens—positions where the model was uncertain during generation. The argument is intuitive: if the model is already confident about a token, there's little to be gained from modifying it; RLVR should focus its capacity on the ambiguous decisions that actually affect reasoning correctness.

• Divergence-based analyses (Huan et al., 2025) corroborate sparsity by measuring the per-token KL divergence between $\pi_{\text{Base}}$ and $\pi_{\text{RL}}$. The divergence distribution is sharply peaked near zero, indicating that for most tokens, the two distributions are nearly identical. Only a small fraction of tokens exhibit meaningful distributional shift.

• Gradient-based analyses (Yang et al., 2025b; Deng et al., 2025) trace sparsity back to the training dynamics itself, showing that policy gradient updates selectively concentrate on specific tokens rather than distributing learning evenly across sequences. The gradient norm is not uniform; it spikes at particular positions.

These converging findings have shaped the field's understanding: RLVR works by making surgical edits to the model's output distribution rather than broadly retraining it. The natural follow-up question becomes: which tokens receive these edits, and how do we identify them?

Where Current Understanding Falls Short: The Magnitude Trap

The critical limitation of prior work is its nearly exclusive focus on the magnitude of change. Whether measuring entropy reduction, KL divergence, or gradient norm, existing analyses ask "how much did the distribution change at this position?" This perspective implicitly treats RLVR's effect as a scalar phenomenon—the model simply pushes harder in some places than others.

The paper identifies this as a fundamental blind spot through a revealing visualization (Figure 1b). When comparing histograms of entropy or KL divergence measured on tokens generated by $\pi_{\text{Base}}$ versus tokens generated by $\pi_{\text{RL}}$, the distributions are nearly indistinguishable. Both cluster sharply near zero (confirming sparsity), but there is no visible separation between the two models' output characteristics. In other words, if you were handed a token and told only its entropy or its KL divergence, you could not tell whether it came from the base model's generation or the RLVR model's generation.

This is not a failure of measurement precision—it is a conceptual limitation of magnitude-only metrics. Entropy and divergence are unsigned: they collapse increases and decreases in probability into the same scale. A token whose probability rose from 0.01 to 0.1 and a token whose probability fell from 0.1 to 0.01 both contribute similarly to a divergence metric. But from the perspective of what RLVR is trying to accomplish—produce better reasoning chains—these two changes are opposites. One represents a token the model learned to favor (potentially a reasoning-enhancing word like "therefore" or "step"), while the other represents a token the model learned to suppress (potentially a reasoning shortcut or error-prone transition).

The consequence is that magnitude-based metrics conflate the signal of RLVR's learning with the noise of benign distributional fluctuations, making it impossible to precisely isolate which tokens are actually responsible for improved reasoning performance.

Why Direction Matters: Separating Signal from Noise

The paper's core methodological contribution is the shift from analyzing "how much" to analyzing "how much and in which direction." The signed log-probability difference:

$\Delta\log p(y_t | x, y_{<t}) = \log \pi_{\text{RL}}(y_t | x, y_{<t}) - \log \pi_{\text{Base}}(y_t | x, y_{<t})$

captures exactly this directional information. Positive values indicate tokens that the RLVR model learned to prefer (probabilities increased); negative values indicate tokens it learned to avoid (probabilities decreased). As shown in Figure 1b, the Δlog p histogram exhibits a clear bimodal pattern with two distinct tails—a positive tail corresponding to RLVR-favored tokens and a negative tail corresponding to base-favored tokens. This directional signature is completely absent from entropy or divergence distributions.

The practical significance of this directional decomposition becomes apparent when we consider the implicit question RLVR is answering during training. At each token position, some choices lead toward correct reasoning chains and some lead toward incorrect chains. RLVR's verifier signal pushes the model to increase the probabilities of tokens that appeared in successful trajectories and decrease the probabilities of tokens from unsuccessful ones. The Δlog p metric aligns with this underlying objective: a large positive Δlog p indicates a token that the verifier consistently rewarded, while a large negative Δlog p indicates one it consistently penalized. In contrast, a token with high entropy but near-zero Δlog p was already ambiguous for both models—RLVR didn't meaningfully change the distribution there, so it's unlikely to be the source of improved reasoning.

The Practical Imperative: From Understanding to Improvement

Beyond the analytical motivation, the paper is driven by a pragmatic question: can we exploit our understanding of RLVR's mechanism to build better systems? The progression from analysis to method follows a direct arc:

1. If Δlog p correctly identifies the reasoning-critical tokens RLVR modifies (and the token replacement experiments demonstrate it does, recovering RLVR accuracy with ~10% replacements versus 15–30% for divergence or entropy), then...

2. Perhaps we can amplify these changes beyond what RLVR naturally achieves—treating Δlog p not just as a diagnostic but as a learned "reasoning direction" that can be extrapolated at test time.

3. And perhaps we can intervene during training itself to strengthen the learning signal on the tokens that Δlog p would ultimately target, making the RLVR process more efficient.

This chain of reasoning distinguishes the paper from purely analytical contributions. Rather than stopping at "RLVR updates are sparse and directional," the authors use that insight to design two practical methods (test-time extrapolation and training-time reweighting) that demonstrably improve reasoning performance over strong RLVR baselines.

How This Paper Positions Itself Relative to Prior Work

The paper situates itself at the intersection of two active research threads:

Thread 1: Understanding RLVR effects. The line of work from Wang et al. (2025b), Huan et al. (2025), and Yang et al. (2025b) converges on the sparsity of RLVR updates. The paper directly builds on these findings but identifies a critical gap: all these analyses emphasize the magnitude of change, not its direction. Critically, the paper does not dismiss magnitude-based sparsity findings—it incorporates them (confirming that most tokens have near-zero Δlog p). Instead, it argues that direction is the missing dimension that distinguishes RLVR's meaningful updates from benign distributional variation.

The paper's relationship to Yang et al. (2025b) is particularly instructive. That work analyzed the update direction (increase or decrease) of probabilities at each individual gradient step. The current paper extends the notion of direction to the full distributional shift from base to final RLVR model—a more holistic and practically useful framing, since it captures the accumulated effect of training rather than per-step fluctuations.

Thread 2: Methods for token-level intervention. The token replacement experiment (Algorithm 1) builds directly on Meng et al. (2026), who originally proposed bidirectional token swapping to verify RL's sparsity. The current paper repurposes this technique as a diagnostic tool: by comparing how different metrics guide replacement, it produces a quantitative ordering of their precision in isolating reasoning-critical tokens. This is a methodological innovation—token replacement becomes a benchmark for evaluating analysis metrics rather than just a validation technique.

The test-time extrapolation (Section 4.1) connects to the reward-guided decoding literature (Khanov et al., 2024; Liu et al., 2024; Xu et al., 2025), where external reward functions re-weight the model's output distribution. The paper positions Δlog p as an implicit token-level reward learned during training, eliminating the need for separate reward models or verifiers at test time.

The training-time reweighting (Section 4.2) engages with competing hypotheses about which tokens RLVR should prioritize. Deng et al. (2025) argue for upweighting low-perplexity responses, while Yang et al. (2025b) argue that low-probability tokens over-dominate training and propose counterbalancing them. The paper stakes out a middle ground: low-probability tokens are indeed critical (consistent with Yang et al.'s observation that they dominate gradient updates), but rather than dampening them, the training process should be reinforced on these tokens because they are the positions that ultimately yield high Δlog p—the reasoning-enhancing changes in the final model.

The Central Tension the Paper Resolves

Underlying the paper's contributions is a resolution of an apparent tension in the literature. On one hand, low-probability tokens receive disproportionately large gradient updates (Lemma 3.1, Figure 3a). On the other hand, Yang et al. (2025b) observe that allowing low-probability tokens to dominate can destabilize training. The paper's resolution is to distinguish between spurious low-probability tokens (noise in the sampling process) and reasoning-critical low-probability tokens (positions where the base model was uncertain but a clear correct answer exists). The latter are precisely the tokens with high final Δlog p, and they should be upweighted. The former should be addressed through standard RL stabilization techniques (clipping, KL penalties). This distinction explains why the paper's reweighting method (Equation 8) improves performance while a blanket anti-domination approach (Yang et al., 2025b) can reduce exploration and Pass@k scores (Table 3, Figure 5).

Concrete Goals of the Paper

With this context established, the paper's specific objectives are:

1. Demonstrate that signed Δlog p provides strictly more information than magnitude-based metrics for identifying reasoning-critical RLVR updates, using both statistical comparison (histograms) and causal intervention (token replacement experiments, Figure 2).

2. Explain why these updates are sparse through gradient analysis (Lemma 3.1, Figure 3), tracing the sparsity of final Δlog p back to the concentration of policy gradient updates on low-probability tokens during training.

3. Exploit this understanding through two practical methods: test-time extrapolation along the Δlog p direction (Section 4.1) and training-time advantage reweighting focused on low-probability tokens (Section 4.2), with both methods validated on AIME-24, AIME-25, and AMC benchmarks across multiple model families (Qwen2.5, Qwen3) and training algorithms (DAPO, ORZ, UniReason).

The paper's overarching thesis is deceptively simple but well-supported: diagnose and improve RLVR by its update direction, not just its magnitude. This reframing opens both a new analytical lens for understanding what RLVR does and a practical toolkit for improving it.

3. Technical Approach

3.1 Reader Orientation

This paper is primarily an empirical analysis paper with two derived methods. Its core idea is that the signed, token-level log-probability difference between a base model and its RLVR-fine-tuned counterpart—a metric the authors call $\Delta\log p$—provides a strictly more informative diagnostic for understanding RLVR's effects than magnitude-based metrics like entropy or KL divergence, and that this diagnostic can be directly exploited to improve reasoning performance through both test-time and training-time interventions.

The problem being solved is identifying and exploiting the precise mechanism by which RLVR transforms a base model into a reasoning specialist. The "shape" of the solution is a three-stage pipeline: first, diagnose which tokens RLVR modifies using the directional $\Delta\log p$ metric; second, validate that these tokens are causally responsible for reasoning improvements through controlled token-replacement experiments; third, exploit this understanding through a test-time extrapolation (amplifying the learned direction on critical tokens) and a training-time reweighting (focusing the policy gradient on tokens that will ultimately yield high $\Delta\log p$).

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major analysis components and two practical extensions:

1. Statistical Analysis Pipeline — computes token-level metrics ($\Delta\log p$, entropy, KL divergence) on generations from paired $\pi_{\text{Base}}$ and $\pi_{\text{RL}}$ models, producing histograms that reveal whether each metric can distinguish which model generated a given token.

2. Token Replacement Experiment (Algorithm 1) — a causal intervention that generates responses from $\pi_{\text{Base}}$, then selectively replaces tokens at positions identified by each metric with tokens sampled from $\pi_{\text{RL}}$. The replacement rate is swept to compare how efficiently each metric recovers RLVR-level reasoning accuracy.

3. Gradient Analysis (Lemma 3.1) — derives the relationship between a token's probability during training and the magnitude of the policy gradient update it receives, explaining why low-probability tokens dominate the learning signal and why they correspond to high $\Delta\log p$ in the final model.

4. Top-p Filtering Experiment (Figure 3c) — a causal training intervention that excludes low-probability tokens during RLVR rollout to verify they are essential, not merely correlated with performance.

5. Test-Time Selective Extrapolation (Section 4.1) — amplifies the learned $\Delta\log p$ direction on critical tokens during inference by re-weighting the RLVR model's distribution, pushing it further along the base-to-RLVR shift.

6. Training-Time Advantage Reweighting (Section 4.2) — modifies the DAPO policy gradient objective to upweight advantages on low-probability tokens (those with the largest $1 - \pi_{\theta_{\text{old}}}(y_{i,t})$), strengthening the learning signal on positions that will become the high-$\Delta\log p$ tokens in the final model.

Information flows as follows: a mathematical reasoning prompt enters $\pi_{\text{Base}}$ → the base model generates candidate tokens autoregressively → each token's log-probability under both $\pi_{\text{Base}}$ and $\pi_{\text{RL}}$ is recorded → $\Delta\log p$ is computed as the difference → tokens with large negative $\Delta\log p$ (those RLVR learned to suppress) are identified as reasoning-critical → these positions guide either token replacement (for diagnostic validation) or selective extrapolation (for performance improvement). In the training-time method, the same principle runs in reverse: during RLVR training, low-probability tokens (which will become high-$\Delta\log p$ tokens) receive amplified advantage signals, concentrating the gradient on reasoning-critical positions from the start.

3.3 Roadmap for the Deep Dive

• First, the core diagnostic metric $\Delta\log p$ — its definition, computational properties, and how it differs from entropy and divergence. This is the analytical foundation everything else builds on.

• Second, the statistical comparison methodology — how histograms are constructed across model pairs to demonstrate that $\Delta\log p$ captures directional shifts invisible to magnitude metrics.

• Third, the token replacement experiment — the causal intervention protocol (Algorithm 1), the three metric-specific criteria functions, and what the replacement-rate-versus-accuracy curves tell us about each metric's precision.

• Fourth, the gradient-based explanation for sparsity — Lemma 3.1 and its proof sketch, showing that policy gradient norms scale with $1 - \pi_\theta(y_{i,t})$, naturally concentrating updates on low-probability tokens. This connects the final model's $\Delta\log p$ pattern back to training dynamics.

• Fifth, the training-time top-p intervention — a causal experiment (Figure 3c) that filters out low-probability tokens during rollout to verify their necessity for RLVR performance gains.

• Sixth, test-time selective extrapolation — the construction of $\pi^\gamma_{\text{Extra}}$ (Equation 7), the selective gating mechanism, and the theoretical justification via Natural Policy Gradient in a tabular bandit setting (Theorem 4.1).

• Seventh, training-time advantage reweighting — the modification to DAPO's objective (Equation 8), the connection to the $\Delta\log p$ pattern, and comparison with competing reweighting hypotheses (PPL-based, anti-domination).

3.4 Detailed, Sentence-Based Technical Breakdown

This is an empirical analysis paper with two derived methods. The core analytical contribution is the $\Delta\log p$ metric and the experimental demonstration that it more precisely identifies reasoning-critical RLVR updates than magnitude-based alternatives. The derived contributions are test-time extrapolation and training-time reweighting, both exploiting the identified directional learning pattern.

The Directional Diagnostic: $\Delta\log p$

The paper's central analytical tool is the signed, token-level log-probability difference between the final RLVR-trained model and its base counterpart:

$\Delta\log p(y_t | x, y_{<t}) = \log \pi_{\text{RL}}(y_t | x, y_{<t}) - \log \pi_{\text{Base}}(y_t | x, y_{<t})$

where $y_t$ is the token being evaluated at position $t$, $x$ is the input prompt, $y_{<t}$ is the prefix of tokens generated so far, $\pi_{\text{RL}}(y_t | x, y_{<t})$ is the RLVR-trained model's predicted probability for token $y_t$ given the context, and $\pi_{\text{Base}}(y_t | x, y_{<t})$ is the base model's predicted probability for the same token under the same context.

What it computes: for every token position in a generated response, this metric takes the logarithm of the probability the RLVR model assigns to the token that was actually generated, subtracts the logarithm of the probability the base model assigns to that same token, and returns a signed scalar. A positive value means the RLVR model increased the probability of that token relative to the base model (it "favors" that token more). A negative value means the RLVR model decreased the probability (it "disfavors" or "suppresses" that token relative to the base model). A value near zero means the two models assign nearly identical probability to that token.

Why this form: the signed difference captures directional information that magnitude-based metrics deliberately discard. Consider a concrete example: suppose at some position in a mathematical reasoning chain, the base model assigns probability 0.15 to the token "therefore" and 0.05 to the token "however." After RLVR training on problems where logical deduction is critical, the RLVR model might assign 0.40 to "therefore" (a large positive $\Delta\log p = \log(0.40) - \log(0.15) \approx 0.98$) and 0.001 to "however" (a large negative $\Delta\log p = \log(0.001) - \log(0.05) \approx -3.91$). The entropy at this position might be similar for both models (both distributions are somewhat peaked), and the KL divergence might be large but unsigned (it tells you that the distribution changed but not which tokens were favored). The $\Delta\log p$ metric explicitly decomposes the change into a per-token signal: RLVR learned to prefer "therefore" at this position and to avoid "however," which directly encodes the reasoning behavior improvement (logical connectors over contrastive ones).

Key property separating it from magnitude metrics: the signed nature means that $\Delta\log p$ measured on tokens generated by $\pi_{\text{RL}}$ will tend positive (since the RLVR model is more likely to generate tokens it assigned higher probability), while measured on tokens generated by $\pi_{\text{Base}}$ will tend negative (since those tokens come from a distribution the RLVR model has moved away from). This creates the bimodal histogram pattern in Figure 1b that magnitude metrics cannot replicate—the distribution of $\Delta\log p$ is different depending on which model generated the tokens, providing a diagnostic signature of RLVR's effect. Entropy and divergence histograms, being unsigned and based on the shape of the distribution rather than its shift, look nearly identical regardless of which model's tokens are being evaluated.

Computational cost: computing $\Delta\log p$ requires maintaining both the base and RLVR models and evaluating each generated token under both. This is the standard cost of any comparative analysis and does not require additional training or specialized infrastructure. The paper notes in its limitations that the requirement of two models is a practical constraint for the test-time extrapolation method (which needs both models at inference), though the training-time reweighting method does not.

Statistical Comparison Methodology

The paper conducts a systematic statistical comparison across three RLVR-base model pairs: ORZ-32B (trained from Qwen2.5-32B via PPO), DAPO-32B (trained from the same Qwen2.5-32B base via DAPO), and UniReason-14B (trained from Qwen3-14B-Base via GRPO).

Generation protocol: for each model pair, the authors sample 32 responses per question from the AIME-24 dataset (30 competition-level math problems) using top-p=0.7, temperature=1.0, and a maximum generation length of 20,000 tokens. Both the base and RLVR models generate responses to the same prompts. This produces two sets of token sequences: one from $\pi_{\text{Base}}$ and one from $\pi_{\text{RL}}$.

Metric computation: for each token in each generated sequence, the following metrics are computed:

• Entropy of the base model's distribution at that position: $H_{\pi_{\text{Base}}}(y_t) = \mathbb{E}_{y \sim \pi_{\text{Base}}(\cdot|x, y_{<t})}[-\log \pi_{\text{Base}}(y | x, y_{<t})]$. This is the expected negative log-probability under $\pi_{\text{Base}}$ at that context, measuring how uncertain the base model is. The RLVR model's entropy $H_{\pi_{\text{RL}}}$ is computed analogously.

• KL divergence between the two models' distributions at that position: $D_{\text{KL}}^{\pi_{\text{RL}}, \pi_{\text{Base}}}(y_t) = \mathbb{E}_{y \sim \pi_{\text{RL}}(\cdot|x, y_{<t})}[\log(\pi_{\text{RL}}(y|x, y_{<t}) / \pi_{\text{Base}}(y|x, y_{<t}))]$. The paper also computes the reverse KL $D_{\text{KL}}^{\pi_{\text{Base}}, \pi_{\text{RL}}}$ and the averaged KL $\bar{D}_{\text{KL}} = \frac{1}{2}(D_{\text{KL}}^{\pi_{\text{RL}}, \pi_{\text{Base}}} + D_{\text{KL}}^{\pi_{\text{Base}}, \pi_{\text{RL}}})$ to avoid asymmetry bias where one direction might overweight certain distributional differences.

• Log-probability difference as defined in Equation 1, computed directly as the difference of log-probabilities for the token actually generated (not an expectation over the vocabulary).

Histogram construction: for each metric, the authors plot separate histograms for tokens generated by $\pi_{\text{Base}}$ and tokens generated by $\pi_{\text{RL}}$ (Figure 1b, Figures 12-15 in Appendix E). The y-axis is on a log scale to emphasize the tail behavior. The key visual diagnostic is whether the two histograms (base-generated vs. RLVR-generated tokens) are distinguishable. If a metric captures the effect of RLVR, the distribution of that metric should look different depending on which model's tokens are being evaluated.

Observed pattern for magnitude metrics: the entropy and KL divergence histograms are sharply peaked near zero for both base-generated and RLVR-generated tokens (confirming the sparsity finding from prior work—most tokens are nearly unchanged), but the two histograms overlap almost completely. For example, on the UniReason model pair, 82.0% of tokens have RLVR model entropy less than 0.1 when evaluating RLVR-generated sequences, versus 86.9% when evaluating base-generated sequences (Figure 13). The distributions are nearly identical. The same holds for KL divergence variants (Figures 13-15). This is because entropy and divergence measure the shape and scale of the probability distribution, not whether specific tokens are more or less favored. A distribution can be high-entropy (flat) or low-entropy (peaked) entirely independently of which tokens occupy the probability mass. RLVR can dramatically reshape which tokens receive probability without changing the overall entropy or divergence from the base distribution at that position.

Observed pattern for $\Delta\log p$: in contrast, the $\Delta\log p$ histogram exhibits a clear bimodal pattern with separation between the two generating models. For tokens generated by $\pi_{\text{RL}}$, the distribution has a positive tail (tokens whose probability RLVR increased), while for tokens generated by $\pi_{\text{Base}}$, the distribution has a negative tail (tokens that RLVR learned to suppress or that the base model favored more). For the DAPO model pair, 90.9% of RLVR-generated tokens have $\Delta\log p$ within $[-0.5, 0.5]$ (concentrated near zero, confirming sparsity), but the positive tail beyond $+2.0$ is populated almost exclusively by RLVR-generated tokens, while the negative tail below $-2.0$ is dominated by base-generated tokens (Figure 12b). The fraction of extreme values differs substantially: 97.3% of base-generated tokens are within $[-0.25, 0.25]$ for the UniReason pair, versus 92.2% for RLVR-generated tokens (Figure 12a). This 5.1 percentage point difference in concentration near zero is a directional signal absent from magnitude metrics.

Why this matters for downstream analysis: the statistical separation directly enables the token-replacement experiment and selective extrapolation. If $\Delta\log p$ can distinguish which model generated a token, then selecting tokens by $\Delta\log p$ threshold should identify positions where the RLVR model made a meaningful change—not just a large change (which could be noise or distributional fluctuation) but a directional change aligned with the verifier's training signal.

Token Replacement Experiment: Causal Validation

The paper adapts the cross-sample token replacement technique from Meng et al. (2026) into a diagnostic benchmark for comparing analysis metrics. The experiment is formalized in Algorithm 1.

Algorithm 1: Selective Token Replacement (reproduced from Section 3.2):

Require: Base and RLVR models π_Base, π_RL, prompt x, criterion function f^τ(·) ∈ {0, 1}
1: Initialize response: t ← 0, y_{≤0} ← ""
2: while y_t ≠ "<EOS>" do
3:     t ← t + 1
4:     Sample from base: y_t ∼ π_Base(· | x, y_{<t})
5:     if f^τ(y_t | x, y_{<t}) = 1 then
6:         Replace the token: y_t ∼ π_RL(· | x, y_{<t})
7:     end if
8: end while
9: return y_{≤t}

What happens step by step: the model generates text autoregressively. At each position $t$, a token is first sampled from the base model's distribution $\pi_{\text{Base}}$. Then, a criterion function $f^\tau$ specific to the metric being evaluated checks whether this position meets the replacement condition. If it does (the function returns 1), the token is discarded and a new token is sampled from the RLVR model's distribution $\pi_{\text{RL}}$ at the same context. If it does not, the base model's token is kept. The threshold $\tau$ is a hyperparameter that controls how selective the criterion is: lower thresholds mean fewer tokens are replaced (the criterion is stricter), higher thresholds mean more tokens are replaced. By sweeping $\tau$, the experiment produces a curve of accuracy versus replacement rate for each metric.

The three criterion functions:

1. Entropy-based: $f^\tau_H(y_t | x, y_{<t}) = \mathbb{I}(H_{\pi_{\text{Base}}}(\cdot | x, y_{<t}) > \tau)$. This replaces tokens at positions where the base model's entropy exceeds a threshold. The motivation comes from Wang et al. (2025b)'s hypothesis that RLVR concentrates updates on high-entropy positions. The indicator function $\mathbb{I}(\cdot)$ returns 1 if the condition is true and 0 otherwise.

2. KL divergence-based: $f^\tau_D(y_t | x, y_{<t}) = \mathbb{I}(\bar{D}_{\text{KL}}(\cdot | x, y_{<t}) > \tau)$. This replaces tokens at positions where the averaged KL divergence between the two models exceeds a threshold, targeting positions where the magnitude of distributional change is largest regardless of direction. The paper uses the averaged KL $\bar{D}_{\text{KL}}$ for the main comparison to avoid asymmetry bias, with forward and reverse KL variants evaluated in ablation (Figure 6).

3. Log-probability difference-based: $f^\tau_{\text{logp}}(y_t | x, y_{<t}) = \mathbb{I}(\Delta\log p(y_t | x, y_{<t}) < \tau)$. This replaces tokens where the signed log-probability difference falls below a threshold. Recall that a large negative $\Delta\log p$ means the RLVR model substantially decreased the probability of the token the base model generated—the base model chose a token that the RLVR model learned to avoid. The criterion targets these positions specifically, on the hypothesis that RLVR's improvements come from suppressing certain base-model choices and replacing them with RLVR-favored alternatives.

Why negative $\Delta\log p$ for replacement: the logic is directional. If the base model generates a token $y_t$ at some position, and $\Delta\log p(y_t)$ is large and positive, that means the RLVR model also favors that token—no replacement is needed because the base model already made the "right" choice from the RLVR perspective. If $\Delta\log p(y_t)$ is large and negative, the base model generated a token the RLVR model learned to avoid—this is precisely where replacement should help. The threshold $\tau$ is set to negative values (e.g., -0.3 to -0.5 in the experiments), meaning only tokens with $\Delta\log p$ below that negative threshold are replaced.

Random baseline: $f^\tau_{\text{rand}}(\cdot) = \mathbb{I}_{\rho \sim U[0,1]}(\rho < \tau)$. This replaces tokens uniformly at random with probability $\tau$, providing a lower bound on how much replacement helps without any metric-based selection.

Experimental sweep: for each metric and each model pair (ORZ, DAPO, UniReason), the threshold $\tau$ is varied to produce replacement rates ranging from approximately 2% to 30% of all generated tokens. For each threshold, 32 responses per AIME-24 question are generated and the average accuracy (Avg@32) is computed. The replacement rate is measured empirically as the fraction of tokens that pass the criterion and are replaced.

Reading the results (Figure 2): the key output is a set of accuracy-versus-replacement-rate curves. All metric-based replacement curves lie above the random baseline, confirming that selective replacement of a minority of tokens can recover RLVR-level performance. The horizontal dashed line marks the RLVR model's accuracy (e.g., 46.15 for ORZ-32B, 52.50 for DAPO-32B, 54.58 for UniReason-14B). The vertical position of each curve at a given replacement rate indicates how much RLVR performance has been recovered.

The critical comparison is which metric reaches the RLVR accuracy line at the lowest replacement rate. For DAPO-32B, $\Delta\log p$-based replacement reaches 52.50% accuracy at approximately 10-11% replacement rate. The KL divergence-based replacement requires approximately 15-20% replacement to reach the same level. The entropy-based replacement requires 25-30% replacement. The random baseline never reaches RLVR accuracy within the tested range (it plateaus well below).

This ordering—$\Delta\log p$ most efficient, then divergence, then entropy—holds consistently across all three model pairs (Figure 2) and across all divergence and entropy variants tested in ablation (Figure 6, Appendix A.2). The forward KL $D_{\text{KL}}^{\pi_{\text{RL}},\pi_{\text{Base}}}$, reverse KL $D_{\text{KL}}^{\pi_{\text{Base}},\pi_{\text{RL}}}$, and RLVR model's entropy $H_{\pi_{\text{RL}}}$ all underperform $\Delta\log p$, though they generally outperform random replacement.

Interpretation of the ordering: the fact that $\Delta\log p$ recovers RLVR accuracy with approximately half the replacements of KL divergence (10% vs. 15-20%) directly quantifies the precision gain from directional information. KL divergence identifies positions where the distribution changed, but many of those changes may be irrelevant to reasoning performance—benign shifts in token probabilities that don't affect whether the final answer is correct. $\Delta\log p$ identifies positions where the change was aligned with the verifier's training signal (tokens the model learned to suppress because they appeared in unsuccessful trajectories). The entropy criterion is least efficient because high-entropy positions are simply positions where the base model was uncertain—RLVR may or may not have changed the distribution there, and even if it did, the change may not be reasoning-relevant.

Per-problem analysis (Appendix A.2, Figure 8): the paper also reports per-problem accuracy curves for the $\Delta\log p$-based replacement. Two notable patterns emerge. First, some problems (approximately 3-5 out of 30) have accuracy that remains at zero across all replacement rates—these are genuinely hard problems that neither the base nor the RLVR model can solve, so no amount of targeted replacement helps. Second, for a small number of problems, accuracy initially drops when replacement is introduced at very low rates, then recovers as the replacement rate increases. The authors hypothesize that this occurs when a small number of RLVR tokens are inserted into a base-model context, creating a mismatch that the base model cannot handle coherently—essentially, the generation is "half-base, half-RLVR" and neither model can complete it properly. As replacement becomes more extensive, the generation becomes more RLVR-like and coherence recovers.

Top replacement tokens (Figure 7): at a matched replacement rate of approximately 8%, the paper visualizes the top 50 tokens selected by each criterion. Entropy-based selection favors logical transition words (e.g., "Thus," "need," "can," "To," "Now"), which are indeed high-entropy positions—the model is choosing among many possible connectors. KL divergence-based selection and $\Delta\log p$-based selection favor more specific mathematical reasoning tokens, including mathematical symbols (e.g., "\(", "\)", "$", "=") and reasoning-specific vocabulary (e.g., "approach," "denote," "find"). The `$\Delta\log p$` criterion selects the highest proportion of mathematical symbols, which the authors interpret as evidence that it more precisely identifies the reasoning-critical positions.

Gradient-Based Explanation for Sparsity

Having established that RLVR produces sparse, directional updates best identified by $\Delta\log p$, the paper traces this sparsity back to the training dynamics of the policy gradient algorithm. The analysis focuses on the DAPO objective (Equation 4) because it is the primary baseline algorithm.

The DAPO gradient decomposition: for a single token $y_{i,t}$ in a generated response $y_i$ (the $i$-th response in a group of $G$ responses to prompt $x$), assuming the token is not affected by the clipping operator, the gradient of the DAPO objective with respect to the policy parameters $\theta$ is:

$\nabla_\theta J_{\text{DAPO}}(y_{i,t}) = w_{i,t} \cdot \nabla_\theta \log \pi_\theta(y_{i,t} | x, y_{i,<t})$

where $w_{i,t} = r_{i,t}(\theta) \cdot \hat{A}_{i,t}$ combines the importance sampling ratio $r_{i,t}(\theta) = \pi_\theta(y_{i,t}) / \pi_{\theta_{\text{old}}}(y_{i,t})$ and the group-relative advantage $\hat{A}_{i,t}$ defined in Equation 2. This is the standard policy gradient form: the log-probability of the chosen token is scaled by a weight factor.

Why the gradient concentrates on low-probability tokens: the key insight is that $\nabla_\theta \log \pi_\theta(y_{i,t})$ is not uniform across tokens. For a softmax-parameterized policy with logits $z$, Lemma 3.1 provides the exact $\ell_1$-norm of the gradient with respect to the logits:

Lemma 3.1. For a softmax-parameterized LLM policy with logits vector $z$ for the output token $y_{i,t}$, the $\ell_1$-norm of the DAPO objective's gradient with respect to $z$ is given by:

$\|\nabla_z J_{\text{DAPO}}(y_{i,t} | x, y_{i,<t})\|_1 = 2|w_{i,t}| \cdot \left(1 - \pi_\theta(y_{i,t} | x, y_{i,<t})\right)$

where $\|\cdot\|_1$ is the $\ell_1$-norm (sum of absolute values of the gradient vector components), $|w_{i,t}|$ is the absolute value of the combined importance sampling ratio and advantage weight, and $1 - \pi_\theta(y_{i,t} | x, y_{i,<t})$ is one minus the probability the current policy assigns to the token that was actually sampled.

What the $\ell_1$-norm captures: the $\ell_1$-norm of the gradient with respect to the logits measures the total magnitude of the update that will be applied to the model's output distribution at this position. A larger $\ell_1$-norm means the policy parameters will shift more aggressively in response to this token.

The critical factor $1 - \pi_\theta(y_{i,t})$: this term grows as the probability of the sampled token decreases. If the token was highly probable under the current policy ($\pi_\theta(y_{i,t}) \approx 1$), then $1 - \pi_\theta(y_{i,t}) \approx 0$ and the gradient norm is small—the model already assigns high probability to this token, so there is little to learn. If the token was improbable ($\pi_\theta(y_{i,t}) \approx 0$), then $1 - \pi_\theta(y_{i,t}) \approx 1$ and the gradient norm approaches $2|w_{i,t}|$—the model is surprised by this token and will make a large update to account for it.

Why this form rather than alternatives: this is a direct consequence of the softmax parameterization combined with the log-likelihood gradient. For a token with index $k$ in the vocabulary $\mathcal{V}$, the partial derivative of $\log \pi_\theta(y_{i,t})$ with respect to logit $z_j$ is:

$\frac{\partial}{\partial z_j} \log \pi_\theta(y_{i,t}) = \begin{cases} 1 - \pi_\theta(\mathcal{V}_j) & \text{if } j = k \\ -\pi_\theta(\mathcal{V}_j) & \text{if } j \neq k \end{cases}$

Summing the absolute values over all vocabulary indices $j$ gives $(1 - \pi_\theta(y_{i,t})) + \sum_{j \neq k} \pi_\theta(\mathcal{V}_j) = 2(1 - \pi_\theta(y_{i,t}))$. The factor of 2 arises because increasing the logit for the chosen token (the $1 - \pi_\theta$ term for the correct index) also implicitly decreases the probabilities of all other tokens (the $\pi_\theta$ terms for all other indices), and both effects contribute to the gradient norm. Alternative parameterizations (e.g., a linear output layer without softmax) would not have this property—the gradient norm would be constant regardless of token probability. The softmax naturally creates larger gradients for surprising tokens.

Empirical verification (Figure 3a): the paper plots the probability distribution of tokens in DAPO's rollout data alongside the gradient norm coefficient $1 - \pi_\theta(\cdot)$ from an intermediate training step. The results are striking: 59.48% of tokens have probability $P \geq 0.98$ (highly probable, receiving negligible gradient), while only 3.17% of tokens have probability $P < 0.02$ (low-probability, receiving maximal gradient). Yet, when plotting the gradient norm coefficient ratio (the fraction of total gradient mass attributed to each probability bin), the low-probability bin dominates despite containing only 3.17% of tokens. This is the quantitative demonstration of "sparse, high-magnitude gradient updates": a small fraction of tokens receive the vast majority of the learning signal.

Connection to $\Delta\log p$ in the final model (Figure 3b): the paper then links gradient concentration during training to the $\Delta\log p$ pattern in the final model. Tokens are binned by their final $\Delta\log p$ value (e.g., top 0-4%, 4-8%, etc.), and within each bin, the average token probability under both the base and RLVR models is computed. The results show a clear monotonic relationship: higher $\Delta\log p$ bins correspond to lower token probabilities in both models. Moreover, the token probability substantially increases from base to RLVR within these high-$\Delta\log p$ bins. This completes the causal chain: during training, low-probability tokens receive the largest gradient updates → the model's probability on these tokens increases → in the final model, these same tokens exhibit large positive $\Delta\log p$. The sparsity of final $\Delta\log p$ (most tokens near zero) is thus a direct consequence of the sparsity of gradient updates, which is itself a consequence of the softmax parameterization's inherent focus on low-probability tokens.

The full gradient norm bound: the paper also references Yang et al. (2025b)'s formal result that the full gradient norm (not just the logit-level norm in Lemma 3.1) is tightly bounded by $1 - \pi_\theta(\cdot)$. This means that even when considering all model parameters (not just the output logits), the magnitude of the update at each token position is dominated by the $1 - \pi_\theta$ factor. The sparsity of updates is not an artifact of analyzing only the final layer; it propagates through the entire network via backpropagation.

Training-Time Top-p Intervention: Causal Necessity

To verify that low-probability tokens are not merely correlated with gradient updates but are essential for RLVR's performance gains, the paper conducts a training-time intervention experiment (Section 3.3, Figure 3c).

Experimental design: the Qwen2.5-Math-7B base model is trained using the standard DAPO recipe (same hyperparameters as the paper's main DAPO baseline), but with a modification to the rollout sampling strategy. During training, when generating the group of $G = 16$ responses per prompt, the sampling uses nucleus (top-p) filtering with varying thresholds: top-p = 0.7, 0.9, 0.95, and 1.0 (the default, no filtering). Top-p sampling truncates the probability distribution to the smallest set of tokens whose cumulative probability mass exceeds $p$, then samples only from those tokens. A top-p of 0.7 means that at each generation step, only the most probable tokens comprising 70% of the probability mass are considered; the remaining 30% (including all low-probability tokens) are assigned zero probability and cannot be sampled. This directly prevents the model from ever generating the low-probability tokens that Lemma 3.1 predicts would receive the largest gradient updates.

What the top-p filter removes: the tokens excluded by top-p filtering are precisely the low-probability tokens identified in Figure 3a as receiving the largest gradient norm coefficients. If these tokens are essential for learning, excluding them during training should degrade performance. If they are incidental (e.g., if the model can learn reasoning behaviors equally well from high-probability tokens), performance should be insensitive to top-p.

Results (Figure 3c): the experiment produces a clear monotonic degradation. With top-p = 1.0 (default DAPO, no filtering), AIME-24 Avg@32 reaches approximately 0.35 by step 140. With top-p = 0.95, performance drops noticeably (reaching approximately 0.30). With top-p = 0.9, performance drops further (approximately 0.25). With top-p = 0.7, performance collapses to approximately 0.20. The more aggressive the filtering of low-probability tokens, the worse the final performance. This is a causal demonstration: low-probability tokens are not merely passive recipients of large gradients; removing them from the training distribution breaks RLVR's ability to learn reasoning.

Why this strengthens the gradient-based explanation: the top-p experiment closes a potential alternative explanation. One could have argued that low-probability tokens receive large gradients but are not actually important—perhaps the model would learn the same behaviors from higher-probability tokens, just more slowly. The top-p results rule this out: even aggressive exclusion of low-probability tokens (top-p = 0.7, removing 30% of probability mass) substantially impairs final performance, indicating that the reasoning behaviors RLVR learns require low-probability token experiences during training. You cannot learn to produce correct mathematical reasoning chains while only practicing on tokens the model is already confident about.

Connection to later methods: this finding directly motivates the training-time reweighting method (Section 4.2). If low-probability tokens are essential and receive proportionally large gradients, then explicitly amplifying their gradient signal (rather than just letting the natural softmax gradient concentration do its work) should accelerate or improve learning. The reweighting method (Equation 8) does exactly this by multiplying the advantage $\hat{A}_{i,t}$ by a factor that increases as $1 - \pi_{\theta_{\text{old}}}(y_{i,t})$ grows.

Test-Time Selective Extrapolation

The token replacement experiment demonstrated that selectively replacing base-model tokens with RLVR-model tokens at positions identified by $\Delta\log p$ can recover RLVR-level accuracy. The extrapolation method (Section 4.1) asks: can we go beyond mere recovery and surpass the RLVR model by amplifying its learned directional updates?

The extrapolated policy: the core construction is a token-level re-weighting of the RLVR model's output distribution. For each token position, the extrapolated log-probability is defined as:

$\log \pi^\gamma_{\text{Extra}}(y_t | x, y_{<t}) = \log \pi_{\text{RL}}(y_t | x, y_{<t}) + \gamma \cdot \Delta\log p(y_t | x, y_{<t}) + z(x, y_{<t})$

where $\gamma \geq 0$ is the extrapolation strength hyperparameter, $\Delta\log p(y_t | x, y_{<t})$ is the signed log-probability difference (Equation 1), and $z(x, y_{<t})$ is the log-partition function (normalization constant) ensuring that the resulting probabilities sum to 1 across the vocabulary.

What this computes in probability space: exponentiating both sides and incorporating the partition function into a proportionality, the extrapolated distribution becomes:

$\pi^\gamma_{\text{Extra}}(y_t | x, y_{<t}) \propto \pi_{\text{RL}}(y_t | x, y_{<t}) \cdot \exp\left(\gamma \cdot \Delta\log p(y_t | x, y_{<t})\right)$

This is a token-level re-weighting: each token's probability under $\pi_{\text{RL}}$ is multiplied by a factor $\exp(\gamma \cdot \Delta\log p)$. For tokens with positive $\Delta\log p$ (those RLVR already favors), the factor is $\exp(\gamma \cdot \text{positive}) > 1$, increasing their probability further. For tokens with negative $\Delta\log p$ (those RLVR disfavors), the factor is $\exp(\gamma \cdot \text{negative}) < 1$, decreasing their probability further. The extrapolation thus pushes the distribution further along the learned base-to-RLVR direction, amplifying the very changes that RLVR discovered during training.

Alternative algebraic form: substituting $\Delta\log p = \log \pi_{\text{RL}} - \log \pi_{\text{Base}}$ into the definition yields:

$\log \pi^\gamma_{\text{Extra}}(y_t | x, y_{<t}) = (1 + \gamma) \cdot \log \pi_{\text{RL}}(y_t | x, y_{<t}) - \gamma \cdot \log \pi_{\text{Base}}(y_t | x, y_{<t}) + z(x, y_{<t})$

This form reveals the extrapolation as a weighted combination of the two models' log-probabilities. When $\gamma = 0$, the extrapolated policy is identical to $\pi_{\text{RL}}$. As $\gamma$ increases, the base model's log-probability is subtracted with increasing weight, pushing the distribution away from $\pi_{\text{Base}}$ and toward regions of token space that RLVR learned to prefer but may not have fully committed to.

Why selective application: the paper does not apply extrapolation globally to all tokens. Instead, it uses the same gating mechanism from the token replacement experiment: only positions where $\Delta\log p(y_t | x, y_{<t}) < \tau$ receive extrapolation (the base model generated a token that RLVR learned to suppress). At these positions, instead of replacing the token with one from $\pi_{\text{RL}}$ (as in token replacement), the token is sampled from $\pi^\gamma_{\text{Extra}}$. At all other positions, the base model's token is kept without modification. This selective approach is motivated by the sparsity finding: most positions have near-zero $\Delta\log p$, and applying extrapolation globally would distort well-calibrated tokens where RLVR made no meaningful change.

Why this form rather than alternatives: the extrapolation $\exp(\gamma \cdot \Delta\log p)$ is a natural generalization of the token replacement operation. Token replacement is equivalent to sampling from a hard-switched distribution: base tokens at most positions, RLVR tokens at selected positions. Extrapolation is a soft interpolation: at selected positions, the distribution is a continuous push beyond $\pi_{\text{RL}}$ rather than a discrete switch to $\pi_{\text{RL}}$. The parameter $\gamma$ controls how far beyond $\pi_{\text{RL}}$ the distribution is pushed. This connects to the reward-guided decoding literature (Khanov et al., 2024; Liu et al., 2024; Xu et al., 2025), where external reward functions re-weight the output distribution. Here, $\Delta\log p$ serves as an implicit token-level reward learned during RLVR training—no separate reward model or verifier is needed at test time. The method requires access to both $\pi_{\text{Base}}$ and $\pi_{\text{RL}}$ (to compute $\Delta\log p$), which the paper acknowledges as a practical limitation.

Hyperparameters in practice (Table 4): the reported results use $\gamma = 0.1$ for ORZ-32B and UniReason-14B, and $\gamma = 0.05$ for DAPO-32B. The selection threshold $\tau$ is set to -0.4, -0.35, and -0.3 respectively, corresponding to replacement rates of 10.1%, 7.5%, and 11.4%. These were chosen to approximately match the token replacement endpoint where $\Delta\log p$ first reaches RLVR-level accuracy in Figure 2.

Sensitivity analysis (Appendix A.3, Table 5): the paper systematically sweeps both $\gamma \in \{0.05, 0.1\}$ and $\tau \in \{-0.5, -0.4, -0.3, -0.2, -0.1\}$ (and denser for UniReason). Across nearly all configurations, extrapolation outperforms the replace-only baseline. The gain is robust: for DAPO, the best extrapolation configuration (55.42 Avg@32 at $\gamma=0.05, \tau=-0.3$) beats the best replace-only configuration (52.71 at $\tau=-0.2$) by 2.71 points, and the RLVR baseline (52.50) by 2.92 points. For ORZ, the best extrapolation (47.50 at $\gamma=0.1, \tau=-0.4$) beats the best replace-only (46.15 at $\tau=-0.3$) by 1.35 points, and the RLVR baseline (46.15) by 1.35 points. For UniReason, the best extrapolation (55.83 at $\gamma=0.1, \tau=-0.35$) beats the best replace-only (54.06 at $\tau=-0.35$) by 1.77 points, and the RLVR baseline (54.58) by 1.25 points.

Direct extrapolation on $\pi_{\text{RL}}$ (Table 1): as an alternative, the paper applies selective extrapolation directly on the RLVR model rather than on the base model (i.e., sample from $\pi_{\text{RL}}$ at all positions, then apply $\pi^\gamma_{\text{Extra}}$ at selected positions). For DAPO-32B with $\gamma=0.1$, performance improves from the RLVR baseline of 52.50 to 53.96 at 1.8% replacement (threshold $\tau = -0.5$), to 55.31 at 5.2% replacement ($\tau = -0.2$), and then plateaus at 55.10 at 20.0% replacement ($\tau = 0.0$). This pattern reinforces the sparsity finding: amplifying a limited set of the most directionally changed tokens is effective, but aggressive intervention across many positions yields diminishing returns because most positions lack meaningful $\Delta\log p$ signal.

Theoretical justification (Theorem 4.1): the paper provides a theoretical analysis in a simplified tabular softmax bandit setting to justify why extrapolating along the learned direction should improve expected reward. The setting assumes a policy $\pi_\theta(y|x) \propto \exp(\theta_{x,y})$ with individually parameterized logits for each prompt-response pair $(x, y)$, trained via Natural Policy Gradient (NPG) with backtracking. NPG is used because its update rule resembles the controlled optimization of PPO-based algorithms like DAPO.

In this setting, the NPG update rule simplifies to $\theta^{t+1}_{x,y} - \theta^t_{x,y} = \eta \cdot A^t(x, y)$, where $\eta$ is the step size and $A^t(x, y)$ is the advantage function at training step $t$. The extrapolated policy is defined as $\pi_{\omega(\theta^t, \gamma)}$ where $\omega(\theta^t, \gamma) = \theta^t + \gamma(\theta^t - \theta^0)$—pushing the logits further along the accumulated update direction from initialization to the current policy.

Theorem 4.1. For a given prompt $x$, if a tabular softmax policy $\pi_{\theta^t}$ is updated via natural policy gradient, then the extrapolated policy $\pi_{\omega(\theta^t, \gamma)}$ satisfies:

$\exists \gamma > 0, \quad \mathbb{E}_{y \sim \pi_{\omega(\theta^t, \gamma)}(\cdot|x)}[R_{x,y}] \geq \mathbb{E}_{y \sim \pi_{\theta^t}(\cdot|x)}[R_{x,y}]$

Equality holds if and only if the reward $R_{x,y}$ is constant for all $y$.

What the theorem states: there exists some positive extrapolation strength $\gamma$ such that the expected reward under the extrapolated policy is at least as high as under the current policy. In other words, moving the logits further along the accumulated NPG update direction cannot hurt (in expectation) and strictly improves unless all responses have identical reward (in which case there is nothing to learn).

Why this holds (proof sketch in Appendix D): the proof proceeds by computing the directional derivative of the expected reward $J(\theta_x) = \mathbb{E}_{y \sim \pi_{\theta_x}(\cdot)}[R_{x,y}]$ at $\theta^t$ along the extrapolation direction $d^t_x = \theta^t_x - \theta^0_x$. For a softmax policy, $\partial J / \partial \theta_{x,y} = \pi_{\theta_x}(y)(R_{x,y} - \pi_{\theta_x}^\top R_x)$, which is the policy probability times the advantage. The directional derivative becomes $\frac{1}{\|d^t_x\|} \sum_y \pi_{\theta^t_x}(y) \cdot A^t(x, y) \cdot d^t_{x,y}$.

The critical observation is that, under NPG, the order of $d^t_{x,y}$ across responses $y$ matches the order of rewards $R_{x,y}$ (proved by induction on the NPG update rule). Similarly, the order of advantages $A^t(x, y)$ matches the order of rewards. Since $d^t_{x,y}$ and $A^t(x, y)$ have the same ordering (both are larger for higher-reward responses), the Chebyshev sum inequality applies: the covariance between advantage and logit-difference is non-negative. Coupled with the fact that the expected advantage is zero ($\sum_y \pi_{\theta^t_x}(y) A^t(x, y) = 0$), this yields a non-negative directional derivative, meaning movement along $d^t_x$ does not decrease $J$. For non-constant rewards, the inequality is strict.

Why the theorem motivates selectivity: the theorem relies on the assumption that the logit updates $\theta^t - \theta^0$ are consistently increasing with reward, which holds in the idealized NPG setting with monotonic learning. However, the empirical analysis has shown that RLVR's updates concentrate on only a minority of tokens—most positions have near-zero $\Delta\log p$. At these positions, the $\Delta\log p$ direction does not satisfy the theorem's assumption of consistent reward-aligned updates (there was essentially no update at all). Extrapolating where $\Delta\log p \approx 0$ would amount to amplifying noise. The selective gating $f^\tau_{\text{logp}}$ restricts extrapolation to positions with significant directional signal, aligning the practical method with the theoretical conditions for improvement.

Extrapolation on the Minerva benchmark (Appendix C, Figure 10): to test generalization beyond pure math, the method is evaluated on Minerva (272 undergraduate STEM problems spanning chemistry, astronomy, and other subjects). Using Avg@8 (fewer samples due to the larger dataset), selective extrapolation consistently outperforms both the RLVR model and selective replacement across all three model pairs. For DAPO-32B, the RLVR baseline achieves 56.63 Avg@8, selective replacement achieves 56.63, and selective extrapolation (with $\gamma=0.1, \tau=-1.0$) achieves 58.27. For ORZ-32B, the gains are smaller but consistent: RLVR 56.41, replacement 56.41, extrapolation 57.17 at $\gamma=0.05$. For UniReason-14B, RLVR 54.23, replacement 54.14, extrapolation 56.16 at $\gamma=0.1$. These results confirm that the $\Delta\log p$ direction generalizes across reasoning domains, not just the mathematical problems on which RLVR was trained.

Training-Time Advantage Reweighting

While test-time extrapolation amplifies the learned $\Delta\log p$ direction after training, the training-time reweighting method (Section 4.2) proactively strengthens the learning signal on tokens that will ultimately become the high-$\Delta\log p$ positions. The method is grounded in the correlation established in Figure 3b: high-$\Delta\log p$ tokens in the final model correspond to low-probability tokens during training.

The reweighted advantage: the method modifies only the advantage term in the DAPO objective (Equation 4). Instead of using the standard group-relative advantage $\hat{A}_{i,t}$ (Equation 2), each token's advantage is scaled by a probability-dependent factor:

$\tilde{A}_{i,t} = \left(1 + \alpha \cdot \left(1 - \pi_{\theta_{\text{old}}}(y_{i,t} | x, y_{i,<t})\right)\right) \cdot \hat{A}_{i,t}$

where $\alpha \geq 0$ is a hyperparameter controlling the reweighting strength, $\pi_{\theta_{\text{old}}}(y_{i,t} | x, y_{i,<t})$ is the probability the old policy (the policy used to generate the rollout data) assigned to the token that was actually sampled, and $\hat{A}_{i,t}$ is the standard group-relative advantage from Equation 2.

What this computes: for each token in each generated response, the factor $(1 - \pi_{\theta_{\text{old}}}(y_{i,t}))$ measures how "surprising" the token was to the old policy. If the token was highly probable ($\pi_{\theta_{\text{old}}} \approx 1$), the factor is near zero and the advantage is essentially unchanged ($\tilde{A}_{i,t} \approx \hat{A}_{i,t}$). If the token was improbable ($\pi_{\theta_{\text{old}}} \approx 0$), the factor is near 1 and the advantage is amplified ($\tilde{A}_{i,t} \approx (1 + \alpha) \cdot \hat{A}_{i,t}$). The parameter $\alpha$ controls the maximum amplification: with $\alpha = 0.2$ (the value used for Qwen2.5-Math-7B), a maximally surprising token has its advantage multiplied by 1.2.

Why this form: the factor $1 - \pi_{\theta_{\text{old}}}(y_{i,t})$ appears in Lemma 3.1 as the term governing the gradient norm. The natural gradient magnitude is already proportional to $1 - \pi_{\theta_{\text{old}}}$ through the softmax derivative. This reweighting further amplifies the advantage component of the gradient weight, making the combined gradient weight proportional to $(1 + \alpha(1 - \pi_{\theta_{\text{old}}})) \cdot |\hat{A}_{i,t}| \cdot (1 - \pi_{\theta_{\text{old}}}) = |\hat{A}_{i,t}| \cdot ((1 - \pi_{\theta_{\text{old}}}) + \alpha(1 - \pi_{\theta_{\text{old}}})^2)$. The linear plus quadratic dependence on $(1 - \pi_{\theta_{\text{old}}})$ means that surprising tokens receive a super-linear boost in gradient magnitude relative to common tokens.

Why not a different functional form: the paper implicitly contrasts this with two alternatives tested in the experiments. The "Dominate" method (Yang et al., 2025b) uses $\tilde{A}^{\text{dom}}_{i,t} = [\alpha \cdot \pi_\theta(y_{i,t}) + 1 - \alpha] \cdot \hat{A}_{i,t}$, which upweights high-probability tokens and downweights low-probability tokens (the opposite of the paper's approach). The "PPL" method (Deng et al., 2025) uses $\tilde{A}^{\text{ppl}}_{i,t} = [1 - \alpha \cdot w_{\text{ppl}}(y_i)] \cdot \hat{A}_{i,t}$ where $w_{\text{ppl}}(y_i)$ is a normalized log-perplexity weight that privileges low-perplexity (high-probability) responses. The paper's method is motivated by the $\Delta\log p$ analysis showing that low-probability tokens are the ones that become the reasoning-critical updates, making direct amplification of their learning signal the natural choice.

Implementation in DAPO: the modification is minimal—only the advantage term is changed. All other DAPO components remain fixed: the double clip ratios ($\epsilon_{\text{low}} = 0.2$, $\epsilon_{\text{high}} = 0.28$), the learning rate ($1 \times 10^{-6}$ with 10-step warmup), the batch configuration (512 prompts per step, 16 responses per prompt, processed in mini-batches of 32 prompts yielding 16 gradient updates per step), and the maximum generation lengths (8k for Qwen2.5-Math-7B with 4k overlong penalty, 20k for Qwen3-8B-Base with 16k overlong penalty). The only new hyperparameter is $\alpha$, set to 0.2 for Qwen2.5-Math-7B and 0.1 for Qwen3-8B-Base.

Connection to the top-p experiment: the reweighting method can be understood as the inverse of the top-p filtering experiment. Top-p filtering reduces the learning signal on low-probability tokens by preventing them from being sampled, which Figure 3c shows degrades performance. The reweighting method amplifies the learning signal on the same low-probability tokens when they are sampled (they are still sampled at their naturally low rates; the amplification happens through the advantage multiplier, not through increased sampling frequency). This preserves the exploration benefits of occasionally sampling low-probability tokens (which top-p filtering destroys) while strengthening the gradient update when they happen to appear.

Experimental setup for evaluation: the method is evaluated on three mathematical reasoning benchmarks: AIME-24, AIME-25, and AMC. Following DAPO's protocol, evaluation uses top-p=0.7 for sampling and reports Avg@32 (accuracy averaged over 32 samples per problem) and Pass@16 (the probability that at least 1 of 16 samples is correct, estimated from 32 samples using the standard unbiased estimator from Chen et al., 2021). The choice of Pass@16 rather than Pass@32 when 32 samples are available is a stability consideration: the estimator for Pass@k has lower variance when $k$ is less than the number of samples.

Comparison with competing reweighting methods (Table 3): on Qwen2.5-Math-7B, the paper's method achieves the highest average performance (43.75 Avg@32, 62.33 Pass@16 across all three benchmarks). The PPL method achieves 41.38 Avg@32 and 61.08 Pass@16—strong but lower, particularly on AIME25 (16.46 vs. 18.54 Avg@32). The Dominate method achieves 43.11 Avg@32 but only 53.63 Pass@16. The low Pass@16 for Dominate is attributed to reduced exploration: the method's $\epsilon_{\text{high}} = 0.24$ (lower than the default 0.28, following the recommended value in Yang et al., 2025b) restricts the policy from deviating far from the old policy, which stabilizes training but limits the diversity of generated solutions. The training curves in Figure 5 confirm this: Dominate exhibits clearly lower entropy during training, indicating less exploration. The paper notes that using the default $\epsilon_{\text{high}} = 0.28$ with the Dominate method resulted in unstable training, suggesting that the anti-domination approach requires restrictive clipping that inherently limits exploration.

Training dynamics (Figure 5): the paper's reweighting method shows steadily increasing AIME-24 Avg@32 throughout training (reaching and sustaining approximately 0.38-0.39), accompanied by increasing response length (from approximately 800 to 1,300 tokens) and stable, non-decreasing entropy. This simultaneous increase in performance and length is characterized as a "key pattern in effective reasoning RLVR training" (citing Guo et al., 2025), suggesting that the model is learning to produce longer, more thorough reasoning chains rather than shortcuts. In contrast, DAPO baseline shows lower accuracy (plateauing around 0.35) with similar length growth. The PPL method shows accuracy between the paper's method and DAPO baseline. The Dominate method shows competitive accuracy but with noticeably lower entropy.

Generalization across model families (Table 2): the method is evaluated on two distinct base models: Qwen2.5-Math-7B (a math-specialized model) and Qwen3-8B-Base (a general-purpose base model). On Qwen2.5-Math-7B, the method improves average performance from DAPO's 42.12 Avg@32, 57.86 Pass@16 to 43.75 Avg@32, 62.33 Pass@16. On Qwen3-8B-Base, improvements are from 44.26 Avg@32, 69.19 Pass@16 to 46.78 Avg@32, 72.52 Pass@16. The consistent gains across both model families suggest that the mechanism (low-probability tokens are reasoning-critical) is not specific to math-pretrained models but is a general property of RLVR training dynamics.

Reproducibility (Appendix B, Figure 9): to account for random variation in RL training, four independent runs of the reweighting method are conducted on Qwen2.5-Math-7B. All four runs converge to similar final performance (AIME-24 Avg@32 reaching the reported value of 39.06 by the end of training), with consistent entropy and length trajectories. This confirms that the gains are not due to a lucky random seed.

Generalization to STEM (Appendix C, Table 6): on the Minerva benchmark (272 undergraduate STEM problems across chemistry, astronomy, physics), the reweighting method continues to outperform DAPO. For Qwen2.5-Math-7B, the method achieves 49.72 Avg@32 and 70.37 Pass@16 versus DAPO's 46.43 Avg@32 and 69.44 Pass@16. For Qwen3-8B-Base, the method achieves 56.57 Avg@32 versus DAPO's 55.04 Avg@32, with comparable Pass@16 (76.78 vs. 76.98). These results are notable because Minerva includes subjects not present in the RLVR training data (which was math-focused), suggesting that the reasoning improvements from better learning on low-probability tokens transfer across STEM domains.

Summary of Design Choices and Their Justifications

• $\Delta\log p$ as the primary metric over entropy or KL divergence: the signed difference captures directional information (which tokens RLVR favors vs. suppresses) that unsigned magnitude metrics discard. The statistical separation in histograms (bimodal with model-dependent tails) and the token replacement efficiency advantage (reaching RLVR accuracy with ~10% vs. 15-30% replacements) provide converging evidence for this choice.

• Selective rather than global application of both test-time methods: the sparsity analysis shows that most tokens have near-zero $\Delta\log p$. Applying extrapolation or replacement globally would distort well-calibrated tokens where RLVR made no meaningful change. The gating function $f^\tau_{\text{logp}}$ restricts intervention to positions with substantial directional signal.

• Negative $\Delta\log p$ as the selection criterion for replacement/extrapolation: tokens the base model generates that RLVR learned to suppress (large negative $\Delta\log p$) are precisely the positions where RLVR's improved reasoning should manifest. Replacing or amplifying at these positions aligns the generation with the learned improvements.

• $1 - \pi_{\theta_{\text{old}}}(y_{i,t})$ as the reweighting factor: this term appears naturally in the gradient norm decomposition (Lemma 3.1) as the factor governing update magnitude per token. Amplifying by this same factor during training reinforces the natural gradient concentration rather than working against it, and the empirical correlation with high final $\Delta\log p$ (Figure 3b) confirms it targets the tokens most responsible for RLVR's reasoning improvements.

• Additive rather than multiplicative reweighting (the $1 + \alpha(1 - \pi)$ form): the additive form ensures that tokens with zero advantage (on-policy correct responses, or responses with average reward) receive no gradient regardless of their probability. A purely multiplicative $(1 - \pi_{\theta_{\text{old}}})$ reweighting of the advantage could assign non-zero weight to zero-advantage tokens, introducing noise. The $1 + \alpha(1 - \pi_{\theta_{\text{old}}})$ factor preserves the zero when $\hat{A}_{i,t} = 0$ while amplifying when $\hat{A}_{i,t} \neq 0$.

• DAPO as the RLVR baseline rather than GRPO or PPO: DAPO is state-of-the-art among open-source RLVR algorithms (as of the paper's writing) and incorporates several techniques (clip-higher, dynamic sampling, token-level loss, overlong punishment, no KL penalty) that represent current best practices. Improvements over DAPO therefore carry more weight than improvements over simpler baselines.

• AIME-24/25 and AMC as evaluation benchmarks: these are standard, challenging mathematical reasoning datasets used across the RLVR literature. AIME (American Invitational Mathematics Examination) problems are particularly difficult (high-school competition level, requiring multi-step reasoning), making them a stringent test of reasoning capability. AMC provides additional problems at a similar difficulty tier.

• Avg@32 and Pass@16 as evaluation metrics: Avg@32 provides a stable estimate of the model's expected accuracy (averaging over 32 independent samples reduces variance). Pass@16 estimates the probability of solving a problem given 16 attempts, measuring exploration capability—whether the model can ever produce a correct solution, even if it's not the most likely one. This dual-metric evaluation captures both exploitation (Avg@32) and exploration (Pass@16).

4. Key Insights and Innovations

Innovation 1: Direction, Not Magnitude, Is the Missing Analytical Dimension for Understanding RLVR

The paper's most fundamental conceptual contribution is the reframing of RLVR analysis from "how much did the model change?" to "in which direction did it change, and on which tokens?" Prior work had already established that RLVR-induced updates are sparse (Wang et al., 2025b; Huan et al., 2025; Yang et al., 2025b; Deng et al., 2025), but the analytical toolkit used to reach that conclusion—entropy, KL divergence, gradient norm—collapsed all changes into unsigned scalars. The field had implicitly accepted that sparsity is the key property worth studying, and that magnitude-based metrics are sufficient to characterize it.

This paper demonstrates that this acceptance was premature and conceptually limiting. The critical observation is not that magnitude metrics are wrong about sparsity (they correctly identify that most tokens change little), but that they are insufficient to distinguish RLVR's meaningful updates from benign distributional variation. The histograms in Figure 1b make this point devastatingly: entropy and KL divergence distributions are nearly identical whether measured on tokens generated by the base model or the RLVR model. You cannot tell which model produced a token by looking at its entropy or divergence. In contrast, Δlog p exhibits a clear bimodal separation with model-dependent tails—the RLVR model's tokens cluster in the positive tail, the base model's in the negative tail.

This is a fundamental reframing, not an incremental refinement. It shifts the analytical question from "identify sparse changes" (which multiple prior metrics already do) to "identify directional changes aligned with the verifier's training signal" (which only a signed metric can do). The distinction matters practically: the token replacement experiment (Figure 2) shows that Δlog p identifies the reasoning-critical minority with roughly twice the precision of KL divergence (reaching RLVR accuracy at ~10% replacement vs. 15–20% for KL and 25–30% for entropy). This efficiency gap is not a minor hyperparameter effect—it reflects the fundamental difference between knowing that a distribution changed (KL divergence) and knowing which way it shifted relative to the training objective (Δlog p).

The significance of this reframing extends beyond the paper's immediate findings. It establishes direction as a first-class analytical dimension for future RLVR research, much as sparsity became a first-class dimension after Wang et al. (2025b). Future work can no longer claim to characterize RLVR's effects by measuring only divergence or entropy; the signed direction must be accounted for. This is analogous to how the RLHF literature recognized that reward over-optimization requires measuring calibration, not just reward magnitude—it changes what counts as a valid analysis.

Innovation 2: RLVR's Update Sparsity Has a Mechanistic Origin in the Softmax Gradient, Not Just a Training Artifact

Prior work had observed RLVR's update sparsity but treated it as an emergent training property—something that happens because of how advantages distribute across responses, or because of clipping, or because of KL regularization. Yang et al. (2025b) came closest to a mechanistic explanation by analyzing gradient norms per token, but focused on the consequences of sparsity (domination by low-probability tokens) rather than its architectural origin.

This paper makes a stronger claim: sparsity is baked into the softmax parameterization itself, independent of training hyperparameters. Lemma 3.1 shows that the ℓ₁-norm of the policy gradient with respect to the output logits is exactly 2|w_{i,t}| · (1 - π_θ(y_{i,t})). The factor 1 - π_θ(y_{i,t}) is not a training heuristic—it is the mathematical derivative of the softmax log-likelihood. This means that any policy gradient method using a softmax output layer will necessarily concentrate updates on low-probability tokens, regardless of the specific RL algorithm (PPO, GRPO, DAPO), the clipping parameters, or the KL penalty weight. The mechanism is architectural, not algorithmic.

This is a conceptual advance, not a new algorithm. It explains why sparsity has been observed so consistently across different RLVR implementations and model families—it's not a quirk of any particular training recipe but a direct consequence of how language models parameterize their output distributions. The empirical demonstration (Figure 3a) that 59.48% of tokens during DAPO training have probability ≥ 0.98 (receiving negligible gradient) while only 3.17% have probability < 0.02 (receiving maximal gradient) grounds this theoretical insight in concrete numbers.

The theoretical contribution is deepened by connecting this training-time mechanism to the final model's Δlog p pattern (Figure 3b). Tokens with the highest Δlog p in the trained model are precisely those that had low probability during training—the gradient concentrated learning on them, and the model's probability on them increased as a result. This closes the loop from training dynamic to final model behavior in a way that prior work had not.

The top-p filtering experiment (Figure 3c) provides the causal evidence that elevates this from correlation to mechanism. Filtering out low-probability tokens during training (top-p = 0.7, 0.9, 0.95) systematically degrades final reasoning performance, demonstrating that these tokens are not merely correlated with gradient concentration but are necessary for RLVR to learn reasoning. The monotonic degradation (lower top-p → worse performance) shows dose-response causality.

This insight has direct implications for RLVR algorithm design. Attempts to "fix" the domination of low-probability tokens by dampening their gradient signal (as in Yang et al., 2025b's anti-domination approach) are working against the fundamental learning mechanism, which explains why those methods reduce exploration (lower Pass@k in Table 3, lower training entropy in Figure 5) without improving final accuracy. The paper's reweighting method (Equation 8) instead amplifies the natural gradient concentration on low-probability tokens, working with the softmax gradient rather than against it, and achieves the best overall performance in Table 2.

Innovation 3: The Learned Δlog p Direction Can Be Exploited as an Implicit Reward for Test-Time Optimization

The paper's third conceptual move is to treat the difference between two models' log-probabilities not merely as a diagnostic but as a learned optimization direction that can be exploited at test time without additional training. This is conceptually distinct from both the diagnostic contribution (Innovation 1) and the training-time mechanism explanation (Innovation 2)—it takes a quantity that prior work would have viewed as an analysis artifact and repurposes it as an actionable control signal.

The key insight is that Δlog p = log π_RL - log π_Base encodes the verifier's training signal in a token-level form. During RLVR training, the verifier rewards responses that reach correct answers. The policy gradient propagates this reward back to individual tokens, increasing the probability of tokens that appeared in successful trajectories and decreasing the probability of tokens from unsuccessful ones. The final Δlog p is the accumulated effect of these per-token reward signals. By treating Δlog p as a token-level reward, the extrapolated policy π^γ_Extra ∝ π_RL · exp(γ · Δlog p) becomes a form of reward-guided decoding (Khanov et al., 2024; Liu et al., 2024; Xu et al., 2025) but with an implicit reward learned during training rather than an external reward model.

This is a practical innovation with conceptual depth. It reframes the base-to-RLVR shift as a "reasoning vector" in probability space that can be extrapolated, analogous to how weight-space extrapolation has been used in model editing and controlled generation. The theoretical justification (Theorem 4.1) connects this to Natural Policy Gradient, showing that in the idealized bandit setting, extrapolating along the accumulated NPG update direction cannot decrease expected reward. The selectivity mechanism (gating by negative Δlog p) bridges the gap between the idealized theory (which assumes uniform, reward-aligned updates) and the empirical reality (where most tokens have near-zero Δlog p).

The significance of this innovation is that it demonstrates a new way to extract value from already-trained RLVR models without additional training. Selective extrapolation raises AIME-24 Avg@32 by 1–3 points over the RLVR model across all tested model pairs (Figure 4), with the gains holding on out-of-domain STEM problems (Minerva, Figure 10). These are modest but consistent gains that come essentially for free (the only cost is computing Δlog p, which requires running both models, but no additional training or data is needed). The method establishes a new paradigm: RLVR training produces not just a better model but a directional signal that can be further exploited at inference time.

Innovation 4: A Unified Framework for Analyzing, Explaining, and Exploiting RLVR Through a Single Diagnostic Quantity

The paper's fourth contribution is architectural: it demonstrates that a single quantity—Δlog p—can serve as a diagnostic tool (identifying reasoning-critical tokens, Section 3.1–3.2), an explanatory mechanism (connecting training dynamics to final model behavior, Section 3.3), and a practical intervention vector (test-time extrapolation, Section 4.1; training-time reweighting, Section 4.2). This unification is more than a convenience; it makes a methodological argument that productive analysis and practical improvement should be grounded in the same conceptual framework.

The field before this paper had separate threads for RLVR analysis and RLVR improvement. Analysis work (Wang et al., 2025b; Huan et al., 2025; Yang et al., 2025b) characterized what RLVR does but did not translate insights into better training methods. Improvement work (DAPO, GRPO variants, reward shaping) developed better algorithms but was not grounded in a precise understanding of which tokens matter and why. The two threads operated largely independently, with analysis serving a retrospective explanatory role rather than a prospective design role.

This paper demonstrates that analysis can be directly causal to improvement. The chain of reasoning is explicit: Δlog p identifies the tokens RLVR modifies (analysis) → those tokens correspond to low-probability positions during training (mechanism) → therefore, amplifying the learning signal on low-probability tokens should improve training (design) → and it does (Table 2). Similarly: Δlog p identifies the direction of RLVR's distributional shift (analysis) → extrapolating along that direction should improve expected reward in a bandit setting (theory) → therefore, pushing the distribution further at selected positions should help at test time (design) → and it does (Figure 4).

This is a methodological contribution to how RLVR research should be conducted. It argues implicitly that good analysis should not just describe what happened but should suggest concrete interventions that can be validated. The paper's structure—three analysis sections feeding into two method sections, all connected through Δlog p—embodies this philosophy. The fact that both derived methods work (test-time extrapolation beating RLVR, training-time reweighting beating DAPO) validates the premise that Δlog p-based analysis is not merely descriptive but predictive of what interventions will help.

The significance of this unification extends beyond the paper's specific methods. Future RLVR work can adopt the same pattern: identify a diagnostic quantity that captures something fundamental about the training process, validate it causally (e.g., through token replacement or filtering experiments), then design interventions that exploit it at either training or inference time. The paper provides a template for closing the analysis-to-improvement loop that had been open in prior work.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The primary benchmark is AIME-24 (American Invitational Mathematics Examination 2024), consisting of 30 competition-level mathematical reasoning problems. The paper also evaluates on AIME-25 (similar difficulty), AMC (American Mathematics Competition, broader in scope), and Minerva (272 undergraduate STEM problems spanning chemistry, astronomy, and physics; Lewkowycz et al., 2022). For RLVR training, the DAPO recipe uses a standard math reasoning training set (details in Appendix B), not the evaluation benchmarks themselves.

• Base model(s). Three RLVR-base model pairs are analyzed: ORZ-32B (Hu et al., 2025a) trained from Qwen2.5-32B via PPO, DAPO-32B (Yu et al., 2025) trained from the same Qwen2.5-32B base via DAPO, and UniReason-14B (Huan et al., 2025) trained from Qwen3-14B-Base via GRPO (Yang et al., 2025a; Team, 2024). For the training-time reweighting experiments, RLVR is run from scratch on Qwen2.5-Math-7B (Yang et al., 2024) and Qwen3-8B-Base (Yang et al., 2025a). These models span two families (Qwen2.5, Qwen3), two scales (7-8B, 14B, 32B), and three RLVR algorithms (PPO, DAPO, GRPO), testing the generality of findings.

• Metrics. The primary evaluation metric is Avg@32: the fraction of 32 independent sampled solutions that are correct, averaged over all problems in the benchmark. This provides a stable estimate of expected per-problem accuracy. Pass@k (specifically Pass@16 when 32 samples are available) estimates the probability that at least one of k samples is correct, computed using the unbiased estimator from Chen et al. (2021), measuring exploration/coverage capability. For the token replacement experiments, the key metric is the replacement rate (fraction of generated tokens replaced) versus Avg@32 accuracy.

• Baselines. The primary RLVR baselines are the published ORZ-32B, DAPO-32B, and UniReason-14B models evaluated with standard sampling (top-p=0.7, temperature=1.0, 32 samples). For the training-time reweighting experiments, the baseline is standard DAPO training (Yu et al., 2025) with recommended hyperparameters (double clip ratios ϵ_low=0.2, ϵ_high=0.28). For reweighting comparisons (Table 3, Figure 5), two competing methods are implemented: PPL (Deng et al., 2025), which upweights low-perplexity responses via Ã^{ppl}_{i,t} = [1 - α · w_ppl(y_i)] · Â_{i,t} with α=0.01; and Dominate (Yang et al., 2025b), which counteracts low-probability token domination via Ã^{dom}_{i,t} = [α · π_θ(y_{i,t}) + 1 - α] · Â_{i,t} with α=0.1 and reduced clip-higher ratio ϵ_high=0.24. The token replacement experiments include a random baseline that replaces tokens uniformly at random with probability τ, providing a lower bound on the benefit of selective replacement.

• Generation budget / compute accounting. For both the analytical experiments (token replacement, statistical analysis) and the test-time extrapolation method, compute is measured in terms of standard generation with top-p=0.7 and temperature=1.0, producing 32 samples per problem. No explicit FLOP counting is performed—all comparisons are at matched generation budgets. For the training-time reweighting experiments, each RLVR step consists of 512 prompts with 16 sampled responses each, processed in mini-batches of 32 prompts yielding 16 gradient updates per step. Maximum generation lengths are 8k tokens for Qwen2.5-Math-7B and 20k for Qwen3-8B-Base, with overlong penalty thresholds at 4k and 16k respectively.

• Cross-validation / statistical protocol. For the training-time reweighting experiments, four independent training runs are conducted on Qwen2.5-Math-7B to account for random variation in RL training (Appendix B, Figure 9). All four runs converge to similar final performance, confirming reproducibility. For the statistical analysis (histograms in Section 3.1, Figures 12-15), metrics are computed over all tokens across 32 samples × 30 problems = 960 responses per model. For the token replacement experiments, replacement rate is controlled by sweeping threshold τ across multiple values (e.g., τ ∈ {-0.5, -0.4, -0.3, -0.2, -0.1}) and measuring empirical replacement rate on the full AIME-24 dataset. No cross-validation is used for selectable hyperparameters in the test-time methods—the extrapolation strength γ and threshold τ are chosen to approximately match the token replacement endpoint where Δlog p first reaches RLVR-level accuracy, and sensitivity is analyzed post-hoc (Appendix A.3, Table 5).

Main Quantitative Results

Statistical Analysis: Directional vs. Magnitude-Based Metrics (Figure 1b, Figures 12-15)

The statistical comparison across three model pairs (ORZ-32B, DAPO-32B, UniReason-14B) reveals a consistent pattern:

"Across all metrics, the histograms are sharply peaked near zero... confirming that RLVR-induced changes are sparse. However, the entropy and KL divergence distributions are nearly identical for both the base and RLVR model outputs."

For the UniReason pair (Figure 12a): 97.3% of base-generated tokens have Δlog p within [-0.25, 0.25], versus 92.2% for RLVR-generated tokens—a 5.1 percentage point difference in concentration near zero that reflects the directional signal. In contrast, the entropy distribution (Figure 13b,d) shows 82.0% of RLVR-generated tokens have entropy < 0.1 when measured on RLVR generations, versus 86.9% when measured on base generations—a 4.9 point difference in the same direction but without the signed bimodality that distinguishes model origin. The KL divergence histograms (Figure 13a,c) overlap almost completely regardless of which model generated the tokens, confirming that divergence is insensitive to the source of the distribution being compared—it quantifies difference magnitude but not direction.

For the DAPO pair (Figure 12b): the Δlog p histograms show 90.5% of base-generated tokens within [-0.5, 0.5] versus 90.9% for RLVR-generated tokens—similar central concentration but with clearly distinct tail behavior. The positive tail beyond +2.0 is populated almost exclusively by RLVR-generated tokens, while the negative tail below -2.0 is dominated by base-generated tokens. This pattern holds across all three model pairs (Figure 12a-c).

The entropy and divergence distributions (Figures 13-15), measured for both base and RLVR models on tokens from both generators, are "nearly indistinguishable" per the authors. This observation is the empirical foundation for the paper's central claim: magnitude metrics cannot capture what RLVR actually changes because they collapse directional shifts into unsigned quantities.

Token Replacement: Recovering RLVR Performance (Figure 2, Figure 6)

The token replacement experiment (Algorithm 1) produces accuracy-versus-replacement-rate curves for each metric across three model pairs. The headline result:

"Selectively replacing a minority of base models' tokens can recover RLVR performance. ... Logp difference > divergence > entropy in identifying RLVR-learned reasoning patterns."

On DAPO-32B (AIME-24): The base model achieves 6.67 Avg@32, the RLVR model achieves 52.50 Avg@32. Δlog p-based replacement reaches RLVR-level accuracy (~52.50) at approximately 10-11% replacement rate. KL divergence (averaged D̄_KL) reaches the same level at approximately 15-20% replacement. Entropy (H_πBase) requires approximately 25-30% replacement. The random baseline never reaches RLVR accuracy within the tested range of up to 30% replacement, plateauing well below (Figure 2, middle panel).

On ORZ-32B: Base achieves 3.02 Avg@32, RLVR achieves 46.15 Avg@32. Δlog p-based replacement reaches RLVR-level accuracy at approximately 10% replacement. KL divergence requires approximately 15% replacement. Entropy requires approximately 25% replacement (Figure 2, left panel).

On UniReason-14B: Base achieves 11.46 Avg@32, RLVR achieves 54.58 Avg@32. Δlog p-based replacement reaches RLVR-level accuracy at approximately 7% replacement. KL divergence requires approximately 12% replacement. Entropy requires approximately 17% replacement (Figure 2, right panel). The lower absolute replacement rates for UniReason reflect that the base model's initial accuracy is higher (11.46 vs. 3.02-6.67), so fewer changes are needed to reach the RLVR ceiling.

The efficiency ordering—Δlog p > KL divergence > entropy—is consistent across all three model pairs. Additional ablation with divergence variants (forward KL, reverse KL, RLVR model entropy) and forward/reverse KL evaluated independently (Figure 6, Appendix A.2) shows the same ordering, with all magnitude metrics requiring more replacements than Δlog p. The forward KL D_KL^{π_RL, π_Base} slightly outperforms the reverse KL, but both trail Δlog p substantially.

"This ordering—logp difference highest, followed by divergence, then entropy—remains stable across different divergence and entropy variants, further validating the superiority of logp difference in isolating the most influential positions."

Qualitative analysis of top replacement tokens at matched 8% replacement rate (Figure 7): entropy-based selection favors logical transition words (Thus, need, can, To, Now—highest fraction approximately 5-6% for "the"), while KL divergence and Δlog p selection favor mathematical reasoning tokens including symbols ("\", "\)", "$", "=") and reasoning-specific vocabulary (approach, denote, find). The Δlog p criterion has the highest proportion of mathematical symbols, consistent with its superior efficiency in recovering reasoning accuracy.

Gradient Analysis: Explaining the Sparsity (Figure 3a-b)

Figure 3a plots token probability and gradient norm coefficient 1 - π_θ(·) from an intermediate DAPO training step. The distribution is extremely skewed: 59.48% of tokens have probability P ≥ 0.98 (receiving negligible gradient), while only 3.17% of tokens have probability P < 0.02 (receiving maximal gradient). When plotting the gradient norm coefficient ratio (fraction of total gradient mass attributed to each probability bin), the low-probability bin dominates despite containing only 3.17% of tokens. This empirically confirms Lemma 3.1's prediction that sparse, high-magnitude gradient updates naturally concentrate on rare tokens.

Figure 3b links training-time token probability to final Δlog p. Tokens are binned by their Δlog p percentile (top 0-4%, 4-8%, etc., up to 16-20%). For each bin, the average probability under the base model and RLVR model is computed. The monotonic relationship is clear: higher Δlog p bins correspond to lower token probabilities in both models. Within the highest Δlog p bin (top 0-4%), the base model probability is approximately 0.5 and the RLVR model probability rises to approximately 0.65—a substantial increase. In the lowest bin (16-20%), probabilities are near 1.0 for both models and nearly identical (both close to 0.95-1.0), confirming that tokens RLVR leaves unchanged are those the base model was already confident about.

Top-p Filtering Causal Experiment (Figure 3c)

Training Qwen2.5-Math-7B with DAPO under varying top-p sampling thresholds during rollout produces a clear dose-response relationship:

• top-p = 1.0 (default, no filtering): AIME-24 Avg@32 reaches approximately 0.35 by step 140.
• top-p = 0.95: Performance drops to approximately 0.30.
• top-p = 0.9: Performance drops further to approximately 0.25.
• top-p = 0.7: Performance collapses to approximately 0.20.

"Even a mild filter (e.g., top-p=0.95) leads to a substantial drop in performance compared to the default setting (top-p=1.0). As the filter becomes more aggressive, performance degrades sharply."

The monotonic degradation causally demonstrates that low-probability tokens are necessary for RLVR to learn reasoning—excluding them systematically impairs final performance.

Test-Time Selective Extrapolation (Figure 4, Table 1)

The extrapolated policy π^γ_Extra is evaluated on AIME-24 with 32 samples per problem, compared against the RLVR model and selective replacement at matched gating thresholds.

"On AIME-24, Selective Extrapolation yields higher Avg@32 than π_RL across ORZ-32B, DAPO-32B, and UniReason-14B under matched gates."

On DAPO-32B (Figure 4, middle bar group): RLVR baseline achieves 52.50 Avg@32. Selective replacement at τ = -0.3 (11.4% replacement) achieves 51.67 Avg@32 (slightly below RLVR at this threshold). Selective extrapolation with γ = 0.05 achieves 55.42 Avg@32—a gain of +2.92 over the RLVR baseline and +3.75 over selective replacement at the same gate.

On ORZ-32B (left bar group): RLVR baseline achieves 46.15 Avg@32. Selective replacement at τ = -0.4 (10.1% replacement) achieves 43.33 Avg@32. Selective extrapolation with γ = 0.1 achieves 47.50 Avg@32—a gain of +1.35 over RLVR.

On UniReason-14B (right bar group): RLVR baseline achieves 54.58 Avg@32. Selective replacement at τ = -0.35 (7.5% replacement) achieves 54.06 Avg@32. Selective extrapolation with γ = 0.1 achieves 55.83 Avg@32—a gain of +1.25 over RLVR.

For all three models, selective replacement underperforms or matches RLVR at the chosen thresholds, while extrapolation consistently surpasses RLVR by 1.25-2.92 points.

Table 1 applies selective extrapolation directly on π_RL (rather than on π_Base with replacement) for DAPO-32B with γ = 0.1:

• 0.0% replacement (no intervention): 52.50 Avg@32
• 1.8% replacement (τ = -0.5): 53.96 Avg@32
• 5.2% replacement (τ = -0.2): 55.31 Avg@32
• 20.0% replacement (τ = 0.0): 55.10 Avg@32

"As the threshold τ in f^τ_logp increases, the AIME-24 performance improves up to a moderate intervention ratio, after which gains plateau."

The plateau at 20% replacement reinforces the sparsity finding: beyond ~5% of most directionally changed tokens, additional interventions provide diminishing returns.

Sensitivity analysis (Appendix A.3, Table 5): sweeping γ ∈ {0.05, 0.1} and τ across multiple values for each model pair shows that extrapolation consistently outperforms replacement across nearly all configurations. For DAPO, the best configuration (γ=0.05, τ=-0.3) achieves 55.42 ± ~2 points across τ values. For ORZ, the best (γ=0.1, τ=-0.4) achieves 47.50. For UniReason, the best (γ=0.1, τ=-0.35) achieves 55.83. The gains are robust to hyperparameter choice.

Generalization to Minerva (Appendix C, Figure 10, Table 7): evaluating with Avg@8 on 272 STEM problems:

• DAPO-32B: RLVR 56.63, replacement 56.63, extrapolation (γ=0.1, τ=-1.0) 58.27
• ORZ-32B: RLVR 56.41, replacement 56.41, extrapolation (γ=0.05, τ=-1.0) 57.17
• UniReason-14B: RLVR 54.23, replacement 54.14, extrapolation (γ=0.1, τ=-0.9) 56.16

Extrapolation yields consistent gains of 1.4-1.9 points over RLVR on out-of-domain STEM problems.

Training-Time Advantage Reweighting (Table 2, Table 3, Figure 5)

The reweighting method (Equation 8) is evaluated against DAPO baseline on two model families across three math benchmarks.

Qwen2.5-Math-7B (Table 2, top): The base model achieves 14.79 (AIME24), 6.67 (AIME25), 40.62 (AMC) Avg@32. DAPO training reaches 35.73, 17.6, 73.04 Avg@32 respectively, averaging 42.12 across benchmarks. The reweighting method (α=0.2) reaches 39.06, 18.54, 73.64 Avg@32, averaging 43.75—a gain of +1.63 average points. Pass@16 improves from DAPO's average of 57.86 to 62.33 (+4.47 points).

Qwen3-8B-Base (Table 2, bottom): Base model achieves 5.42, 5.73, 27.64 Avg@32. DAPO reaches 36.98, 26.67, 69.13 Avg@32, averaging 44.26. Reweighting (α=0.1) reaches 38.13, 31.15, 71.05 Avg@32, averaging 46.78—a gain of +2.52 average points. Pass@16 improves from 69.19 to 72.52 (+3.33 points).

"enhancing low-probability tokens' weight consistently improves reasoning accuracy across all tested models and datasets. Notably, this enhanced accuracy (Avg@32) doesn't come at the cost of exploration ability (often measured by Pass@k); in fact, the average Pass@16 also increases over the DAPO baseline."

The per-benchmark gains are not uniform. On AIME25 with Qwen3-8B-Base, the jump from DAPO's 26.67 to 31.15 Avg@32 (+4.48 points) is particularly large, suggesting the reweighting is most beneficial on problems where DAPO's baseline exploration is insufficient to discover correct reasoning chains.

Comparison with competing reweighting methods (Table 3): On Qwen2.5-Math-7B:

The paper's method achieves the best overall Avg@32 and Pass@16. The PPL method achieves competitive Avg@32 on AMC (79.97, the highest single-benchmark score) but its average is lower (41.38 vs. 43.75) and its Pass@16 is substantially lower (53.63 vs. 62.33). The Dominate method achieves 43.11 Avg@32 but only 53.63 Pass@16—the large gap between Avg@32 and Pass@16 (10.52 points) versus the paper's method (18.58 points gap, indicating more exploration headroom) is attributed to reduced training entropy from the restrictive ϵ_high = 0.24 setting.

Training dynamics (Figure 5): Four curves track AIME-24 Avg@32 over 250 RLVR steps. The paper's method (blue) consistently outperforms DAPO (red) from approximately step 50 onward, reaching ~0.39 versus ~0.35. Response length increases from ~800 to ~1,300 tokens for the paper's method, following a similar trajectory to DAPO but slightly higher. Training entropy for the paper's method remains in the 0.3-0.4 range, comparable to DAPO and PPL, while the Dominate method shows clearly lower entropy (0.2-0.3). The simultaneous increase in accuracy, response length, and sustained entropy is highlighted as characteristic of effective reasoning RLVR.

Reproducibility (Appendix B, Figure 9): Four independent runs of the reweighting method on Qwen2.5-Math-7B show all runs reaching the reported AIME-24 Avg@32 of ~39.06 by step 250, with consistent entropy (~0.35-0.40) and length (~1,200-1,400 tokens) trajectories. The curves are tightly clustered, confirming that the gains are not due to a favorable random seed.

Generalization to Minerva (Appendix C, Table 6):

• Qwen2.5-Math-7B: DAPO achieves 46.43 Avg@32, 69.44 Pass@16. Reweighting achieves 49.72 Avg@32 (+3.29), 70.37 Pass@16 (+0.93).
• Qwen3-8B-Base: DAPO achieves 55.04 Avg@32, 76.98 Pass@16. Reweighting achieves 56.57 Avg@32 (+1.53), 76.78 Pass@16 (essentially tied).

The gains on Minerva confirm that the reasoning improvements from better learning on low-probability tokens transfer across STEM domains beyond pure math.

Ablation Studies and Robustness Checks

Δlog p vs. additional magnitude metric variants (Figure 6, Appendix A.2): The paper evaluates forward KL D_KL^{π_RL, π_Base}, reverse KL D_KL^{π_Base, π_RL}, and RLVR model entropy H_π_RL as additional criteria for DAPO's token replacement experiment. All magnitude-based metrics require more replacements to reach RLVR accuracy than Δlog p. Forward KL is the best among magnitude metrics (reaching RLVR accuracy at ~15% replacement), followed by averaged KL (~18%), reverse KL (~22%), and entropy variants (~25-30%). The RLVR model's entropy H_π_RL performs similarly to the base model's entropy H_π_Base—neither entropy variant is competitive with divergence or Δlog p. This confirms that the advantage of Δlog p is robust across the choice of magnitude metric and is not an artifact of using a suboptimal KL variant.

Token-level analysis of selected positions (Figure 7): At matched ~8% replacement rate for DAPO-32B, the top 50 replacement tokens by each criterion are visualized. Entropy-selected tokens are dominated by common function words and transition phrases (the, thus, need, can, To, Now), with the top token "the" accounting for ~5% of replacements. KL divergence-selected tokens include more mathematical symbols (", \)", $, =, +, -) and reasoning vocabulary (approach, denote, from, since, Thus). Δlog p-selected tokens show the highest proportion of mathematical symbols and domain-specific tokens ($, \, \)", $\, =, 1, 2, 3, +, -, Rew). The paper interprets this as evidence that mathematical reasoning tokens, not just generic transition words, are the more efficient targets for improving reasoning accuracy.

Per-problem accuracy during replacement (Figure 8, Appendix A.2): Per-problem accuracy curves (30 subplots for AIME-24) using Δlog p-based replacement show that: (1) some problems (approximately 3-5 out of 30) remain at zero accuracy across all replacement rates—these are genuinely hard problems unsolvable by either model; (2) for most problems, accuracy increases monotonically with replacement rate; (3) a few problems (e.g., Problem 0 of DAPO, Problem 2 of ORZ) show an initial accuracy drop at low replacement rates before recovering at higher rates. The authors attribute this to coherence disruption: a small number of RL tokens inserted into a base-model context creates a generation that neither model can complete coherently.

Selective vs. global extrapolation (Table 1): Extrapolation applied globally to π_RL (no selectivity) is not evaluated directly, but the plateau in Table 1 (performance stops improving beyond ~5% replacement) implies that global extrapolation would be ineffective or harmful—amplifying directions at positions with near-zero Δlog p would amplify noise rather than signal.

Extrapolation on π_RL vs. π_Base (Table 1 vs. Figure 4): Two deployment modes are tested: extrapolation as part of base-model generation with selective replacement (Figure 4), and extrapolation applied directly to π_RL's own tokens (Table 1). Both modes yield improvements over their respective baselines (π_RL for Table 1, selective replacement for Figure 4), with the direct π_RL extrapolation reaching 55.31 Avg@32 at 5.2% replacement (vs. RLVR's 52.50) and the base-replacement extrapolation reaching 55.42 Avg@32 at 11.4% replacement (vs. base's 6.67). The similar ceiling (~55.3-55.4) suggests that extrapolation converges to a comparable improved distribution regardless of the starting point.

Hyperparameter sensitivity for extrapolation (Table 5, Appendix A.3): Full grid search over γ ∈ {0.05, 0.1} and τ ∈ {-0.5, -0.4, -0.3, -0.2, -0.1} for DAPO and ORZ (and denser τ grid for UniReason) shows:

• DAPO: across 10 (γ, τ) combinations, 8 of 10 extrapolation runs outscore the best replacement run (52.71), and 9 of 10 outscore the RLVR baseline (52.50). The performance range is 51.88-55.42.
• ORZ: across 10 combinations, 7 of 10 extrapolation runs outscore the best replacement run (46.15), and 2 of 10 outscore the RLVR baseline (46.15). The performance range is 43.44-47.50.
• UniReason: across 10 combinations, 9 of 10 extrapolation runs outscore the best replacement run (54.06), and 8 of 10 outscore the RLVR baseline (54.58). The performance range is 51.88-55.83.

The gains are robust but not universal across all hyperparameter settings—some (γ, τ) combinations underperform, underscoring that the method requires tuning.

Comparison of reweighting formulations (Table 3, Figure 5): The paper tests three reweighting strategies:

• PPL (Deng et al., 2025) reweights by response-level perplexity: Ã^{ppl}_{i,t} = [1 - α · w_ppl(y_i)] · Â_{i,t}. This achieves 41.38 Avg@32 and 61.08 Pass@16 (average across benchmarks). It excels on AMC (79.97 Avg@32, the highest single score) but underperforms on AIME25 (16.46 vs. paper's 18.54 Avg@32).
• Dominate (Yang et al., 2025b) reweights toward high-probability tokens: Ã^{dom}_{i,t} = [α · π_θ(y_{i,t}) + 1 - α] · Â_{i,t}. This achieves 43.11 Avg@32 but only 53.63 Pass@16. The low Pass@16 is attributed to reduced exploration entropy (Figure 5, approximately 0.2-0.3 vs. 0.3-0.4 for other methods) caused by the restrictive ϵ_high = 0.24.
• The paper's method achieves 43.75 Avg@32 and 62.33 Pass@16, the best on both metrics. The balanced improvement in both exploitation (Avg@32) and exploration (Pass@16) distinguishes it from the Dominate method (which sacrifices exploration) and the PPL method (which has unstable Pass@16 across benchmarks).

Model scale and family generalization (Table 2): The reweighting method is tested on two distinct model families at parameters 7B-8B. Consistent gains of 1.6-2.5 average points over DAPO are observed across both families, suggesting the mechanism (low-probability tokens are reasoning-critical) generalizes beyond any specific pretraining distribution.

Benchmark generalization (Tables 2, 6, Figure 10): Performance improvements are demonstrated on four distinct benchmarks: AIME-24, AIME-25, AMC (math reasoning), and Minerva (broad STEM). The gains are consistent across all benchmarks, with Minerva results (Table 6, Figure 10) being particularly notable since Minerva includes subjects (chemistry, astronomy) not present in the math-focused RLVR training data.

Critical Assessment

The paper makes three core claims: (1) Δlog p is more effective than magnitude-based metrics for identifying reasoning-critical RLVR updates; (2) this insight can be exploited for test-time improvement via selective extrapolation; (3) this insight can be exploited for training-time improvement via advantage reweighting. Each requires scrutiny against the experimental evidence.

Claim 1—Δlog p superior to magnitude metrics: The evidence is strong and multi-faceted. The statistical separation in Figure 1b and Figures 12-15 is visually unambiguous: entropy and divergence distributions overlap regardless of which model generated the tokens, while Δlog p exhibits model-dependent bimodality. The token replacement experiment (Figure 2) provides causal validation: Δlog p recovers RLVR accuracy with approximately half the replacements of the next-best metric (KL divergence), and this ordering is consistent across three model pairs and six magnitude metric variants (Figure 6). The gradient analysis (Lemma 3.1, Figure 3a-b) provides a mechanistic explanation linking training dynamics to the final Δlog p pattern, and the top-p filtering experiment (Figure 3c) causally confirms that the low-probability tokens identified as high-Δlog p are necessary for RLVR's performance.

However, the evidence is limited to mathematical reasoning on AIME-24. While the Minerva results (Figure 10) show the extrapolation method generalizes to broader STEM, the core diagnostic claim (Δlog p identifies reasoning-critical tokens better than KL/entropy) is validated only on AIME-24 token replacement curves. The paper does not run token replacement experiments on Minerva or other non-math benchmarks. The claim's generality to code generation, logical reasoning, or other RLVR domains is untested.

Additionally, the AIME-24 dataset has only 30 problems. The per-problem analysis (Figure 8) reveals substantial heterogeneity—some problems show monotonic improvement with replacement, others show initial degradation, and a few are unsolvable. With only 30 problems, the aggregate replacement curves (Figure 2) could be influenced by a small number of problems. The paper does not report confidence intervals on the replacement curves, making it difficult to assess whether Δlog p's advantage over KL divergence (e.g., 10% vs. 15% replacement to reach RLVR accuracy on DAPO) is statistically significant or within the noise of the 30-problem sample.

The replacement experiment design has an intrinsic limitation: replacing tokens from π_RL while keeping the context generated by π_Base creates a hybrid generation that may not be coherent (as acknowledged in the per-problem analysis). When 10% of tokens are replaced, the remaining 90% of context is from the base model's distribution, which the RLVR model was not trained to condition on. This distribution shift could artificially inflate the replacement rate needed to reach RLVR accuracy—the hybrid generation may need more RLVR tokens to compensate for the mismatched context. If so, the 10% Δlog p replacement rate is an upper bound on the true fraction of reasoning-critical tokens. The paper does not have a way to estimate this distortion.

Claim 2—Test-time extrapolation improves over RLVR: The evidence shows consistent but modest gains (+1.25 to +2.92 Avg@32 on AIME-24, Figure 4; +1.4 to +1.9 Avg@8 on Minerva, Figure 10). The sensitivity analysis (Table 5) confirms robustness to hyperparameter choice. However, several limitations temper the practical significance:

• Requires both base and RLVR models at inference time. The paper acknowledges this as a primary limitation. Extrapolation doubles the model memory footprint and compute for log-probability evaluation. For many deployment scenarios, this is prohibitively expensive. The paper does not explore distillation or parameter-efficient approaches to reduce this cost.

• The gains are small relative to RLVR training itself. RLVR training takes DAPO-32B from 6.67 to 52.50 Avg@32—a gain of +45.83 points. Extrapolation adds +2.92 points to that, or about 6.4% of the RLVR gain. Whether this incremental improvement justifies the doubled inference cost depends on the application's accuracy-sensitivity. The paper frames the gains as "essentially free" (no additional training), but they are not computationally free—inference cost doubles.

• Hyperparameter selection requires a validation set. The optimal (γ, τ) combination varies by model (γ=0.05 for DAPO, γ=0.1 for ORZ and UniReason; τ varies from -0.4 to -0.3). The paper selects these based on AIME-24 performance (the same dataset used for evaluation, though for the sensitivity analysis this appears to be post-hoc). In a true deployment scenario, selecting hyperparameters without access to the evaluation set would require a separate validation benchmark.

• Theorem 4.1 provides theoretical motivation but is idealized. The NPG bandit setting with individual logit parameters per prompt-response pair abstracts away sequence structure, context dependence, and the softmax coupling across tokens that makes real LLM extrapolation challenging. The theorem proves existence of a beneficial γ but provides no guidance on its magnitude, and the proof relies on the monotonicity of NPG updates (which clipping and mini-batching in DAPO violate).

Claim 3—Training-time reweighting improves over DAPO: The evidence in Table 2 shows consistent gains: +1.63 average points for Qwen2.5-Math-7B, +2.52 for Qwen3-8B-Base. These are meaningful improvements over a strong DAPO baseline. The reproducibility runs (Figure 9) confirm the gains are not due to random seed. The Pass@16 gains (+4.47 and +3.33 average points) suggest the method improves exploration as well as exploitation.

However, the absolute performance levels are modest. The state-of-the-art for 7-8B math models is not established in the paper—DAPO's 42.12 average and the paper's 43.75 average on Qwen2.5-Math-7B are presented without comparison to published results from other 7B-scale math reasoning models. It's unclear whether these are competitive numbers or substantially below the frontier for this model scale. The paper compares only against DAPO and two alternative reweighting methods, not against other RLVR algorithms at similar scale (e.g., GRPO, Dr.GRPO, Open-Reasoner-Zero at 7B). The claim is that reweighting improves over DAPO, which is supported, not that it achieves state-of-the-art, which is untested.

Comparison of reweighting methods (Table 3) reveals methodological tensions. The Dominate method (Yang et al., 2025b) achieves the second-highest Avg@32 (43.11) but the lowest Pass@16 (53.63). The paper attributes this to reduced exploration from ϵ_high = 0.24, but notes that using the default ϵ_high = 0.28 caused unstable training. This raises an unresolved question: if the anti-domination approach genuinely requires reduced clipping to stabilize, then the low exploration may be an inherent cost of that strategy rather than a hyperparameter choice. The paper does not explore whether its own reweighting method would benefit from different ϵ_high settings or whether the DAPO baseline's ϵ_high = 0.28 is optimal.

The causal chain from Δlog p analysis to reweighting design has a logical gap. The paper argues: high-Δlog p tokens are low-probability during training (Figure 3b) → therefore, amplifying advantages on low-probability tokens should help. However, Figure 3b shows a correlation between final Δlog p and training-time token probability, not a causal relationship. It's possible that low-probability tokens naturally have high Δlog p because they start from a low baseline, not because they are inherently reasoning-critical. The top-p experiment (Figure 3c) shows that low-probability tokens are necessary in aggregate, but does not distinguish whether the specific tokens that the reweighting amplifies are the right ones. The reweighting method amplifies all low-probability tokens, including potentially spurious ones that happened to appear in high-advantage trajectories by chance. A more targeted reweighting based on estimated future Δlog p (e.g., using an auxiliary predictor) could potentially outperform the current approach.

Missing experiments that would strengthen the paper:

• Token replacement on the reweighted model: Does the reweighting-trained model exhibit a different Δlog p pattern than the DAPO baseline? If the reweighting works by strengthening learning on tokens that will become high-Δlog p, the reweighted model should show more concentrated Δlog p (fewer tokens with moderate Δlog p, sharper bimodality). This would close the loop between the analysis and the method.

• Scaling to larger models: All reweighting experiments use 7-8B models. The paper's analysis uses 14B and 32B models for the diagnostic studies. Whether the reweighting gains persist or diminish at larger scales is unknown. Larger models have different pretraining distributions, different entropy landscapes, and potentially different sensitivity to low-probability token amplification.

• Longer training horizons: Figure 5 shows training curves for 250 steps. It's possible that the reweighting method converges faster but DAPO catches up given more steps. The paper does not establish whether the gain is in final asymptote or convergence speed.

• Ablation on the specific form of the reweighting factor: The paper uses 1 + α(1 - π) but does not test alternatives like (1 - π)^β with varying β, or a thresholded version that only amplifies tokens below a probability cutoff. These ablation choices would clarify whether the specific functional form matters or just the general principle of upweighting low-probability tokens.

• Combining test-time extrapolation with reweighting-trained models: The two methods are presented as independent contributions. Testing whether a reweighting-trained model benefits from additional extrapolation (or whether its Δlog p direction is "stronger" and yields larger extrapolation gains) would unify the paper's narrative. The current separation leaves open whether the two methods are complementary or partially redundant.

On the paper's framing of Yang et al. (2025b): The paper positions its reweighting method as the opposite of Yang et al.'s anti-domination approach (upweighting high-probability tokens to counter low-probability domination). The experimental comparison (Table 3) shows the paper's method outperforming Dominate on both Avg@32 and Pass@16, which the paper uses to argue that "low-probability tokens are not over-dominating; they are the essential learning signal." However, Yang et al. (2025b) studied a different setting (potentially different base models, training data, and hyperparameters). The Dominate implementation in this paper uses ϵ_high = 0.24 per Yang et al.'s recommendation, and the paper notes that ϵ_high = 0.28 caused instability—this instability may be specific to the interaction of Dominate with DAPO's other components, not a fundamental property of anti-domination. The comparison is fair as-implemented but does not resolve the underlying tension about optimal token-weighting strategy.

Summary of evidential strength: Claim 1 (Δlog p diagnostic superiority) is well-supported for math reasoning on AIME-24 but limited by the small dataset (30 problems), lack of non-math token replacement experiments, and the hybrid-generation confound. Claim 2 (extrapolation improvement) shows consistent but modest gains that must be weighed against the doubled inference cost. Claim 3 (reweighting improvement) shows meaningful gains over DAPO on 7-8B models but the comparison set is narrow, the causal link to Δlog p analysis is correlational rather than demonstrated, and scaling to larger models is unexplored. The paper's experimental program is thorough within its scope but leaves open the question of whether the Δlog p perspective enables gains beyond the specific methods and settings tested.

6. Limitations and Trade-offs

Difficulty Estimation and the Unseen Cost of Computing Δlog p

The assumption or constraint. The test-time extrapolation method (Section 4.1) and the token replacement diagnostic (Section 3.2) both require computing Δlog p by evaluating every generated token under both π_Base and π_RL. The paper implicitly assumes this dual-model inference cost is acceptable, and the headline extrapolation gains (+1.25 to +2.92 Avg@32 on AIME-24, Figure 4) are reported without factoring in the cost of producing Δlog p. The paper acknowledges this in Section 6 (Limitations and future work):

"One primary limitation of our extrapolation method is the requirement of two models; future work could integrate this with parameter-efficient finetuning to reduce computational cost."

However, the acknowledgment is brief and does not quantify the magnitude of the overhead.

The consequence. In practice, selective extrapolation doubles the model memory footprint and roughly doubles the inference FLOPs at each token position where Δlog p must be computed. For the selective approach (gating by f^τ_logp), the base model generates all tokens (full inference cost), and at positions flagged by the criterion, the RLVR model must also run a forward pass to sample from π^γ_Extra. Since Δlog p itself requires log-probabilities from both models, computing whether a token meets the criterion Δlog p < τ already requires evaluating both models at every position—you cannot know which tokens to replace without first computing Δlog p everywhere. The gating only saves the cost of sampling from π^γ_Extra, not the cost of evaluating Δlog p. The actual inference cost is therefore approximately 2× the baseline model's inference cost (both models must run forward passes for every generated token), not the ~1.1× that the replacement ratio alone would suggest. A practitioner deploying this method faces a direct accuracy-vs-cost tradeoff: +1-3 Avg@32 points for ~2× the inference compute. Whether this is worthwhile depends entirely on the application's sensitivity to accuracy gains at that scale.

What evidence exists in the paper. None. The paper never reports the wall-clock time, FLOP count, or memory footprint of the extrapolation method compared to the RLVR baseline or the base model. The efficiency of Δlog p-based selection is measured only in terms of token replacement rate (e.g., "10% of tokens replaced"), which understates the actual computational cost by an order of magnitude because it counts only the tokens where sampling is redirected, not the tokens where Δlog p must be computed to make the gating decision. Table 5 (Appendix A.3) reports hyperparameter sweeps across thresholds and γ values but never includes compute cost as a dimension of comparison.

Mitigation status. The paper acknowledges the two-model requirement as a limitation and gestures at future work on "parameter-efficient finetuning" to reduce the cost, but provides no concrete method or analysis. A natural mitigation—using a distilled or quantized version of one model to reduce the per-token cost—is not explored. The training-time reweighting method (Section 4.2) does not suffer from this limitation, as it modifies only the training objective and produces a single model at inference time, but the paper does not frame it as the "efficient" alternative to extrapolation.

The Hardest Problems Gain Nothing from Either Method

The assumption or constraint. The paper's central thesis—that RLVR's reasoning gains come from sparse, directional updates on a minority of tokens, which can be identified and amplified—implicitly assumes that the base model already has some non-trivial probability of producing correct reasoning chains. When the base model's pass rate on a problem is near zero, there are no "reasoning-critical tokens" for RLVR to learn, because there are no successful trajectories to propagate verifier signal from. The paper does not explicitly state this capability boundary as an assumption, but the pattern is visible throughout the results.

The consequence. Both the test-time extrapolation and the training-time reweighting methods provide zero benefit on problems where the base model fundamentally lacks the capability to reason correctly. In the token replacement experiment (Appendix A.2, Figure 8), approximately 3-5 out of 30 AIME-24 problems remain at zero accuracy across all replacement rates—no amount of targeted token substitution can create correct reasoning where neither model knows how to produce it. The training-time top-p experiment (Figure 3c) shows that all top-p configurations degrade relative to the unfiltered baseline, but even the best configuration (top-p=1.0) plateaus at ~0.35 Avg@32—meaning ~65% of AIME-24 problems are not solved by the 7B model regardless of training configuration. The reweighting method improves this (Table 2: Qwen2.5-Math-7B reaches 39.06 Avg@32 on AIME-24), but the majority of problems remain unsolved. The method amplifies reasoning behavior the model can already produce, but cannot create reasoning capability de novo. This is the same fundamental boundary observed in the test-time compute scaling literature (where the hardest difficulty bins show near-zero improvement regardless of budget), and it applies here: Δlog p-based methods amplify existing capability but do not create it.

What evidence exists in the paper. The per-problem accuracy curves in Figure 8 (Appendix A.2) show a subset of problems with flat zero-accuracy lines across all replacement rates—the clearest evidence. The AIME-24 Avg@32 numbers for the 7-8B models (Table 2) top out at 38-39% even after reweighting, confirming that ~60% of problems remain unsolved. The paper does not break down performance by problem difficulty or analyze whether the improvement from extrapolation/reweighting is concentrated on problems that the base model already had some chance of solving, which would directly quantify this capability boundary. The Minerva generalization results (Appendix C) similarly show gains but leave most problems unsolved (best Avg@32 is 56.57 for Qwen3-8B-Base).

Mitigation status. The paper does not address this limitation. There is no difficulty-stratified analysis, no discussion of which problem types benefit from the methods and which do not, and no acknowledgment that the methods are bounded by the base model's pretrained capabilities. The paper's framing—"diagnose and improve RLVR by its update direction"—implicitly assumes there exist meaningful directional updates to diagnose, which fails at the capability frontier. Future work on combining Δlog p-based methods with continued pretraining or knowledge augmentation might address this, but the current methods offer no path forward for genuinely novel reasoning beyond the base model's reach.

Single Domain (Mathematical Reasoning) with No Evidence Beyond STEM

The assumption or constraint. All token replacement experiments, statistical analyses, and training-time reweighting evaluations are conducted on mathematical reasoning benchmarks (AIME-24, AIME-25, AMC for training; Minerva for generalization testing). The paper's claims about Δlog p identifying "reasoning-critical" tokens are validated exclusively in the context of mathematical problem-solving. The Minerva results (Appendix C, Figures 10, Tables 6-7) extend to broader STEM (chemistry, astronomy, physics) but remain within formal, verifiable reasoning with objective correctness criteria. The paper does not test on code generation, logical reasoning, commonsense QA, multi-hop inference from text, or any domain where "correctness" might be ambiguous, multi-dimensional, or subjective.

The authors do not explicitly claim generality beyond math, but the paper's language consistently frames findings in domain-neutral terms: "reasoning-critical tokens," "reasoning accuracy," "reasoning performance." The Δlog p metric itself is domain-agnostic (it computes only log-probability differences, requiring no domain-specific verifier), which might lead readers to assume the diagnostic and methods transfer to any RLVR-trained model.

The consequence. RLVR for mathematical reasoning has a specific structure that may not generalize: the verifier provides a clean binary signal (correct/incorrect final answer) with no partial credit, and successful reasoning chains follow a relatively constrained logical structure (step-by-step deduction toward a unique answer). In domains like code generation, the verifier might be multi-dimensional (compilation success, test pass rate, execution efficiency) or noisy (flaky tests). In dialogue or creative writing, RLVR is often applied with learned reward models rather than verifiable rewards, introducing reward model bias and over-optimization dynamics that mathematical verifiers avoid. The Δ\log p distribution for a dialogue model trained via RLHF might look fundamentally different from the bimodal pattern observed for math RLVR models—the verifier signal is less crisp, so the directional concentration may be weaker or noisier. The token replacement experiment's finding that Δ\log p recovers RLVR accuracy with ~10% token replacements might be specific to math, where a small number of critical reasoning tokens (mathematical operators, logical connectors, equation terms) can deterministically change the reasoning trajectory. In code generation, a single token error (wrong variable name, off-by-one in an index) can break correctness, but there may be many such critical tokens scattered throughout the output rather than concentrated in a sparse few, changing the replacement dynamics.

What evidence exists in the paper. The Minerva results provide the only out-of-distribution generalization test. The selective extrapolation gains on Minerva (+1.4 to +1.9 Avg@8, Figure 10) are comparable in relative magnitude to AIME-24 gains, and the reweighting method also transfers (Table 6: +3.29 Avg@32 for Qwen2.5-Math-7B, +1.53 for Qwen3-8B-Base). This suggests the methods generalize across STEM subjects with verifiable answers. However, Minerva remains within the formal reasoning paradigm—there is no test on a qualitatively different RLVR application. The paper does not include any experiment on non-STEM tasks, any analysis of whether the Δ\log p bimodal pattern holds for non-math models, or any discussion of domain-specific failure modes.

Mitigation status. The paper does not claim generality and does not discuss domain limitations. The Minerva results provide some evidence of STEM-level generalization but do not address the broader question. A cautious practitioner would assume the findings are validated only for formal, verifiable mathematical reasoning and would seek additional evidence before applying the methods to code, dialogue, or other RLVR domains.

Small Evaluation Sets with No Statistical Significance Testing

The assumption or constraint. The paper's primary diagnostic experiment (token replacement, Figure 2) and primary method evaluation (selective extrapolation, Figure 4) are conducted on AIME-24, which contains only 30 problems. The selective extrapolation sensitivity analysis (Table 5, Appendix A.3) reports performance variations across hyperparameter choices without any measure of statistical uncertainty. The per-problem accuracy analysis (Figure 8) reveals substantial heterogeneity—some problems show monotonic improvement, others show initial degradation, and a few are unsolvable—yet aggregate results are reported as point estimates without confidence intervals.

The AIME-25 and AMC benchmarks used for reweighting evaluation contain 30 and 40 problems respectively (AIME-25 has 30, AMC-10/12 has 40-50), which are also small sample sizes. The Minerva benchmark (272 problems) is larger, providing stronger statistical grounding for the generalization claims.

The consequence. With only 30 problems, a single problem can shift Avg@32 by ~3.3 percentage points (1/30 of the maximum possible accuracy). The observed gains from extrapolation (+1.25 to +2.92 Avg@32, Figure 4) could be driven by improvement on a small number of problems (perhaps 3-5 out of 30) while the majority show no change or slight degradation. The per-problem curves in Figure 8 are consistent with this: several problems show flat or declining accuracy at the replacement rates used for extrapolation. The lack of confidence intervals makes it impossible to distinguish a robust, widespread small improvement from a concentrated large improvement on a lucky subset of problems. For a practitioner deciding whether to deploy selective extrapolation, this distinction matters: if the gains are concentrated on a few "easy" problems that the RLVR model already mostly solves, the practical value is limited compared to a method that helps on the problems the model currently fails.

The training-time reweighting results (Table 2) are also reported without uncertainty quantification, though the reproducibility runs (Figure 9, four independent training runs on Qwen2.5-Math-7B) provide some evidence of stability—all four runs converge to similar AIME-24 Avg@32 (~39.06). However, this only addresses training variance, not evaluation variance from the small test set.

What evidence exists in the paper. The paper provides per-problem breakdowns for the token replacement experiment (Figure 8, 30 subplots showing individual problem trajectories) but does not aggregate this into per-problem improvement statistics for the extrapolation method. The sensitivity analysis (Table 5) reports point estimates across hyperparameter grids but no standard errors or confidence intervals. The reproducibility runs (Figure 9) show training convergence stability but are evaluated on the same fixed 30-problem AIME-24 set, so they do not quantify test-set sampling variance. The paper does not use bootstrap confidence intervals, does not report standard deviations across the 32 samples per problem, and does not apply any statistical test to compare methods.

Mitigation status. The paper does not acknowledge this limitation. The use of multiple benchmarks (AIME-24, AIME-25, AMC, Minerva) and multiple model pairs (ORZ, DAPO, UniReason; Qwen2.5-Math-7B, Qwen3-8B-Base) provides some cross-validation—consistent gains across settings suggest the effect is real—but the small individual test sets mean that any single comparison is noisy. The Minerva results (272 problems) partially address this by providing a larger sample, and the gains there are consistent with AIME-24 results, which strengthens confidence. However, the paper never quantifies uncertainty or acknowledges the small-sample limitation of its primary evaluations.

The Reweighting Method's Connection to Δlog p Is Correlational, Not Causal

The assumption or constraint. The training-time reweighting method (Section 4.2) is motivated by the correlation shown in Figure 3b: tokens with high final Δlog p had low probability during training (π_θold). The reweighting amplifies advantages on all low-probability tokens during training, on the premise that these are the tokens that will become the high-Δlog p, reasoning-critical positions. However, the paper never demonstrates that the specific tokens amplified by the reweighting are the ones that ultimately contribute to improved reasoning. The relationship is a post-hoc correlation observed in the final DAPO model, not a causal mechanism verified under the reweighting intervention.

The consequence. The reweighting method amplifies all low-probability tokens during training—including tokens that are improbable for good reason (they lead to incorrect reasoning) and tokens that are improbable due to random sampling noise. The standard DAPO advantage mechanism already handles the former: tokens from incorrect trajectories receive negative advantages, so amplifying their weight drives probability down (desirable). But tokens from correct trajectories that were improbable by chance (e.g., an unusual but valid equation format) receive positive advantages, and amplifying their weight might overfit to spurious patterns in the training data. The method provides no mechanism to distinguish "good" low-probability tokens (genuinely reasoning-critical, as indicated by high final Δlog p) from "accidental" low-probability tokens (correct but idiosyncratic).

The paper's comparison with the Dominate method (Yang et al., 2025b) in Table 3 illustrates this tension implicitly. Yang et al. argue that low-probability tokens over-dominate training and should be dampened to prevent overfitting to rare token sequences. The paper's method does the opposite—amplifying them further—and achieves better performance, which the paper interprets as evidence that "low-probability tokens are the essential learning signal." But this interpretation assumes that all low-probability tokens are beneficial to amplify, which the paper's own Δlog p analysis would contradict: many low-probability tokens during training will end up with near-zero final Δlog p (they were not meaningfully updated), suggesting they were not reasoning-critical. The reweighting amplifies these tokens too, potentially adding noise to the gradient.

What evidence exists in the paper. There is no experiment that would close this causal gap. The paper does not:

• Compute Δlog p for the reweighting-trained model and compare its distribution to the DAPO baseline (does reweighting produce more concentrated, higher-magnitude Δlog p?).
• Analyze whether the tokens most amplified by reweighting (those with highest (1 - π_θold) and largest positive advantage) correspond to tokens that exhibit high Δlog p in the final model.
• Ablate the reweighting to only amplify low-probability tokens that also meet some additional criterion (e.g., above-median advantage, or belonging to responses that ultimately received positive reward), which would test whether selectivity improves over uniform amplification.

The training curves in Figure 5 show that reweighting increases accuracy without reducing entropy, which is consistent with the method not overfitting, but doesn't isolate the mechanism.

Mitigation status. The paper does not acknowledge this as a limitation. The narrative arc—Figure 3b shows correlation, therefore reweighting by 1 - π_θold should help—is presented as a direct implication, when it is in fact a hypothesis that the reweighting experiments support but do not prove. A practitioner implementing this method should understand that it is grounded in a plausible but unverified causal model, not a rigorously established mechanism. The method works empirically (Tables 2, 3, 6), which is the primary justification, but the conceptual link to Δlog p analysis remains correlational. Future work that directly tests whether reweighting specifically amplifies the tokens destined for high Δlog p would strengthen the paper's central thesis that "diagnosing Δlog p enables improvement."

Test-Time Extrapolation Gains Are Modest Relative to RLVR Training and Require Tuning

The assumption or constraint. The paper presents selective extrapolation as a method that "surpasses RLVR model's performance without further training" (Section 4.1), with gains of +1.25 to +2.92 Avg@32 on AIME-24 (Figure 4). The framing implies that these gains are practically meaningful and come at low cost. However, the paper does not benchmark against the obvious alternative: spending the same computational resources on additional RLVR training rather than on inference-time extrapolation.

The consequence. The RLVR training that produced the DAPO-32B model improved AIME-24 Avg@32 from 6.67 (base) to 52.50 (RLVR)—a gain of +45.83 points. Extrapolation adds +2.92 points to this, representing a 6.4% relative improvement over what RLVR already achieved. Whether this is "worth it" depends on context: if a practitioner has already invested substantial compute in RLVR training, a ~6% accuracy boost from a test-time wrapper might be attractive. But the cost of extrapolation (effectively 2× inference compute, as discussed above) means the same resources could instead be used to generate more samples from the RLVR model (majority voting at 2× samples) or to continue RLVR training for additional steps. The paper does not compare extrapolation against these natural baselines: what is the RLVR model's majority-vote accuracy at 64 samples (matching the 2× compute of extrapolation at 32 samples)? Does additional RLVR training beyond the published checkpoint yield gains larger or smaller than extrapolation? Without these comparisons, a practitioner cannot determine whether extrapolation is the most compute-efficient way to spend their inference budget.

The sensitivity analysis (Table 5, Appendix A.3) shows that extrapolation performance varies with both γ and τ, and that some hyperparameter combinations underperform the RLVR baseline or selective replacement. For ORZ-32B, extrapolation with γ=0.05, τ=-0.5 achieves 47.19 (beating RLVR's 46.15), but with γ=0.1, τ=-0.5 achieves 43.75 (worse than RLVR). For DAPO-32B, γ=0.1, τ=-0.4 achieves 53.33 (beating RLVR's 52.50), but γ=0.1, τ=-0.3 achieves 53.85 (barely beating). The method is not "plug-and-play"—it requires per-model tuning of two hyperparameters using a validation set, and the optimal values differ across models. This tuning cost (in compute and in access to labeled validation data) is not accounted for in the paper's framing of the method as "without further training."

What evidence exists in the paper. The sensitivity analysis in Table 5 provides the raw data showing hyperparameter-dependent performance. The paper does not report the cost of hyperparameter search (how many (γ, τ) combinations were tested to find the reported values, and on what validation data). The extrapolation results on Minerva (Table 7, Appendix C) use different τ values than AIME-24 (τ = -1.0 or -0.9 for Minerva versus -0.3 to -0.4 for AIME-24), indicating that the optimal thresholds are dataset-dependent as well as model-dependent. The paper never compares extrapolation against 2× majority voting or against further RLVR training, making it impossible to assess the method's compute-efficiency relative to alternatives.

Mitigation status. The paper acknowledges the hyperparameter issue in Section 6: "The extrapolation also introduces additional hyperparameters, and future work can explore combining the selection threshold and extrapolation strength for a more adaptive extrapolation." However, this framing understates the practical burden: two hyperparameters that vary across models and datasets require non-trivial validation infrastructure. There is no discussion of the cost-vs-alternative-baselines question. A practitioner adopting this method would need to budget for hyperparameter tuning on their specific model and task, and should independently verify that the gains exceed what could be achieved by simply generating more samples or training longer.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a conceptual reframing of RLVR analysis that is likely to shift how the field thinks about—and measures—what RLVR training does to language models. The shift is not a paradigm overthrow (the paper does not claim RLVR works differently than previously understood), but it is more than an incremental refinement: it identifies a missing analytical dimension (direction) and demonstrates that incorporating it yields both sharper diagnostics and actionable improvements. The magnitude of this shift can be calibrated by comparing it to the sparsity finding that preceded it. When Wang et al. (2025b) established that RLVR updates are sparse, they changed what questions analysts ask: no longer "does RLVR change the model globally?" but "which minority of tokens does it modify?" This paper changes the question again: no longer "which tokens changed most?" but "which tokens changed in the verifier-aligned direction, and can we exploit that direction explicitly?"

The most important conceptual contribution is the decoupling of distributional change magnitude from reasoning relevance. Prior work had implicitly assumed that because RLVR updates are sparse, identifying the high-magnitude tokens should identify the reasoning-critical ones. The paper's statistical comparison (Figure 1b) demonstrates why this assumption fails: entropy and KL divergence distributions look identical regardless of which model generated the tokens, meaning that high-magnitude changes include many tokens that are irrelevant to improved reasoning. The signed log-probability difference Δlog p separates model-origin in a way magnitude metrics cannot, and this separation maps onto reasoning performance (Figure 2: Δlog p reaches RLVR accuracy with ~10% replacement versus 15-30% for KL/entropy). The practical implication is that future RLVR analyses must account for direction—reporting only divergence or entropy is now insufficient, much as reporting only average reward without calibration metrics would be insufficient in RLHF analysis.

This reframing resolves an apparent tension in the literature about low-probability tokens. Yang et al. (2025b) observed that low-probability tokens dominate RLVR's gradient updates and proposed dampening them to stabilize training. The paper's gradient analysis (Lemma 3.1, Figure 3a) confirms the domination—59.48% of tokens have probability ≥ 0.98 and receive negligible gradient, while 3.17% have probability < 0.02 and receive maximal gradient—but reinterprets it as the mechanism of learning rather than a training pathology. The top-p filtering experiment (Figure 3c) causally demonstrates that removing low-probability tokens during training breaks RLVR's ability to learn reasoning. The reweighting method (Table 2) shows that amplifying these tokens (rather than dampening them) improves performance. This does not falsify Yang et al.'s observation that low-probability tokens can cause instability (the Dominate method in Table 3 required reduced clipping to avoid training collapse) but it reframes the response: rather than dampening low-probability tokens, the solution is to ensure they are channeled productively through proper advantage estimation and clipping, not suppressed.

The paper also establishes a new paradigm for closing the analysis-to-improvement loop in RLVR research. Before this work, RLVR analysis and RLVR improvement were largely parallel threads. Analysts characterized sparsity, entropy dynamics, and gradient concentration; practitioners developed better algorithms (DAPO, GRPO variants, reward shaping). The two threads rarely intersected causally—analysis explained what happened retrospectively but did not directly generate design principles. This paper demonstrates that a single diagnostic quantity (Δlog p) can serve as both an analysis tool and a direct blueprint for method design. The training-time reweighting method is Δlog p-motivated (amplify the tokens that will become high-Δlog p) and the test-time extrapolation method is Δlog p-defined (push the distribution along the learned Δlog p direction). Both work. This pattern—identify a fundamental quantity, validate it causally, then build methods that exploit it—provides a template that future RLVR research can adopt. The paper implicitly argues that good analysis should be predictive of what interventions will help, not merely descriptive of what already happened.

The practical magnitude of the findings, however, tempers any claim of a transformative breakthrough. The training-time reweighting improves average reasoning accuracy by ~1.6-2.5 points over DAPO (Table 2) on 7-8B models—meaningful but not dramatic. The test-time extrapolation adds ~1.25-2.92 points over the RLVR model (Figure 4)—again meaningful but modest, and at a ~2× inference cost. The paper has not demonstrated that Δlog p-based methods close the gap to larger models, enable fundamentally new capabilities, or solve hard problems that RLVR alone cannot. The landscape shift is therefore primarily analytical and methodological: the paper provides a better diagnostic toolkit and a demonstration that this toolkit enables improvement, but the absolute performance frontier remains governed by the base model's pretrained capabilities.

Research directions that become more attractive after this paper:

• Directional analysis as a standard component of RLVR papers. The paper's histograms (Figure 1b, Figures 12-15) provide a simple, replicable protocol: compare Δlog p distributions for base-generated and RLVR-generated tokens, and check for bimodal separation. This takes hours of compute (generate from both models, compute per-token log-probs, plot histograms) and provides a diagnostic that magnitude metrics cannot. Future RLVR papers that report only KL divergence or entropy should face the question: does Δlog p tell you something those metrics miss?

• Implicit reward signals learned during training. The framing of Δlog p as a token-level implicit reward (Section 4.1: π^γ_Extra ∝ π_RL · exp(γ · Δlog p)) opens the door to test-time optimization methods that leverage the training signal directly rather than requiring external verifiers or reward models. This connects RLVR analysis to the reward-guided decoding literature in a way that was not obvious before.

• Token-level understanding of RLVR's effect on reasoning structure. The qualitative finding that Δlog p-selected tokens include more mathematical symbols and reasoning-specific vocabulary (Figure 7) while entropy-selected tokens are dominated by function words suggests that RLVR's reasoning improvements have a specific syntactic signature. Analyzing which types of tokens exhibit high Δlog p could reveal what reasoning behaviors RLVR is actually modifying.

Research directions that become less attractive after this paper:

• Pure magnitude-based RLVR analysis without directional decomposition. A paper that reports only KL divergence between base and RLVR models, or only entropy reduction on RLVR outputs, now has a clear limitation: it cannot distinguish verifier-aligned changes from benign distributional variation, and the paper's evidence (Figure 1b, Figure 2) shows this distinction matters for identifying reasoning-critical updates. The bar for RLVR analysis has been raised.

• Anti-domination approaches that dampen low-probability tokens uniformly. The paper's reweighting method (Table 2) and Yang et al.'s Dominate method (Table 3) represent opposite responses to the same observation (low-probability tokens dominate gradients). The paper's method wins on both Avg@32 and Pass@16 (43.75 vs. 43.11 Avg@32, 62.33 vs. 53.63 Pass@16), and the top-p filtering experiment (Figure 3c) shows that excluding low-probability tokens breaks learning. Uniform dampening is likely the wrong response; future work should focus on selective reweighting or stabilization that preserves the essential learning signal.

Follow-Up Research This Work Enables

Training a lightweight Δlog p predictor to enable cheap, real-time test-time steering without retaining the base model. The practical barrier to the extrapolation method is the requirement of both π_Base and π_RL at inference time—computing Δlog p doubles the inference cost. A natural follow-up is to train an auxiliary head on π_RL that predicts Δlog p directly from the RLVR model's hidden states, without requiring a forward pass through π_Base. The training signal would be the actual Δlog p computed from paired model outputs (as done in this paper's statistical analysis). The key evaluation would compare downstream extrapolation performance using predicted versus true Δlog p: if the predictor can match 80%+ of the extrapolation gain with no additional inference cost, the method becomes immediately practical for deployment. The paper's statistical analysis (Figures 12-15) provides the training data pipeline—generate 32 responses per question from both models, compute per-token Δlog p, and train the auxiliary head on this supervised signal. A negative result (the predictor fails to capture enough directional signal to maintain gains) would suggest that Δlog p depends on fine-grained probability differences the RLVR model's representations do not encode.

Measuring whether reweighting-trained models produce more concentrated Δlog p distributions, closing the causal loop between analysis and method design. The paper's training-time reweighting is motivated by the post-hoc observation that high-Δlog p tokens correspond to low-probability tokens during training (Figure 3b), but the paper never verifies that reweighting actually produces a different Δlog p pattern. A direct follow-up would train both DAPO and reweighting models from the same base checkpoint, compute Δlog p against the shared base model for both, and compare the distributions. The hypothesis is that reweighting-trained models should show more extreme bimodality: a sharper positive tail (stronger reasoning-enhancing updates) and a more concentrated near-zero mass (less noise amplification). If the distributions are indistinguishable despite performance differences, the mechanism of improvement is not what the paper claims, and future work would need to look elsewhere (e.g., better exploration dynamics, different advantage scaling effects). This experiment requires only the models already trained in the paper plus the Δlog p computation pipeline used in Section 3.1.

Difficulty-stratified analysis of which problems benefit from Δlog p-based methods and which do not. The per-problem accuracy curves in Figure 8 (Appendix A.2) show that ~3-5 out of 30 AIME-24 problems remain at zero accuracy across all token replacement rates, and some problems show initial accuracy drops before recovering. A systematic follow-up would bin problems by the base model's pass@1 rate (as done in the test-time compute scaling literature) and evaluate both extrapolation and reweighting gains within each difficulty bin. The paper's central claim—that RLVR's gains come from amplifying reasoning behaviors the base model can already produce—predicts that gains should concentrate on easy-to-medium problems and vanish on the hardest problems (where the base model's pass@1 is near zero). Confirming this would establish a clear capability boundary for Δlog p-based methods; disconfirming it (gains on hard problems without base-model capability) would be a major finding suggesting the methods can create new reasoning behaviors, not just amplify existing ones. The experiment requires no new training—only re-evaluating the paper's existing models and token replacement data with difficulty binning.

Applying Δlog p-guided replacement and extrapolation to code generation RLVR to test domain generality. The paper's token replacement experiment (Figure 2) and extrapolation method (Figure 4) are evaluated exclusively on mathematical reasoning, where the verifier provides a binary correctness signal and reasoning chains follow constrained logical structures. A strong generalizability test would replicate the experiment on a code generation RLVR model (e.g., trained on APPS, CodeContests, or LiveCodeBench via GRPO/DAPO with compiler-based verifiers). The key open question is whether Δlog p's efficiency advantage over magnitude metrics (reaching RLVR accuracy with ~10% vs. 15-30% replacement in math) transfers to code, where critical tokens might be more diffusely distributed (a single wrong variable name, off-by-one index, or missing edge-case check can break correctness, and these span many positions rather than concentrating in a few). The experiment would train or obtain a paired base-and-RLVR code model, run the token replacement protocol on HumanEval or LiveCodeBench, and compare replacement-efficiency curves. A null result (entropy or KL divergences catch up or surpass Δlog p in code) would reveal that the directional diagnostic is domain-specific, and would motivate domain-specific analysis metrics.

Combining Δlog p-guided test-time extrapolation with majority voting or best-of-N selection to test whether the methods are complementary or redundant. The paper's extrapolation method is evaluated at 32 samples (Avg@32) and the RLVR baseline is also evaluated at 32 samples. A natural question is whether extrapolation's gains are orthogonal to the gains from additional parallel sampling. The experiment would produce 64 samples from the RLVR model and 64 samples from the extrapolated policy (at matched compute, accounting for the ~2× cost of extrapolation), then compare majority-vote accuracy at various sample budgets. If extrapolation's gains disappear when majority voting is applied, the method is effectively a more sample-efficient way to achieve the same result (useful, but not additive). If gains persist or compound, the Δlog p direction captures reasoning improvements that parallel sampling cannot replicate. This experiment would use the paper's existing models and extrapolation hyperparameters (Table 4), and would directly inform deployment decisions.

Practical Applications and Downstream Use Cases

Cost-efficient batch inference for mathematical reasoning evaluation at scale. Organizations that run large-scale mathematical reasoning evaluations (e.g., automated grading systems, math tutoring platforms, or benchmark evaluation pipelines) face an accuracy-vs-cost tradeoff: the RLVR model solves ~52% of AIME-24 problems (DAPO-32B, Avg@32), but generating 32 samples per problem is computationally intensive. The paper's selective extrapolation offers a concrete path to higher accuracy at a fixed budget: at 32 samples, extrapolation provides +2.92 Avg@32 points over the RLVR model on AIME-24 for DAPO-32B (Figure 4), and the gains hold on broader STEM (Minerva: +1.64 Avg@8, Figure 10). For a batch pipeline evaluating thousands of problems, a ~3 point accuracy improvement from a test-time wrapper (no retraining, no data collection) could translate to hundreds of additional correctly solved problems. The practical tradeoff is the ~2× inference cost, which for batch processing can be absorbed through parallelism. The reduced selection threshold τ = -0.3 to -0.4 (Table 4) makes the method relatively insensitive to exact tuning within a range of 10-11% replacement rate, simplifying deployment.

Improved RLVR training recipes with minimal code changes. The training-time reweighting method (Equation 8) modifies a single line of the DAPO advantage computation: multiply the advantage by 1 + α(1 - π_θold(y_{i,t})). This is a drop-in change to any DAPO (or GRPO) training code that has access to per-token old-policy probabilities (which DAPO already computes for the importance sampling ratio). The paper demonstrates consistent gains on Qwen2.5-Math-7B (+1.63 average Avg@32, +4.47 Pass@16, Table 2) and Qwen3-8B-Base (+2.52 average Avg@32, +3.33 Pass@16) across three benchmarks. For a team already running DAPO training, the cost of adopting this method is a single α hyperparameter sweep (the paper uses α=0.2 for Qwen2.5, α=0.1 for Qwen3). The gains do not come at the cost of exploration (Pass@16 improves), and training stability is maintained (Figure 5: entropy remains healthy, Figure 9: all 4 reproducibility runs converge). The reweighting method is strictly better than the Dominate alternative tested in Table 3, which sacrifices Pass@16 (-8.7 points vs. the paper's method) for a similar Avg@32, and the PPL alternative, which is less consistent across benchmarks (AIME25 drops to 16.46 vs. 18.54 for the paper's method).

Diagnostic toolkit for RLVR model development and debugging. Teams developing RLVR-trained models currently lack a standard diagnostic for whether training is producing the right kind of changes. The paper's Δlog p histogram protocol (Section 3.1) provides a concrete, reproducible check: after training, generate 32 responses from both base and RLVR models on a held-out set, compute per-token Δlog p, and examine whether the histograms show clear bimodal separation (positive tail for RLVR-generated tokens, negative tail for base-generated tokens). An RLVR training run that fails to produce this bimodality—e.g., one where Δlog p is symmetric and centered at zero for both model's generations—has likely failed to learn directional, verifier-aligned updates, even if magnitude metrics (KL divergence, entropy reduction) look normal. The token replacement experiment (Algorithm 1, Figure 2) provides a quantitative benchmark: a well-trained model should reach RLVR accuracy with ≤15% Δlog p-guided token replacements. If substantially more replacements are needed, the training run likely learned diffuse rather than sparse directional updates. This diagnostic requires no labeled data beyond the base and RLVR model weights, and the computational cost (~1,000 generations for a 30-problem diagnostic set) is negligible compared to the training run itself.

Targeted model patching for reasoning errors without full retraining. The token replacement experiment (Figure 2) demonstrates that replacing only ~10% of a base model's tokens with RLVR choices recovers full RLVR accuracy. This suggests a model-patching workflow: identify a set of reasoning problems where the base model consistently fails, diagnose the specific token positions where Δlog p is most negative (i.e., where the base model's token choices diverge most from RLVR's learned preferences), and apply targeted test-time correction only at those positions. Unlike full RLVR retraining, which requires thousands of GPU-hours, this approach requires only the base and RLVR model weights plus targeted inference. The practical scenario is a model provider that has trained a general RLVR model but deploys a smaller, cheaper base model for most queries; when a query is flagged as high-stakes or difficult, the system can selectively route token decisions to the (more expensive) RLVR model at the most critical positions identified by Δlog p, providing RLVR-level accuracy at a fraction of the cost. The paper's ~10% replacement figure provides the quantitative foundation: the cost overhead would be approximately 10% of the full RLVR inference cost rather than 100%, assuming the Δlog p computation can be amortized or predicted (per the auxiliary predictor follow-up above).