Sparse but Critical: A Token-Level Analysis of Distributional Shifts in RLVR Fine-Tuning of LLMs

🎯 Pitch

RLVR fine-tuning for reasoning is not a global rewrite of the model—inserting RL tokens at fewer than 4% of positions into base-model generations recovers nearly all performance gains, while swapping them back collapses performance entirely. These critical tokens act like targeted edits that redirect reasoning without introducing novel vocabulary, primarily re-ranking candidates the base model already considered.

1. Executive Summary

This paper presents a systematic empirical study analyzing how reinforcement learning with verifiable rewards (RLVR) reshapes token-level predictive distributions in large language models, using Qwen2.5-32B with DAPO and SimpleRL (GRPO) training on the AIME 2024 and AIME 2025 benchmarks. The authors characterize RLVR as a sparse, targeted refinement process that induces meaningful distributional shifts at only a small fraction of token positions — fewer than 4% of tokens suffices to recover full RL performance gains when injected into base-model generations (forward cross-sampling), while reverting a similarly small fraction of RL tokens back to base-model choices collapses performance to base levels (reverse cross-sampling). They further demonstrate that even at these high-divergence positions, RLVR primarily reallocates probability mass within an existing candidate set rather than introducing novel tokens, with over 80% of RL top-1 tokens already ranked in the base model's top-3 — establishing that RLVR acts as a targeted editing mechanism that steers reasoning trajectories through sparse, high-impact local decisions, rather than globally rewriting model behavior.

2. Context and Motivation

The Core Problem: We Don't Understand How RLVR Changes Model Behavior at Inference Time

The fundamental question this paper tackles is deceptively simple: when reinforcement learning with verifiable rewards (RLVR) improves an LLM's reasoning ability, what exactly changes inside the model? This matters because RLVR — encompassing methods like GRPO (Shao et al., 2024) and DAPO (Yu et al., 2025) — has become the de facto standard for post-training reasoning improvements, driving models like DeepSeek-R1 and Qwen's math-specialized variants to state-of-the-art performance on competition-level benchmarks. Yet despite this empirical success, the mechanism through which RLVR modifies model behavior remains largely opaque.

Current evaluation of RL fine-tuning focuses overwhelmingly on aggregate response-level metrics: accuracy, average reward, response length, pass@k. The paper explicitly calls out this gap in Section 1:

"Most evaluations of RL fine-tuning focus on aggregate response-level metrics such as accuracy, reward, and response length. While informative, these provide only a high-level view of improvement and offer limited insight into the mechanisms by which model behavior changes."

This gap is significant for several reasons. First, without understanding how RLVR works, practitioners cannot reliably diagnose failures, anticipate when fine-tuning will succeed or fail, or design better training objectives. Second, the lack of mechanistic understanding makes it difficult to distinguish between genuine reasoning improvements and superficial pattern matching that may not generalize. Third, as RLVR is increasingly deployed in production systems, interpretability becomes crucial for safety and reliability — we need to know why a model made a particular decision, not just that it got the right answer.

Where Prior Approaches Fall Short

The paper identifies several strands of prior work that touch on aspects of this question but leave the central gap unaddressed.

High-level comparisons between RL and SFT. A growing body of work argues metaphorically that RL fine-tuning acts as a "scalpel rather than a hammer" (Rajani et al., 2025; Chu et al., 2025; Shenfeld et al., 2025), amplifying existing capabilities through localized changes rather than making broad modifications like supervised fine-tuning. These studies provide evidence at the level of domain-transfer performance, catastrophic forgetting patterns, and overall KL divergence from the base model. However, as the paper notes, they do not provide a token-level distributional view of change — they cannot tell us which specific tokens are being modified, how probability mass is being reallocated, or whether the changes that matter for reasoning are concentrated or diffuse.

Token-level analyses that remain incomplete. Several recent works have begun examining RLVR through token-level statistics:

Wang et al. (2025) attribute a substantial portion of RL gains to "high-entropy minority tokens" — tokens where the base model's predictive distribution has high uncertainty — and argue these positions are disproportionately important for learning.
Cheng et al. (2025) connect high-entropy tokens to exploratory reasoning steps, suggesting that RLVR encourages the model to consider alternative reasoning branches at points of uncertainty.
Cui et al. (2025) highlight entropy collapse risks (where RL training makes the model overly deterministic) and propose token-level regularization mechanisms to counteract this.
Huan et al. (2025) analyzes RL-induced changes using token-level KL divergence and rank-shift statistics, providing quantitative measures of how individual token predictions change.
Chen et al. (2026) shows that RL training modifies a sparse subset of tokens when viewed through rank-shift statistics and develops a theoretical analysis based on reasoning patterns.

While these works provide valuable insights, the paper argues they leave an incomplete picture. Specifically, prior token-level analyses answer where and how much change occurs (e.g., at high-entropy tokens) but do not address a cluster of deeper questions:

Distribution structure: How are shifts structured across positions and contexts? Is entropy the right lens, or do low-entropy changes also matter? How does probability mass get reallocated across candidate tokens at modified positions?
Functional importance: Which token-level changes actually drive downstream reasoning gains, as opposed to being epiphenomenal? Correlating divergence with gains is not the same as establishing causation.
Mechanism of change: At positions where distributions shift, does RLVR introduce fundamentally new candidate tokens, or does it primarily reorder and amplify tokens that were already plausible under the base model?
Training dynamics: How do these shifts evolve over the course of RL training? Do they appear early and stabilize, or emerge late in training?

The paper's explicit framing of this gap appears in the introduction:

"a more detailed distributional view of change remains missing: how such shifts are structured across positions and contexts, how probability mass is reallocated across candidate tokens, how they evolve over training, and to what extent they are responsible for RLVR's performance gains."

The Conflicting Signals in Prior Evidence

The paper is also motivated by a subtle tension in existing findings. On one hand, the "scalpel not hammer" narrative suggests RLVR makes small, targeted changes. On the other hand, work on entropy collapse (Cui et al., 2025) and the observation that certain RLVR methods like DAPO remove KL regularization entirely suggest that training could be making large, unconstrained modifications to the policy. Which is it?

This tension is resolved by the paper's central empirical finding — that RLVR induces highly sparse, targeted shifts even without KL regularization — but prior to this work, no study had directly measured the distribution and functional impact of token-level changes at the granularity needed to reconcile these perspectives.

How This Paper Positions Itself

The authors frame their contribution as a systematic empirical study organized around three complementary analyses (Section 1), each addressing a different facet of the central question:

Token-level characterization (Section 2): What is the structure of distributional shifts? Through JS divergence histograms, percentile curves, positional concentration analysis, entropy-divergence relationships, and semantic identity investigation, the paper builds a comprehensive picture of where and what kind of changes RLVR induces.
Functional importance through cross-sampling (Section 3): Are these sparse changes actually responsible for performance gains? The forward and reverse cross-sampling experiments are the paper's key methodological innovation, directly testing causality by selectively swapping token choices between base and RL models and measuring the impact on downstream accuracy.
Fine-grained distribution mechanics (Section 4): At positions that do change, how does probability mass move? Through top-k overlap analysis, rank reordering statistics, low-probability token behavior, and training evolution tracking, the paper shows that RLVR refines within existing candidate sets rather than introducing novel tokens.

Importantly, the paper does not propose a new RLVR training method as its primary contribution. Rather, it positions itself as providing a diagnostic toolkit and conceptual framework for understanding existing methods, with the divergence-weighted advantage experiment (Section 5) serving as a proof-of-concept that such understanding can inform better training objectives. The authors are explicit about this framing in the conclusion:

"our work offers a perspective for designing future RL objectives and diagnostics that further incorporate distributional structure, opening avenues for more effective, interpretable, and controllable LLM post-training."

The Cross-Sampling Innovation and Its Conceptual Roots

The cross-sampling framework introduced in Section 3.1 is the paper's most distinctive methodological contribution and deserves close attention because it enables causal claims about the importance of specific token-level changes — something prior correlational analyses could not do.

The key insight is this: if RLVR's gains are truly concentrated in sparse, high-divergence token decisions, then we should be able to achieve two things:

Forward cross-sampling: Generate primarily under the base model, but at positions where the base and RL policies diverge substantially (measured by JS divergence), sample the next token from the RL model instead. If performance approaches RL levels with only a small fraction of such interventions, this demonstrates that the RL model's advantage stems from those specific token choices, and that the base model is otherwise capable of completing reasoning trajectories correctly once steered in the right direction.
Reverse cross-sampling: Generate primarily under the RL model, but at high-divergence positions, revert to the base model's token choices. If performance collapses to base levels with only a small fraction of reversions, this demonstrates that the RL model's gains are fragile — dependent on preserving those specific token-level decisions.

The paper connects this framework to speculative decoding and BiLD (Big Little Decoder; Kim et al., 2023), but the motivation is fundamentally different. Speculative decoding and BiLD use a small model for most steps and fall back to a large model only when the small model is uncertain, with the goal of latency reduction while preserving the large model's output distribution. In contrast, the paper's cross-sampling uses the same model size for both primary and intervention policies (just the base vs. RL-tuned versions of the same architecture) and manipulates intervention budgets specifically to measure functional importance rather than to optimize inference efficiency.

The divergence-based switching rule (intervene when $JS(\pi_{base}(\cdot|x_{<t}) \| \pi_{RL}(\cdot|x_{<t})) > \varepsilon$ ) is also a deliberate choice. By using the same JS divergence that the paper uses to characterize distributional shifts (Section 2), the cross-sampling experiments create a direct link between the descriptive analysis ("only N% of tokens show high divergence") and the causal analysis ("those same high-divergence tokens are the ones that matter for performance"). This unified metric is key to the paper's argumentative coherence.

Why the AIME Benchmark and Qwen2.5 Models?

The paper's choice of experimental setting is deliberate and worth unpacking. The AIME (American Invitational Mathematics Examination) 2024 and 2025 benchmarks are competition-level math problems requiring multi-step symbolic reasoning. This domain is particularly well-suited to studying RLVR's mechanisms because:

Clean verifiability: Math problems have unambiguous correct answers, enabling reliable reward signals and accuracy evaluation. This eliminates confounds from reward model noise that plague RLHF studies.
Multi-step reasoning: Problems require chains of logical deduction where individual token choices (e.g., which variable substitution to make, which theorem to apply) can have cascading effects on downstream reasoning — exactly the kind of trajectory sensitivity the cross-sampling experiments aim to measure.
Non-trivial but improvable: The base Qwen2.5-32B model achieves approximately 8% on AIME 2024 (as shown in Table 1), while RLVR fine-tuning raises this to ~25% (SimpleRL) or ~44% (DAPO). This leaves a large performance gap to attribute, making it easier to isolate the token-level changes responsible.

The Qwen2.5 model family is chosen because it represents a strong but not saturated base model for mathematical reasoning, and because the availability of multiple RLVR variants (DAPO with clip-higher mechanism, SimpleRL with GRPO, and SFT baselines) enables controlled comparisons across training paradigms.

The Unexplored Tension: Sparse Changes vs. Large Performance Gains

Underlying the entire paper is a conceptual puzzle that motivates the investigation. Intuitively, one might expect that moving from 8% to 44% accuracy on AIME 2024 (the DAPO improvement) would require substantial, widespread changes to the model's behavior — the model is getting more than 5× as many problems correct, which feels like it should require a fundamentally different policy. The paper's central discovery — that only ~7.8% of tokens show meaningful divergence, and that injecting these tokens into base-model generations recovers the full gain — challenges this intuition and suggests a more nuanced picture: reasoning performance is bottlenecked by a small number of critical decision points, and the base model is actually capable of completing correct reasoning trajectories if guided past those bottlenecks. This is a specific, testable hypothesis that the cross-sampling experiments are designed to validate.

3. Technical Approach

3.1 Reader Orientation

This paper is a systematic empirical analysis — not a new training method — that builds a diagnostic toolkit for understanding how RLVR fine-tuning modifies an LLM's behavior at the level of individual token predictions. The core problem it solves is the opacity of RLVR's mechanism: we know RLVR improves math reasoning accuracy (often dramatically), but we don't know which token-level decisions changed and which of those changes actually cause the improvement. The solution takes the form of three layered analyses: (1) a descriptive characterization of where and how much token distributions shift between base and RL models, (2) a causal intervention framework (cross-sampling) that isolates which of those shifts are functionally necessary for performance gains, and (3) a fine-grained mechanical analysis of what kind of probability reallocation happens at the shifted positions.

3.2 Big-Picture Architecture (Diagram in Words)

The system comprises five major components, though three are models and two are analysis procedures:

Base model ( $\pi_{\text{base}}$ ) — the pretrained LLM (primarily Qwen2.5-32B) before any RL fine-tuning. Serves as the reference point against which all distributional shifts are measured.
RL-fine-tuned model ( $\pi_{\text{RL}}$ ) — the same architecture after RLVR training with either DAPO or SimpleRL (GRPO). This is the model whose improved reasoning behavior we aim to explain.
SFT baseline model — a supervised fine-tuned version of the same base model, trained on identical data, used as a control to determine whether observed sparsity patterns are specific to RLVR or generic to any fine-tuning.
Distributional analysis pipeline (Section 2) — takes RL-generated sequences as reference trajectories, evaluates both $\pi_{\text{base}}$ and $\pi_{\text{RL}}$ at every token position, computes token-level JS divergence, entropy, and positional statistics at each position, and aggregates these into histograms, percentile curves, and binned analyses.
Cross-sampling intervention engine (Section 3) — a controlled generation procedure that mixes $\pi_{\text{base}}$ and $\pi_{\text{RL}}$ during decoding according to a divergence-thresholded switching rule, with an adjustable intervention budget (maximum number of swaps), used to measure the causal impact of high-divergence token decisions on downstream accuracy.
Fine-grained mechanics analyzer (Section 4) — examines, at positions where divergence is high, what happens to top-k candidate overlap, token rank ordering, tail probability behavior, and how these statistics evolve over training checkpoints.

Information flows as follows: a prompt enters → the RL model generates a complete response (the "reference trajectory") → for each token position in that trajectory, both base and RL next-token distributions are computed → JS divergence and entropy are calculated per position → these per-position statistics are aggregated across the dataset for the descriptive analyses (Section 2) → the same divergence thresholds are used to define intervention points for cross-sampling experiments (Section 3) → at the intervened positions, the mechanics of probability reallocation are examined (Section 4).

3.3 Roadmap for the Deep Dive

First, the formal framework for comparing distributions: how next-token distributions are evaluated, why Jensen-Shannon divergence is chosen over KL, and how the top-p truncation interacts with divergence computation.
Second, the descriptive characterization pipeline: how divergence histograms, percentile curves, positional concentration, entropy-divergence relationships, and semantic identity analyses are constructed from per-token statistics.
Third, the cross-sampling framework in full detail: the mathematical formulation of mixed policies, the switching rule, forward vs. reverse interventions, intervention budgets, and the connection to speculative decoding.
Fourth, the fine-grained mechanics tools: top-k overlap measurement, rank reordering analysis, low-probability token tracking, and training evolution analysis via intermediate checkpoints.
Fifth, the divergence-weighted advantage experiment as a diagnostic probe and the SFT comparison setup as a control.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical analysis paper whose core idea is that RLVR induces sparse, targeted distributional shifts at the token level, and that these sparse shifts — not global policy changes — are causally responsible for reasoning improvements. The authors build a multi-layered diagnostic framework to characterize, causally validate, and mechanistically explain these shifts.

The Core Distributional Comparison Framework

The foundation of all analyses in the paper is a direct token-level comparison between the base model's next-token distribution and the RL-fine-tuned model's next-token distribution, evaluated on identical sequence contexts. The authors make a deliberate methodological choice: rather than having each model generate its own sequences independently (which would compare distributions over different contexts), they use the RL model's generated sequences as reference trajectories and evaluate both models at every token position along those trajectories. This means the prefix $x_{<t}$ is always the same for both models at position $t$ — it is the prefix that the RL model actually generated. The base model's distribution $\pi_{\text{base}}(\cdot \mid x_{<t})$ therefore answers the counterfactual question: "if the base model were at this exact point in the RL model's reasoning trajectory, what would it predict next?"

This design choice matters for several reasons. First, it ensures that observed divergences are not confounded by the models taking different paths through the reasoning space (which would make it hard to tell whether divergence at position $t$ is due to a true policy difference or simply due to being in a different context). Second, it aligns the analysis with the cross-sampling experiments (Section 3), where the same context-matching logic applies: interventions happen at positions where, given the same prefix, the two models would make different predictions. Third, it provides a natural interpretation of what RLVR does: it identifies positions along the RL model's own trajectories where the base model would have made a different decision.

Formal definition of per-token divergence. At each token position $t$ with prefix $x_{<t}$ , the authors compute the Jensen-Shannon divergence between the two conditional distributions:

$JS(x_{<t}) = D_{JS}(\pi_{\text{base}}(\cdot \mid x_{<t}) \parallel \pi_{\text{RL}}(\cdot \mid x_{<t})) = \frac{1}{2} D_{KL}(\pi_{\text{base}}(\cdot \mid x_{<t}) \parallel M_t) + \frac{1}{2} D_{KL}(\pi_{\text{RL}}(\cdot \mid x_{<t}) \parallel M_t)$

where $M_t = \frac{1}{2}(\pi_{\text{base}}(\cdot \mid x_{<t}) + \pi_{\text{RL}}(\cdot \mid x_{<t}))$ is the midpoint distribution, and $D_{KL}$ is the Kullback-Leibler divergence.

What it computes: Jensen-Shannon divergence measures the distance between two probability distributions over the vocabulary $V$ . It first constructs the average distribution $M_t$ (element-wise mean of the two probability vectors), then computes the KL divergence from each original distribution to this average, and finally takes the arithmetic mean of those two KL divergences. The result is a single scalar in $[0, \log 2]$ (approximately $[0, 0.693]$ with natural log) that is zero when the distributions are identical and approaches $\log 2$ when they are maximally different (disjoint support).

Why this form over KL divergence: The authors provide three specific justifications for choosing JS over KL (Section 2.1). First, JS divergence is symmetric: $D_{JS}(P \parallel Q) = D_{JS}(Q \parallel P)$ , whereas KL divergence depends on which distribution is the reference. This matters because the authors compare base and RL models without a privileged direction — there is no reason to treat one as the "true" distribution and the other as the approximation. Second, JS divergence is bounded in $[0, \log 2]$ , which prevents extreme values from dominating aggregate statistics like mean divergence across sequences. KL divergence is unbounded above, so a small number of positions where the two distributions have near-disjoint support could produce arbitrarily large values and skew averages. Third, JS divergence remains well-defined even when the distributions lack absolute continuity — specifically, when using top-p truncated distributions (where probability mass is zeroed out for tokens below the nucleus), KL divergence may be undefined because $\pi_{\text{RL}}(v \mid x_{<t})$ could be non-zero for a token $v$ where $\pi_{\text{base}}(v \mid x_{<t}) = 0$ , making $\log(0/\text{non-zero})$ undefined. JS divergence avoids this issue because the midpoint distribution $M_t$ assigns non-zero probability wherever either original distribution does.

Top-p truncation and its interaction with divergence. Unless otherwise stated, all divergences in the paper are computed on top-p truncated distributions using the same sampling configuration employed during generation (nucleus sampling with $p = 0.7$ and temperature $= 1$ ). Top-p truncation works by sorting tokens by probability, accumulating from highest to lowest, and keeping only those tokens whose cumulative probability is $\leq p$ ; all remaining tokens have their probabilities set to zero (with the distribution renormalized). The authors explicitly state (Section 2.1) that this choice is made so that "comparisons reflect the models' effective differences under the actual sampling regime." In other words, if the model would never actually sample tokens below the nucleus threshold during generation, those tokens' probabilities are irrelevant to the behavioral difference between models.

To verify that truncation does not distort the findings, the authors provide robustness checks in Appendix A.5 (Figures 30 and 32) computing JS divergence with both the truncated and estimated full distributions ( $p = 1$ ), finding that "the sparsity patterns remain consistent across different top-p settings." This is an important validation because truncation could theoretically hide divergence in the tail of the distribution.

Entropy and probability computation. Critically, while divergence is computed on truncated distributions, entropy and individual token probabilities are computed from the full estimated distributions (Section 2.1). This means the entropy value $H_{\pi}(x_{<t})$ at a position reflects the model's complete uncertainty, including tokens that would be excluded by nucleus sampling. The justification is that entropy "grounds" the analysis in the model's complete output distribution even though the effective sampling behavior only considers the nucleus. This asymmetry — truncated divergence, full-distribution entropy — is a deliberate design choice that reflects different goals: divergence measures what differs in practice, entropy measures what the model "knows."

The Descriptive Characterization Pipeline (Section 2)

The first major analysis component characterizes the structure of distributional shifts without yet asking whether they matter for performance. Given a set of RL-generated sequences (32 responses per problem, sampled with temperature = 1, top-p = 0.7), the pipeline computes per-token JS divergence, entropy, and positional information for every token position, then aggregates using several complementary views.

Divergence histogram and percentile curves (Figure 2). For a given model pair (e.g., Qwen2.5-32B base vs. DAPO-fine-tuned) and dataset (e.g., AIME 2024), the authors collect all per-token JS divergence values across all generated sequences into a single distribution. They then plot:

A log-scaled histogram showing the frequency of token positions at each divergence level. The log scale is crucial because the distribution is extremely skewed — most positions cluster near zero, and a linear scale would make it impossible to see the tail.
A percentile curve that plots divergence value against cumulative percentile, making it easy to read statements like "X% of tokens have divergence below Y."

From these plots, the paper extracts key quantitative claims: under DAPO, "more than 83% of token positions exhibit near-zero divergence," while under SimpleRL, "this proportion exceeds 98%." The "near-zero" threshold is not formally defined here but can be inferred from the histogram spike at zero — it means divergence values indistinguishable from zero given the estimation precision.

Positional concentration analysis (Figure 3). To understand where in a sequence divergence tends to occur, the authors aggregate divergence as a function of normalized token position — the token index divided by the total sequence length, so that position 0.0 is the start of the response and position 1.0 is the end. For each normalized position bin, they compute the mean and median divergence across all sequences that have a token at that relative position, along with percentile bands (e.g., 25th–75th percentile) showing the spread.

The normalization by sequence length is essential because different problems produce responses of widely varying lengths; without normalization, positional patterns would be washed out by length variation. The percentile bands reveal that even when the mean divergence is low in the middle of sequences, individual sequences can still have high-divergence positions scattered throughout — a point the paper emphasizes by noting the "wide percentile spread" and the example trajectories in Figure 1a and 1b.

Divergence–entropy relationship (Figure 4 and Appendix Figures). This analysis asks: are high-divergence positions associated with high or low model uncertainty? The procedure is:

Binarize positions into low-divergence ( $JS < 0.1$ ) and high-divergence ( $JS > 0.1$ ) bins.
Within each bin, plot the distribution of token-level entropy for both the base model and the RL model separately (kernel density estimates or histograms).
Overlay the two entropy distributions to see whether the RL model increases or decreases entropy at these positions.

The threshold 0.1 for "high divergence" is a pragmatic choice that the authors use consistently throughout the paper for binarization purposes. It is not derived theoretically but serves as a reasonable operational split between "essentially unchanged" and "meaningfully different."

The interpretation of these plots (described in the paper's text) reveals a key pattern: low-divergence positions are overwhelmingly low-entropy (both models are confident and agree), while high-divergence positions span the full entropy range under DAPO but concentrate in high base-entropy regions under SimpleRL. This difference is attributed to DAPO's clip-higher mechanism, which allows the model to modify even initially confident (low-entropy) predictions.

Semantic identity analysis (Figure 5). To understand which types of tokens appear at high-divergence positions, the authors construct word clouds where token size is proportional to frequency. Tokens are partitioned into those sampled from high-divergence distributions ( $JS > 0.1$ ) vs. low-divergence distributions ( $JS < 0.01$ ). However, the paper immediately cautions against over-interpreting token identity, using the word "the" as an example: it appears among the most frequent high-divergence tokens, yet its full divergence distribution across all occurrences is "overwhelmingly concentrated in the lower regime" (Figure 23). This motivates the paper's conclusion that "token identity alone is insufficient to characterize divergence" and that context is essential — the same token type can be a critical decision point in one context and routine in another.

SFT comparison as a control. The paper replicates the same analyses on an SFT-fine-tuned version of Qwen2.5-32B, trained on 42k instances from the AM-DeepSeek-R1-Distilled-1.4M dataset for 5 epochs with full parameter fine-tuning using DeepSpeed ZeRO-3. This serves as a control to test whether the sparsity pattern is unique to RLVR or a generic property of any fine-tuning. The SFT model shows "substantially larger high-divergence set and a broader divergence distribution" (Figure 6, 12), demonstrating that RLVR's extreme sparsity is not simply a consequence of fine-tuning — it is specific to the RL objective.

Cross-model and cross-dataset validation. The paper extends the descriptive analysis to Qwen2.5-Math-7B (with two DAPO variants at different clip-upper settings of 0.28 and 0.2), Qwen3-8B-Base with DAPO, and Mistral-Small-24B with SimpleRL, evaluated on AIME 2024, AIME 2025, GPQA-Diamond, and the models' respective fine-tuning datasets. These extensions (Appendix A.4, A.5) "confirm that our findings generalize across models, datasets, and training paradigms."

The Cross-Sampling Intervention Framework (Section 3)

This is the paper's key methodological innovation and the component that elevates the analysis from descriptive to causal. The idea is to create a mixed decoding policy that blends $\pi_{\text{base}}$ and $\pi_{\text{RL}}$ during autoregressive generation, selectively choosing which model to sample from at each token position based on their local divergence, and then measuring how reasoning accuracy changes as a function of how many interventions are performed.

Formal framework: mixed policies and switching rules. Let $(X_t)_{t \geq 1}$ be the sequence of tokens generated during decoding, with stopping time $\tau = \inf\{t \geq 1 : X_t = \text{EOS}\} \land T_{\text{max}}$ , where $T_{\text{max}}$ is set to 8000 tokens (the generation budget used throughout the paper; see Table 1). The primary policy $\pi_{\text{prim}}$ governs generation by default, while the intervention policy $\pi_{\text{int}}$ is used only at selected positions.

A switching rule $S : V^{<\mathbb{N}} \to \{0, 1\}$ is a binary function that, given the prefix $x_{<t}$ , decides whether to sample from the intervention policy ( $S = 1$ ) or the primary policy ( $S = 0$ ). The mixed policy is then:

$X_t \sim \pi_{\text{mix}}^{\text{(prim, int)}}(\cdot \mid x_{<t}) = (1 - S_t) \pi_{\text{prim}}(\cdot \mid x_{<t}) + S_t \pi_{\text{int}}(\cdot \mid x_{<t})$

where $S_t = S(x_{<t})$ .

What it computes: This is a stochastic mixture at each step — with probability depending on the switching rule's output (which is deterministic given the prefix), the next token is drawn either entirely from the primary policy or entirely from the intervention policy. This is not a probabilistic blend of the two distributions (which would average their probabilities). Instead, at each position, exactly one model is responsible for generating the token.

Why this form: The authors chose a hard-switching mechanism rather than a soft mixture (e.g., sampling from a weighted combination of the two distributions) because the goal is to isolate the decision-making difference between the models. A soft mixture would blur the distinction and make it impossible to attribute performance changes to specific token choices. Hard switching means that when the intervention model is used, the token is what that model would have chosen — all probability mass from the primary model is ignored.

Divergence-based switching rule. The switching rule is defined directly in terms of the Jensen-Shannon divergence between the two models' distributions at the current prefix:

$S(x_{<t}) = \mathbf{1}\{D_{JS}(\pi_{\text{prim}}(\cdot \mid x_{<t}) \parallel \pi_{\text{int}}(\cdot \mid x_{<t})) > \varepsilon\}$

where $\varepsilon \geq 0$ is a fixed threshold. In plain language: if the two models would make substantially different predictions at this point (divergence above $\varepsilon$ ), use the intervention model; otherwise, use the primary model.

This rule creates a direct operational link between the descriptive analysis (which identified sparse high-divergence positions) and the causal analysis. The same JS divergence statistic that was used to characterize "only N% of positions show high divergence" now defines which positions receive interventions.

Forward cross-sampling. In the forward direction, $\pi_{\text{prim}} = \pi_{\text{base}}$ and $\pi_{\text{int}} = \pi_{\text{RL}}$ . The generation begins with the base model as default, but at positions where the RL model would have made a substantially different prediction (high JS divergence), the RL model's token is sampled instead. After that intervention, generation continues under the base model until the next high-divergence position or termination. This tests the hypothesis: can the base model complete correct reasoning trajectories if it is "nudged" at critical decision points to follow the RL model's choices?

Reverse cross-sampling. In the reverse direction, $\pi_{\text{prim}} = \pi_{\text{RL}}$ and $\pi_{\text{int}} = \pi_{\text{base}}$ . The generation follows the RL model by default, but at high-divergence positions, the base model's token is sampled instead. This tests the hypothesis: are the RL model's gains fragile — dependent on maintaining its specific token choices at exactly those positions where it differs from the base model?

Intervention budgets and progressive steering. To study how performance changes gradually with the number of interventions, the authors introduce an intervention budget $k$ that limits the total number of cross-sampling swaps. The switching rule is modified to enforce:

$S_t = 0 \quad \text{for all } t \text{ such that } \sum_{s=1}^{t-1} S_s \geq k$

This means that after $k$ interventions have been performed, all subsequent tokens are sampled from the primary policy regardless of divergence. The paper then measures accuracy for a fixed threshold $\varepsilon$ as $k$ increases from 0 to some maximum, producing the accuracy curves in Figures 7 and 8.

Why progressive measurement matters. If RLVR's gains were due to a single critical token decision (a "butterfly effect" where one choice determines the entire trajectory), we would see a sharp, non-monotonic jump in accuracy at some intervention count. If the gains are due to many small, cumulative improvements, we would see a smooth increase. The paper finds the latter — "accuracy improves steadily as additional RL-sampled tokens are introduced, with no sharp threshold" — which supports the interpretation that RLVR's gains are "sparsely distributed across multiple decision points rather than requiring all RL-induced changes."

Effective token counting. The paper carefully distinguishes between "cross-sampled tokens" and "effective tokens." A cross-sampled token is one where the switching rule triggered and the intervention policy was used. However, sometimes the intervention policy and primary policy would have sampled the same token anyway (an "identity swap"). The paper excludes these identity swaps when reporting effective token counts and percentages (Table 1), presenting both the effective and total numbers. For example, on AIME 2024 with SimpleRL forward cross-sampling, "3.86% effective tokens" corresponds to 38 effective token substitutions per response on average, with a total of 75 cross-sampling events (meaning roughly half of the interventions resulted in a different token than the primary policy would have chosen). Token percentages are "computed at the sequence level" — that is, as a fraction of the average sequence length.

Choice of divergence thresholds. The thresholds $\varepsilon$ used for cross-sampling are model- and direction-specific and are provided in Appendix A.2.2. For example:

Qwen2.5-32B SimpleRL on AIME 2024: forward $\varepsilon = 0.03$ , reverse $\varepsilon = 0.05$
Qwen2.5-32B DAPO on AIME 2024: forward $\varepsilon = 0.08$ , reverse $\varepsilon = 0.06$

These thresholds were likely selected empirically to produce a manageable number of intervention points (enough to span a range of budgets, not so many that every position triggers). The paper does not describe an explicit threshold selection procedure, but the variation across configurations reflects the different divergence scales (DAPO produces higher overall divergence than SimpleRL, hence higher thresholds).

Connection to speculative decoding and BiLD. The paper explicitly notes that the cross-sampling framework is "conceptually related" to speculative decoding (Leviathan et al., 2023) and BiLD (Kim et al., 2023), but identifies three key differences. First, speculative decoding uses a small model as primary and a large model as intervention to reduce latency while preserving the large model's exact output distribution through rejection sampling; cross-sampling uses same-size models and does not attempt to preserve any particular target distribution — it deliberately creates a new mixed distribution to probe functional importance. Second, BiLD triggers fallback based on small-model confidence (max-probability threshold) and uses rollback based on cross-entropy discrepancy, whereas cross-sampling triggers on JS divergence and investigates how many interventions are needed rather than optimizing for efficiency. Third, BiLD proposes prediction alignment by fine-tuning the small model on large-model outputs to reduce disagreements, whereas cross-sampling intentionally exploits the remaining disagreements as the object of study.

Sequence-level divergence bounds (Appendix A.1). The paper provides formal bounds relating the token-level switching rule to sequence-level divergences between the mixed policy and the intervention policy. These bounds serve as theoretical grounding for the empirical observation that a small number of token-level interventions can bring the mixed policy close to the intervention policy in distribution. Specifically, Proposition A.2 shows that when the switching rule uses a token-level KL threshold $\varepsilon$ , the sequence-level KL divergence is bounded by $\varepsilon \cdot \mathbb{E}[N_0]$ where $N_0$ is the number of non-intervention steps. In plain language: the total divergence from the mixed policy to the intervention policy is at most the per-step divergence bound times the expected number of steps that didn't use the intervention policy. Since most steps have low divergence (Section 2), $\varepsilon$ can be small, and the bound is tight in practice.

Fine-Grained Distribution Mechanics (Section 4)

Once the cross-sampling experiments establish that high-divergence positions are causally important, the natural next question is: what kind of change happens at those positions? Section 4 provides three complementary lenses on this question.

Top-k overlap analysis (Figure 9 and 22). At each high-divergence position ( $JS > 0.1$ ), the authors extract the top-k tokens (ranked by probability) from both the base and RL models, then compute the fraction of overlap:

$\text{Overlap}_k = \frac{|\text{top-}k(\pi_{\text{base}}) \cap \text{top-}k(\pi_{\text{RL}})|}{k}$

This is computed for $k \in \{1, 2, 3, 4, 5, 10, 20, 50\}$ and then averaged across all high-divergence positions. The procedure is repeated for multiple divergence thresholds (0.01, 0.05, 0.1, 0.2) to show that the pattern holds across different stringency levels.

What it measures: Top-k overlap tells us whether the set of tokens considered plausible is similar between the two models, regardless of their exact ranking within that set. High overlap at $k = 1$ would mean the models agree on the single most likely token; high overlap at $k = 50$ means the broader candidate pool is similar even if the top choice differs.

The key pattern: Overlap jumps sharply from $k = 1$ to $k = 2$ and reaches 80%+ by $k = 3$ for SimpleRL. This means that at most high-divergence positions, the RL model's new top-1 token was already the base model's #2 or #3 choice — it was promoted rather than invented.

Rank reordering analysis (Figure 22 and Appendix). For each high-divergence position, the authors take the RL model's top-3 tokens and look up their ranks in the base model's sorted distribution. They then plot the distribution of these "base-model ranks" across all high-divergence positions. For example, if the RL model's top-1 token is "therefore" and "therefore" was the base model's 3rd-ranked token, that contributes a value of 3 to the distribution.

The paper reports that "around 30% of RL top-1 tokens are already ranked first under the base model" (i.e., the models agree on the top choice even at positions classified as high-divergence — this seems paradoxical but occurs because divergence is a distributional measure, not just a top-1 measure; the models can agree on the argmax while disagreeing on the full distribution shape), and "over 80% (DAPO) and 90% (SimpleRL) fall within the base top-3."

Low-probability token behavior (Figure 10 and Appendix). This analysis asks: at high-divergence positions, does the RL model's new top choice come from tokens that were highly unlikely under the base model? For each high-divergence position, the authors take the RL model's top-1 token and record its probability under the base model. They then plot the cumulative fraction of such positions where this base probability falls below a threshold, sweeping the threshold from 0 to 1.

The key finding is that "under DAPO, only about 5% (among high-divergence positions) of RL top-1 tokens have base probability below 0.01, while under SimpleRL this fraction is nearly zero." This is strong evidence that RLVR selects from tokens the base model already considered plausible, rather than "inventing" new candidates.

Training evolution analysis (Figure 11). Using intermediate checkpoints from DAPO training on Qwen2.5-Math-7B (200 total steps), the authors track how the divergence distribution evolves. The critical design choice is that they condition on the final model's outputs as reference trajectories — meaning they evaluate all intermediate checkpoints on the same set of token sequences (the ones the fully-trained model generates). This isolates the effect of policy change from the effect of taking different paths through the reasoning space.

They plot:

JS divergence percentiles (10th, 25th, 50th, 75th, 90th, 95th, 99th) as a function of training step. The widening gap between high and low percentiles indicates increasing concentration of change.
Jaccard overlap between each checkpoint's high-divergence set and the final model's high-divergence set, measuring how stable the identity of divergent positions is over training.

The paper reports that "Jaccard overlap increases gradually before rising sharply near the end of training," suggesting that the specific token positions that will ultimately diverge are not identifiable early in training but crystallize late.

Experimental Configurations and Hyperparameters

Models. The primary analysis uses Qwen2.5-32B as base model with two RLVR variants: DAPO (trained with clip ratios $\epsilon_{\text{low}} = 0.2$ and $\epsilon_{\text{high}} = 0.28$ , no KL penalty, dynamic sampling) and SimpleRL (GRPO with stricter KL constraints). Additional analyses use Qwen2.5-Math-7B, Qwen3-8B-Base, and Mistral-Small-24B. The SFT baseline was trained on 42k instances from AM-DeepSeek-R1-Distilled-1.4M for 5 epochs with full parameter fine-tuning.

Training hyperparameters (Appendix A.2.3). For DAPO training on Qwen2.5-Math-7B: learning rate $1 \times 10^{-6}$ , 10-step warmup with AdamW, no explicit reference-KL penalty, 512 prompts per step with 16 sampled responses each, split into mini-batches of 32 prompts (yielding 16 gradient updates per step). Maximum generation length 8k tokens, overlong-penalty threshold 4k tokens.

Inference configuration (Section 2.1, Appendix A.2.1). All generation uses nucleus sampling with top-p = 0.7 and temperature = 1, implemented via vLLM for efficiency. Divergence computation uses the same top-p truncated distributions, while entropy uses full distributions. On fine-tuning data, top-p = 1 is used to reflect the training sampling distribution.

Divergence-weighted advantage (Section 5, Appendix A.2.3). For the diagnostic experiment, training uses sigmoid-weighted advantages with $s = 0.3$ and direction-specific annealing. In the high-KL boost configuration, $\alpha$ increases linearly from 0 to 50 starting at step 100; in the low-KL boost configuration, $\alpha$ decreases linearly from 0 to -50 starting at step 150. The KL divergence is computed with respect to the old policy using the k3 estimator, and divergences are detached from the computation graph to ensure the weighting only affects the advantage signal, not the gradient through the divergence computation itself.

Evaluation protocol. On AIME 2024 and AIME 2025, the authors sample 32 responses per problem and report Mean@32 accuracy (the fraction of problems for which at least one of the 32 responses is correct, equivalent to pass@1 with 32 samples). This multi-sample evaluation reduces variance from stochastic sampling, which is important when studying token-level effects that may be sensitive to individual sampling decisions.

Cross-sampling thresholds (Appendix A.2.2). The specific thresholds used are listed per configuration:

Qwen2.5-32B SimpleRL AIME 2024: forward $\varepsilon = 0.03$ , reverse $\varepsilon = 0.05$
Qwen2.5-32B SimpleRL AIME 2025: forward $\varepsilon = 0.05$ , reverse $\varepsilon = 0.05$
Qwen2.5-32B DAPO AIME 2024: forward $\varepsilon = 0.08$ , reverse $\varepsilon = 0.06$
Qwen2.5-32B DAPO AIME 2025: forward $\varepsilon = 0.1$ , reverse $\varepsilon = 0.08$
Mistral-Small-24B SimpleRL AIME 2024: forward $\varepsilon = 0.002$ , reverse $\varepsilon = 0.02$

The wide variation in thresholds (from 0.002 for Mistral to 0.1 for DAPO) reflects the different baseline divergence scales across models and training methods. The paper does not describe how these thresholds were chosen, but the cross-sampling curves (scanning across intervention budgets for a fixed threshold) are the primary result, so the threshold choice primarily affects the range of budgets that can be explored — a threshold that is too high would produce too few intervention points to span the curve, while a threshold that is too low would make the effective and total token counts indistinguishable.

Summary of Design Choices and Their Justifications

JS divergence over KL divergence: symmetry, boundedness, and well-definedness under top-p truncation — all three properties are essential for the analysis.
RL-generated sequences as reference trajectories: ensures the two models are compared at identical contexts, eliminating path-dependence confounds and aligning with the cross-sampling logic.
Truncated distributions for divergence, full distributions for entropy: divergence reflects effective behavioral differences under the actual sampling regime; entropy reflects the model's complete uncertainty.
Hard-switching cross-sampling (not soft mixture): enables clean attribution of performance changes to specific token choices — a soft mixture would blur the causal signal.
Progressive intervention budgets (not binary before/after): reveals that performance changes accumulate gradually across multiple decision points, not at a single critical juncture — this is an empirical finding that the design enables discovering.
Intermediate checkpoint analysis conditioned on final-model trajectories: isolates policy change from path change, distinguishing "the model would now make different choices at the same point" from "the model takes different paths so it never reaches that point."
SFT control: demonstrates that the observed sparsity is specific to RLVR, not a generic fine-tuning property — this is essential for the paper's central claim that RLVR is a qualitatively different kind of update.
Multiple models and datasets in appendix: establishes generalizability without cluttering the main narrative — the main text focuses on the clearest exemplars (Qwen2.5-32B on AIME) while the appendix demonstrates robustness.

4. Key Insights and Innovations

Innovation 1: A Causal Framework for Attributing Reasoning Gains to Specific Token-Level Decisions

The paper's most fundamental conceptual contribution is not the finding that RLVR induces sparse changes — several prior works had observed sparsity through rank-shift statistics or overall KL divergence (Huan et al., 2025; Chen et al., 2026). Rather, it is the cross-sampling intervention framework that transforms this descriptive observation into a causal claim: that the specific sparse positions identified are directly responsible for reasoning performance, not merely correlated with it.

Prior to this work, the field's toolkit for analyzing RLVR's mechanisms was limited to correlation-based methods. One could observe that certain token positions show high divergence between base and RL models (Section 2), or that high-entropy tokens are disproportionately important for learning (Wang et al., 2025; Cheng et al., 2025), but one could not rule out the possibility that these correlations were epiphenomenal — that the real drivers of improved reasoning were diffuse changes across many positions, with some showing higher divergence simply because they were more sensitive to measurement. The cross-sampling framework breaks this confound by directly manipulating which model's token choices are used at which positions, then measuring the downstream accuracy impact. This is a genuinely causal methodology applied to a problem where prior work had been limited to observational statistics.

What makes this framework distinctive at the idea level is its operationalization of counterfactual reasoning about generation trajectories. The question "does the base model already know how to solve this problem, and just needs steering at a few critical junctures?" becomes testable through forward cross-sampling: generate primarily under the base model, but at high-divergence positions, substitute RL token choices, then let generation continue under the base model. If accuracy approaches RL levels with only a handful of such substitutions (Table 1: fewer than 4% effective tokens for SimpleRL, fewer than 8% for DAPO), then the base model must be capable of completing correct reasoning trajectories — it just wasn't finding them without the RL model's guidance at those specific decision points. This is a stronger claim than "RL changes are sparse" — it asserts that the base model's capabilities are largely sufficient, and RLVR primarily helps the model access those capabilities through better local decisions.

The complementary reverse cross-sampling direction provides a convergence test: if RL gains are truly concentrated in these positions, then reverting those same positions to base-model choices should erase them. The finding that substituting fewer than 5–10% of effective tokens collapses accuracy from RL levels (~25–44%) to base levels (~5–8%) provides the symmetric verification. The fact that both directions converge on the same conclusion — that a small set of token decisions carries disproportionate weight — substantially strengthens the causal interpretation beyond what either direction alone would establish.

This innovation is fundamental, not incremental. It introduces a new class of diagnostic experiment (controlled mixing of model policies during decoding with divergence-based switching) that can be applied to any pair of models — base vs. fine-tuned, small vs. large, different RL checkpoints — to isolate the behavioral differences that matter. The conceptual connection to speculative decoding and BiLD (Section 3.1) is noted but the motivation is inverted: rather than using a cheaper model to approximate an expensive one, the framework uses two same-architecture models with different training to measure the functional importance of their disagreements. This inversion — from efficiency mechanism to diagnostic tool — is a genuinely novel conceptual move.

The progressive budget experiments (Figures 7, 8) add further depth to this innovation by revealing that performance changes accumulate gradually rather than at a single critical juncture. If accuracy had jumped from base to RL levels at some threshold number of interventions, that would imply a "butterfly effect" where one crucial token choice determines the entire trajectory. The observed smooth, near-monotonic curves instead demonstrate that RLVR's gains are distributed across multiple decision points, each contributing incrementally — a finding that would have been inaccessible without the budget-sweeping methodology.

Innovation 2: RLVR as a Targeted Steering Mechanism Rather Than a Global Policy Rewrite

The paper's second major conceptual contribution is the evidence-based reframing of what RLVR is as a training process. The dominant implicit model in the field — reflected in concerns about entropy collapse (Cui et al., 2025), the removal of KL regularization in methods like DAPO, and the substantial accuracy improvements on benchmarks — was that RLVR might be fundamentally reshaping the model's output distribution, either by making it dramatically more deterministic or by teaching it qualitatively new reasoning strategies. The paper systematically dismantles this assumption and replaces it with a more precise picture: RLVR is a sparse trajectory editor that makes targeted, local modifications to a largely intact base policy.

Several converging lines of evidence support this reframing, each of which would be individually suggestive but which together make a compelling case:

The sparsity of divergence (Section 2.2). The finding that 83%+ (DAPO) to 98%+ (SimpleRL) of token positions exhibit near-zero JS divergence between base and RL models — even for DAPO, which removes KL regularization entirely — establishes that the RL policy is overwhelmingly similar to the base policy at the token-by-token level. This is not a model that has been globally rewritten; it is a model that has been edited at a small fraction of positions. The SFT comparison (Figures 6, 12) is crucial here: SFT produces a substantially broader divergence distribution, confirming that RLVR's sparsity is not simply "fine-tuning makes small changes" but is a distinctive property of the RL objective.

The promotion-not-invention mechanism (Section 4). At the positions that do change, the fine-grained analysis reveals that RLVR almost never promotes tokens that were highly unlikely under the base model. With over 80% of RL top-1 tokens at high-divergence positions already ranked in the base model's top-3 (Figure 22), and fewer than 5% of RL top-1 tokens (DAPO) having base probability below 0.01 (Figure 10), the evidence strongly indicates that RLVR is selecting from an existing candidate set rather than learning to consider new tokens. This is a conceptual reframing of what RLVR "learns" — it is not acquiring new knowledge about which tokens are plausible in mathematical reasoning contexts; it is learning better priority ordering within the plausibility set it already possesses. The base model already knew that the promoted token was a reasonable choice; RLVR teaches it that this choice is better than the previous default.

The trajectory-steering interpretation (Section 3.2). The cross-sampling results reveal that the base model can complete correct reasoning trajectories after only a few RL-guided token substitutions. The forward cross-sampling finding is particularly revealing: injecting RL tokens into base generations and then letting the base model continue demonstrates that the base model has the capability to finish reasoning correctly — it just doesn't navigate to those trajectories on its own. The paper articulates this as: "Rather than introducing entirely new reasoning behaviors, RLVR refines a sparse set of local token choices that reliably steer generation toward more effective reasoning trajectories that remain accessible to the base model." This is a fundamentally different picture than "RL makes the model smarter" — it suggests RL makes the model better at navigating a reasoning space that it already understands.

This innovation is fundamental because it changes the conceptual vocabulary with which the field discusses RLVR. Instead of asking "what new capabilities does RL teach?", the paper suggests we should ask "which existing capabilities does RL help the model access?" Instead of viewing accuracy improvements as evidence of learning, we might view them as evidence of better metacognition — the model learning to select reasoning paths that were always within its competence but not its default behavior. This reframing has direct implications for training methodology (discussed in Innovation 4) and for understanding the limits of RLVR: if the model lacks certain capabilities entirely, RLVR cannot compensate, because it works by reordering existing options, not creating new ones.

The "scalpel not hammer" metaphor (Rajani et al., 2025; Chu et al., 2025) captured this intuition at a high level, but the paper provides the precise, token-level characterization that transforms metaphor into mechanism. It identifies what the scalpel cuts: a small set of high-divergence decision points. It identifies how it cuts: by reordering already-plausible candidates within an existing support. And it identifies why cutting there matters: because those positions steer trajectories toward reasoning paths the model already knows how to complete.

Innovation 3: Fine-Grained Characterization of DAPO vs. SimpleRL as Different Regimes of Exploration-Exploitation

The paper's third contribution is a nuanced empirical characterization of how different RLVR training configurations — specifically DAPO's clip-higher mechanism vs. SimpleRL's more constrained GRPO updates — produce qualitatively different patterns of token-level change. This goes beyond the simple observation that "DAPO performs better" (which is already known; Yu et al., 2025) to provide a mechanistic explanation of why DAPO's greater exploration translates to stronger performance.

The key insight emerges from synthesizing multiple distributional statistics:

Divergence sparsity and magnitude. SimpleRL concentrates nearly all token distributions (>98%) at near-zero divergence, while DAPO spreads divergence more broadly (~17% above near-zero). However, even DAPO maintains extreme sparsity — the "clear spike at zero" and the steep percentile curves (Figure 2) show that its broader exploration still leaves the vast majority of the policy intact. This resolves a tension in the literature: DAPO removes KL regularization, which might suggest it could make arbitrarily large changes, yet the empirical distribution of divergence shows it remains highly conservative at most positions.

Entropy regime of modification. The divergence-entropy analysis (Figure 4 vs. Appendix Figure 21) reveals a striking qualitative difference: DAPO modifies token distributions across the full entropy range, including initially low-entropy (confident) predictions, while SimpleRL concentrates its modifications primarily in higher base-entropy regions. This means DAPO can override the base model's confident but incorrect choices — for example, changing a firmly-predicted arithmetic operation or variable substitution when that choice leads to a dead end. SimpleRL, constrained by its KL penalty, is largely limited to modifying positions where the base model was already uncertain. The Qwen2.5-Math-7B comparison with different clip-upper settings (0.28 vs. 0.2) confirms this as a parametric effect of the clip-higher mechanism: the more restrictive 0.2 clip concentrates modifications more strongly in high-entropy regions (Appendix Figure 35).

Rank reordering depth. At high-divergence positions, DAPO produces more extensive reordering among top candidates (Figure 9: lower top-k overlap at small k), while SimpleRL's reorderings are shallower — it typically swaps the top-1 with the top-2 rather than reshuffling the top-5. However, DAPO's reorderings remain within the shared candidate set (high overlap at k > 2): it reshuffles aggressively but doesn't introduce fundamentally new tokens.

Tail behavior. DAPO more frequently promotes tokens from the low-probability tail of the base distribution (Figure 10): about 5% of RL top-1 tokens at high-divergence positions have base probability below 0.01, compared to near-zero for SimpleRL. While still rare, these tail promotions may be consequential — the paper notes that "although such low-probability promotions remain rare overall, they may still be important for improved reasoning performance."

The synthesis of these patterns yields a coherent picture: DAPO operates in a broader exploration regime where it can override confident base-model errors and occasionally elevate truly dispreferred alternatives, while SimpleRL operates in a conservative exploitation regime that sharpens and reorders within the existing high-probability set. The stronger performance of DAPO (~44% vs. ~25% on AIME 2024) correlates with its ability to make these deeper modifications, particularly at low-entropy error positions where the base model was confidently wrong.

This is an incremental but practically significant contribution. It does not propose new mechanisms but rather provides the first systematic evidence for how a specific algorithmic choice (clip-higher) translates into distributional behavior. This enables practitioners to reason about the tradeoffs of different RLVR configurations in a principled way: if a task requires overriding deeply ingrained but incorrect model behaviors, DAPO-like exploration is necessary; if the task primarily requires sharpening existing capabilities, more conservative methods may suffice with less risk of distributional collapse. The comparison also suggests diagnostic metrics (entropy distributions at high-divergence positions, top-k overlap curves, tail-promotion fractions) that could be used to monitor training dynamics and detect when a method is making the right kind of changes for a given task.

Innovation 4: Divergence-Weighted Advantages as a Proof-of-Concept for Distribution-Aware Training Objectives

The paper's fourth contribution is more speculative but conceptually important: the demonstration that incorporating token-level divergence information into the RL training objective — through divergence-weighted advantages — can improve performance over standard uniform weighting (Table 2). This is not presented as a new state-of-the-art method (the gains are modest and the authors are appropriately cautious about generalization), but rather as a diagnostic validation of the paper's central thesis: if sparse, high-divergence tokens are indeed the primary drivers of reasoning improvement, then explicitly focusing the learning signal on those tokens should be beneficial.

The experimental design is elegant in its symmetry. The paper tests both directions: high-KL boost (amplifying advantages at positions where the policy has already changed substantially, α > 0) and low-KL boost (amplifying advantages at positions where the policy has changed less, α < 0). Both improve over the DAPO baseline (35.9–36.7% vs. 33.6% on AIME 2024, averaged over 3 runs), with the high-KL boost showing slightly stronger performance. The fact that both directions help — rather than one being clearly harmful — suggests that the relationship between divergence and optimal learning signal is more nuanced than a simple "focus on changing tokens" heuristic. Low-KL boost might help by encouraging updates in regions that the model has been neglecting, preventing the policy from becoming overly narrow, while high-KL boost might help by concentrating capacity on the most impactful decisions.

This contribution is incremental as a method but conceptually significant as a validation. The paper explicitly frames the divergence-weighted advantage experiment as a "diagnostic intervention" and a "complementary diagnostic tool that may inform future refinements of token-level training strategies." The authors do not claim it as a ready-to-deploy training recipe; they note that "effective divergence weighting across training configurations may require model-specific paradigms or adaptive scheduling mechanisms to stabilize learning dynamics." The value of this experiment lies not in the specific accuracy numbers but in the proof-of-concept: it demonstrates that the token-level distributional structure the paper has characterized is actionable — it can inform better training objectives. This closes the loop from descriptive analysis (Section 2) → causal validation (Section 3) → mechanistic understanding (Section 4) → improved training (Section 5), providing a complete narrative arc for the paper's methodology.

The connection to the fine-grained mechanics analysis (Section 4) adds depth: the paper notes that future work might explore weighting strategies that "more effectively promote the rare actions" where RLVR elevates initially low-probability tokens. Since these rare promotions appear more frequent in stronger models (DAPO vs. SimpleRL, and within DAPO variants with higher clip-upper), they may represent a particularly valuable class of updates that standard uniform weighting underserves. The divergence-weighted advantage framework provides a template for testing such hypotheses.

Innovation 5: SFT vs. RLVR Distinguished Through Token-Level Distribution Structure

The paper's fifth contribution is the systematic comparison of supervised fine-tuning and RLVR at the token-distribution level, which provides a mechanistic grounding for the increasingly common claim that "SFT memorizes, RL generalizes" (Chu et al., 2025). While prior work had established this distinction through behavioral metrics (performance on held-out tasks, catastrophic forgetting patterns), the paper provides the first direct evidence that RLVR and SFT produce qualitatively different patterns of token-level change — and that these patterns are consistent with their different training objectives.

The key empirical contrast (Figures 6, 12, 16–18) reveals several dimensions of difference:

Sparsity. RLVR concentrates changes in a far smaller fraction of token positions than SFT. This is not a trivial consequence of "RL makes smaller changes" — the weight-level analysis in Appendix A.3 (Table 3) shows that the relative gap ratio σ is 0.00143 for RL vs. 0.00347 for SFT on Qwen2.5-32B, confirming that RL makes smaller parameter changes as well. But the token-level analysis goes further: it shows that even the changes RL does make are concentrated in fewer positions proportionally than SFT's changes, indicating a fundamentally different allocation of update budget.

Candidate-set fidelity. At high-divergence positions, RLVR maintains substantially higher top-k overlap with the base model than SFT does (Figure 16). This means RLVR's edits are more likely to be reorderings within the existing candidate set, while SFT more frequently promotes tokens that were not in the base model's top candidates. The rank-reordering analysis (Figure 17) confirms this: SFT promotes more tokens from outside the base top-3 into the fine-tuned model's top-3.

Tail promotion. SFT more frequently elevates tokens that had very low probability under the base model (Figure 18). This is consistent with SFT's objective: it learns to directly imitate target outputs, which may require using tokens that the base model dispreferred, whereas RLVR optimizes a reward signal on self-generated trajectories, which constrains it to operate within the distribution of tokens it would naturally sample.

Positional structure. Both methods show increased divergence at the start of responses (Figure 14), but SFT's divergence remains elevated across the entire sequence, while RLVR's drops to near-zero in the middle. This suggests RLVR's edits are more contextually targeted — concentrated at initial branching points and answer-formatting regions — while SFT makes more uniformly distributed changes.

These contrasts provide a distributional-level explanation for why RLVR may generalize better than SFT on held-out reasoning problems: by making sparse, targeted edits that reorder within existing candidate sets, RLVR preserves the base model's broad distributional knowledge while refining specific decision points. SFT's denser, more globally distributed changes risk overwriting capabilities that were useful for out-of-distribution problems. The finding that RLVR's sparsity is not simply a generic property of "less aggressive training" — it is distinctive to the RL objective — is the crucial conceptual insight. The parameter-level analysis in Appendix A.3 reinforces this: RL's smaller σ values show it makes less extensive weight modifications than SFT, even when trained on identical data for identical steps, suggesting the RL objective itself induces more targeted updates.

This is an incremental but illuminating contribution. It doesn't introduce new methodology but rather provides a mechanistic, distributional-level explanation for a behavioral phenomenon (RL's better generalization) that had been observed but not mechanistically understood. It also suggests a diagnostic for distinguishing RLVR-trained models from SFT-trained ones: examine the sparsity of token-level divergence from the base model. A model with extremely concentrated divergence is likely RLVR-trained; one with more diffuse divergence is likely SFT-trained.

5. Experimental Analysis

Evaluation Methodology

Dataset. The primary evaluation uses the AIME (American Invitational Mathematics Examination) 2024 and AIME 2025 benchmarks. These are competition-level mathematics problems requiring multi-step symbolic reasoning, with unambiguous ground-truth answers enabling reliable accuracy evaluation. The exact number of problems is not specified in the paper's main text (AIME typically contains 30 problems per year), but the authors sample 32 responses per problem for robustness. Additional evaluations use GPQA-Diamond and the models' respective fine-tuning datasets (Appendix A.5).
Base model(s). The primary base model is Qwen2.5-32B (Qwen et al., 2025), a 32-billion parameter LLM achieving approximately 8% accuracy on AIME 2024 before fine-tuning. The paper argues this model is in a useful regime — non-trivial but far from saturation — leaving room for RLVR to make measurable improvements. Extended analyses use Qwen2.5-Math-7B (two DAPO variants with clip-upper settings of 0.28 and 0.2), Qwen3-8B-Base with DAPO, and Mistral-Small-24B with SimpleRL to demonstrate cross-architecture generalizability.
Metrics. The primary metric throughout is Mean@32 accuracy — the fraction of problems for which at least one response among 32 sampled completions is correct, equivalent to pass@1 with 32 samples. This multi-sample protocol reduces variance from stochastic sampling, which is important when studying token-level effects sensitive to individual sampling decisions. All generation uses nucleus sampling with top-p = 0.7 and temperature = 1, implemented via vLLM (Kwon et al., 2023), with a maximum sequence length of 8000 tokens.
Baselines. The paper compares several model configurations: (1) the base Qwen2.5-32B model without any fine-tuning (the "base model" reference); (2) the same model after DAPO fine-tuning (Yu et al., 2025); (3) the same model after SimpleRL/GRPO fine-tuning (Zeng et al., 2025); and (4) a supervised fine-tuned (SFT) version trained on 42k instances from the AM-DeepSeek-R1-Distilled-1.4M dataset for 5 epochs with full parameter fine-tuning. The SFT baseline serves as a control to determine whether observed distributional patterns are specific to RLVR or generic to any fine-tuning.
Generation budget / compute accounting. The primary unit of comparison across methods is Mean@32 accuracy at fixed sampling budgets. For cross-sampling experiments, "compute" is measured in terms of the number of cross-sampling interventions (token substitutions), with the distinction between "effective tokens" (excluding identity swaps where both models would have chosen the same token) and "total tokens" (all interventions). Token percentages are computed at the sequence level relative to average response length. The divergence-weighted advantage experiments (Section 5) compare at fixed training steps with identical data and hardware.
Cross-validation / statistical protocol. The paper does not employ traditional cross-validation. Instead, for the divergence-weighted advantage experiments (Table 2), results are "each averaged over 3 runs" to account for training stochasticity, with standard deviations reported (e.g., 42.48 ± 1.35 for the DAPO baseline). For cross-sampling experiments, the same divergence thresholds are used across all evaluated problems, and results are reported as Mean@32 accuracy across the full test set without train/test splits, since the models being compared are already trained and the cross-sampling procedure is a zero-shot intervention.

Main Quantitative Results

Token-Level Distributional Shift Characterization

The descriptive analysis in Section 2 establishes the foundational quantitative claim of the paper: RLVR induces highly sparse token-level distributional shifts. Figure 2 provides the headline numbers: under DAPO, more than 83% of token positions exhibit near-zero JS divergence (defined operationally as the clear spike at zero in the log-scaled histogram), while under SimpleRL, this proportion exceeds 98%. The "near-zero" characterization is supported by the steep rise of the percentile curves — the 80th percentile under DAPO corresponds to JS divergence of approximately 0.02 (barely above zero on the [0, log 2] scale), while under SimpleRL the 98th percentile marks the transition out of the near-zero regime. The complementary percentile curves show that even at the 99th percentile, JS divergence values remain modest (well below the theoretical maximum of ~0.693), confirming that the changes, while concentrated, are not extreme in magnitude.

Positional concentration (Figure 3) reveals a consistent pattern across both DAPO and SimpleRL: average divergence peaks at normalized positions near 0.0–0.2 (the beginning of the response) and increases modestly again near 0.8–1.0 (the end). The early concentration is attributed to initial high-level branching decisions (e.g., which approach to take), while the late increase corresponds to adjustments in answer formatting and termination behavior. However, the wide percentile bands (25th–75th percentile spread) indicate that individual sequences can exhibit high divergence sporadically throughout the middle of responses — the aggregate trend masks substantial per-sequence variability.

Divergence–entropy relationship (Figure 4 and Appendix Figure 21) quantifies how distributional shifts interact with model uncertainty. For DAPO (Figure 4), low-divergence positions (JS < 0.1) are overwhelmingly low-entropy under both base and RL models, with entropy concentrated below ~1.0 nat, but the RL model's entropy is shifted slightly lower (sharper peaks). High-divergence positions (JS > 0.1) span a broad entropy range under DAPO, from near-zero to over 4.0 nats, indicating DAPO modifies both initially confident (low-entropy) and uncertain (high-entropy) predictions. For SimpleRL (Appendix Figure 21), high-divergence positions concentrate more strongly in the higher base-entropy regime (above ~2.0 nats), reflecting a more conservative update pattern that avoids overriding confident base-model predictions.

SFT comparison (Figures 6, 12) provides the critical control experiment. The SFT-fine-tuned Qwen2.5-32B exhibits a substantially broader divergence distribution than RLVR: the zero-spike is less pronounced, and the percentile curve rises more gradually, with the 80th percentile reaching JS divergence values approximately 2–3× higher than DAPO's. The top-k overlap analysis (Figure 16) shows SFT achieves consistently lower overlap with the base model at all k, particularly at small k (k = 1–3), indicating more aggressive re-ranking that frequently promotes tokens outside the base model's top candidates. Figure 14 confirms that SFT's divergence remains elevated across the entire response sequence rather than concentrating at the start and end like RLVR.

Cross-model validation (Appendix A.5) extends the sparsity finding across configurations. Qwen2.5-Math-7B with DAPO at two clip-upper settings (0.28 and 0.2) on AIME 2024 and AIME 2025 (Figure 33) shows that the more restrictive 0.2 clip produces sparser shifts (higher proportion near-zero) but with higher divergence at the upper percentiles — fewer token positions change, but those that do change more dramatically. Qwen3-8B-Base with DAPO (Figure 43) confirms sparsity persists even after nearly twice the standard training horizon. Mistral-Small-24B with SimpleRL (Figure 48) replicates the >98% near-zero pattern on a different architecture. GPQA-Diamond results (Figure 28) show consistent sparsity patterns on a non-math reasoning benchmark.

Cross-Sampling: Causal Importance of Divergent Tokens

The cross-sampling experiments (Section 3) translate the descriptive sparsity finding into a causal claim. Table 1 provides the headline quantitative summary, with Figures 7 and 8 showing the full accuracy curves as a function of intervention budget.

Forward cross-sampling (Figure 7, Table 1): Injecting RL-sampled tokens into base-model generations progressively recovers RL-level performance. For Qwen2.5-32B with SimpleRL on AIME 2024, using only 3.86% effective tokens (38 effective substitutions per response on average, out of an average sequence length of approximately 980 tokens based on the total intervention counts in Table 1) is sufficient to close the performance gap from base accuracy (~8.23%) to RL-level accuracy (>25%). On AIME 2025, the effect is even more dramatic: 1.53% effective tokens (13 substitutions) raises accuracy from ~5.3% to >14%, actually exceeding the standalone RL policy's performance. For DAPO, the required token fractions are larger — 7.8% effective tokens on AIME 2024 (280 substitutions) to go from ~8.23% to >44%, and 6.47% effective tokens on AIME 2025 to go from ~5% to >33% — reflecting DAPO's substantially larger performance gains.

Reverse cross-sampling (Figure 8, Table 1): Reverting RL token choices back to base-model choices at high-divergence positions causes near-monotonic performance degradation. For SimpleRL on AIME 2024, 5% effective reversions (29 substitutions) collapses accuracy from ~25.5% to below 8.3%. On AIME 2025, 4.73% effective reversions (31 substitutions) degrades accuracy from ~12.7% to below 4%, which is actually below base-level performance. For DAPO, 10.1% effective reversions (173 substitutions) on AIME 2024 erases the gain from ~44.8% to below ~8.5%, while 9.89% effective reversions (181 substitutions) on AIME 2025 collapses performance from ~32% to below ~4.5%.

Progressive trajectory shaping (Figures 7, 8): The accuracy curves as a function of intervention budget reveal smooth, near-monotonic changes. In forward cross-sampling, accuracy increases steadily as additional RL-sampled tokens are introduced, with no sharp threshold — each intervention contributes incrementally on average. In reverse cross-sampling, performance degrades similarly smoothly as RL token choices are reverted. The paper emphasizes that "even just the first few RL-sampled tokens can already yield measurable performance gains in forward cross-sampling," demonstrating that small, early edits can redirect generation toward different reasoning trajectories that the primary policy then continues.

Random baseline (Appendix Figure 55): To verify that the cross-sampling gains are not an artifact of simply changing tokens, the paper includes a random baseline where token substitutions are made at random positions rather than at high-divergence positions. The random baseline "shows slow performance improvement," with accuracy increasing far less steeply as a function of the number of replaced tokens, confirming that targeted RL token selection at high-divergence positions is critical for performance gains.

Effective vs. total tokens (Table 4): The paper carefully distinguishes between total cross-sampling interventions and effective interventions (excluding identity swaps). For SimpleRL forward cross-sampling on AIME 2024, 3.86% effective tokens corresponds to 7.58% total tokens (75 total interventions, of which 38 produced a different token than the base model would have chosen). This ~50% effective-to-total ratio indicates that at roughly half of the high-divergence positions, the two models happen to sample the same token despite having different distributions.

Cross-sampling token pairs (Appendix Figure 52): Histograms of (primary-sampled token, intervention-sampled token) pairs at intervention points reveal that the substitutions are "mostly plausible and semantically reasonable" — the tokens being swapped are not obviously wrong; they are locally sensible alternatives that nonetheless lead to different downstream reasoning trajectories.

Sequence-level divergence bounds (Appendix A.1): The theoretical analysis provides formal support for the empirical observation. Proposition A.2 shows that when the switching rule uses a token-level KL threshold ε, the sequence-level KL divergence between the mixed policy and the intervention policy is bounded by ε · E[N₀], where N₀ is the number of non-intervention steps. Since most steps have low divergence (Section 2), ε can be small and the bound is tight, meaning that a small number of interventions suffices to bring the mixed policy close to the intervention policy in distribution. An analogous bound is provided for JS divergence (Proposition A.6), though it requires a history-dependent skew Jensen-Shannon divergence.

Fine-Grained Distribution Mechanics

The fine-grained analysis (Section 4) characterizes the kind of change occurring at the sparse high-divergence positions identified in Section 2 and validated as causally important in Section 3.

Top-k overlap (Figure 9): At high-divergence positions (JS > 0.1), overlap between the base and RL models' top-k token sets remains high for k ≥ 2. For SimpleRL, average overlap exceeds 80% at k = 2 (often above 85%), while DAPO shows slightly lower but still substantial overlap. Both methods exhibit a sharp increase from k = 1 to k = 2, indicating that while the top-1 token often changes, the replacement was typically already among the base model's top-2 or top-3 candidates. Results are shown across multiple divergence thresholds (0.01, 0.05, 0.1, 0.2) to demonstrate robustness.

Rank reordering (Figure 22): For DAPO, approximately 30% of RL top-1 tokens at high-divergence positions are already ranked first under the base model (this seems paradoxical but occurs because JS divergence is a distributional measure, not just a top-1 measure — the models can agree on argmax while disagreeing on the full distribution shape). Over 80% of RL top-1 tokens fall within the base model's top-3. For SimpleRL, the alignment is even stronger: over 90% within top-3. RL top-2 tokens typically lie within the base top-3–4 range.

Low-probability token behavior (Figure 10, Appendix Figure 24): At high-divergence positions, the RL model's new top-1 token rarely comes from the base model's probability tail. Under DAPO (Figure 10a), only about 5% of RL top-1 tokens (among high-divergence positions) have base probability below 0.01, and this fraction drops sharply as the threshold increases — fewer than 15% have base probability below 0.1. Under SimpleRL (Appendix Figure 24), the fraction with base probability below 0.01 is nearly zero, and even at a 0.1 threshold, fewer than 2% qualify. Figure 10b shows the RL probability distribution for the subset of tokens with base probability <0.01, revealing that even when RL promotes a low-base-probability token, it does not assign it extremely high probability — the distribution is concentrated in the 0.05–0.30 range.

Clip-higher comparison (Appendix Figures 37–39): The two DAPO variants on Qwen2.5-Math-7B reveal systematic differences. The 0.28 clip-upper setting (Figure 37, top row) shows approximately 3–4× more RL top-1 tokens with base probability below 0.01 compared to the 0.2 clip setting (bottom row). In top-k overlap (Figure 38), the 0.28 setting has lower overlap at small k (more aggressive re-ranking of top candidates) but higher overlap at larger k (the broader candidate set is better preserved), while the 0.2 setting shows the opposite pattern — higher overlap at small k (top-1 often unchanged) but lower overlap at larger k (narrower, more concentrated distribution). The rank-shift analysis (Figure 39) confirms: under 0.2 clip, the RL model's 3rd-ranked token occasionally originates from much lower base ranks (positions 10–50), but given the substantially lower RL entropy in this setting, such promoted tokens carry relatively little probability mass.

Training evolution (Figure 11): Using intermediate DAPO checkpoints on Qwen2.5-Math-7B (200 total steps), conditioned on the final model's outputs, the divergence percentiles (Figure 11a) show that JS divergence increases monotonically throughout training, with higher percentiles (95th, 99th) growing faster than lower ones (10th, 25th). The widening gap over training steps indicates that distributional change becomes increasingly concentrated in a small subset of tokens. Figure 11b shows the Jaccard overlap between each checkpoint's high-divergence token set and the final model's set: overlap increases gradually from approximately 0.3 at step 25 to roughly 0.55 at step 150, then rises sharply to ~0.8 by step 200, suggesting the specific token positions that will ultimately diverge are not identifiable early in training but crystallize late.

SFT comparison on fine-grained metrics (Appendix Figures 16–18): SFT consistently shows lower top-k overlap at all k values (Figure 16), more extensive rank reordering (Figure 17: a larger fraction of RL top-3 tokens originate from outside the base top-3), and more frequent promotion of low-base-probability tokens (Figure 18: the cumulative curve rises earlier and reaches higher values). These patterns are consistent with SFT's objective of directly imitating target outputs, which requires more extensive distributional changes.

Divergence-Weighted Advantage

The exploratory experiment in Section 5 investigates whether incorporating token-level divergence into the RL training objective can improve performance. Table 2 reports results on Qwen2.5-Math-7B evaluated on AIME 2024, AIME 2025, and AMC, with Mean@32 scores averaged over 3 runs.

Baseline DAPO achieves 33.61% on AIME 2024, 18.75% on AIME 2025, 75.08% on AMC, and an overall average of 42.48 ± 1.35%.

Low-KL boost (α < 0, amplifying advantages at positions with low divergence from the old policy) improves to 35.90% on AIME 2024 (+2.29), 19.90% on AIME 2025 (+1.15), 78.97% on AMC (+3.89), and an overall average of 44.92 ± 0.05 (+2.44 over baseline).

High-KL boost (α > 0, amplifying advantages at positions with high divergence from the old policy) achieves 36.74% on AIME 2024 (+3.13), 20.00% on AIME 2025 (+1.25), 78.40% on AMC (+3.32), and an overall average of 45.05 ± 0.79 (+2.57 over baseline).

Both divergence-weighting configurations outperform the baseline, with the high-KL boost showing marginally higher overall average (45.05 vs. 44.92) though the difference is well within one standard deviation. The standard deviations for the weighted variants (0.05 and 0.79) are notably smaller than the baseline (1.35), suggesting more stable training dynamics. The improvement on AMC (+3.32–3.89) is proportionally larger than on AIME (+1.15–3.13), though this may reflect the higher baseline accuracy making absolute gains easier.

Training configuration (Appendix A.2.3): Under high-KL boost, s = 0.3 and α increases linearly from 0 to 50 starting at step 100. Under low-KL boost, s = 0.3 and α decreases linearly from 0 to -50 starting at step 150. The KL divergence is computed with respect to the old policy (π_{θ_old}) using the k3 estimator (Schulman, 2020), and divergences are detached from the computation graph.

Ablation Studies and Robustness Checks

Robustness to top-p truncation for JS divergence (Appendix A.5, Figures 30 and 32): JS divergence percentile curves computed with different top-p sampling settings (0.7, 0.8, 0.9) and with the estimated full distribution (top-p = 1) show consistent sparsity patterns. Figure 30 demonstrates this for DAPO on AIME 2024 across top-p ∈ {0.8, 0.9}, while Figure 32 shows the top-p = 1 comparison for both DAPO and SimpleRL on AIME 2025. The fraction of near-zero tokens remains above 80% (DAPO) and 98% (SimpleRL) regardless of truncation level.

Generalizability across datasets (Appendix Figures 28, 29, 31): The sparsity and positional concentration patterns replicate on GPQA-Diamond (Figure 28) and AIME 2025 (Figure 31). On GPQA-Diamond, DAPO shows similar percentile curves with >80% near-zero divergence, and positional concentration at the start and end of responses persists. Entropy distributions across divergence bins (Figure 29) match the AIME patterns: low-divergence positions are low-entropy, high-divergence positions span a broader range.

Generalizability across model architectures and scales (Appendix A.5): Qwen2.5-Math-7B (Figures 33–39), Qwen3-8B-Base (Figures 43–46), and Mistral-Small-24B (Figures 48–51) all exhibit the sparse divergence pattern. For Qwen3-8B-Base with DAPO, trained for nearly twice as many steps as Qwen2.5-Math-7B, "sparse distributional shifts persist even under extended training" (Figure 43). For Mistral-Small-24B with SimpleRL, the >98% near-zero pattern is replicated, and the positional concentration and entropy patterns match those observed for Qwen2.5-32B.

SFT as a control for sparsity (Figures 6, 12–15): The SFT-fine-tuned Qwen2.5-32B produces substantially broader divergence distributions (Figures 6, 12), elevated divergence across the entire response sequence (Figure 14), and divergent tokens that more frequently concentrate in high base-entropy regions (Figure 15). On AIME 2025 (Figure 19), the same pattern holds — SFT's divergence is less sparse than RLVR's. These comparisons demonstrate that RLVR's extreme sparsity is not a generic consequence of fine-tuning but is specific to the RL objective.

Effect of clip-higher settings (Appendix A.5.2): Comparing Qwen2.5-Math-7B DAPO variants with clip-upper 0.28 vs. 0.2 reveals a parametric tradeoff between breadth of changes and their magnitude. The 0.2 clip produces sparser overall divergence (more tokens near-zero, Figure 33c,d vs. 33a,b), but at the high-divergence tail, JS values are higher (the 95th and 99th percentiles are elevated). This means the more restrictive setting concentrates its limited update budget into fewer but more extreme changes. The 0.28 setting spreads changes more broadly but with lower maximum divergence.

Random baseline for cross-sampling (Appendix Figure 55): A random token substitution baseline (substituting at random positions rather than at high-divergence positions) shows "slow performance improvement" compared to the divergence-targeted cross-sampling. This confirms that the observed gains are not an artifact of simply replacing tokens — the selection of which positions to intervene at is critical.

Cross-sampling on additional model configurations (Appendix A.6): Forward and reverse cross-sampling results are replicated for DAPO on AIME 2024 (Figure 54), DAPO on AIME 2025 (Figure 53), and Mistral-Small-24B with SimpleRL on AIME 2024 (Figure 56). In all cases, forward cross-sampling progressively recovers RL accuracy while reverse cross-sampling progressively degrades it, with effective token fractions in the 1–10% range.

Weight-level analysis (Appendix A.3, Table 3): The relative gap ratio σ (measuring cumulative weight divergence between original and fine-tuned models) is substantially smaller for RL than for SFT across both model scales. For Qwen2.5-32B: σ = 0.00143 after RL vs. 0.00347 after SFT. For Qwen2.5-Math-7B: σ = 0.00136 after RL vs. 0.00944 after SFT. This parameter-level analysis corroborates the token-level finding that RL induces more targeted, less extensive modifications.

Critical Assessment

Does the paper demonstrate that RLVR induces sparse token-level distributional shifts? Yes, this claim is robustly supported across multiple models (Qwen2.5-32B, Qwen2.5-Math-7B, Qwen3-8B, Mistral-Small-24B), multiple datasets (AIME 2024, AIME 2025, GPQA-Diamond), multiple training methods (DAPO, SimpleRL, with varying hyperparameters), and multiple evaluation protocols (different top-p values, full vs. truncated distributions). The SFT comparison provides a genuine control showing the sparsity is RL-specific. However, the paper does not define "near-zero divergence" with a formal threshold — it relies on visual inspection of the histogram spike at zero and the steepness of the percentile curves. A reader looking for a precise definition (e.g., "JS < 0.001") will not find one. This is a minor weakness but does not undermine the central claim, since the quantitative statements about specific percentiles (83%, 98%) are well-supported.

Does the paper demonstrate that sparse divergent tokens are causally responsible for reasoning gains? This is the paper's strongest claim and is well-supported by the cross-sampling experiments, but with important qualifications. The forward cross-sampling results (Figure 7, Table 1) show convincingly that injecting RL token choices at high-divergence positions into base-model generations recovers RL-level performance with only a small fraction of substitutions. The reverse cross-sampling results (Figure 8, Table 1) provide a convergent test showing that reverting those same positions erases RL gains. The progressive nature of the accuracy curves (smooth, near-monotonic) rules out the possibility that a single critical token drives the entire effect.

However, there are several limitations to the causal interpretation. First, the cross-sampling procedure is performed with a fixed divergence threshold and the budget is swept, but the specific thresholds (ε values in Appendix A.2.2) vary substantially across configurations without a documented selection procedure. If the thresholds were tuned post-hoc to produce compelling curves, this would weaken the causal claim, though the consistency across configurations and the progressive budget sweep (which shows the effect is robust to threshold choice within a range) partially mitigate this concern. Second, the cross-sampling experiments are conducted only on the same test set used for the descriptive analysis (AIME 2024, 2025). There is no out-of-distribution test to verify that the identified divergent positions generalize to new problems. Third, the cross-sampling framework demonstrates that RL token choices at high-divergence positions are sufficient to recover RL performance in the forward direction and necessary to maintain it in the reverse direction, but it does not rule out the possibility that other tokens (below the divergence threshold) also contribute meaningfully in ways that the coarse binarization misses. The progressive budget curves address this to some degree by showing that even very few interventions have measurable effects, but a finer-grained analysis (e.g., testing whether low-divergence positions contribute zero marginal benefit or just a smaller one) is absent.

Does the paper demonstrate that RLVR primarily reallocates probability mass within existing candidate sets rather than introducing new tokens? This claim is well-supported by the top-k overlap (Figure 9), rank reordering (Figure 22), and low-probability token behavior (Figure 10) analyses. The numbers are striking: >80% of RL top-1 tokens at high-divergence positions are in the base top-3, and <5% (DAPO) to ~0% (SimpleRL) have base probability below 0.01. However, there is an important interpretive nuance the paper acknowledges but does not fully resolve: the "rare cases of substantial re-ranking and promotion of initially low-probability tokens may still be important for the observed improvements in reasoning performance." The paper cannot determine from these analyses whether the handful of tail promotions are the critical changes or incidental — the cross-sampling experiments don't separate "re-ranking within top-3" interventions from "promoting from the tail" interventions. An ablation that cross-samples only at positions where the RL top-1 token was already top-3 in the base model vs. only at positions where it was promoted from the tail would address this. This is a genuine open question that the paper identifies but does not answer.

Does the divergence-weighted advantage experiment support the paper's central thesis? Yes, but the support is suggestive rather than definitive. Both low-KL and high-KL boost improve over baseline (Table 2), which aligns with the idea that token-level divergence structure is a useful signal for modulating training. However, the fact that both directions improve — focusing on already-changing tokens and focusing on not-yet-changed tokens — complicates the interpretation. If concentrating learning on high-divergence tokens were straightforwardly optimal, low-KL boost should have been harmful, not helpful. The paper acknowledges this ambiguity: "the optimal choice between these approaches, and indeed whether divergence weighting provides benefits at all, may depend on the specific models and training methods used." The experiment is described as a "diagnostic intervention," and the paper does not claim to have found a universally better training objective. The small number of training runs (3 per configuration), the restriction to a single model scale (7B), and the modest absolute gains (+2.4–2.6 percentage points overall average) are appropriate to the experiment's diagnostic framing.

Weaknesses and missing experiments. Several limitations deserve mention:

The paper never combines the three analyses into a single integrated experiment. The forward cross-sampling recovers RL performance, but does it do so through the specific re-ranking mechanism characterized in Section 4? One could imagine cross-sampling only at positions where the base model's top-3 contains the RL model's top-1, and separately only at positions where the RL model's top-1 comes from outside the base top-3. This would distinguish whether re-ranking within the candidate set or tail promotion is the dominant causal mechanism.
All experiments are on math reasoning benchmarks. The authors extend to GPQA-Diamond, which is a step toward generalizability, but all datasets involve reasoning with verifiable answers. Whether the findings transfer to open-ended generation, dialogue, or tasks without clean reward signals is untested.
The test sets are small. AIME 2024 and 2025 each contain 30 problems. The effective sample size for Mean@32 accuracy is determined by the number of problems, and 30 is small enough that confidence intervals on the cross-sampling curves would be wide. The paper does not report confidence intervals for any of the cross-sampling results, which makes it difficult to assess whether the difference between, say, 3.86% and 7.8% effective tokens (SimpleRL vs. DAPO forward cross-sampling on AIME 2024) is statistically reliable or within sampling noise. The divergence-weighted advantage experiments (Table 2) do report standard deviations over 3 runs, but this captures training variance, not test-set variance.
The difficulty estimation question is unaddressed. In a deployment setting, one would need to know in advance which token positions are high-divergence for a new problem. The cross-sampling experiments use oracle knowledge (both models' full distributions at every position along the RL-generated trajectory) to decide where to intervene. The paper does not investigate whether a lightweight model could predict divergence positions without full distribution access, which limits the practical applicability of the cross-sampling findings. This is analogous to the difficulty estimation problem in the reference paper's compute-optimal scaling — the analysis assumes access to information that would be expensive to obtain in practice.
No analysis of failure modes or when the sparse targeting breaks down. The paper shows that ~10% effective reversions erase DAPO's gains, but does not characterize the cases where more reversions are needed or where the RL model is making changes that the cross-sampling framework cannot isolate (e.g., if RL performance depends on consistent distributional shifts across many positions that individually fall below the threshold).
The divergence-weighted advantage uses old-policy KL rather than base-policy JS divergence. The paper's central descriptive metric is JS divergence between base and RL models. The training experiment uses KL divergence with respect to the old policy (from the previous update iteration). These are different quantities — old-policy KL measures how much the policy has changed in one update step, while base-policy JS measures cumulative change from the pretrained model. The paper does not test whether base-policy JS divergence weighting would be more or less effective than old-policy KL weighting, which would be the more direct translation of the descriptive findings into a training objective.
The SFT comparison uses different data than the RLVR models. The SFT baseline was trained on AM-DeepSeek-R1-Distilled-1.4M, which is a different data source than the DAPO/SimpleRL training data. This confounds the RL-vs-SFT comparison — differences in divergence patterns could be due to data differences rather than objective function differences. The paper acknowledges that the training data are not identical but does not control for this. A stronger comparison would train SFT and RL on the same set of problems with the same base model generations.

Overall assessment. The paper's central empirical claims — that RLVR induces sparse distributional shifts, that these shifts are causally important for reasoning performance, and that the shifts primarily involve re-ranking within existing candidate sets — are well-supported by the reported experiments. The cross-sampling framework is genuinely novel and provides evidence for causality that goes beyond prior correlational analyses. The main limitations are the restriction to math reasoning benchmarks, small test sets without confidence intervals, the oracle nature of the divergence information used in cross-sampling, and the lack of integration between the three analysis axes in a single unified experiment. The divergence-weighted advantage experiment is appropriately scoped as a diagnostic rather than a method contribution, and its results are suggestive but preliminary. The paper's claims would be strengthened by: (1) cross-sampling on a larger and more diverse test set, (2) an ablation distinguishing re-ranking interventions from tail-promotion interventions, (3) investigation of whether divergence positions can be predicted without full distribution access, and (4) replication on non-math, non-verifiable-reward tasks.

6. Limitations and Trade-offs

Limitation 1: Oracle Access to Full Token Distributions Makes Cross-Sampling Impractical for Deployment

The assumption or constraint. The cross-sampling framework — the paper's central causal methodology — requires computing the full next-token distributions $\pi_{\text{base}}(\cdot \mid x_{<t})$ and $\pi_{\text{RL}}(\cdot \mid x_{<t})$ at every position along the RL model's generated trajectory, calculating their Jensen-Shannon divergence, and using this divergence to decide which token decisions to intervene on. This is an oracle-level information requirement: in a deployment setting, one does not have access to the base model's distribution at arbitrary prefixes along trajectories the RL model (or a cross-sampled policy) is generating, nor does one have a pre-computed divergence threshold calibrated to the specific problem instance.

The paper acknowledges this indirectly by flagging the difficulty estimation cost in Section 3.2 and the connection to speculative decoding's efficiency motivation, but does not quantify the computational overhead of the divergence computation itself. Crucially, the cross-sampling experiments (Figures 7, 8) use the RL model's own generated sequences as reference trajectories to identify high-divergence positions — meaning the analysis is retrospective: it first generates with the RL model, then computes divergences along that fixed path, and then performs cross-sampling interventions. This is fundamentally different from online cross-sampling, where one would need to compute divergences during generation without a pre-existing reference trajectory.

The consequence. The headline finding — that fewer than 4–10% of effective token substitutions suffice to recover or erase RL performance — is an existence proof that the RL model's advantage is concentrated in sparse decisions, but it does not provide a deployable method for achieving those gains without the oracle divergence computation. A practitioner cannot use forward cross-sampling to improve a base model's accuracy on a new problem without first generating RL trajectories, computing divergences, and selecting thresholds — a process that is more expensive than simply using the RL model directly. The $4\times$ and $10\times$ intervention efficiency ratios (e.g., 3.86% effective tokens recovering full RL performance) compare intervention count to total tokens in the RL trajectory, but they do not account for the cost of identifying which tokens to intervene on.

This also creates a gap between the paper's descriptive analysis (Section 2) and its causal claims. The descriptive analysis shows where divergence occurs; the cross-sampling shows that those positions matter. But the cross-sampling does not demonstrate that one can find those positions without first generating the full RL trajectory — it assumes the answer to the very question it implicitly raises about practical deployment.

What evidence exists in the paper. The thresholds for cross-sampling vary substantially across configurations (Appendix A.2.2): $\varepsilon = 0.03$ for SimpleRL forward vs. $\varepsilon = 0.1$ for DAPO forward on AIME 2025. These are post-hoc selections calibrated to the specific model-dataset-direction combinations. There is no experiment testing whether a single threshold generalizes across problems within a dataset, let alone across datasets. There is no experiment where the divergence is estimated from partial information (e.g., using a smaller draft model, as in speculative decoding). The random baseline (Appendix Figure 55) controls for the importance of which tokens are swapped, but not for the feasibility of identifying those tokens without oracle access.

Mitigation status. The paper does not provide a practical method for online divergence estimation or position prediction. The divergence-weighted advantage experiment (Section 5) takes a step toward making the insight actionable — it uses divergence signals during training to modulate the learning objective — but this is a training-time intervention, not an inference-time deployment strategy. The paper explicitly flags the speculation-decoding connection (Section 3.1) but does not develop it into a deployable system. The authors do not suggest future work on predicting divergence positions, which is a notable omission given that this is the primary barrier between the paper's diagnostic framework and practical application.

Limitation 2: All Experiments Are on Math Reasoning Benchmarks with Verifiable, Closed-Form Answers

The assumption or constraint. Every experiment in the paper — the descriptive characterization (Section 2), the cross-sampling causal analysis (Section 3), the fine-grained mechanics (Section 4), and the divergence-weighted advantage training (Section 5) — is conducted on benchmarks where the task is to produce a single, verifiably correct answer: AIME 2024, AIME 2025, GPQA-Diamond, and the models' fine-tuning datasets. These are all multiple-choice or short-answer reasoning problems with ground-truth labels that enable unambiguous accuracy evaluation and reward computation.

The paper acknowledges this limitation implicitly through its choice of benchmarks but does not discuss it as a limitation. The extension to GPQA-Diamond (Appendix A.5, Figures 28–29) is presented as a generalization check, but GPQA-Diamond is also a closed-form QA benchmark — it shares the same structural property of having verifiable answers, just in a different domain (graduate-level science rather than competition math). The authors state that the additional evaluations "confirm that our findings generalize across models, datasets, and training paradigms" (Section 2.1), but "datasets" here means different math/science QA benchmarks, not different task types.

The consequence. The paper's central findings — sparsity of distributional shifts, concentration of functional importance in a small fraction of token decisions, and the promotion-not-invention mechanism — are all measured under conditions where the model's task is to produce a single, objectively correct answer through multi-step reasoning. It is unclear whether these findings transfer to open-ended generation tasks (dialogue, creative writing, summarization), tasks without verifiable ground truth (subjective evaluation, stylistic judgment), or tasks where correctness is multi-dimensional (code generation with multiple valid solutions, planning with multiple acceptable plans).

This matters because the mechanism the paper identifies — RLVR refines local token choices to steer trajectories toward reasoning paths the base model already knows — depends on the existence of better paths that can be steered toward. In math reasoning, better paths lead to correct answers. In open-ended tasks, "better" may be ill-defined or multi-modal, and the sparse-steering mechanism may not apply. The finding that RLVR mainly reorders within existing candidate sets (Section 4) is particularly dependent on the base model having a well-formed plausibility space for the task — in domains where the base model's token distributions are diffuse or multi-modal, reordering may not be sufficient.

Furthermore, the cross-sampling methodology itself relies on having a clear performance metric (accuracy) against which to measure the effect of interventions. Extending the framework to tasks where quality is measured by human preference or automatic metrics with lower precision would require different evaluation protocols that the paper does not address.

What evidence exists in the paper. The experiments are comprehensive within the math/science QA domain: five models (Qwen2.5-32B, Qwen2.5-Math-7B at two clip settings, Qwen3-8B, Mistral-Small-24B), two RLVR methods (DAPO, SimpleRL), one SFT baseline, and three benchmarks (AIME 24, AIME 25, GPQA-Diamond). But there is zero evidence outside the verifiable-answer paradigm. This is a domain coverage limitation, not a model coverage limitation.

Mitigation status. The paper does not discuss generalization to non-verifiable tasks or open-ended generation. The authors do not flag this as a limitation or suggest future work in this direction. The conclusion describes RLVR as "a sparse intervention on a small set of high-impact decision points that steer generation trajectories" (Section 7), which is stated as a general claim but supported only by evidence from closed-form reasoning tasks.

Limitation 3: The Cross-Sampling Causal Claim Depends Heavily on the Choice of Reference Trajectories and Divergence Thresholds

The assumption or constraint. The cross-sampling experiments use the RL model's own generated sequences as reference trajectories: for each problem, the authors first generate complete responses from $\pi_{\text{RL}}$ , then compute JS divergences between $\pi_{\text{base}}$ and $\pi_{\text{RL}}$ at every position along those trajectories, and finally use those divergences to decide where to intervene when generating under the mixed policy. The divergence thresholds $\varepsilon$ are fixed per configuration (Appendix A.2.2) and vary substantially: from $\varepsilon = 0.002$ (Mistral-Small forward) to $\varepsilon = 0.1$ (DAPO forward on AIME 2025).

This creates a potential circularity in the forward cross-sampling direction: the mixed policy $\pi_{\text{mix}}^{\text{(base, RL)}}$ is defined using divergence thresholds computed from trajectories generated entirely by the RL model before any intervention. But the cross-sampled trajectories are different from the reference trajectories — they include base-model tokens at non-intervention positions. The divergence at position $t$ along a cross-sampled trajectory may differ from the divergence computed at the same position in the reference RL trajectory, because the prefix $x_{<t}$ is different (it contains base-model tokens from earlier in the sequence). The paper does not verify that the pre-computed divergence positions remain high-divergence along the actual cross-sampled paths, nor does it report how often the switching rule triggers at positions that were not identified as high-divergence in the reference trajectory.

The consequence. If the divergence landscape shifts substantially along cross-sampled trajectories compared to pure RL trajectories, then the intervention points selected using the reference trajectory are suboptimal — some high-divergence positions along the actual cross-sampled path may be missed (because they were low-divergence on the RL trajectory), and some interventions may be applied at positions that are no longer high-divergence (reducing the effective-to-total token ratio). This would mean the reported effective token percentages (Table 1) are underestimates of the true number of interventions needed if one could optimally select positions online — the paper's headline claim that "only 3.86% of tokens suffice" is measuring efficacy under a potentially misaligned intervention policy.

The choice of threshold $\varepsilon$ further compounds this issue. The thresholds vary by a factor of 50× across configurations (0.002 to 0.1), and the paper does not describe a selection procedure — it is unclear whether these were chosen to produce the cleanest cross-sampling curves, to match some quantile of the divergence distribution, or through some other criterion. If thresholds were tuned per-configuration to optimize the cross-sampling curves, the reported effective token percentages would be a best-case result rather than an inherent property of the model pair, and would not generalize to new configurations without similar tuning.

What evidence exists in the paper. The paper provides the full list of thresholds in Appendix A.2.2, but no threshold sensitivity analysis — no experiment scanning across multiple $\varepsilon$ values per configuration to show that the effective token percentages are robust to threshold choice. The progressive budget curves (Figures 7, 8) sweep across intervention counts for a fixed threshold, not across thresholds. The paper does not report how many high-divergence positions exist at each threshold versus how many are actually intervened on during cross-sampling, nor the effective-to-total token ratio as a function of threshold.

The random baseline (Appendix Figure 55) is the only control, and it tests random positions (within the set of all positions), not random tokens at the same positions. A stronger control would be to intervene at high-divergence positions identified from a different set of RL trajectories (e.g., from a different sampling run) to measure how stable the divergence positions are across stochastic realizations.

Mitigation status. The paper does not address the reference trajectory circularity or provide threshold sensitivity analysis. The theoretical bounds in Appendix A.1 provide an upper bound on sequence-level divergence in terms of per-step divergences, but these bounds assume the switching rule uses the online divergence at each step (computed during generation), not a pre-computed map from a reference trajectory. The bounds thus describe an idealized version of cross-sampling that the experiments do not implement. The authors flag the connection to speculative decoding (Section 3.1) but stop short of implementing an online version where divergence is computed during generation.

Limitation 4: Small Test Sets Without Confidence Intervals Undermine the Precision of the Quantitative Claims

The assumption or constraint. The primary benchmarks — AIME 2024 and AIME 2025 — each contain 30 problems. The Mean@32 accuracy metric computes the fraction of these 30 problems for which at least one of 32 sampled responses is correct. The effective sample size for accuracy comparisons is therefore at most 30 (the number of problems), and the paper does not report confidence intervals for any of the cross-sampling accuracy curves (Figures 7, 8) or the effectiveness percentages (Table 1).

The paper also does not report the variance of the effective token counts across problems. Table 1 provides average effective token counts (e.g., 38 tokens for SimpleRL forward on AIME 2024), but these averages are across 30 data points with potentially high variance — if a few problems require many more interventions than typical, the average could be skewed. The standard deviation of the effective token count per problem is not reported.

The consequence. The paper's headline quantitative claims — "3.86% effective tokens suffices," "7.8% effective tokens for DAPO," "10.1% effective reversions erases gains" — are point estimates without uncertainty quantification. On a 30-problem test set, the 95% binomial confidence interval for an accuracy difference of, say, 20 percentage points (base vs. RL) is approximately ±15–18 percentage points. The differences between SimpleRL (3.86%) and DAPO (7.8%) effective token requirements — a factor of ~2× — may or may not be statistically distinguishable. The claim that forward cross-sampling on AIME 2025 with SimpleRL "actually exceeds the standalone RL policy's performance" (Section 3.2) is based on a difference of roughly 1–2 percentage points (RL accuracy ~12.7% vs. mixed policy accuracy >14%), which is well within the noise floor for a 30-problem test set.

The divergence-weighted advantage experiments (Table 2) report standard deviations over 3 training runs (e.g., 42.48 ± 1.35 for baseline DAPO), which addresses training variance but not evaluation variance — the standard deviation of the accuracy estimate itself over different test sets is not captured.

This limitation particularly affects the paper's comparative claims. The smooth, near-monotonic shape of the cross-sampling curves (Figures 7, 8) is visually compelling but without error bars, a reader cannot tell whether the apparent monotonicity is real or whether the curve could plausibly be flat in some regions, non-monotonic, or have a sharp threshold that is obscured by sampling noise. The claim that "performance changes progressively and near-monotonically" (Section 3.2, takeaways) is an interpretation of point estimates.

What evidence exists in the paper. The test set sizes are documented implicitly — AIME is known to have 30 problems per year, and 32 samples per problem is stated in Section 2.1. The paper does not mention the sample size when interpreting quantitative differences. The only standard deviations reported are for the training-run replicates in Table 2. The cross-sampling curves (Figures 7, 8, 53, 54, 56) all show point estimates without error bars or shaded regions.

Mitigation status. No confidence intervals, error bars, or statistical tests are reported for the cross-sampling experiments. The paper does not discuss the precision of its estimates or acknowledge this as a limitation. The use of 32 samples per problem reduces the variance of the per-problem estimate (by making it more likely to detect a correct answer if one exists), but it does not increase the effective sample size for comparing models, which is bounded by the number of problems. A standard approach for small test sets — bootstrapping or binomial confidence intervals — is not employed.

Limitation 5: The Divergence-Weighted Advantage Does Not Use the Same Divergence Metric as the Descriptive Analysis

The assumption or constraint. The paper's central descriptive and causal analyses (Sections 2–4) all use Jensen-Shannon divergence between the base model and the RL-fine-tuned model ( $D_{JS}(\pi_{\text{base}} \parallel \pi_{\text{RL}})$ ) as the metric for identifying where meaningful distributional shifts occur. This is the quantity that reveals sparsity, that defines the cross-sampling intervention points, and that frames the fine-grained mechanics analysis.

However, the divergence-weighted advantage experiment (Section 5) uses a fundamentally different divergence quantity: KL divergence with respect to the old policy ( $D_{KL}(\pi_{\theta_{\text{old}}} \parallel \pi_{\theta})$ ), where $\pi_{\theta_{\text{old}}}$ is the policy from the previous update iteration, not the pretrained base model. The old-policy KL measures how much the policy changed in a single training step, while the base-policy JS measures the cumulative change from the starting point. These are different objects with different interpretations: a token position that has high old-policy KL at step 150 may have already accumulated substantial base-policy JS by step 100 and be changing only incrementally by step 150, while a position with low old-policy KL may have zero base-policy JS (never changed at all).

The paper acknowledges this implicitly by calling it "old-policy KL" and noting it "quantifies the magnitude of recent policy updates at each token position" (Section 5.2), but does not discuss whether this metric aligns with the base-policy JS divergence that the rest of the paper is built around.

The consequence. The divergence-weighted advantage experiment is presented as a validation of the paper's central thesis — "if only a small fraction of tokens drive improvements, can training be more effectively guided by modulating token-level learning signals according to these divergences?" (Section 5 preamble). But the experiment tests a different divergence than the one the thesis is about. If old-policy KL and base-policy JS identify different token positions as "high-divergence," then the experiment does not directly test whether weighting by the sparsity pattern documented in Section 2 improves training. It tests whether weighting by recent policy change improves training — a related but distinct hypothesis.

The results in Table 2 show that both high-KL boost and low-KL boost outperform the baseline. This symmetric improvement is difficult to interpret under the paper's sparsity thesis. If the sparse, high-base-JS tokens are the key drivers of improvement, then focusing on them (high-KL boost) should help, and focusing away from them (low-KL boost) should hurt. The fact that both help suggests either: (1) old-policy KL does not align well with base-policy JS (so high and low old-policy KL positions are both different from the "critical sparse positions" identified in Sections 2–4), or (2) the relationship between divergence and optimal learning signal is genuinely bidirectional (sharpening at changing positions helps for some reasons, exploring at unchanged positions helps for others), in which case the paper's framing of "sparse positions are the key drivers" is oversimplified.

What evidence exists in the paper. The paper does not compare old-policy KL distributions to base-policy JS distributions. It does not measure whether the token positions identified as high-divergence by the two metrics overlap. The $\alpha$ annealing schedules (Section 5.2, Appendix A.2.3) start at different steps for the two configurations (step 100 for high-KL, step 150 for low-KL), which adds a confounding factor — the training dynamics at step 100 may differ from step 150, making it hard to attribute performance differences purely to the divergence weighting.

Mitigation status. The paper does not experiment with base-policy JS divergence weighting, which would be the direct operationalization of Sections 2–4's findings. Computing base-policy JS during training would require maintaining a frozen copy of the base model and evaluating distributions at each step, which is computationally expensive but technically feasible. The authors present the divergence-weighted advantage as a "diagnostic intervention" and "complementary diagnostic tool" (Section 5.2) rather than a definitive method, but the gap between the concept tested (old-policy KL) and the concept analyzed throughout the paper (base-policy JS) weakens the claimed connection between the diagnostic and the central thesis.

Limitation 6: The Paper Does Not Distinguish Whether Cross-Sampling Works via Re-Ranking or via Rare Tail Promotions

The assumption or constraint. The fine-grained mechanics analysis (Section 4) establishes that at high-divergence positions, RLVR primarily reallocates probability mass within an existing candidate set — over 80% of RL top-1 tokens are already in the base model's top-3 (Figure 22), and fewer than 5% (DAPO) to ~0% (SimpleRL) of RL top-1 tokens have base probability below 0.01 (Figure 10). However, the cross-sampling experiments (Section 3) do not disentangle these two mechanisms. When the cross-sampling intervention substitutes an RL-sampled token at a high-divergence position, the paper does not report whether that token was a "re-ranking" (already in the base top-3) or a "tail promotion" (base probability < 0.01). The effective token counts in Table 1 aggregate both types of substitutions.

The paper explicitly acknowledges this as an open question in the takeaways of Section 4: "Although substantial reshuffling or promotion of low-probability tokens occurs only rarely, these events may still be important and contribute meaningfully to improved reasoning performance, a view consistent with the observation that stronger-performing models exhibit such behaviors more frequently." But it does not test this hypothesis experimentally.

The consequence. The paper cannot determine whether the functional importance of high-divergence positions comes primarily from re-ranking within already-plausible candidates, or from the rare promotions of initially unlikely tokens. This distinction matters for both scientific understanding and practical methodology. Scientifically, if rare tail promotions are the critical changes that drive reasoning improvements (with re-rankings being incidental or easier-to-achieve byproducts), then the "promotion-not-invention" framing in Section 4 is misleading — RLVR's mechanism would be better described as "mostly re-ranking, but the rare tail promotions are what actually matter." Practically, if tail promotions are the key drivers, then training methods should focus on enabling and identifying these rare events, which are undersampled in standard RLVR objectives (as the paper notes in Section 5.2 when discussing future work on "effectively promoting the rare actions").

Conversely, if re-rankings are entirely sufficient — and tail promotions are incidental — then the paper's central claim that RLVR operates within the base model's existing capability is fully vindicated, and training methods can safely ignore the tail. The current analysis cannot distinguish between these two interpretations.

What evidence exists in the paper. The correlation between model strength and tail promotion frequency provides suggestive evidence: DAPO (44% accuracy) promotes more tail tokens than SimpleRL (25% accuracy), and DAPO with clip-upper 0.28 promotes more tail tokens than clip-upper 0.2 (Appendix Figure 37). However, correlation does not establish that the tail promotions cause the improved performance — the broader exploration enabled by clip-higher could improve performance through more effective re-ranking at a larger number of positions, with the tail promotions being a side effect of the same exploration mechanism.

The cross-sampling experiments could address this with a stratified analysis: separate the high-divergence intervention positions into "re-ranking positions" (RL top-1 was in base top-3) and "tail-promotion positions" (RL top-1 had base probability < 0.01), and run cross-sampling with interventions only at one type or the other. If interventions at re-ranking positions alone recover RL performance, tail promotions are not necessary. If interventions at tail-promotion positions are disproportionately impactful per-token, they are the critical changes. If both are needed, the mechanism is mixed. The paper does not perform this experiment.

Mitigation status. The paper identifies this as an open question but does not attempt to resolve it. The divergence-weighted advantage experiment (Section 5) hints at a possible approach — weighting schemes that "more effectively promote the rare actions" — but does not implement a tail-promotion-specific weighting. The authors flag this as a direction for future work: "These include approaches that aggregate information from token-level divergences into better signals, as well as those that more effectively promote the rare actions discussed in Section 4" (Section 5.2). This is a legitimate identification of an open problem, but it means the paper's mechanistic picture is incomplete — we know that sparse high-divergence positions matter, but not which aspect of the distributional change at those positions is driving the effect.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper fundamentally reframes how the field should think about RLVR fine-tuning: not as a process that teaches models new capabilities, but as a trajectory-steering mechanism that selectively amplifies decisions the base model already knows how to make. This is a conceptual shift of genuine significance — it replaces the dominant implicit model ("RL makes the model smarter at math") with a precise, empirically grounded alternative ("RL teaches the model to navigate its existing reasoning space more effectively through sparse, local edits to its predictive distribution").

The magnitude of this shift is substantial but not paradigm-breaking. The paper does not introduce new training algorithms or achieve state-of-the-art benchmark performance. Instead, it provides a diagnostic toolkit and conceptual vocabulary that changes how researchers frame questions about RLVR. Before this work, the natural question was "how much does RL improve accuracy?" After this work, the natural question becomes "which specific token decisions did RL change, and did those changes steer trajectories toward reasoning paths the base model already had access to?" This is a qualitative change in the granularity and causal precision of the questions the field can ask.

Reconciling prior contradictions. The paper resolves several tensions in the existing literature. The observation that RL can produce dramatic accuracy improvements (e.g., ~8% to ~44% on AIME 2024 with DAPO) while making minimal parameter-level changes (Appendix Table 3: relative gap ratio σ = 0.00143) had been noted but not explained. The token-level sparsity finding — that >83% of token distributions show near-zero divergence even under DAPO, which removes KL regularization entirely — provides the mechanism: the model remains overwhelmingly similar to its base version at most positions, and the accuracy gains come from a concentrated set of high-impact edits. This reconciles the "RL as targeted refinement" narrative (Rajani et al., 2025; Chu et al., 2025) with the concern that removing KL constraints could lead to unbounded policy drift (Yu et al., 2025) — the drift is bounded not by an explicit penalty, but by the inherent structure of the RL objective, which apparently provides implicit regularization toward the base distribution.

The paper also reconciles two competing perspectives in the token-level analysis literature. Wang et al. (2025) and Cheng et al. (2025) emphasized high-entropy tokens as the primary locus of RL-driven change, while Cui et al. (2025) warned about entropy collapse at other positions. The paper's divergence-entropy analysis (Section 2.4) shows that both perspectives capture part of the truth, but the full picture depends on the RLVR method. SimpleRL concentrates its modifications in high-entropy regions (consistent with Wang et al.), while DAPO modifies both high- and low-entropy positions (modifying confident but incorrect base-model predictions). The paper thus provides a unified framework where the entropy-change relationship is a parameter of the training method, not a fixed property of RLVR.

Research directions that become more attractive.

Token-level diagnostics for RLVR training monitoring. The paper's descriptive toolkit — divergence histograms, percentile curves, positional concentration, divergence-entropy plots — provides concrete, computable metrics for monitoring whether RL training is making the right kind of changes. A training run where divergence is diffuse rather than sparse, or where high-divergence positions are concentrated in low-entropy rather than high-entropy regions, may indicate hyperparameter problems before accuracy degradation becomes apparent. This makes token-level monitoring a viable complement to reward-curve watching.
Understanding when the base model doesn't have the right answers in its candidate set. The paper's finding that RLVR primarily re-ranks within existing candidate sets (Section 4) raises a question that was previously invisible: for problems where the base model's pass@1 is near zero, does its top-k candidate set at any position contain the tokens needed for a correct solution? If not, RLVR cannot help regardless of budget — the model needs new knowledge, which is a fundamentally different requirement than trajectory steering. This reframes the limits of RLVR in terms of the base model's distributional coverage rather than its sampling behavior.
Verifier design informed by distributional structure. The cross-sampling results show that base and RL models differ primarily at positions where token choices steer reasoning trajectories. This suggests that process reward models (PRMs) trained to evaluate reasoning steps could be made more efficient by focusing their capacity on the types of positions that empirically matter — those where local distributional divergence correlates with downstream trajectory divergence. The paper does not explore verifier training, but the connection is natural.

Research directions that become less attractive.

Global KL regularization as the primary mechanism for controlling RLVR. The paper shows that DAPO, which removes KL regularization entirely, maintains near-zero divergence at >83% of positions. This strongly suggests that KL penalties to a reference model are not the primary constraint preventing RLVR from drifting — the policy remains close to the base model through the implicit regularization of the RL objective itself (likely because the reward signal provides no gradient toward tokens that don't improve outcomes). Research focused on tuning KL penalty coefficients may be addressing a less important lever than previously assumed.
Architectural modifications to "make room" for RL-driven learning. The finding that RLVR modifies a tiny fraction of token distributions suggests that the bottleneck is not model capacity (the base model already has the right tokens in its top-k) but rather decision-making priority. Approaches that add new parameters or modules specifically for RL fine-tuning may be over-engineered relative to the problem — the existing model has sufficient representational capacity; it just needs better local decision rules.
Treating all token positions as equally important for training. The uniform token-level averaging used in standard GRPO and DAPO objectives applies the same learning signal to every position. The paper's finding that performance depends on sparse, high-divergence decisions — and the exploratory success of divergence-weighted advantages (Table 2) — suggests that uniform weighting is leaving efficiency on the table. Research that continues to use uniform token weighting without considering distributional structure is optimizing a suboptimal objective.

Follow-Up Research This Work Enables

Causal decomposition of re-ranking vs. tail promotion in cross-sampling. The paper's fine-grained analysis (Section 4) shows that RLVR's distributional changes are a mixture of common re-rankings (the RL top-1 token was already in the base top-3) and rare tail promotions (the RL top-1 token had base probability <0.01). The cross-sampling experiments establish that high-divergence positions are causally important in aggregate, but cannot determine whether the functional impact comes primarily from the frequent re-rankings or the rare tail promotions. A direct follow-up would stratify intervention positions by the base-model rank of the RL model's top-1 token: run forward cross-sampling with interventions only at positions where the RL top-1 was already in the base top-3, and separately only at positions where it was promoted from below rank 10 or below probability 0.01. Comparing the accuracy-vs-budget curves for these two strata would reveal whether the per-token impact of a tail promotion is larger than a re-ranking, and whether recovering full RL performance requires the rare events or can be achieved through re-rankings alone. A negative result where tail-promotion-only interventions fail to recover performance would strengthen the paper's "RLVR operates within existing candidate sets" thesis; a positive result where tail promotions are disproportionately impactful would require revising that thesis.

Predicting divergence positions without full distribution access. The paper's cross-sampling framework relies on oracle knowledge of token-level divergences — one must compute both models' full next-token distributions at every position to identify intervention points. For the framework to become practically deployable, one needs a method for predicting which positions will be high-divergence without generating the full RL trajectory or evaluating the full distribution at every step. A natural approach: train a lightweight classifier on the base model's hidden states (or on simple distributional statistics like entropy, top-1 probability, or top-2 probability gap) to predict whether the base-vs-RL JS divergence at that position will exceed a threshold. The training signal would come from the paper's existing divergence annotations on AIME problems. The key evaluation would be: if cross-sampling interventions are guided by the predicted divergence positions (using only base model information available at inference time), what fraction of the oracle-guided cross-sampling gain is recovered? Even recovering 50% of the gain with zero RL-model distribution access would make the approach practically relevant, since the base model is cheap to run and the divergence predictor could be a small distilled model.

Online cross-sampling with divergence computed during generation. The current cross-sampling experiments use a retrospective design: generate with the RL model first, compute divergences along that fixed reference trajectory, then perform cross-sampling using those pre-computed divergence positions. This means the cross-sampled trajectories inherit the divergence map from a trajectory they do not actually follow. An online version would compute $D_{JS}(\pi_{\text{base}}(\cdot \mid x_{<t}) \parallel \pi_{\text{RL}}(\cdot \mid x_{<t}))$ at each step $t$ during generation, using the actual prefix generated by the mixed policy so far, and intervene if the online divergence exceeds a threshold. This online protocol would test whether the divergence positions are stable under the distribution shift caused by earlier interventions — if the online version requires significantly more interventions to achieve the same accuracy, the paper's effective token percentages are underestimates of the true cost. The theoretical bounds in Appendix A.1 describe exactly this online setting, so the experiment would also validate those bounds empirically. The computational cost is higher (both models must be evaluated at every step), but modern inference engines with KV-cache sharing across model variants could make this feasible at moderate batch sizes.

Cross-sampling on non-math, non-verifiable-reward tasks. Every experiment in the paper is on benchmarks with clean, verifiable ground-truth answers. The cross-sampling framework is not inherently tied to verifiable rewards — it only requires two models ( $\pi_{\text{base}}$ and $\pi_{\text{RL}}$ ) and a performance metric. A stress-test of the framework's generalizability would apply it to an RLHF-tuned model on an open-ended generation task (e.g., summarization evaluated by ROUGE or human preference, dialogue evaluated by human judgment, or creative writing evaluated by automatic metrics like BERTScore). The key question is whether the "sparse steering" mechanism persists when the RL training objective is a learned reward model rather than a verifiable ground-truth checker, and when the evaluation metric is less precise than exact-match accuracy. A negative finding — that divergence is distributed broadly rather than sparsely, or that cross-sampling at high-divergence positions does not recover RL-level quality — would indicate that the paper's central mechanism is specific to verifiable-reward domains with sharp correctness signals. A positive finding would substantially expand the paper's claimed scope.

Divergence-weighted training with base-policy JS rather than old-policy KL. The divergence-weighted advantage experiment (Section 5) uses KL divergence with respect to the old policy (from the previous update iteration), which measures recent change rather than cumulative divergence from the base model. The paper's central descriptive metric throughout Sections 2–4 is JS divergence between base and RL models — the quantity that reveals sparsity and defines cross-sampling intervention points. A direct test of whether the paper's distributional characterization translates into better training would replace the old-policy KL with base-policy JS in the weighting scheme: $w_t = 1 + s(\sigma(\alpha \cdot D_{JS}(\pi_{\text{base}}(\cdot \mid x_{<t}) \parallel \pi_{\theta}(\cdot \mid x_{<t})) - 0.5))$ . Computing base-policy JS during training requires maintaining a frozen copy of the base model and evaluating both distributions at each step, which is memory-intensive but feasible for 7B-scale models with parameter-efficient fine-tuning or by computing distributions only at a subset of positions. The prediction: if the paper's thesis is correct that sparse base-policy divergence positions are the key drivers of learning, base-policy JS weighting should outperform both the uniform baseline and the old-policy KL weighting. A null result — base-policy JS weighting does not improve over old-policy KL weighting — would indicate that recent change (measured by old-policy KL) is a better signal for modulating learning than cumulative change (measured by base-policy JS), which would complicate the paper's narrative that cumulative sparsity is the essential structural feature.

Difficulty-stratified analysis of divergence sparsity. The paper treats all AIME problems uniformly when computing divergence statistics, but problems vary substantially in difficulty and in the base model's baseline accuracy. A natural extension is to bin problems by the base model's pass@1 rate (similar to the difficulty estimation approach used in compute-optimal test-time scaling analyses) and examine whether the sparsity of RLVR-induced divergence varies with difficulty. The prediction from the paper's trajectory-steering framework: on problems where the base model has non-trivial pass@1 (it sometimes finds correct solutions), RLVR should primarily refine existing reasoning paths and divergence should be sparse. On problems where the base model's pass@1 is near zero, RLVR may need to make more extensive changes — either more divergent positions, or higher divergence at each position — because there are no "pre-existing correct paths" to steer toward. If this prediction holds, it would establish a boundary condition for the paper's mechanism and connect it to the broader literature on when test-time compute is vs. is not effective relative to pretraining. The MATH dataset's built-in difficulty labels (Levels 1–5) provide a natural stratification, though the paper correctly notes that model-specific difficulty (pass@1 rate) is more predictive than human-assigned difficulty.

Practical Applications and Downstream Use Cases

Token-level monitoring dashboards for RLVR training runs. The paper's descriptive toolkit — divergence histograms, percentile curves, positional concentration plots, and divergence-entropy relationships — can be operationalized as training-run diagnostics. A production RLVR training system could compute these statistics periodically (e.g., every 50 steps) on a fixed validation set and generate a dashboard showing: (a) the fraction of token positions with near-zero divergence (should remain high; a sudden drop indicates the policy is drifting broadly rather than editing sparsely), (b) the 95th and 99th percentile divergence values (should increase slowly and plateau; rapid growth indicates over-optimization), (c) positional concentration (should remain concentrated at sequence starts/ends; mid-sequence spread indicates the policy is making changes at routine rather than strategic positions), and (d) the entropy distribution of high-divergence positions (should span a range under DAPO-like methods; concentration in only the high-entropy tail indicates the method is not overriding confident errors, while concentration in only the low-entropy regime indicates entropy collapse). These diagnostics would allow practitioners to detect training pathologies — over-optimization, entropy collapse, loss of sparsity — hours or days before they manifest as reward degradation, enabling early hyperparameter intervention.

Efficient inference through selective model switching. The finding that the base model's behavior is essentially identical to the RL model's at >83–98% of token positions suggests a practical inference optimization: generate most tokens using the base model (which may be smaller, faster, or more memory-efficient) and switch to the RL model only at positions predicted to be high-divergence. This is a direct inversion of speculative decoding — instead of using a small model to approximate a large one, use the base model as the default and the RL model as a targeted corrector. The engineering challenge (and the key missing piece) is predicting high-divergence positions without computing the RL model's full distribution, as discussed in the follow-up research directions above. If a lightweight divergence predictor can identify even 50% of the oracle high-divergence positions using only base-model features, the resulting system would generate responses that are significantly closer to RL-model quality than pure base-model generation, while using the expensive RL model for only a small fraction of tokens. For deployments where the RL model is substantially larger or more expensive than the base model (e.g., a 70B RL model vs. a 7B base model distilled from it), this could translate to 5–10× latency and cost reductions.

Targeted data augmentation for SFT and distillation. The paper's demonstration that RLVR's gains are concentrated in sparse token-level decisions has implications for how training data should be constructed for distillation or supervised fine-tuning. Rather than generating full RL-model trajectories and training on all tokens equally (which includes >80% of positions where the RL model agrees with the base model), one could generate trajectories and only train on the high-divergence positions — either by masking the loss at low-divergence positions, or by constructing contrastive training pairs that juxtapose the base model's and RL model's choices at high-divergence positions. This "sparse distillation" approach would focus the student model's capacity on exactly the decisions that drive reasoning improvements, potentially achieving comparable downstream accuracy with less training data and compute. The paper's cross-sampling results provide direct evidence that this would work: if injecting only ~4–8% of RL tokens into base-model generations recovers full RL performance, then a student model that learns to make RL-like choices at only those positions should similarly recover most of the RL model's advantage. The key parameter to tune would be the divergence threshold for deciding which positions to train on — the paper's cross-sampling threshold variation (Appendix A.2.2) provides a starting point, but a systematic sweep across thresholds measuring downstream student accuracy would be needed.

Improved reward shaping in RLVR. The paper's divergence-entropy analysis reveals that DAPO modifies both high-entropy and low-entropy positions, while SimpleRL concentrates modifications in high-entropy regions. This suggests that different RLVR methods have implicit "exploration biases" — they are more or less willing to override confident base-model predictions. A practical application is to use token-level divergence and entropy statistics to design adaptive reward shaping: for domains where the base model is known to have systematic confident errors (e.g., consistently applying the wrong formula to a specific problem type), increase the effective clip-upper or add an auxiliary reward for modifying low-entropy predictions, explicitly counteracting the conservative bias that prevents SimpleRL-like methods from fixing confident mistakes. Conversely, for domains where the base model's confident predictions are mostly correct and improvements come from resolving uncertainty at ambiguous steps, maintain a conservative update regime that concentrates changes in high-entropy regions. The paper's Section 4 provides concrete metrics (top-k overlap at high-divergence positions, fraction of RL top-1 tokens with low base probability) that could be monitored during training to determine whether the current exploration bias is appropriate for the task.