Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning

ArXiv: 2506.01939

🎯 Pitch

RL training for LLM reasoning can be made dramatically more efficient by updating only the ~20% of tokens with high entropy—the critical 'forking' points—while ignoring the rest. Surprisingly, on a 32B model this sparse method actually outperforms standard full-gradient training by over 11 points on AIME’25, establishing a new state-of-the-art for base-model RLVR. The gain scales with model size, and training solely on low-entropy tokens actively harms performance.

1. Executive Summary

This paper analyzes the mechanisms of Reinforcement Learning with Verifiable Rewards (RLVR) for LLM reasoning through a novel token-entropy perspective, examining how different tokens in Chain-of-Thought (CoT) trajectories influence reasoning performance during training with Qwen3-8B, Qwen3-14B, and Qwen3-32B models on the MATH benchmark suite. The authors identify high-entropy minority tokens — a small fraction (~20%) of tokens that exhibit high generation entropy and function as "forking tokens" at critical reasoning decision points (e.g., logical connectors like "however," "since," "suppose") — while the remaining 80% of tokens are low-entropy and primarily complete linguistic structures. By restricting policy gradient updates in DAPO to only these forking tokens, the approach achieves performance matching or exceeding full-gradient RLVR — most notably delivering +11.04 points on AIME'25 and +7.71 on AIME'24 for the 32B model (setting new SoTA scores of 63.5 and 56.7 respectively for base-model-trained reasoning models under 600B parameters) — while training exclusively on the 80% lowest-entropy tokens causes severe degradation, establishing that RLVR's efficacy derives almost entirely from optimizing these high-entropy decision points and that the benefit scales with model size.

2. Context and Motivation

The Core Problem: We Don't Know Which Tokens Actually Matter in RLVR

The fundamental question this paper tackles is deceptively simple: when we train an LLM to reason using reinforcement learning with verifiable rewards (RLVR), which tokens in its Chain-of-Thought actually drive the performance improvements? This matters because, despite the explosive recent success of RLVR in producing state-of-the-art reasoning models like DeepSeek R1, Kimi K1.5, and Qwen3, the underlying mechanism remains poorly understood. Prior work has been remarkably successful at the algorithmic level — developing better RL objectives (GRPO, DAPO, VAPO), stabilizing training dynamics (clip-higher, dynamic sampling), and scaling up infrastructure — but operates under an implicit assumption: that all tokens in a reasoning trajectory contribute roughly equally to learning, and therefore all tokens should receive policy gradient updates.

The paper argues this assumption is almost certainly wrong. CoT trajectories are not homogeneous streams of symbols. They contain varied linguistic elements: logical connectors ("therefore," "however," "suppose"), mathematical expressions, word completions, code fragments, and natural language scaffolding. Some tokens represent critical decision points where the model chooses between genuinely different reasoning directions. Others are deterministic continuations of already-committed reasoning paths, generated with near-certainty given the preceding context. The paper's central contention is that treating all these tokens identically during RL training is suboptimal, and that identifying which tokens actually matter could lead to both better performance and deeper understanding.

This gap is significant for several practical reasons the authors highlight throughout the paper:

• Training efficiency: If 80% of tokens are irrelevant to reasoning improvements, computing gradients on them wastes substantial computation. While the paper doesn't focus on wall-clock speedups, the principle that a fraction of tokens drives nearly all gains has direct implications for how we allocate compute during RLVR.

• Understanding generalization: Chu et al. (2025) demonstrated empirically that RL generalizes better than supervised fine-tuning (SFT) on reasoning tasks, and the paper hypothesizes this may be directly tied to how RL handles high-entropy tokens — potentially offering a mechanistic explanation for one of the most important empirical findings in modern LLM post-training.

• Model scaling: The paper shows that the benefit of focusing on high-entropy tokens increases with model size, suggesting that larger models have more to gain from selective gradient application. This has implications for how we design RLVR pipelines for frontier-scale models.

• Algorithm design: If certain tokens are critical while others are noise (or actively harmful), current RLVR algorithms are leaving performance on the table by diluting the learning signal across all tokens uniformly.

Conflicting Intuitions and the Blind Spot in Prior Work

The paper enters a landscape where RLVR has been a spectacular empirical success but a theoretical puzzle. On one side, the results are undeniable: DeepSeek R1 demonstrated that pure RL on base models can produce sophisticated reasoning with emergent "aha moments," and Qwen3 showed that RLVR pushes reasoning performance well beyond what's achievable through supervised fine-tuning alone. On the other side, a growing body of analysis work has raised uncomfortable questions about what exactly RLVR is learning:

• Gandhi et al. (2025) found that the presence of reasoning behaviors (format, structure), rather than answer correctness, is the key driver of performance improvements in RL — implying the model learns to "look like it's reasoning" rather than necessarily learning correct reasoning content.

• Li et al. (2025) showed that structure in long CoTs matters more than content, with models easily learning to reason from demonstrations where the structure was preserved but the actual tokens were randomized.

• Vassoyan et al. (2025) and Lin et al. (2024) independently identified "critical tokens" where models are prone to errors — but their criterion for what makes a token critical was based on downstream outcome correctness (whether changing a particular token alters the final answer), not on the model's own uncertainty.

These findings collectively suggest that RLVR might be learning something more superficial than we'd hope — the form rather than the substance of reasoning. But they also reveal a critical blind spot: none of these analyses examined how the model represents uncertainty at the token level during generation. The concept of "token importance" was always defined externally (does the output change? is the answer correct?) rather than internally (does the model itself exhibit high uncertainty here?).

Where Existing Approaches Fall Short

The paper identifies several specific limitations in how prior work has approached both RLVR itself and the analysis of reasoning in LLMs:

RLVR treats all tokens uniformly. In PPO, GRPO, DAPO, and their derivatives, the policy gradient loss (Equations 2 and 4) sums over all tokens in a generated response. Every token contributes equally to the gradient, with the advantage $\hat{A}_t$ modulating the magnitude but not the fact of inclusion. This is a direct carryover from traditional RL — in continuous control, every action in a trajectory matters for learning the value function. But CoT trajectories in language are fundamentally different: many tokens are deterministic completions (suffixes of words, standard mathematical notation, boilerplate text) where the model has near-zero uncertainty. Applying policy gradients to these tokens is, at best, computational waste and, at worst, noise that interferes with learning at the truly critical decision points.

The paper makes this concrete: as shown in Figure 2(a), over 50% of tokens in Qwen3-8B's CoT outputs have entropy below $10^{-2}$ — meaning the model is essentially deterministic at those positions. Only about 20% of tokens have entropy above 0.672. The current paradigm of uniform gradient application means that 80% of the computation is spent on tokens the model already "knows."

Entropy analysis in RLVR has been aggregate, not token-level. Prior works (Yang et al., 2025; Yu et al., 2025; Yue et al., 2025b) have noted that generation entropy matters during CoT reasoning and that maintaining some level of exploration is important. But these analyses treated entropy as a summary statistic — the average entropy over all tokens, or the entropy of the final answer distribution. They did not examine the distribution of entropy across individual token positions within a trajectory. This aggregation masks the heterogeneity the paper uncovers: a uniform entropy increase might help at high-entropy forks but actively harm performance at low-entropy continuations (as the temperature experiments in Figure 3 demonstrate).

The "entropy bonus" approach from traditional RL is misguided for LLMs. In standard RL, adding an entropy bonus to the training loss is a well-established method to encourage exploration (Schulman et al., 2017; Mnih et al., 2016). This uniformly increases the entropy of all actions. The paper argues this is precisely the wrong approach for LLM CoTs, because it would increase entropy at deterministic continuation tokens that should be low-entropy to maintain linguistic coherence. The paper's temperature manipulation experiments (Figure 3) provide direct evidence: increasing $T_{\text{low}}$ (temperature for low-entropy tokens) causes nonsensical outputs and degrades performance, while selectively increasing $T_{\text{high}}$ (temperature for high-entropy forking tokens) improves performance. This suggests that current RLVR implementations that either omit entropy bonuses entirely or apply them uniformly are missing an important asymmetry.

"Critical token" identification has relied on outcome-based signals, not model-internal uncertainty. The most closely related prior work identifies important tokens by their causal effect on outputs — if replacing a token changes the final answer, it's "critical" (Lin et al., 2024), or if the model is error-prone at certain decision points, those are "critical" (Vassoyan et al., 2025). While these approaches identify genuinely important positions, they require either ground-truth knowledge (which answer is correct?) or counterfactual interventions (what if we change this token?). The paper proposes a fundamentally different criterion: token entropy, which is a property of the model's own output distribution — no external correctness signal needed. The claim is that entropy at token position $t$ serves as a more "native" signal that reflects the model's inherent uncertainty about reasoning direction, potentially capturing important decision points even when the model ultimately produces a correct (or incorrect) answer.

The exploration-exploitation dynamics of RLVR are poorly characterized. While it's known that RLVR training involves a balance between exploring new reasoning strategies and exploiting known good ones, the token-level mechanism of this balance is not understood. The paper's analysis reveals something subtle: RLVR training primarily adjusts the entropy of already-high-entropy tokens, while low-entropy tokens remain stable (Figure 4 and Figure 5). This suggests that exploration happens not uniformly but at specific "exploration sites" (the forking tokens) that are to some extent pre-determined by the base model's pretrained distribution. This challenges the assumption that RL can freely explore any aspect of reasoning — it may be constrained to exploring primarily at points where the base model already exhibits uncertainty.

How This Paper Positions Itself

The paper frames its contribution as bridging the gap between the empirical power of RLVR and the lack of mechanistic understanding of what RLVR actually does. It doesn't propose a new RL algorithm (it uses DAPO as-is, merely with gradient masking). It doesn't claim to discover a new training technique that hadn't been tried. Instead, it proposes a new analytical lens — token entropy patterns — and uses this lens to:

1. Characterize what makes certain tokens special in CoT reasoning (Section 3: the entropy distribution is bimodal, high-entropy tokens are logical forks, and maintaining their high entropy matters for performance).

2. Analyze how RLVR interacts with these patterns (Section 4: RLVR largely preserves the base model's entropy patterns, primarily adjusting already-high-entropy tokens upward).

3. Validate the importance of forking tokens by showing that training on them alone matches or exceeds full training (Section 5), while training on low-entropy tokens causes degradation.

The paper explicitly positions its contribution as analytical rather than algorithmic. The key result is demonstrating that high-entropy minority tokens are the causal drivers of RLVR's effectiveness, not proposing a new method per se (though the gradient-masking approach is new, it's presented more as an experimental tool to test the hypothesis than a method to be adopted wholesale).

There are two important theoretical stakes the paper raises that go beyond the immediate experimental results:

First: a potential reconciliation of the "RL generalizes, SFT memorizes" finding. Chu et al. (2025) showed this pattern empirically but didn't provide a mechanism. The paper's Discussion 1 hypothesizes that SFT, by pushing output logits toward one-hot distributions, collapses entropy at forking tokens — destroying the model's flexibility to explore alternative reasoning paths. RLVR, by contrast, preserves or increases entropy at these forks, maintaining reasoning flexibility that generalizes to novel problems. This is a mechanistic hypothesis that, if validated, would be a significant step toward understanding why post-training methods differ in their generalization properties.

Second: an explanation for why LLM CoTs have distinct entropy patterns compared to traditional RL trajectories. In traditional RL, actions are sampled from distributions with uniform uncertainty (e.g., Gaussian policies with fixed standard deviation). LLM CoTs, by contrast, show a sharp entropy disparity: most tokens are near-deterministic, a few are high-entropy. The paper attributes this to the combination of (a) large-scale pretraining on language data, which forces most tokens to conform to learned linguistic structures (producing low entropy), and (b) the inherent ambiguity at logical transitions and decision points, where the pretraining distribution is multimodal (producing high entropy). This framing connects the paper's token-entropy analysis to fundamental questions about how language model pretraining shapes the "action space" available to RL at reasoning time.

The paper also positions itself as extending rather than contradicting findings like those of Gandhi et al. (2025) and Li et al. (2025). Those works argued that RLVR learns format and structure more than content. The paper's token-entropy analysis doesn't refute this — but it adds precision: if RLVR is learning format, it's learning it through specific high-entropy tokens that govern structure, not through the low-entropy tokens that fill in the determined content. The forking tokens (words like "however," "suppose," "therefore") are precisely the tokens that define reasoning structure — so the finding that these are what RLVR optimizes is consistent with the format-learning hypothesis, while providing a more granular, token-level mechanism.

The Immediate Technical Question

With this context established, the paper's technical program becomes clear. It proceeds in three stages: (1) establish that high-entropy minority tokens exist and function as reasoning forks (Section 3), (2) show that RLVR training preserves and reinforces the base model's entropy patterns rather than creating new ones (Section 4), and (3) demonstrate through gradient ablation experiments that these forking tokens are causally responsible for performance gains in RLVR (Section 5). The first stage answers "what are these tokens?", the second "how does RLVR interact with them?", and the third "do they actually matter?" The Discussion (Section 6) then steps back to consider broader implications — why SFT and RL differ, why entropy bonuses are suboptimal, and how pretrained knowledge shapes the RL exploration landscape.

3. Technical Approach

3.1 Reader Orientation

This paper is primarily an analytical investigation rather than a new method paper — it develops a conceptual framework for understanding which tokens in a Chain-of-Thought reasoning trajectory actually drive learning during RLVR, and then validates this framework through carefully designed gradient ablation experiments. The core idea is that in CoT reasoning, only a small fraction (~20%) of tokens exhibit high generation entropy and function as "forking tokens" at critical decision points, while the vast majority are deterministic continuations; the paper hypothesizes that these forking tokens are the causal drivers of RLVR's effectiveness, and tests this by showing that restricting policy gradient updates to only these tokens matches or exceeds full-gradient training while training on the remaining low-entropy tokens causes severe degradation.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components that interact in a pipeline:

1. Base LLM (Qwen3-8B/14B/32B) — the pretrained language model that generates Chain-of-Thought reasoning responses. It produces both the token sequences and their associated logit distributions (from which entropy is computed), serving as the starting point for all analysis and the substrate for RLVR training.

2. Entropy Analyzer — a computational module that takes the logit outputs from the base model (or intermediate RLVR checkpoints) during generation, computes per-token entropy using the standard definition $H_t = -\sum_{j=1}^{V} p_{t,j} \log p_{t,j}$, and partitions tokens into high-entropy (forking) and low-entropy (continuation) groups based on a percentile threshold. This component produces both the statistical characterization of entropy patterns (Section 3) and the gradient masks used during training (Section 5).

3. RLVR Training Pipeline (DAPO) — the reinforcement learning system that trains the model using verifiable outcome rewards (correct/incorrect on math problems). It generates groups of responses per question, computes token-level advantages using group-relative normalization, and applies policy gradient updates. The key modification in this paper is the addition of an entropy-based gradient mask that zeroes out the policy gradient loss for tokens whose entropy falls below a batch-specific threshold $\tau^B_\rho$.

4. Verification Module — the automated correctness checker that compares model outputs to ground-truth answers using the is_equivalent() function, assigning binary rewards (1.0 for correct, 0.0 for incorrect) that feed into the advantage computation.

Information flows as follows: a math question enters the system → the current policy (base model or RLVR checkpoint) generates multiple responses with temperature $T = 1.0$ → the entropy analyzer computes per-token entropies from the policy's output logits → the verification module scores each response's final answer → the DAPO pipeline computes token-level advantages from these binary rewards using group statistics → the entropy-based mask filters which tokens contribute to the policy gradient loss → only high-entropy tokens (top $\rho = 20\%$ within the batch) receive gradient updates → the model parameters are updated → repeat for the next training batch.

3.3 Roadmap for the Deep Dive

• First, the entropy analysis methodology (Section 3 of the paper) — how token entropy is computed, what statistical patterns emerge, and how the forking-token hypothesis is quantitatively validated through temperature-manipulation experiments. This establishes the phenomenon that motivates everything else.

• Second, the entropy evolution tracking (Section 4 of the paper) — how the positions and magnitudes of high-entropy tokens change (or don't change) during RLVR training, using overlap ratios and percentile trajectories. This reveals that RLVR preserves the base model's entropy patterns rather than creating new ones.

• Third, the gradient-masked DAPO formulation (Section 5.1 of the paper) — the precise mathematical modification to the DAPO objective that restricts policy gradient updates to high-entropy tokens, including the thresholding mechanism and normalization adjustment. This is the experimental apparatus for testing causality.

• Fourth, the training and evaluation setup — the hyperparameters, datasets, benchmarks, and evaluation protocol. Understanding these is essential because the paper's claims about performance depend on specific implementation choices (model sizes, response lengths, RL algorithm details).

• Fifth, the temperature manipulation experiments — the controlled decoding experiments that independently vary temperature for high-entropy and low-entropy tokens to confirm the functional role of forking tokens. These provide the causal evidence linking entropy to reasoning quality before the RLVR training results.

3.4 Detailed, Sentence-Based Technical Breakdown

This is an analytical paper whose central mechanistic claim is that high-entropy minority tokens in Chain-of-Thought trajectories function as "forks" that determine reasoning direction, and these tokens alone drive the performance improvements observed during RLVR training. The paper validates this claim through a sequence of observational analyses (characterizing entropy patterns in base model outputs and tracking their evolution during RLVR), controlled experiments (manipulating decoding temperature for different token groups), and causal interventions (training with gradient masks that zero out low-entropy token contributions). The methodology is built on top of an existing state-of-the-art RLVR algorithm (DAPO) and uses it as-is except for the addition of the entropy-based gradient mask.

Token Entropy Definition and Computation

The fundamental quantity throughout the paper is token-level generation entropy, defined for each position $t$ in a generated sequence as the entropy of the model's output probability distribution over the vocabulary at that position.

$H_t := -\sum_{j=1}^{V} p_{t,j} \log p_{t,j}$

where $V$ is the vocabulary size, and $p_{t,j}$ is the $j$-th component of the probability vector $\mathbf{p}_t \in \mathbb{R}^V$ defined by:

$\mathbf{p}_t = \pi_\theta(\cdot \mid q, o_{<t}) = \text{Softmax}\left(\frac{\mathbf{z}_t}{T}\right)$

where $\pi_\theta$ denotes the LLM parameterized by $\theta$, $q$ is the input query, $o_{<t} = (o_1, o_2, \ldots, o_{t-1})$ represents the previously generated tokens, $\mathbf{z}_t \in \mathbb{R}^V$ denotes the pre-softmax logits at time step $t$, $\mathbf{p}_t \in \mathbb{R}^V$ is the corresponding probability distribution over the vocabulary, and $T \in \mathbb{R}$ is the decoding temperature.

What it computes: The entropy $H_t$ measures the uncertainty or "spread" of the model's token prediction distribution at position $t$. If the model is highly certain (one token has probability near 1, all others near 0), $H_t$ approaches 0. If the model is maximally uncertain (uniform distribution over all $V$ tokens), $H_t$ approaches $\log V$. The computation first applies the softmax with temperature scaling to convert raw logits to probabilities, then sums $-p \log p$ over all $V$ vocabulary entries. The output is a single non-negative scalar per token position.

Why this form: Entropy is the natural information-theoretic measure of uncertainty in a probability distribution. Unlike alternatives such as max probability (which only captures the peak and ignores the shape of the tail) or variance (which is not well-defined for categorical distributions), entropy captures the full distributional spread: it is low when probability mass is concentrated on few tokens and high when mass is spread across many plausible tokens. The temperature $T$ parameter allows controlling the sharpness of the softmax — higher $T$ flattens the distribution (increasing entropy), lower $T$ sharpens it (decreasing entropy). Using $T = 1.0$ throughout most experiments means the entropy reflects the model's natural, unscaled uncertainty.

Critical clarification from the paper: The term "token entropy" refers to the entropy of the generation distribution at index $t$, not a property of the specific token $o_t$ that happens to be sampled. The paper states this explicitly: "'Token entropy' corresponds to the token generation distribution, not a specific token... the token entropy $H_t$ refers to the entropy at index $t$, which is determined by the token generation distribution $p_t$ rather than by any specific token $o_t$ sampled from $p_t$." This distinction matters because the same surface token (e.g., the word "the") could have high entropy in one context and low entropy in another, depending on what alternatives the model considers plausible at that position. In off-policy settings where sequences are generated by a rollout policy $\pi_\phi$ different from the training policy $\pi_\theta$, the entropy is still computed using $\pi_\theta$ — it measures the training policy's uncertainty when evaluating the given sequence, not the generating policy's uncertainty.

Statistical Analysis of Entropy Patterns in CoT Outputs

To characterize the entropy patterns that motivate the entire paper, the authors conduct a large-scale statistical analysis of token entropies in Chain-of-Thought outputs from Qwen3-8B.

Data collection procedure: The authors use Qwen3-8B (Yang et al., 2025) to generate responses for queries from AIME'24 and AIME'25, using a decoding temperature of $T = 1.0$. They enforce "thinking mode" for every question, meaning the model is prompted to produce explicit step-by-step reasoning before giving a final answer. This yields over $10^6$ response tokens for analysis. For each token position in each response, the entropy is computed using Equation (1) from the model's output logits at that position.

Entropy Pattern 1 — The bimodal distribution: The paper reports that "over half the tokens (approximately 50.64%) have entropy below $10^{-2}$, while only 20% of tokens have entropy greater than 0.672." This is visualized in Figure 2(a), which plots frequency against entropy on a log-scale Y-axis. The distribution is sharply bimodal: a massive spike at very low entropy (near zero) and a long, sparse tail extending to higher entropy values. This means that in a typical CoT, roughly half the tokens are generated with near-certainty (the model assigns probability close to 1 to a single token), while a small minority exhibit substantial uncertainty.

Entropy Pattern 2 — The semantic roles: To understand what the high-entropy and low-entropy tokens actually represent linguistically, the authors select the 100 tokens with the highest and lowest average entropy from the $10^6$-token corpus, filtering to only include tokens with frequency above 100 (to mitigate noise). The results are visualized as word clouds in Figure 2(b) and 2(c):

• Highest-entropy tokens (Figure 2b) are logical connectors and reasoning operators: "wait," "however," "unless" (contrasts/shifts), "thus," "also" (progression/addition), "since," "because" (causality), "suppose," "assume," "given," "define" (introducing assumptions or conditions in mathematical derivations). The paper characterizes these as tokens that "bridge the logical connection between two consecutive parts of reasoning."

• Lowest-entropy tokens (Figure 2c) are word suffixes, source code fragments, and mathematical expression components — elements of language that follow deterministically from their prefixes. The paper describes these as tending to "complete the current part of a sentence or finish constructing a word."

The forking token concept: Based on these two patterns, the paper introduces the term "forking tokens" to describe the high-entropy minority. The intuition is that these tokens represent points in the reasoning trajectory where the model faces a genuine choice between multiple plausible directions — a "fork" in the reasoning path. Choosing "however" versus "therefore" steers the logic in opposite directions; choosing "suppose" versus "given" introduces different types of mathematical assumptions. The low-entropy tokens, by contrast, are the deterministic continuations that "follow the path" once a direction has been committed.

Quantitative threshold: The paper establishes the entropy threshold $h_{\text{threshold}} = 0.672$ to separate forking tokens from others, computed as the 80th percentile of the entropy distribution among the $10^6$ sampled tokens. This means exactly 20% of tokens are classified as forking tokens by this criterion.

Temperature Manipulation Experiments — Causal Validation of the Forking Hypothesis

The word cloud analysis in Figure 2 is correlational: it shows that high-entropy tokens tend to be logical connectors, but doesn't prove that high entropy at these positions causes better reasoning. To establish causality, the paper conducts controlled decoding experiments where it independently manipulates the temperature for forking tokens versus other tokens during evaluation.

The manipulation mechanism: The paper modifies the decoding process so that each token position receives a customized temperature based on whether it's classified as a forking token. Specifically, the adjusted probability distribution $\mathbf{p}'_t$ for token $t$ is:

$\mathbf{p}'_t = \text{Softmax}\left(\frac{\mathbf{z}_t}{T'_t}\right)$

where $\mathbf{z}_t \in \mathbb{R}^V$ are the pre-softmax logits for token $t$, and $T'_t \in \mathbb{R}$ is the adjusted temperature defined piecewise:

$T'_t = \begin{cases} T_{\text{high}} & \text{if } H_t > h_{\text{threshold}} \\ T_{\text{low}} & \text{otherwise} \end{cases}$

where $h_{\text{threshold}} = 0.672$ is the forking token classification threshold (the 80th percentile from the prior analysis), $T_{\text{high}} \in \mathbb{R}$ is the temperature applied to forking tokens, and $T_{\text{low}} \in \mathbb{R}$ is the temperature applied to all other tokens. $H_t$ is computed from the unmodified logits (with $T=1.0$) to determine classification before temperature adjustment.

What it computes: For each token position during decoding, the procedure first computes the standard entropy $H_t$ at $T=1.0$ to classify the token as forking or not, then applies a customized temperature that either sharpens or flattens the distribution depending on the classification. If $T'_t < 1$, the distribution becomes peakier (lower entropy, more deterministic); if $T'_t > 1$, it becomes flatter (higher entropy, more random). The operation samples the next token from the resulting adjusted distribution.

Why this form: The piecewise temperature allows cleanly separating the effect of entropy at forking tokens from entropy at continuation tokens. If increasing entropy only at forks improves performance while increasing entropy at continuations degrades it, that provides causal evidence that the forking function is real — the high entropy isn't just a statistical curiosity but a functional necessity for effective reasoning. The classification uses unmodified entropy ($T=1.0$) to avoid circularity (the temperature adjustment would change which tokens appear high-entropy).

Experimental design: The experiments use Qwen3-8B with a maximum response length of 28,672 tokens, evaluated on AIME 2024 and AIME 2025. Two sweeps are conducted:

• Red curve (Figure 3): Fix $T_{\text{low}} = 1.0$ and vary $T_{\text{high}}$ across values: 0.0, 0.05, 0.3, 0.6, 1.0, 2.0, 3.0, 5.0. This tests: what happens when we change only the forking token temperature?
• Blue curve (Figure 3): Fix $T_{\text{high}} = 1.0$ and vary $T_{\text{low}}$ across the same values. This tests: what happens when we change only the continuation token temperature?

Results and interpretation:

• Lowering $T_{\text{high}}$ hurts performance dramatically. When forking tokens are forced to be more deterministic ($T_{\text{high}} < 1.0$), the average AIME score drops from approximately 71-72 at $T_{\text{high}} = 1.0$ to around 62-63 at $T_{\text{high}} = 0.05$, and collapses to approximately 20 at $T_{\text{high}} = 0.0$ (fully deterministic forking tokens). This demonstrates that high entropy at forking tokens is functionally necessary — removing the model's ability to consider multiple alternatives at decision points catastrophically degrades reasoning.

• Increasing $T_{\text{high}}$ improves performance. Raising forking token temperature to moderate values ($T_{\text{high}} = 2.0$ to $3.0$) produces a measurable improvement over the $T_{\text{high}} = 1.0$ baseline, with the average score rising from approximately 71 to 72+. This suggests that the model's natural entropy at forks (at $T=1.0$) is slightly below optimal — giving it more randomness at decision points helps explore better reasoning paths.

• Lowering $T_{\text{low}}$ has relatively mild effects. Reducing temperature for continuation tokens ($T_{\text{low}} < 1.0$) only slightly degrades performance compared to the dramatic effect at forking tokens. The blue curve in Figure 3 shows a much gentler decline.

• Increasing $T_{\text{low}}$ destroys performance. When continuation tokens are made more random ($T_{\text{low}} > 1.0$), performance degrades substantially — and, critically, the paper notes that increasing $T_{\text{low}}$ "can even cause LLMs generating nonsensical outputs." This makes sense: continuation tokens are typically word completions and syntactic scaffolding where randomness disrupts basic language coherence, making the output unreadable regardless of reasoning quality.

The key asymmetry: The experiments reveal an asymmetry that is central to the paper's thesis. Forking tokens benefit from higher temperature (more exploration at decision points is good), while continuation tokens are harmed by higher temperature (randomness at deterministic positions breaks language coherence). This asymmetry directly motivates why a uniform temperature increase (or uniform entropy bonus during RL) is suboptimal — it raises entropy at both useful and harmful locations — and why a selective approach that targets only forking tokens should outperform.

Connection to the 20% threshold: The threshold $h_{\text{threshold}} = 0.672$ (80th percentile) is not arbitrarily chosen — it emerges from the temperature manipulation results. The paper shows that treating the top 20% of tokens differently from the bottom 80% is empirically justified because these two groups exhibit fundamentally different functional relationships between entropy and performance. The 20% number is thus both a descriptive statistic (20% of tokens exceed this entropy) and a prescriptive threshold (this is the fraction whose entropy behavior matters for reasoning quality).

Tracking Entropy Evolution During RLVR Training

Having established that high-entropy minority tokens exist and matter for reasoning in the base model, the paper next investigates how RLVR training changes (or preserves) these entropy patterns. This is Section 4 of the paper and uses the Qwen3-14B base model trained with vanilla DAPO.

Training procedure for entropy tracking: The paper applies DAPO to the Qwen3-14B base model using the standard configuration described in Section 5.2 (training batch size 512, mini-batch size 32, learning rate $10^{-6}$, no learning rate warmup or scheduling, maximum response length 20,480, cache length 4,096, clip-higher with $\epsilon_{\text{high}} = 0.28$ and $\epsilon_{\text{low}} = 0.2$, no KL divergence loss, no entropy loss). At various checkpoints during training (step 0, 16, 112, 160, 480, 800, 864, 840, 1280, 1360), the model is used to generate 16 responses per question across the six evaluation benchmarks (AIME'24, AIME'25, AMC'23, MATH500, Minerva, OlympiadBench). For each token in these responses, logits are computed and entropy is calculated using Equation (1).

Analysis 1 — Position overlap of high-entropy tokens: The paper computes, for each intermediate checkpoint, the overlap ratio between the set of token positions in the top 20% of entropy for that checkpoint and the corresponding sets for (a) the base model (step 0) and (b) the final RLVR model (step 1360). The overlap ratio is the fraction of positions that appear in the top 20% for both models being compared — essentially "do the same token positions remain high-entropy throughout training?"

Overlap Ratio with Base Model (Step 0): 100% → 98.92% → 98.70% → 93.04% → 93.02% → 93.03% → 87.45% → 87.22% → 87.09% → 86.67%

Overlap Ratio with Final RLVR Model (Step 1360): 86.67% → 86.71% → 86.83% → 90.64% → 90.65% → 90.64% → 96.61% → 97.07% → 97.34% → 100%

What this means operationally: The overlap with the base model starts at 100% (trivially, since step 0 is the base model) and gradually decreases to approximately 86.67% at convergence. Meanwhile, the overlap with the final RLVR model starts at 86.67% (at step 0, only 86.67% of the base model's high-entropy positions will still be high-entropy after training) and increases to 100%. Critically, even at convergence, 86.67% of the high-entropy token positions in the RLVR model are the same positions that were high-entropy in the base model. This is the key insight: RLVR does not fundamentally reorganize which tokens are high-entropy; it largely preserves the base model's pattern of which positions exhibit uncertainty.

Why this matters: If RLVR were learning entirely new reasoning strategies from scratch, we might expect the entropy pattern to shift dramatically — new positions would become uncertain as the model learned new decision points, and old positions would become deterministic. The high overlap (86.67% preserved) suggests instead that RLVR works within the uncertainty landscape already established by pretraining. The remaining ~13% change represents "gradual and relatively minor" adjustments, not wholesale reorganization.

Analysis 2 — Per-percentile entropy changes: To understand how much entropy changes (not just whether positions shift), the paper bins tokens by their initial entropy percentile in the base model and computes the average entropy change after RLVR for each bin. The results are plotted in Figure 4, where the X-axis shows percentile boundaries (0% to 100% in 5% increments), and the Y-axis shows the average absolute change in entropy, on a log scale.

The paper reports: "Tokens with higher initial entropy in the base model tend to exhibit larger increases in entropy after RLVR."

What this means: The function is sharply increasing — tokens that start in the bottom 50% of entropy show minimal change (their entropy stays low), while tokens in the top percentiles (especially 95-100%) show changes that are orders of magnitude larger. This is visible in Figure 4's log-scale Y-axis, where the average change for the 95-100% percentile is approximately $10^{-1}$ while for the 0-5% percentile it's around $10^{-4}$ — a 1000× difference.

Analysis 3 — Entropy percentile trajectory during training: Figure 5 tracks how different percentiles of the entropy distribution evolve over the course of RLVR training. The X-axis shows training steps, and multiple curves show the entropy value at different percentiles (0th, 10th, 20th, ..., 90th, 100th). The paper reports: "as we move from the 0th to the 100th percentile, the range of fluctuations during the whole RLVR training steadily diminishes." In other words, lower-entropy tokens show smaller variance over training; higher-entropy tokens show larger variance.

Synthesis of the evolution analysis: The three analyses collectively paint a consistent picture:

1. RLVR preserves the identity of high-entropy positions (86.67% overlap with base model at convergence).
2. RLVR primarily changes the magnitude of entropy at already-high-entropy positions (the per-percentile change is concentrated in the top percentiles).
3. High-entropy positions show the most dynamic behavior during training (largest variance in the percentile trajectories).

The paper's interpretation is that RLVR training is not creating new exploration sites — it operates on the uncertainty structure inherited from pretraining, amplifying entropy at positions the base model already found ambiguous while leaving deterministic positions largely untouched. This is, in the paper's terms, "RLVR primarily preserves the existing entropy patterns of the base models" and "RLVR primarily adjusts the entropy of high-entropy tokens, while the entropy of low-entropy tokens varies only within a small range."

The Gradient-Masked DAPO Objective

The paper's central causal experiment is modifying the DAPO training objective to exclude low-entropy tokens from policy gradient updates. This is described in Section 5.1.

Baseline: Standard DAPO. The unmodified DAPO objective (Equation 4 in the paper) is:

$J_{\text{DAPO}}(\theta) = \mathbb{E}_{B \sim \mathcal{D}, (q, a) \sim B, \{o_i\}_{i=1}^G \sim \pi_{\theta_{\text{old}}}(\cdot \mid q)} \left[ \frac{1}{\sum_{i=1}^G |o_i|} \sum_{i=1}^G \sum_{t=1}^{|o_i|} \min\left( r_i^t(\theta) \hat{A}_i^t, \text{clip}\left(r_i^t(\theta), 1 - \epsilon_{\text{low}}, 1 + \epsilon_{\text{high}}\right) \hat{A}_i^t \right) \right]$

subject to the constraint that $0 < |\{o_i \mid \text{is\_equivalent}(a, o_i)\}| < G$ (each batch must contain at least one correct and one incorrect response).

Here:

• $\mathcal{D}$ is the training dataset of queries $q$ and ground-truth answers $a$
• $B$ is a batch sampled from $\mathcal{D}$
• $\{o_i\}_{i=1}^G$ is a group of $G$ responses generated by the rollout policy $\pi_{\theta_{\text{old}}}$ for query $q$
• $|o_i|$ is the length (number of tokens) of response $i$
• $r_i^t(\theta) = \frac{\pi_\theta(o_i^t \mid q, o_i^{<t})}{\pi_{\theta_{\text{old}}}(o_i^t \mid q, o_i^{<t})}$ is the importance sampling ratio for token $t$ of response $i$, measuring how much more (or less) likely the current policy is to generate this token compared to the old policy
• $\hat{A}_i^t$ is the estimated advantage for token $t$ of response $i$, computed as: $\hat{A}_i^t = \frac{R_i - \text{mean}(\{R_i\}_{i=1}^G)}{\text{std}(\{R_i\}_{i=1}^G)}$ where $R_i = 1.0$ if the response is correct and $0.0$ otherwise
• $\epsilon_{\text{low}} = 0.2$ and $\epsilon_{\text{high}} = 0.28$ are the clipping bounds for the importance ratio
• The outer $\frac{1}{\sum_{i=1}^G |o_i|}$ normalizes by the total number of tokens across all responses in the group

What it computes: For each question in a batch, the model generates $G$ responses. The verifier checks each response's final answer against the ground truth, assigning binary rewards (1.0 for correct, 0.0 for incorrect). The advantage $\hat{A}_i^t$ is computed by normalizing each response's reward relative to the group: subtract the group mean and divide by the group standard deviation. This means correct responses get positive advantages (above the group average) and incorrect responses get negative advantages (below average). For each token in each response, the policy gradient is the product of the advantage and the importance ratio $r_i^t(\theta)$, clipped to prevent too-large updates. The objective is the average of these per-token clipped policy gradients, normalized by the total token count. The constraint ensures each batch has both positive and negative examples (at least one correct and one incorrect response), which is necessary for the group-relative advantage to provide a meaningful learning signal.

Why this form: DAPO builds on GRPO (Group Relative Policy Optimization) but makes several key changes that the paper inherits: it removes the KL penalty term (allowing the policy to drift further from the reference model), uses clip-higher ($\epsilon_{\text{high}} > \epsilon_{\text{low}}$) which allows the importance ratio to increase more than it decreases — the paper notes this is important because high-entropy tokens tend to have high importance ratios, and clip-higher allows them to contribute more to the gradient. The token-level loss (summing over individual tokens rather than averaging at the response level) allows differential treatment of tokens, which is precisely what the entropy-based mask exploits.

The modified objective: DAPO with forking tokens only. The paper's modification (Equation 6) introduces an indicator function that gates the gradient contribution:

$J^{\mathcal{B}}_{\text{HighEnt}}(\theta) = \mathbb{E}_{B \sim \mathcal{D}, (q, a) \sim B, \{o_i\}_{i=1}^G \sim \pi_{\theta_{\text{old}}}(\cdot \mid q)} \left[ \frac{1}{\sum_{i=1}^G \sum_{t=1}^{|o_i|} \mathbb{I}\left[H_i^t \geq \tau^\mathcal{B}_\rho\right]} \sum_{i=1}^G \sum_{t=1}^{|o_i|} \mathbb{I}\left[H_i^t \geq \tau^\mathcal{B}_\rho\right] \cdot \min\left( r_i^t(\theta) \hat{A}_i^t, \text{clip}\left(r_i^t(\theta), 1 - \epsilon_{\text{low}}, 1 + \epsilon_{\text{high}}\right) \hat{A}_i^t \right) \right]$

subject to the same constraint $0 < |\{o_i \mid \text{is\_equivalent}(a, o_i)\}| < G$.

The two differences from Equation (4), highlighted in red in the paper, are:

• Addition of the indicator: $\mathbb{I}[H_i^t \geq \tau^\mathcal{B}_\rho]$ multiplies the per-token policy gradient term. This indicator function evaluates to 1 if the entropy $H_i^t$ of token $t$ in response $i$ meets or exceeds the batch-specific threshold $\tau^\mathcal{B}_\rho$, and 0 otherwise. Tokens with entropy below the threshold simply do not contribute to the loss — their gradients are masked to zero.
• Adjusted normalization: The denominator changes from $\sum_{i=1}^G |o_i|$ (total token count) to $\sum_{i=1}^G \sum_{t=1}^{|o_i|} \mathbb{I}[H_i^t \geq \tau^\mathcal{B}_\rho]$ (count of high-entropy tokens only). This ensures the loss is properly normalized — it computes the average over only the tokens being trained on, not all tokens.

Here:

• $H_i^t$ is the entropy of token $t$ in response $i$, computed from the current policy $\pi_\theta$ using Equation (1)
• $\rho \in (0, 1]$ is a predefined ratio specifying what fraction of tokens to retain (set to $\rho = 20\%$ for main experiments)
• $\tau^\mathcal{B}_\rho$ is the entropy threshold computed per batch such that exactly the top $\rho$ fraction of all tokens in that batch have entropy $\geq \tau^\mathcal{B}_\rho$

What it computes operationally: For each training batch, the procedure works as follows. First, the model generates $G$ responses per question and computes the token-level advantages $\hat{A}_i^t$ exactly as in standard DAPO. Second, for every token in every response, the model's current logits are used to compute entropies $H_i^t$. Third, all entropies across all tokens in the batch are pooled together, sorted, and the $\rho$-th percentile is computed as the threshold $\tau^\mathcal{B}_\rho$. Fourth, only tokens with entropy above this threshold contribute to the policy gradient loss — the indicator function zeros out the gradient for all others. Fifth, the loss is normalized by the count of retained tokens (not total tokens). The constraint ensuring mixed correct/incorrect responses per batch remains unchanged.

Why per-batch thresholding? Computing the threshold per batch rather than using a fixed absolute threshold (like $h_{\text{threshold}} = 0.672$ from the statistical analysis) is an important design choice. The paper's justification (implicit in the formulation) is that the entropy distribution shifts during training (as shown in Figure 5) — a fixed absolute threshold calibrated on the base model's entropy distribution would become misaligned as training progresses and the entropy distribution changes. By computing the threshold dynamically per batch, the approach adapts to the evolving entropy landscape and always selects exactly the top $\rho$ fraction of tokens regardless of how the absolute entropy values change. This is a kind of distributional robustness mechanism.

Why this indicates causality: If high-entropy tokens are the causal drivers of RLVR's effectiveness, then training only on them should preserve or even improve performance (by removing "noise" from low-entropy tokens). If they are merely correlated with something else that drives learning, then excluding low-entropy tokens should hurt performance (because the real drivers would be lost). The gradient mask experiment is thus a causal intervention: it breaks the correlation between high entropy and whatever else might matter, and tests whether entropy alone is sufficient to identify the tokens that matter.

Training and Evaluation Setup

The paper uses DAPO as the RLVR algorithm throughout, with all configuration details inherited from Yu et al. (2025) and implemented in the verl framework (Sheng et al., 2024).

Base models: Experiments use three model sizes from the Qwen3 family (Yang et al., 2025): Qwen3-8B, Qwen3-14B, and Qwen3-32B. All are base (pretrained) models, not instruction-tuned variants. The paper also conducts a smaller-scale experiment on Llama-3.1-8B (Grattafiori et al., 2024) as an out-of-family validation, though with a cold-start SFT phase required to achieve non-trivial performance.

Training dataset: All main RLVR experiments use DAPO-Math-17K (Yu et al., 2025), a dataset of 17,000 mathematical reasoning problems with verifiable ground-truth answers. The dataset primarily consists of mathematical problems, which is important for interpreting the generalization results on LiveCodeBench (a coding benchmark).

RLVR hyperparameters (from Section 5.2):

• Algorithm: DAPO with all standard techniques enabled:

  • Clip-higher: $\epsilon_{\text{high}} = 0.28$, $\epsilon_{\text{low}} = 0.2$ (the higher upper bound allows the importance ratio to increase more than decrease, which the paper connects to allowing high-entropy tokens — which tend to have large importance ratios — to contribute more to the gradient)
  • Dynamic sampling: enabled (the exact group size is not specified but is part of the DAPO recipe)
  • Token-level policy gradient loss (per-token rather than per-response averaging)
  • Overlong reward shaping: maximum response length 20,480 tokens, cache length 4,096 tokens (responses exceeding the max length receive shaped penalties rather than being truncated or discarded)

• Optimization:

  • Training batch size: 512 (total prompts per batch)
  • Mini-batch size: 32 (for gradient accumulation; with batch size 512 and mini-batch size 32, this yields 16 gradient steps per training batch)
  • Learning rate: $10^{-6}$ (constant, no warmup, no scheduling)
  • No KL divergence loss term
  • No entropy loss/bonus term

• Generation:

  • Decoding temperature during training: $T = 1.0$ (this is the standard DAPO setting and is used for both rollout generation and entropy computation)
  • Maximum response length: 20,480 tokens for main experiments; extended to 29,696 tokens for the "unlocking more potential" experiment in Section 5.4

• Gradient mask parameter: $\rho = 20\%$ for main experiments (top 20% highest-entropy tokens retained). Ablation experiments also test $\rho = 10\%$, $\rho = 50\%$, and the inverse case of retaining only the bottom 80% (lowest entropy tokens).

Chat template: The prompt format used for Qwen3 models is:

User:\n[question]\nPlease reason step by step, and put your final answer within \boxed{}.\n\nAssistant:\n

where [question] is replaced by the specific math problem, and <|endoftext|> serves as the EOS token. The \boxed{} format is used for answer extraction by the verifier.

Evaluation protocol: The trained models are evaluated on six standard mathematical reasoning benchmarks:

1. AIME'24 — American Invitational Mathematics Examination 2024 problems
2. AIME'25 — AIME 2025 problems
3. AMC'23 — American Mathematics Competition 2023 problems
4. MATH500 — 500 problems from the MATH dataset (Hendrycks et al., 2021)
5. Minerva — a collection of STEM reasoning problems
6. OlympiadBench (He et al., 2024) — olympiad-level problems

All evaluations are conducted in a zero-shot setting. For each question, the model generates 16 independent responses with decoding temperature $T = 1.0$. The paper reports "Acc@16" (average accuracy over the 16 responses — the fraction that are correct) and "Len@16" (average number of tokens per response). The pass@1 metric would be the fraction of questions where at least one of the 16 responses is correct; Acc@16 is the fraction of the 16 responses that are individually correct, which is a stricter metric.

Scaling experiments: The main results train all three model sizes (8B, 14B, 32B) from their respective base models using both vanilla DAPO (all tokens) and DAPO with forking tokens only ($\rho = 20\%$). The scaling trend analysis in Figure 8 compares the performance delta (forking tokens minus all tokens) across model sizes on AIME'24 and AIME'25.

Out-of-distribution generalization test (Section 5.4): To test whether the forking-token approach generalizes beyond the training distribution, the paper evaluates models trained on DAPO-Math-17K (math only) on LiveCodeBench (Jain et al., 2024), a code generation benchmark (version 5, covering August 2024 to February 2025). This tests whether the benefits of selective gradient application transfer to an unseen domain.

Cold-start SFT for Llama-3.1-8B: The paper notes that when DAPO is applied directly to the Llama-3.1-8B base model, "it achieves very low accuracy (approximately 1%) on the training dataset and often generates responses with repetitive words early in the RL training process." To address this collapse, the authors:

1. Use Qwen3-32B as a teacher to generate responses for a subset of DAPO-Math-17K queries
2. Randomly sample 10,000 queries with correct teacher-generated answers as cold-start SFT data
3. Fine-tune the Llama-3.1-8B base model on these 10,000 examples
4. Use the remaining 7,398 queries for subsequent RL training

This cold-start procedure is necessary because Llama-3.1-8B, unlike the Qwen3 models, does not spontaneously generate coherent mathematical reasoning under RLVR optimization — an important practical note about model-dependent effects. Even after cold-start, both vanilla DAPO and the forking-token variant achieve relatively low AIME'24 scores (Figure 11), and the paper acknowledges these results are "less convincing compared to those observed on the Qwen3 models."

Hyperparameter sensitivity experiments: To investigate the optimal retention ratio $\rho$, the paper varies it from the default 20% to 10%, 50%, and 100% (vanilla DAPO, all tokens). It also tests the inverse: retaining only the bottom 80% (lowest-entropy tokens) and masking the top 20%. These experiments are conducted on all three model sizes, with results reported in Figure 7.

Design Choice Analysis: Why Entropy-Based Gradient Masking?

The paper's core methodological innovation is not a new RL algorithm but a new selection criterion for which tokens should receive gradient updates. Understanding why entropy is chosen over alternatives is crucial.

Alternative 1: Uniform gradient application (vanilla DAPO, GRPO, PPO). This is the default in all prior RLVR work. The assumption is that every token in a response carries useful learning signal, modulated by the advantage $\hat{A}_t$ (which can be zero or negative, reducing or reversing the gradient). The paper's results directly refute this assumption by showing that masking 80% of tokens (the low-entropy ones) doesn't harm performance and can even improve it (Figure 6 and Table 2). If 80% of tokens were contributing positively, masking them would cause substantial degradation — the fact that it doesn't (and can help) implies these tokens are at best neutral and at worst harmful.

Alternative 2: Advantage-based token selection. One could imagine keeping only tokens with large absolute advantages $|\hat{A}_t|$, on the theory that these tokens had the largest impact on outcome. The paper does not directly test this, but the temperature manipulation experiments (Figure 3) suggest it would be suboptimal. Tokens with small advantages might still be important for exploration — they represent points where the model is uncertain but hasn't yet learned which direction leads to correct answers. Conversely, tokens with large advantages in early training might represent lucky guesses rather than learnable decision points. Entropy is an input-side signal (model uncertainty before seeing the outcome) while advantage is an output-side signal (how much the outcome differed from expectation). The paper's results suggest the input-side signal is more informative for identifying learning-relevant tokens.

Alternative 3: Surprisal-based token selection. One could select tokens based on $- \log p(o_t \mid q, o_{<t})$ — how "surprising" the actual sampled token is. This differs from entropy in that entropy measures the spread of the distribution while surprisal measures the probability of the specific realization. A token can have high entropy (many plausible alternatives) but low surprisal if the sampled token had high probability among those alternatives. The paper uses entropy rather than surprisal because entropy captures the potential for exploration (the model recognizes multiple valid paths) even when the actual sampled token was unsurprising.

Alternative 4: Attention-based or gradient-based importance. One could identify important tokens using attention weights (tokens that attend broadly to the context are likely decision points) or gradient norms (tokens that would produce large gradient updates if trained on are likely informative). The paper's entropy-based criterion is simpler to compute (just the output logits, no need for backpropagation or attention analysis) and has an information-theoretic interpretation that connects cleanly to exploration behavior.

Why the paper chooses entropy: The choice is theoretically motivated by the connection between entropy and exploration. In reinforcement learning, entropy measures how much the policy "considers" alternative actions — high entropy means the policy is uncertain and could plausibly take different actions, which is precisely the condition for exploration. If the paper's hypothesis is correct that RLVR works primarily by improving exploration at decision points, then entropy is the natural signal for identifying those decision points. The temperature manipulation experiments (Figure 3) provide empirical validation: increasing entropy at high-entropy tokens improves performance, increasing entropy at low-entropy tokens degrades it. This confirms that entropy is not just a correlate but a functional property — the model needs high entropy at forks and low entropy at continuations.

Why per-batch thresholding (not absolute threshold): The paper computes $\tau^\mathcal{B}_\rho$ dynamically per batch rather than using a fixed threshold like $h_{\text{threshold}} = 0.672$. This is important because Figure 5 shows that the entropy distribution shifts during training — the absolute entropy values change. A fixed threshold that selected 20% of tokens at the start of training might select 10% or 30% later. The per-batch approach keeps the selection consistent (always top $\rho$ fraction) regardless of distributional drift.

Why the normalization adjustment matters: Changing the denominator from total token count to high-entropy token count is not a minor detail — it affects the effective learning rate. If the denominator stayed at total token count while only 20% of tokens contributed to the numerator, the effective gradient would be $5\times$ smaller. The adjusted normalization ensures that the average gradient per trained token is the same as in vanilla DAPO, keeping the optimization dynamics comparable. If performance changed purely because of this normalization difference, the results would be an artifact of effective learning rate rather than token selection. By matching the normalization, the paper isolates the effect of which tokens are trained.

Temperature Manipulation as Causal Validation

The temperature manipulation experiments in Section 3 are methodologically important enough to warrant their own breakdown. They serve as the bridge between observing that high-entropy tokens are logical connectors (correlational) and claiming that high entropy at these positions causes better reasoning (causal).

The key insight: The paper doesn't just show that high-entropy tokens are different — it shows that actively changing their entropy has asymmetric effects on performance, and that these effects align with the forking-token hypothesis. This transforms the observation from "high-entropy tokens exist" to "high-entropy tokens are functionally necessary."

Experimental mechanics: The experiment modifies the decoding process at inference time. During autoregressive generation, after computing the logits $\mathbf{z}_t$ for position $t$, the standard procedure would apply softmax with $T = 1.0$ to get probabilities and sample. The paper instead:

1. Computes the entropy $H_t$ of the standard ($T=1.0$) distribution to classify the token as forking or not
2. Applies a different temperature $T'_t$ based on this classification
3. Samples from the temperature-adjusted distribution

This means the classification itself is done with the unmodified distribution — the temperature adjustment doesn't create a feedback loop where changing the temperature changes the classification. The token that gets sampled is drawn from the adjusted distribution, which may differ from what would have been sampled at $T=1.0$.

The two sweeps: The red sweep varies only $T_{\text{high}}$ while keeping $T_{\text{low}} = 1.0$. This asks: "What happens when we change entropy only at the forks?" The blue sweep varies only $T_{\text{low}}$ while keeping $T_{\text{high}} = 1.0$. This asks: "What happens when we change entropy only at the continuations?" Together, they decompose the effect of uniform temperature changes (which would affect both) into separate components.

Interpreting the low-$T_{\text{high}}$ degradation: When $T_{\text{high}} = 0.0$, the forking tokens become fully deterministic — at each fork, the model always picks the single most likely token. The catastrophic drop in AIME score (to ~20) shows that reasoning requires randomness at these decision points. Even $T_{\text{high}} = 0.05$ (very sharp, nearly deterministic) causes a large drop. The interpretation: reasoning involves genuine choice between plausible alternatives, and forcing deterministic selection locks the model into a single, often suboptimal, path that it cannot recover from.

Interpreting the high-$T_{\text{low}}$ degradation: When $T_{\text{low}}$ is increased significantly, continuation tokens become randomized. Since these tokens are typically word completions (e.g., the suffix "-ing" after "reason-"), randomizing them produces nonsensical outputs — the surface form of the language breaks down. This demonstrates that low entropy at continuations is not merely a statistical pattern but a functional requirement for producing coherent text.

Connection to RLVR: During RLVR training, the policy gradient updates change the model's parameters, which changes its output distributions. If training on all tokens uniformly pushed the model toward more deterministic behavior everywhere (as SFT does), it would reduce entropy at forks and hurt reasoning. The paper's finding that RLVR preserves and increases forking token entropy (Figure 4, Figure 5) suggests that RLVR naturally avoids this pitfall — possibly because the advantage-weighted gradient encourages maintaining high probability on tokens that appear in correct responses while not penalizing alternative tokens that could have led to correct responses. The gradient-masking experiment then takes this one step further: by explicitly preventing low-entropy tokens from receiving updates, it ensures the model doesn't accidentally reduce exploration at forks while trying to optimize continuations.

Summary of Key Design Decisions and Their Justifications

• Entropy as the selection criterion over advantage, surprisal, attention, or gradient norms: entropy directly measures the model's uncertainty at each position, which is the property hypothesized to govern exploration value; empirically validated by temperature manipulation experiments showing functional asymmetry between high and low entropy tokens.

• Per-batch percentile thresholding over fixed absolute threshold: adapts to the shifting entropy distribution during training, maintaining consistent selection ratio regardless of how absolute entropy values evolve.

• $\rho = 20\%$ as the retention ratio: emerges from the observation that 20% of tokens exceed the 80th-percentile entropy threshold in base model outputs; validated by ablation showing deviation in either direction (10% or 50%) reduces performance.

• Adjusted normalization (dividing by count of retained tokens rather than total tokens): isolates the effect of which tokens are trained from the effect of effective learning rate changes.

• DAPO as the base algorithm rather than PPO or GRPO: DAPO's clip-higher mechanism ($\epsilon_{\text{high}} = 0.28 > \epsilon_{\text{low}} = 0.2$) is argued to naturally favor high-entropy tokens (which tend to have large importance ratios), making the gradient mask an amplification of DAPO's existing bias rather than a correction to it.

• No entropy bonus in training: the paper explicitly notes that training excludes both KL divergence loss and entropy loss. This is deliberate: the paper argues (Discussion 3) that uniform entropy bonuses are suboptimal because they increase entropy at low-entropy continuations. The gradient mask approach achieves selective entropy increase (only at forks) without an explicit entropy bonus term.

• Maximum response length 20,480 for main experiments: standard DAPO setting providing sufficient space for long CoT reasoning; extended to 29,696 for the "unlocking more potential" experiment to demonstrate that the approach benefits from even longer reasoning chains.

4. Key Insights and Innovations

Innovation 1: Token Entropy as a Principled Diagnostic for Reasoning-Relevant Learning

The paper's most fundamental contribution is the introduction of token-level generation entropy as a model-intrinsic signal for identifying which parts of a reasoning trajectory actually matter for learning, replacing the field's implicit assumption that all tokens in a Chain-of-Thought are equally relevant training targets.

What was the field doing before? Prior work in RLVR treated the policy gradient loss as a flat sum over all tokens in a response — PPO, GRPO, DAPO, and their derivatives all compute $\sum_t \min(r_t(\theta) \hat{A}_t, \text{clip}(...))$ without discriminating between tokens. This is a direct inheritance from traditional RL, where every action in a trajectory matters for credit assignment. When prior work did identify "important" tokens, it used outcome-based criteria: tokens whose modification changes the final answer (Lin et al., 2024), or positions where models are empirically error-prone (Vassoyan et al., 2025). These criteria require either ground-truth knowledge or counterfactual interventions — they identify importance retrospectively, by looking at downstream consequences.

What's the conceptual shift? The paper proposes that the model's own uncertainty at generation time — measured by the entropy of its output distribution $H_t = -\sum_{j} p_{t,j} \log p_{t,j}$ — is a more fundamental signal than downstream outcomes. The intuition hinges on a distinction the field hasn't previously made operational: some tokens are "forks" where the model genuinely considers multiple plausible reasoning directions, while others are deterministic continuations of already-committed paths. Entropy captures this distinction directly from the model's logits, without any external correctness signal. A token with $H_t \approx 0$ is one where the model is certain; a token with $H_t > 0.672$ (the paper's empirical 80th-percentile threshold) is one where it's meaningfully uncertain.

This reframes the problem from "which tokens, if changed, would alter the answer?" (causal importance) to "which tokens does the model itself treat as decision points?" (epistemic importance). The two criteria overlap but are not identical: a model could be uncertain at a position yet coincidentally sample the correct token, or be erroneously certain at a position that actually requires careful reasoning. By using the model's internal uncertainty rather than external outcomes, the entropy criterion identifies potential decision points regardless of whether they were navigated correctly in a particular sampled trajectory.

Why does this matter beyond the performance numbers? This diagnostic concept opens a new direction for analyzing and improving RLVR that doesn't depend on having ground-truth answers available at training time. The entropy signal is available from the policy's own forward pass — it's essentially free to compute alongside generation. This makes it practical to integrate into training pipelines (as the gradient mask does) without the computational overhead of counterfactual rollouts or the circularity of using correctness signals to identify which tokens to train on. The paper demonstrates the diagnostic power of this lens through the temperature manipulation experiments (Figure 3): selectively modifying entropy at high-entropy tokens improves reasoning, while modifying entropy at low-entropy tokens degrades it — a clean functional dissociation that validates entropy as a meaningful category, not just a statistical curiosity.

The significance is that this provides a principled, computationally cheap, model-intrinsic criterion for token importance that generalizes across problems without requiring per-problem correctness signals. It's not merely a new feature — it's a new way of looking at CoT trajectories through the model's own uncertainty landscape.

Innovation 2: The Entropy-Preservation Finding — RLVR Amplifies, Doesn't Create, Exploration Sites

The paper's second key insight is an empirical discovery about RLVR dynamics that contradicts what one might naively expect: RLVR training does not fundamentally reorganize which token positions are high-entropy; it largely preserves the base model's entropy patterns and primarily amplifies the magnitude of uncertainty at positions that were already high-entropy before training.

What would the naive expectation be? A natural hypothesis about RLVR is that it teaches the model new reasoning strategies — discovering novel decision points, learning to be uncertain about things it was previously overconfident about, and becoming confident about things it was previously uncertain about. Under this view, the entropy landscape should shift substantially during training: new positions would become high-entropy (as the model learns to recognize new decision points) and others would become low-entropy (as the model masters aspects of reasoning). This is essentially the "RL teaches new capabilities" model.

What the paper actually found. Table 1 reports that after full RLVR training (1,360 steps on Qwen3-14B), the set of token positions in the top 20% of entropy has an 86.67% overlap with the base model's high-entropy positions. This means that the model at convergence is uncertain about almost exactly the same positions it was uncertain about before training — only about 13% of high-entropy positions shifted. Simultaneously, Figure 4 shows that the magnitude of entropy change is concentrated in the already-high-entropy percentiles, with the top 5% of tokens showing average entropy changes orders of magnitude larger than the bottom 50%. And Figure 5 shows that the variance of entropy during training increases with the percentile — high-entropy positions are the dynamic ones, while low-entropy positions barely move.

Why is this surprising and important? This finding fundamentally reframes what RLVR does. It suggests that RLVR is not creating new exploration sites — the model's uncertainty landscape is largely determined by pretraining, and RLVR works within it. The model comes into RLVR already "knowing" which positions in a reasoning trajectory are decision points (high-entropy) and which are deterministically determined (low-entropy). What RLVR does is amplify exploration at those pre-existing decision points — increasing the entropy at forks so the model more thoroughly explores the alternatives, while leaving the deterministic scaffolding untouched.

This connects to and explains several otherwise puzzling observations in the literature. Gandhi et al. (2025) found that RLVR primarily learns reasoning behaviors (format, structure) rather than content — the entropy-preservation finding provides a mechanism: if the structure of reasoning (where forks occur) is fixed by pretraining, then RLVR can only optimize within that structure. Li et al. (2025) found that structure matters more than content — the entropy finding suggests this is because the structure (which tokens are high-entropy forks) is the scaffold that RLVR operates on, while the content (what specifically gets generated at deterministic continuations) follows from that structure.

The finding also places a sharp constraint on what RLVR can achieve: if a base model has near-zero entropy at all positions for a class of problems (it's completely certain about its (incorrect) reasoning), then RLVR has no high-entropy forks to amplify and cannot improve. This explains the paper's difficulty-bin results mirroring those in prior search/revision literature — RLVR helps most on problems where the base model has some uncertainty at decision points, and fails on problems where it's uniformly confident.

Is this a reframing or an incremental observation? This is a reframing of the RLVR mechanism. The dominant narrative has been that RL "teaches" reasoning — that the model learns new capabilities through trial and error. The paper's finding suggests a more constrained picture: pretraining establishes the set of possible reasoning trajectories and marks the decision points, and RLVR optimizes which of those trajectories the model tends to follow. The "teaching" happens not by creating new decision points but by shaping the probabilities at existing ones. This is a more fundamental claim than just "entropy matters" — it's a claim about the locus of learning in RLVR, and it shifts attention from what the model learns to where it learns.

Innovation 3: The Causal Demonstration That ~20% of Tokens Drive Nearly All Gains — And That Low-Entropy Tokens Are Actively Harmful at Scale

The third innovation is the paper's gradient ablation experiment, which provides causal evidence that the high-entropy minority tokens are not merely correlated with learning but are the primary causal drivers — and goes further by showing that at larger model scales, the low-entropy majority tokens actively interfere with learning.

What distinguishes this from a feature importance analysis? Prior work identifying "critical tokens" (Lin et al., 2024; Vassoyan et al., 2025) used observational or counterfactual methods on fixed models: find tokens where intervention changes the answer, or where models tend to make errors. These methods identify tokens that are currently important for generation. They do not show that these tokens are where learning happens during training — a token could be critical for correct output without being the locus of gradient-driven improvement.

The paper's gradient mask experiment is a training-time causal intervention: it selectively prevents certain tokens from receiving gradient updates and observes whether the model's performance trajectory changes. This directly tests the counterfactual: "if we trained without these tokens, would the model still improve?" The results are striking in their asymmetry:

• Training on only the top 20% highest-entropy tokens preserves or exceeds full-gradient performance across all three model sizes (Table 2, Figure 6). On the 32B model, it produces substantial gains: +11.04 on AIME'25, +7.71 on AIME'24. On the 14B, gains of +5.21 and +4.79. On the 8B, performance is maintained.

• Training on only the bottom 80% lowest-entropy tokens causes severe degradation (Figure 7, bottom-80% curve). Despite these tokens comprising 80% of the training data by count, the model trained on them alone performs dramatically worse than the full-token baseline.

The "beyond 80/20" claim. The paper frames this as "beyond the 80/20 rule" because the 20% high-entropy tokens don't just match 80% of the performance — they match or exceed 100% of the performance. This is more extreme than a typical Pareto distribution where the vital few provide disproportionate but partial contribution. Here, the vital few provide essentially all the benefit, and the trivial many provide none (or negative benefit at larger scales).

Why does the benefit scale with model size? Figure 8 shows that the performance delta between forking-token DAPO and vanilla DAPO increases with model size: minimal on 8B, moderate on 14B, large on 32B. The paper hypothesizes this is because smaller models have "insufficient capacity to benefit from increased exploration" — their reasoning is too constrained by limited pretrained knowledge for the extra exploration at forks to yield better paths. Larger models have richer representations that can productively use the expanded search at decision points. This is an important boundary condition: the forking-token approach's benefits are not universal but are amplified by scale.

The negative result that matters. The finding that low-entropy tokens degrade performance when trained on (rather than being merely neutral) at larger scales is a negative result with significant implications. It suggests that current RLVR algorithms are not just computationally wasteful by training on irrelevant tokens — they are actively injecting noise into the optimization process. The paper doesn't fully characterize the mechanism (why do low-entropy token gradients hurt?), but the result is robust across model sizes and implies that selective gradient application isn't just an efficiency hack — it's a form of denoising that becomes more important as models get larger.

How fundamental is this contribution? The gradient masking technique itself is simple — an indicator function in the loss. The intellectual contribution is the experimental design that uses this simple tool to establish causality, combined with the scaling-dependent finding that the benefit grows with model size. This transforms the paper's earlier observational findings (entropy patterns exist, RLVR preserves them) into a causal argument: high-entropy tokens are not just interesting; they're necessary and sufficient for the benefits of RLVR at sufficient model scale.

Innovation 4: A Unified Mechanism for Why RL Generalizes While SFT Memorizes

The paper's Discussion 1 proposes a mechanistic hypothesis that connects the token-entropy framework to one of the most important empirical findings in modern LLM post-training: the observation by Chu et al. (2025) that RL on outcome rewards generalizes to unseen tasks while supervised fine-tuning tends to memorize training data and generalize poorly.

The hypothesis. The paper argues that the key difference between RL and SFT at the token level is their effect on forking-token entropy. SFT pushes output logits toward one-hot distributions — it maximizes the probability of the specific tokens in the training demonstrations, which necessarily reduces entropy everywhere, including at decision points. RLVR, by contrast, preserves or increases entropy at forking tokens (as shown in Figures 4 and 5) because the policy gradient doesn't penalize alternative tokens that weren't sampled — it only adjusts probabilities based on outcome, not on matching a specific token sequence.

Why this matters for generalization. If reasoning generalization depends on maintaining flexibility at decision points (so the model can adapt to novel problems by considering alternative paths), then SFT's effect of collapsing entropy at forks would destroy this flexibility — the model becomes locked into the specific reasoning paths seen during training. RLVR preserves the fork structure inherited from pretraining, so the model retains the ability to explore alternative reasoning directions when faced with novel problems.

What makes this more than speculation. The paper provides converging evidence that is consistent with this hypothesis without claiming to have proven it:

1. The entropy-preservation finding (Table 1, Figures 4-5) shows RLVR maintains the base model's uncertainty structure
2. The temperature manipulation experiments (Figure 3) show that reducing entropy at forks destroys performance — the exact mechanism by which SFT would hurt generalization
3. The LiveCodeBench experiment (Figure 9) shows that forking-token DAPO trained on math generalizes better to coding than vanilla DAPO — suggesting that preserving the entropy structure at forks (rather than letting low-entropy token gradients potentially interfere) improves cross-domain transfer

Is this incremental or fundamental? This is a mechanistic hypothesis that, if validated by future work, would be a significant step toward understanding why different post-training paradigms produce different generalization behaviors. It connects an observable, measurable property (token-level entropy patterns) to a downstream behavioral difference (generalization vs. memorization), providing a concrete target for future investigation. The paper doesn't claim to have proven this — it's presented as a discussion point — but the evidence accumulated across sections makes it a plausible and testable account rather than pure speculation.

The framework also suggests practical interventions beyond RLVR: if SFT's generalization weakness stems partly from entropy collapse at forks, then SFT procedures that explicitly preserve entropy at high-entropy positions (e.g., through a regularization term or through mixing SFT with entropy-preserving objectives) might recover better generalization while retaining SFT's stability and efficiency advantages. The paper doesn't explore this, but it follows naturally from the hypothesis.

5. Experimental Analysis

Evaluation Methodology

• Dataset. All RLVR experiments use DAPO-Math-17K (Yu et al., 2025), a dataset of 17,000 mathematical reasoning problems with verifiable ground-truth answers. This dataset primarily contains mathematical problems. For evaluation, six standard mathematical reasoning benchmarks are used: AIME'24, AIME'25, AMC'23, MATH500 (Hendrycks et al., 2021), Minerva, and OlympiadBench (He et al., 2024). The out-of-distribution generalization test uses LiveCodeBench (Jain et al., 2024, v5, covering August 2024 to February 2025), a code generation benchmark. The statistical entropy analysis in Section 3 uses AIME'24 and AIME'25 questions to collect over 10^6 response tokens from Qwen3-8B.

• Base model(s). The paper uses three model sizes from the Qwen3 family (Yang et al., 2025): Qwen3-8B, Qwen3-14B, and Qwen3-32B, all base (pretrained) variants. These are chosen as "one of the most recent and capable reasoning models within a comparable parameter scale" — the 8B for statistical analysis and initial experiments, and all three sizes for the scaling analysis. An additional out-of-family validation is conducted on Llama-3.1-8B (Grattafiori et al., 2024), though this requires a cold-start SFT phase since the base model achieves only ~1% accuracy on the training dataset when RLVR is applied directly.

• Metrics. The primary metrics are Acc@16 and Len@16. Acc@16 is the average accuracy over 16 independent responses per question — the fraction of the 16 generated responses that are judged correct by the verifier's is_equivalent() function, which compares extracted boxed answers to ground truth. This is distinct from pass@1 (fraction of questions with at least one correct response) and represents a stricter metric. Len@16 is the average number of tokens per response across the 16 generations. For the difficulty-level analysis in Section 3, the metric is the average of AIME 2024 and AIME 2025 scores, plotted as a continuous function of temperature.

• Baselines. The primary baseline is vanilla DAPO (Yu et al., 2025), which applies policy gradient updates to all tokens in generated responses. This is the "full-gradient" baseline against which the entropy-masked variant is compared. Within ablation experiments, the paper also compares against variants trained on: top 10% highest-entropy tokens, top 50% highest-entropy tokens, bottom 80% lowest-entropy tokens, and 100% of tokens (which is identical to vanilla DAPO). The out-of-distribution experiment on LiveCodeBench compares top 10%, top 20%, and 100% token variants. The Llama-3.1-8B experiment compares DAPO with forking tokens against vanilla DAPO after cold-start SFT.

• Generation budget / compute accounting. The training compute budget is measured in gradient steps — all methods use identical batch sizes (512 training prompts, 32 mini-batch size, 16 gradient steps per batch) and identical hardware configuration. The forking-token variant does not reduce wall-clock training time since logits must still be computed for all tokens (to compute entropy for thresholding), and the forward/backward passes through the model are identical — only the loss computation differs by masking certain token positions. The paper does not report FLOP counts or wall-clock comparisons. For evaluation, the compute budget is measured in number of generated responses (16 per question) and response length (controlled by the maximum response length parameter, typically 20,480 tokens, extended to 29,696 in one experiment).

• Cross-validation / statistical protocol. The paper does not employ cross-validation in the traditional sense — there is no hyperparameter tuning on held-out validation sets. The RLVR training runs use fixed hyperparameters inherited from DAPO (Yu et al., 2025). The evaluation reports Acc@16 across 16 independent samples per question to reduce sampling variance, but does not report confidence intervals, standard errors, or statistical significance tests. The entropy analyses in Sections 3 and 4 are descriptive/observational rather than inferential. The temperature manipulation experiment (Figure 3) sweeps a range of temperature values and reports point estimates without error bars. This absence of statistical rigor is a notable limitation — particularly for the main performance claims where differences of a few points on AIME'24 or AIME'25 (e.g., +1.25 on 8B, Table 2) may not be reliably distinguishable given the modest test set sizes (AIME'24 and AIME'25 each contain only 30 questions).

Main Quantitative Results

The paper's central experimental claim is that restricting RLVR policy gradient updates to the top 20% highest-entropy tokens ("forking tokens") achieves performance matching or exceeding full-gradient training, with the benefit scaling with model size. The results are organized around four axes: overall performance comparison, per-difficulty/proportion analysis, scaling trends, and out-of-distribution generalization.

Overall Performance: Forking-Only DAPO vs. Vanilla DAPO

Headline results across six benchmarks and three model sizes. Table 2 presents the comprehensive comparison between vanilla DAPO (all tokens) and DAPO with only forking tokens (top 20% highest-entropy). The pattern is scale-dependent:

• Qwen3-32B: Forking-token DAPO outperforms vanilla DAPO across all six benchmarks. The most substantial gains are on AIME'25 (+11.04: 56.67 vs. 45.63) and AIME'24 (+7.71: 63.54 vs. 55.83). Smaller but positive gains appear on AMC'23 (+2.34), Olympiad (+2.86), MATH500 (+0.52), and Minerva (+0.12). The average improvement across all benchmarks is +4.10 points on Acc@16. Response lengths also increase substantially: average Len@16 rises from 5,848.80 to 7,227.03 (+1,378.22 tokens on average).

• Qwen3-14B: Forking-token DAPO again outperforms across all six benchmarks. Gains are moderate: AIME'25 (+4.79), AIME'24 (+5.21), AMC'23 (+2.03), Olympiad (+3.48), MATH500 (+1.37), Minerva (+1.03). Average improvement: +2.99. Response lengths increase from 4,752.16 to 7,628.44 (+2,876.28 tokens).

• Qwen3-8B: Forking-token DAPO performs comparably to vanilla DAPO, with a mix of small gains and one slight loss. Acc@16: AIME'24 (+1.25), AIME'25 (+0.83), MATH500 (+0.46), Minerva (+0.48), Olympiad (+0.76), but AMC'23 (−0.625). Average improvement: +0.53. Response lengths increase from 4,021.99 to 5,508.00 (+1,486.01 tokens).

Key interpretation from Table 2. The headline numbers demonstrate three patterns. First, training on only 20% of tokens does not degrade performance — it either matches (8B) or substantially exceeds (14B, 32B) full-token training. This is the paper's central "beyond 80/20" claim: the 20% high-entropy tokens contribute all or more than all of the performance, while the 80% low-entropy tokens contribute nothing positive (and at larger scales, are actively harmful). Second, the benefit increases with model size, a scaling trend the paper emphasizes. Third, response lengths consistently increase under forking-token training, suggesting that focusing gradients on decision points produces more extensive reasoning chains — the paper doesn't fully unpack this mechanism, but it's consistent with the idea that forking-token gradients encourage more thorough exploration at decision points, leading to longer, more detailed CoTs.

Training dynamics (Figure 6). Figure 6 plots AIME'24 scores and response lengths as a function of gradient steps, comparing vanilla DAPO (100% tokens) against forking-token DAPO (top 20%) for each model size:

• 32B (Figure 6a, 6b): The forking-token variant shows both higher final performance and more stable training — the vanilla DAPO curve exhibits a visible performance degradation near step ~800 (a classic RL training instability where performance spikes and then collapses), while the forking-token curve monotonically increases and stabilizes at a higher plateau. The paper quantifies this as "stabilizes training and improves the AIME'24 score by 7.73" (the 7.73 figure refers to the final gap visible in the plot; note this differs slightly from the 7.71 in Table 2, which may reflect different checkpoint selection or averaging). Response length under forking-token training grows steadily and significantly exceeds vanilla DAPO throughout training.

• 14B (Figure 6c, 6d): Similar pattern — forking-token training shows a more stable upward trajectory, eventually outperforming vanilla DAPO by approximately 5.21 points (as in Table 2). Response lengths again increase markedly under the forking-token variant.

• 8B (Figure 6e, 6f): The two curves are largely overlapping — forking-token training roughly tracks vanilla DAPO throughout, consistent with the minimal average gain (+0.53) in Table 2. Response lengths increase slightly under forking-token training but the gap is narrower than for larger models.

Entropy Proportion Ablation: How Much Is "Just Right"?

The 20% sweet spot. Figure 7 presents the most detailed ablation of the paper, varying the retention ratio $\rho$ across five values — top 10%, top 20%, top 50%, bottom 80% (lowest-entropy tokens only), and 100% (vanilla DAPO) — on all three model sizes, tracking both AIME'24 scores and "overall entropy" (average entropy over all tokens) throughout training:

• 8B (Figure 7a, 7b): Performance is relatively insensitive to $\rho$ across the 10%, 20%, and 50% settings — all three track each other closely, with final scores in a narrow range. The 100% setting (vanilla DAPO) is slightly but consistently below the high-entropy variants after the initial training phase. The bottom-80% curve (training only on low-entropy tokens) shows catastrophic degradation — performance collapses and stays near zero throughout training. The overall entropy plot (Figure 7b) reveals why: the 20% setting maintains the highest overall entropy throughout training, while both 10% (too few forking tokens) and 100% (diluted by low-entropy tokens) produce persistently lower overall entropy. The bottom-80% setting drives entropy toward zero.

• 14B (Figure 7c, 7d): The sensitivity to $\rho$ increases compared to 8B. The top-20% setting clearly outperforms top-10% and top-50%, with top-10% showing a slight drop from the optimal and top-50% showing a more substantial decline. The 100% setting underperforms all high-entropy variants by a wider margin than at 8B. The bottom-80% setting again collapses. The overall entropy plot shows the same ranking as 8B: 20% maintains highest entropy, 10% and 50% are below it, 100% is lower still, bottom-80% collapses.

• 32B (Figure 7e, 7f): The sensitivity is most pronounced at this scale. The top-20% setting is clearly optimal and substantially outperforms both top-10% and top-50%, which in turn outperform 100%. The performance ordering (20% > 10% > 50% > 100%) is stark and consistent. The bottom-80% setting collapses as before. The overall entropy plot shows the largest separation between settings at this scale — the 20% setting maintains dramatically higher entropy than all alternatives.

What these ablation results establish. The experiments demonstrate that (a) there is an optimal proportion (~20%) that balances exploration and performance — too few forking tokens (10%) removes useful decision points and weakens exploration, too many (50%, 100%) includes low-entropy tokens that dilute the exploration signal; (b) this optimal proportion's benefit scales with model size — the performance penalty for deviating from 20% is smallest at 8B and largest at 32B; (c) low-entropy tokens are not merely neutral — training on them alone (bottom 80%) completely destroys performance; (d) the mechanism appears to involve maintaining higher overall entropy (exploration) at the optimal proportion, as the entropy curves in Figures 7b, 7d, and 7f consistently show that the 20% setting sustains the highest overall entropy throughout training.

The "bottom 80% collapses" result deserves emphasis. The paper states that "training exclusively on the 80% lowest-entropy tokens results in severe performance degradation." Figure 7 shows this is not just degradation — it's complete collapse to near-zero accuracy. This is the strongest evidence that low-entropy tokens are not irrelevant background but actively harmful when they are the sole locus of training. The mechanism is partially visible in the entropy curves: training on low-entropy tokens drives overall entropy toward zero, eliminating all exploration. The model becomes maximally deterministic and cannot discover new reasoning paths. This is consistent with the paper's Discussion 1 hypothesis about SFT vs. RL — SFT's tendency to push distributions toward one-hot would have a similar entropy-collapsing effect, explaining why SFT generalizes poorly.

Scaling Trends: Bigger Models Benefit More

Figure 8 quantifies the scaling relationship. The bar charts plot the performance delta (forking-token DAPO minus vanilla DAPO) for AIME'24 and AIME'25 across the three model sizes:

• AIME'24 (Figure 8a): Delta grows from approximately +1.25 at 8B, to +5.21 at 14B, to +7.71 at 32B.
• AIME'25 (Figure 8b): Delta grows from approximately +0.83 at 8B, to +4.79 at 14B, to +11.04 at 32B.

The paper interprets this as evidence that "concentrating exclusively on forking tokens in the policy gradient loss may yield greater benefits in larger reasoning models." The proposed explanation is that "the insufficient capacity of the smaller model... restricts its ability to benefit from increased exploration" — larger models have richer internal representations that can productively use the additional exploration at decision points to discover better reasoning paths, while smaller models are capacity-constrained and already exploring about as well as they can at the 20% default.

A note on the 8B near-zero delta. The 8B results in Table 2 show an average improvement of only +0.53 across all six benchmarks, with one negative result (AMC'23: −0.625). The paper frames this as "performance remains unaffected" and "can still achieve competitive reasoning performance." The AIME'24 delta of +1.25 on a test set of 30 questions (where each question contributes ~3.33 points to the score) represents less than half a question's worth of improvement — well within sampling noise. The practical takeaway is that for the 8B model, forking-token DAPO is essentially equivalent to vanilla DAPO, not better. The scaling trend claim is therefore most robust at the 14B and 32B scales, with the 8B results serving as a baseline showing no harm.

Extended Response Length: Unlocking More Potential

The 29,696-token experiment. Section 5.4 reports an experiment where the maximum response length is extended from the standard 20,480 to 29,696 tokens, continuing training from the best 32B forking-token DAPO checkpoint (which achieved 63.54 on AIME'24 at 20,480 tokens). The results are shown in Figure 10:

• AIME'24 score (Figure 10a): Improves from 63.54 to 68.12 — a gain of +4.58 points by simply allowing longer responses during continued training.
• Response length (Figure 10b): Increases substantially, though exact numbers must be read from the plot since the paper only quotes the score improvement numerically.

The paper frames this as evidence that "the full potential of our approach may not yet be realized" and that "with a longer context length or potentially more challenging training data, even greater performance gains could be achieved." This is a forward-looking claim rather than a demonstrated result — the experiment shows that the 20,480-token ceiling was binding for the 32B model's performance, and that lifting it provides additional gains, but doesn't establish whether further ceiling lifts would continue to produce gains or whether a new plateau would be reached.

State-of-the-art claim. The paper states that the 32B model trained with forking tokens achieves "scores of 63.5 on AIME'24 and 56.7 on AIME'25, setting a new state-of-the-art (SoTA) for reasoning models trained directly from base models with fewer than 600B parameters." The extension to 29,696 tokens pushes the AIME'24 score to 68.1, further strengthening this claim. These SoTA claims are relative to the specific category of "trained directly from base models" (not from instruction-tuned models) and "fewer than 600B parameters," which excludes larger models like DeepSeek R1 and models trained from instruction-tuned starting points. Within this defined category, the comparison is reasonable, though the paper does not provide a systematic comparison table against other models in this class to contextualize the SoTA claim.

Out-of-Distribution Generalization: LiveCodeBench

Does the math-trained model generalize to coding? Figure 9 presents results on LiveCodeBench (Jain et al., 2024, v5) for Qwen3-32B models trained on DAPO-Math-17K (mathematics only). The comparison includes top-10%, top-20%, and 100% (vanilla DAPO) token variants:

• Top 20% (orange curve): Achieves the highest accuracy, reaching approximately 27-28% on the smoothed curves by the end of training.
• Top 10% (blue curve): Slightly below top 20% but still clearly above vanilla DAPO.
• 100% / vanilla DAPO (green curve): Consistently lowest accuracy throughout training, plateauing around 24-25%.

The paper notes that "due to the high variance of the accuracy curves, we smooth the curves using window smoothing with a window size of 10." The raw accuracy numbers are not reported, only the smoothed curves — a presentation choice that makes precise comparison difficult but clearly shows the ranking.

Interpretation. The result demonstrates that the forking-token approach's benefits are not domain-specific — training a math-only model with selective gradient application improves performance on a coding benchmark, suggesting the approach enhances general reasoning capabilities rather than overfitting to math-specific patterns. This is consistent with the Discussion 1 hypothesis: preserving entropy at forking tokens maintains reasoning flexibility that transfers across domains. However, the absolute performance levels are modest (27-28% on LiveCodeBench), and the paper does not report results for the 14B and 8B models on this out-of-distribution test, so the scaling of this generalization benefit is uncharacterized.

Llama-3.1-8B: Out-of-Family Validation

The cold-start necessity. Figure 11 presents the only non-Qwen3 experiment. The paper notes that applying DAPO directly to the Llama-3.1-8B base model fails — the model achieves "very low accuracy (approximately 1%) on the training dataset" and produces repetitive text. To make RLVR viable, the authors perform a cold-start SFT phase: Qwen3-32B generates responses for DAPO-Math-17K queries, 10,000 correct-answer responses are randomly sampled as SFT data, and the remaining 7,398 queries are held for subsequent RL.

Results (Figure 11). After cold-start:

• Forking-token DAPO (top 20%): Achieves higher AIME'24 scores and longer response lengths than vanilla DAPO, with both metrics showing improvement over training.
• Vanilla DAPO: Lower but non-zero performance, with shorter responses.

The paper acknowledges these results are "less convincing compared to those observed on the Qwen3 models" — the absolute AIME'24 scores are relatively low (judging from the plots, likely in the 5-15% range). This is an important boundary condition: the forking-token approach depends on the base model having a reasonable starting level of reasoning capability. Llama-3.1-8B, which cannot generate coherent math reasoning without SFT initialization, shows the expected relative benefit (forking > vanilla) but at absolute performance levels too low to draw strong conclusions.

Response Length as an Emergent Property

A consistent finding across all experiments is that forking-token DAPO produces longer responses than vanilla DAPO. This is visible in Table 2 (Len@16 increases for all model sizes and benchmarks), Figure 6 (response length curves are consistently higher for forking-token training), Figure 10 (extending max length produces even longer responses), and Figure 11 (same pattern on Llama). The average response length increase ranges from +1,378 tokens (32B) to +1,486 (8B) in Table 2.

The paper does not deeply analyze this phenomenon but it is thematically important. If high-entropy tokens are decision points where the model chooses between reasoning paths, training on them specifically may encourage the model to spend more tokens exploring branches at these points — producing longer, more thorough reasoning chains. This is consistent with the exploration hypothesis: the model learns that being uncertain and exploring alternatives at forks (rather than committing quickly) improves its chance of reaching a correct answer. The response length increase is thus an observable behavioral signature of the entropy-based training, not a separately engineered property.

Ablation Studies and Robustness Checks

• Retention ratio $\rho$ (Figure 7): The proportion of highest-entropy tokens retained is varied from 10% to 20% to 50% to 100% (all tokens) to bottom-80% (lowest-entropy only). On 8B, performance is insensitive across 10%-50% with slight degradation at 100%. On 14B, 20% is optimal with clear degradation at 10% and 50%. On 32B, 20% is strongly optimal with both 10% (too few) and 50% (too many, including low-entropy tokens) producing notably lower performance. The bottom-80% setting causes catastrophic collapse to near-zero accuracy across all model sizes. The corresponding overall entropy curves (Figures 7b, 7d, 7f) show that the 20% setting consistently maintains the highest average entropy during training, supporting the exploration-based mechanism.

• Low-entropy token training (bottom 80% in Figure 7): Training exclusively on the 80% lowest-entropy tokens (masking the top 20% forking tokens) results in severe performance degradation — accuracy collapses to near zero on all model sizes. This is presented as a negative result demonstrating that low-entropy tokens are not merely irrelevant but actively harmful when they are the sole training target. The entropy curves show this drives overall entropy toward zero, eliminating all exploration.

• Maximum response length extension (Figure 10): Extending the maximum response length from 20,480 to 29,696 for the best 32B forking-token checkpoint improves AIME'24 from 63.54 to 68.12 (+4.58), demonstrating that the standard length ceiling was binding. Response lengths increase correspondingly. This is not a true ablation but a boundary exploration showing the approach benefits from additional generation budget.

• Out-of-distribution generalization (Figure 9): Models trained on DAPO-Math-17K (math only) are evaluated on LiveCodeBench (coding). The top-20% forking-token variant outperforms both top-10% and vanilla DAPO (100%), demonstrating that the benefits transfer across domains and are not specific to the math training distribution. The top-10% variant also outperforms vanilla DAPO, showing the effect is robust to the exact retention proportion in this out-of-distribution setting.

• Model family robustness (Figure 11): The approach is tested on Llama-3.1-8B as an out-of-family validation. Due to the base model's inability to generate coherent math reasoning under RLVR alone (accuracy ~1% on training data), a cold-start SFT phase using Qwen3-32B-generated data is required. After cold-start, forking-token DAPO outperforms vanilla DAPO, suggesting the relative benefit generalizes across model families, but the absolute performance is low, making the result less definitive than the Qwen3 experiments.

• Entropy pattern preservation during RLVR (Table 1, Figures 4, 5): Not a training ablation but a robustness check on the underlying phenomenon. Table 1 shows that the overlap ratio of top-20% entropy positions between intermediate RLVR checkpoints and the base model stays above 86% even at convergence. Figure 4 shows entropy changes are concentrated in high-entropy percentiles. Figure 5 shows higher-entropy percentiles exhibit larger variance during training. These analyses collectively validate that the entropy-pattern stability observed on Qwen3-14B is a robust feature of RLVR training, not an artifact of a particular checkpoint or measurement.

• Temperature sweep validation (Figure 3): On Qwen3-8B with max response length 28,672, varying $T_{\text{high}}$ (forking tokens) and $T_{\text{low}}$ (other tokens) independently demonstrates the functional asymmetry: lowering $T_{\text{high}}$ degrades performance (from ~71-72 to ~20 at $T_{\text{high}} = 0.0$), increasing $T_{\text{high}}$ improves it (to ~72 at $T_{\text{high}} = 3.0$), lowering $T_{\text{low}}$ has mild effects, and increasing $T_{\text{low}}$ causes nonsensical outputs and degradation. This validates the causal role of forking-token entropy in reasoning quality independently of the RLVR training results.

Notable absences in the ablation suite. The paper does not ablate several factors that would strengthen the causal argument: (a) no experiment randomizes which tokens are masked (keeping the fraction at 20% but selecting tokens randomly rather than by entropy) — this would test whether any 20% subset works or whether entropy specifically matters; (b) no experiment uses an alternative token importance criterion (advantage magnitude, attention entropy, gradient norm) as a competing selection mechanism — this would test whether entropy is uniquely informative or merely one of several correlated signals; (c) no experiment varies the group size $G$ in DAPO — the interaction between the number of samples per question and the forking-token proportion is unexplored; (d) no experiment varies the training dataset size — the robustness of the 20% figure to data quantity is unknown.

Critical Assessment

The paper's experimental program aims to support three central claims: (1) high-entropy minority tokens function as reasoning "forks" and their entropy matters causally for reasoning quality, (2) RLVR preserves base-model entropy patterns and primarily adjusts high-entropy tokens, and (3) restricting policy gradient updates to these forking tokens matches or exceeds full-gradient training with benefits scaling with model size. The evidence for each claim warrants careful scrutiny.

Claim 1 (entropy patterns and forking function): Partially supported, with a gap between the statistical characterization and the causal demonstration.

The statistical analysis in Section 3 (Figure 2) convincingly demonstrates that Qwen3-8B CoT outputs exhibit a bimodal entropy distribution where a minority of tokens have high entropy and these tokens are disproportionately logical connectors ("however," "since," "suppose"). This is a robust descriptive finding at the chosen temperature ($T=1.0$) and model (Qwen3-8B). However, the characterization that these are "forks that steer the model toward diverse reasoning pathways" is interpretive rather than directly demonstrated — the word cloud analysis (Figures 2b, 2c) shows correlation between high entropy and logical connector vocabulary, but does not show that different tokens sampled at these positions lead to genuinely different downstream reasoning. A demonstration would require showing that intervening to change the token at a high-entropy position (e.g., replacing "however" with "therefore") produces a different reasoning trajectory and final answer — the paper's temperature manipulation experiments (Figure 3) change the level of randomness but do not trace the consequences of alternative token choices.

The temperature manipulation experiments provide stronger causal evidence: modifying entropy at high-entropy tokens and low-entropy tokens produces asymmetric effects on overall AIME score. This demonstrates that the entropy magnitude at these positions matters for aggregate performance. However, the experiment operates at decoding time on a fixed model — it shows that a trained model uses high-entropy tokens effectively, not that these tokens are where learning occurs during RLVR. The leap from "high-entropy tokens are functionally important at inference" to "high-entropy tokens are where RLVR training improvements occur" requires the RLVR gradient ablation experiments in Section 5, which are logically separate.

Additionally, the 20% threshold ($h_{\text{threshold}} = 0.672$) is an empirical percentile computed from $10^6$ tokens from one model (Qwen3-8B) on one task distribution (AIME). The paper does not report whether this threshold generalizes to other models or tasks — the entropy distribution of Qwen3-32B CoT outputs might have a different 80th-percentile value, and the "correct" threshold for separating forking from continuation tokens might vary with model scale, task difficulty, or decoding parameters.

Claim 2 (RLVR preserves entropy patterns): Well-supported for the specific model and algorithm tested.

Table 1 demonstrates that the overlap ratio between high-entropy token positions in intermediate checkpoints and the base model stays above 86% throughout training on Qwen3-14B with DAPO. Figure 4 shows that entropy changes are concentrated in high-entropy percentiles. Figure 5 shows the dynamic range of entropy is larger at higher percentiles. Together, these analyses provide consistent evidence that RLVR training does not fundamentally reorganize which positions are uncertain — it primarily adjusts the magnitude of uncertainty at already-uncertain positions.

The limitation is that this analysis is conducted on a single model (Qwen3-14B), a single RL algorithm (DAPO), and a single training run. The paper does not report whether the preservation pattern holds for the 8B or 32B models, for alternative RL algorithms (GRPO, PPO), or for different hyperparameter settings (different learning rates, different clip-higher bounds). The claim that "RLVR largely retains the base model's entropy patterns" may be specific to DAPO's training dynamics rather than a general property of RLVR. DAPO's clip-higher mechanism ($\epsilon_{\text{high}} > \epsilon_{\text{low}}$) may specifically preserve high-entropy token behavior by allowing their importance ratios to increase more than decrease — a different RL algorithm without this asymmetry might produce different entropy evolution.

Furthermore, the overlap ratio analysis only looks at the positions of high-entropy tokens, not their ordering or contextual dependence. Two models could have 86% overlap in which token indices are high-entropy while having very different relationships between what gets generated before and after those positions. The overlap metric captures identity preservation but may miss more subtle reorganization of the entropy landscape.

Claim 3 (forking tokens drive RLVR gains, scaling with model size): Supported with qualifications about statistical reliability, model dependence, and effect sizes at small scale.

This is the paper's marquee experimental result and the evidence is strongest at larger model scales. Table 2 and Figure 6 show that forking-token DAPO outperforms vanilla DAPO on Qwen3-32B and Qwen3-14B across all six benchmarks, with gains of +4.10 and +2.99 average Acc@16 respectively. The ablation in Figure 7 shows that the 20% proportion is optimal and that training on low-entropy tokens is catastrophic. These findings are internally consistent and the scaling trend (Figure 8) is clear: benefit increases with model size.

Several qualifications are necessary:

Statistical reliability is not addressed. The AIME benchmarks each contain only 30 questions. An improvement of +1.25 on AIME'24 for the 8B model represents less than half a question's difference in the number of correct responses (30 × 0.0125 = 0.375 questions). The paper reports Acc@16 (average over 16 responses per question), which is more stable than single-sample accuracy but still subject to substantial variance with only 30 questions. Without confidence intervals, standard errors, or significance tests, it is impossible to determine whether the reported differences — particularly the small ones on 8B and the moderate ones on Minerva and MATH500 — are statistically reliable or within sampling noise. The LiveCodeBench results (Figure 9) are explicitly noted to have "high variance" and are presented with smoothing, further indicating that the signal-to-noise ratio is a concern.

The control experiment is missing. The paper never tests what happens when a random 20% of tokens are selected for gradient updates (rather than the top 20% by entropy). This is the most important missing ablation. If random selection also matches or exceeds full-gradient training, then the entropy-based selection criterion is not special — any sparse gradient approach might work, and the paper's entire framing around "forking tokens" would be undermined. Conversely, if random selection performs poorly, that would strongly validate the entropy criterion. This experiment would directly test whether entropy specifically matters or whether the benefit is from gradient sparsity in general (e.g., reduced gradient noise, implicit regularization). Its absence is a significant gap.

Single model family for main results. All strong positive results come from the Qwen3 family. The Llama-3.1-8B experiment (Figure 11) is weakened by the cold-start necessity and low absolute performance. The paper acknowledges this limitation: "we believe the results on Llama-3.1-8B are less convincing compared to those observed on the Qwen3 models." The possibility that Qwen3 models have particularly well-structured entropy patterns that make the forking-token approach effective — while other model families might not — is not ruled out. Replication on additional model families (Mistral, Gemma, DeepSeek) would substantially strengthen the generality claim.

The mechanism is inferred, not directly tested. The paper interprets the results as showing that forking tokens "facilitate exploration" and that the 20% setting "optimally balances exploration and performance." The evidence for the exploration mechanism comes from the overall entropy curves in Figure 7 — the 20% setting maintains higher average entropy than alternatives. However, higher average entropy is an aggregate statistic that could arise for multiple reasons (longer responses, different vocabulary choices, genuinely more uncertainty at decision points) and does not directly demonstrate that the quality of exploration (trying genuinely different reasoning paths) improves. The paper does not measure exploration diversity (e.g., number of distinct reasoning strategies attempted per problem, entropy of the final answer distribution across samples) — it measures only the per-token entropy of the generation distribution. The claim that forking tokens drive effective exploration (rather than just preserving uncertainty) is an interpretation that goes beyond what the experiments directly measure.

The 8B results complicate the narrative. The paper's headline claim is that "high-entropy minority tokens drive effective reinforcement learning." On the 8B model, the benefit is minimal (+0.53 average Acc@16, Table 2) and one benchmark shows a negative result (AMC'23: −0.625). The paper interprets this as "due to the model's lower capacity, the benefits of enhanced exploration appear limited." This is a plausible explanation but it introduces a boundary condition that weakens the universality of the claim: the approach works well only when the model has sufficient capacity to benefit from additional exploration at decision points. For smaller models, the approach is essentially neutral — it doesn't hurt, but it doesn't help much either. This is important for practitioners considering adopting the method: if you're training smaller models (8B and below), the gradient-masking approach may not be worth the implementation complexity.

Response length as a confound. A striking finding across all experiments is that forking-token DAPO produces substantially longer responses (Table 2: Len@16 increases by 1,378-2,876 tokens depending on model size). The paper does not disentangle whether the performance improvement comes from the entropy-based token selection per se or from the increased response length that selection induces. If longer responses generally improve accuracy on math reasoning benchmarks (which is plausible — more tokens allow more thorough verification and error correction), then the forking-token approach might be benefiting performance indirectly by encouraging verbosity. A controlled experiment that caps response length to match the vanilla DAPO lengths would help isolate the mechanism, but is not conducted.

SoTA claims are narrowly scoped. The claim of "new state-of-the-art for reasoning models trained directly from base models with fewer than 600B parameters" is carefully qualified and, within that scope, the 63.5 (and later 68.1) AIME'24 scores are impressive. However, the paper does not provide a comparative table showing where other models in this category stand, making it difficult to assess the magnitude of the claimed lead. The category definition also excludes several highly relevant comparisons (models trained from instruction-tuned checkpoints, models larger than 600B) that would contextualize the results in the broader reasoning-model landscape.

The LiveCodeBench generalization result is promising but thin. Figure 9 shows that forking-token DAPO trained on math outperforms vanilla DAPO on coding. This is a single experiment on a single model size (32B) with smoothed curves and acknowledged high variance. No results are reported for 8B or 14B, and the absolute performance levels are modest. The generalization claim is directionally supported but far from comprehensively established.

Overall, the experimental evidence supports the paper's central narrative — that high-entropy minority tokens are important for RLVR — but the strongest causal evidence is limited to: (a) Qwen3-14B and Qwen3-32B models, (b) the DAPO algorithm, (c) mathematical reasoning (with one suggestive coding result), and (d) moderate-to-large performance differences where statistical reliability is not quantified. The missing random-selection control experiment and the lack of statistical testing are the most significant gaps, as they leave open the possibility that gradient sparsity (rather than entropy-based selection specifically) accounts for the benefits, and that some of the reported improvements are within sampling noise.

6. Limitations and Trade-offs

Model Family and Domain Specificity

The assumption or constraint. All main RLVR experiments demonstrating substantial performance gains use models from the Qwen3 family (8B, 14B, 32B), trained exclusively on mathematical reasoning data (DAPO-Math-17K) and evaluated on mathematical benchmarks. The paper acknowledges this explicitly in its Limitations section: "First, our experiments could be extended to models beyond the Qwen family. Although we attempted to evaluate our approach on LLaMA models, they struggled to achieve meaningful performance on the AIME benchmarks."

The consequence. The paper's headline finding — that restricting gradient updates to the top 20% highest-entropy tokens outperforms full-gradient RLVR — may not transfer to other model families or training distributions. The Llama-3.1-8B experiment (Figure 11) required a cold-start SFT phase using Qwen3-32B-generated data because the base model collapsed to ~1% accuracy under direct RLVR, and even after this intervention, absolute AIME'24 performance remained low. This is direct evidence that the approach's success depends on properties of the base model that are not universal. The Qwen3 models may have particularly well-structured entropy patterns — their pretraining data, architecture, or training recipe may produce cleaner separation between forking and continuation tokens. A model with a less bimodal entropy distribution (where the distinction between high-entropy and low-entropy tokens is fuzzier) might not benefit from the same 20% threshold, or might require a different retention ratio altogether. The paper does not characterize what base-model properties predict whether the approach will work, leaving practitioners with no guidance on whether their model of choice will benefit.

What evidence exists in the paper. The Llama-3.1-8B experiment (Section 5.4, Figure 11) is the only out-of-family test. The paper's own characterization of these results as "less convincing compared to those observed on the Qwen3 models" signals the limitation. No experiments test DeepSeek, Mistral, Gemma, or any non-Qwen3 model at 14B or 32B scale — the scales where the benefit is most pronounced for Qwen3. The out-of-distribution generalization experiment on LiveCodeBench (Figure 9) tests domain transfer (math → coding) but still uses a Qwen3-32B base model, so it does not address model-family generality.

Mitigation status. The paper explicitly flags this as a limitation ("First, our experiments could be extended to models beyond the Qwen family") and suggests it as future work, but does not attempt to characterize which model properties might govern transferability or provide diagnostic tools for practitioners to assess whether their base model is suitable.

Missing Random-Selection Control Experiment

The assumption or constraint. The paper's central causal claim is that entropy specifically — not gradient sparsity in general — identifies the tokens that drive RLVR effectiveness. The experiments demonstrate that training on the top 20% by entropy works well, and that training on the bottom 80% by entropy fails catastrophically. However, the paper never tests whether training on a random 20% of tokens (keeping the fraction constant but selecting tokens uniformly rather than by entropy) would also match or exceed full-gradient training.

The consequence. Without this control, the paper cannot distinguish between two competing explanations for the results: (a) entropy selectively identifies reasoning-critical decision points, as the paper argues, or (b) gradient sparsity itself is beneficial — perhaps because reducing the effective number of gradient-updated tokens per batch acts as implicit regularization, reduces gradient noise, or changes the effective learning rate dynamics in a way that improves optimization. If a random 20% of tokens also outperforms full-gradient training, the entire "forking token" framing would be undermined — entropy would be incidental, not causal. Even if random selection underperforms entropy-based selection but still outperforms full-gradient training, it would suggest that the paper's mechanism is partially correct but incomplete, and that sparsity contributes independently of token identity.

This is the single most important missing experiment in the paper, because the theoretical contribution — understanding RLVR through token entropy — stands or falls on whether entropy specifically matters. The field already knows that regularization and gradient noise reduction can improve RL training stability; the paper's novelty is the claim that which tokens are regularized out (low-entropy continuations) is what matters. Without the random-selection baseline, this claim remains an interpretation of the data rather than a demonstrated causal mechanism.

What evidence exists in the paper. None. The paper tests 10%, 20%, 50%, 100%, and bottom-80% by entropy, but zero experiments with random token selection at any retention ratio. The ablation suite in Figure 7 explores the proportion of entropy-selected tokens but never the selection criterion itself. This absence is not acknowledged as a limitation in the paper.

Mitigation status. Not addressed. The paper does not mention the need for this control, does not conduct it, and does not list it as a limitation or future direction. This is the most significant methodological gap in the experimental design.

Unquantified Statistical Reliability and Small Test Sets

The assumption or constraint. The paper reports all evaluation results as point estimates (Acc@16 scores) without confidence intervals, standard errors, statistical significance tests, or any quantification of uncertainty. The primary evaluation benchmarks — AIME'24 and AIME'25 — each contain only 30 questions. Many of the reported performance differences, particularly on the 8B model, are small in absolute terms and represent fractions of a question's worth of improvement on these small test sets.

The consequence. Several of the paper's specific performance claims may be within sampling noise and are not reliably distinguishable from zero effect. For example:

• Qwen3-8B on AIME'24: +1.25 Acc@16. With 30 questions and 16 samples each (480 total answer evaluations), a 1.25-point swing in average accuracy represents roughly 6 additional correct answers out of 480 — a 1.25% change that could plausibly arise from sampling variation.
• Qwen3-8B on AMC'23: −0.625 Acc@16. This negative result (forking-token DAPO slightly worse than vanilla DAPO) might be noise or might be a genuine signal that the approach hurts on certain benchmark distributions — the paper cannot distinguish these possibilities.
• Qwen3-32B on Minerva: +0.12 Acc@16. This near-zero difference cannot be meaningfully interpreted without uncertainty quantification.
• Qwen3-32B on MATH500: +0.52 Acc@16. Again, a sub-one-point difference on a benchmark where statistical variance is not characterized.

The paper's headline scaling trend (Figure 8) — "as model size increases, the performance gain over vanilla DAPO becomes increasingly significant" — is visually clear at 14B and 32B for AIME'24 and AIME'25, but the 8B data points sit in a statistical gray zone. Without error bars, the reader cannot assess whether the 8B results represent a genuine null effect (forking-token DAPO truly equivalent to vanilla DAPO at this scale) or an effect hidden by measurement noise. The LiveCodeBench results (Figure 9) are explicitly described as having "high variance" and are presented with smoothing — an acknowledgment of noise, but without quantifying it or reporting unsmoothed values.

What evidence exists in the paper. The paper never reports standard errors, confidence intervals, or p-values. The AIME benchmarks are identified as containing 30 questions each (standard for AIME, which has 30 problems per year), but the paper does not discuss the implications of this small test-set size for statistical power. The LiveCodeBench variance is acknowledged verbally ("Due to the high variance of the accuracy curves, we smooth the curves using window smoothing with a window size of 10") but not quantified.

Mitigation status. The paper uses Acc@16 (average over 16 independent responses per question) which reduces sampling variance compared to single-sample accuracy, but does not provide the statistical machinery to translate this into confidence bounds. Re-running evaluations with different random seeds, reporting bootstrap confidence intervals, or using a larger evaluation benchmark for the small-effect claims would address this limitation, but none of these are done.

Difficulty Estimation Overhead and Per-Batch Thresholding Cost

The assumption or constraint. The gradient-masking approach requires computing per-token entropy for every token in every generated response in every training batch, and then computing the batch-specific $\rho$-th percentile threshold $\tau^B_\rho$ to determine which tokens to retain. The paper does not account for the computational overhead of this entropy computation and thresholding in any cost analysis, nor does it compare wall-clock training time between vanilla DAPO and forking-token DAPO.

The consequence. While the paper frames the approach as a potential efficiency win (training on 20% of tokens suggests 5× fewer gradient computations, though this is not the paper's focus), the practical reality may be different. Computing token entropies requires running the full forward pass to obtain logits for all tokens — the same computation that vanilla DAPO performs. The entropy computation itself (summing $-p \log p$ over the vocabulary for each token) is relatively cheap as a post-processing step, but the per-batch threshold requires sorting all token entropies across the entire batch to find the $\rho$-th percentile, which for large batch sizes (512 prompts × responses per prompt × tokens per response) could be non-trivial with naive implementations. More fundamentally, the forward and backward passes through the model are identical — the gradient mask only zeroes out certain token positions in the loss computation. This means the FLOPs per training step are essentially unchanged; the benefit is in which gradients are applied, not in reduced computation.

A practitioner considering adopting this method might reasonably expect training-time speedups from using only 20% of tokens. The paper provides no evidence for or against such speedups, and the approach as implemented (computing entropy for all tokens to then mask 80%) incurs additional overhead relative to vanilla DAPO. If the method does not reduce wall-clock training time and the performance benefit is concentrated at larger model scales (32B >> 14B > 8B), the practical cost-benefit calculation differs substantially from what a casual reading might suggest.

What evidence exists in the paper. None. The paper does not report training wall-clock time, FLOP counts per step, GPU-hours to convergence, or any comparison of computational cost between forking-token DAPO and vanilla DAPO. The experiments use identical batch sizes and optimization hyperparameters, but the overhead of entropy computation and thresholding is unmeasured. The paper's abstract and introduction frame the contribution in terms of understanding mechanisms and improving performance, not computational efficiency, so this is not a misrepresentation — but it is an unaddressed practical consideration.

Mitigation status. The paper does not discuss this as a limitation. The per-batch thresholding design choice is justified in terms of adapting to the shifting entropy distribution during training (which is methodologically sound), but the computational cost of this choice is not analyzed. A practitioner would need to implement and profile the approach on their own infrastructure to determine the net cost.

The 20% Threshold Is Empirically Derived from One Model and Not Theoretically Justified

The assumption or constraint. The retention ratio $\rho = 20\%$ is set based on the observation that 20% of tokens exceed the 80th-percentile entropy threshold ($h_{\text{threshold}} = 0.672$) in $10^6$ CoT tokens generated by Qwen3-8B on AIME problems at $T=1.0$. The paper then applies this same 20% ratio to all three model sizes (8B, 14B, 32B) in the RLVR experiments, and the ablation in Figure 7 confirms that 20% is optimal for all three sizes. However, the paper provides no principled reason why 20% should be optimal beyond the empirical observation, and does not investigate whether a different proportion would be optimal for different models, tasks, or training configurations.

The consequence. A practitioner applying this method to a different model family, a different task domain (e.g., coding, scientific reasoning), or a different RL algorithm cannot assume that $\rho = 20\%$ is optimal — or even reasonable. The paper's own ablation (Figure 7) shows that the sensitivity to $\rho$ varies with model size: on 8B, performance is largely flat from 10% to 50%; on 32B, the 20% setting is clearly superior and deviations cause notable drops. This suggests that the optimal proportion is model-dependent, yet the paper offers no diagnostic for determining it without running the full ablation that Figure 7 represents (training multiple models from scratch at different $\rho$ values — a computationally expensive proposition).

The absence of a theoretical grounding for the threshold also limits the paper's explanatory power. The paper frames forking tokens as "critical decision points that determine the trajectory of reasoning," but if only 20% of tokens serve this role, why 20%? Is this a fundamental property of language (20% of tokens in any coherent text are decision points), a property of mathematical reasoning (math CoTs have a specific ratio of logical-connective to content tokens), an artifact of Qwen3's pretraining, or something else? Without a mechanistic account, the 20% figure remains an empirical regularity whose domain of validity is unknown.

What evidence exists in the paper. The choice of $\rho = 20\%$ is justified in Section 3: "the entropy of over half the tokens (approximately 50.64%) is below $10^{-2}$, while only 20% of tokens have entropy greater than 0.672." This is a descriptive statistic from one model on one task. The paper acknowledges in the Limitations section that "it is possible that the observations and conclusions presented in this paper may not generalize to all RLVR scenarios. For instance, in a different RLVR setting, the effective proportion of 20% observed in our experiments may need to be adjusted to a different value to achieve optimal results." This is an honest acknowledgment, but it leaves the practitioner with a hyperparameter-tuning problem rather than a principle.

Mitigation status. The paper partially mitigates this by conducting the $\rho$ ablation (Figure 7) on all three model sizes, showing that 20% is consistently optimal for Qwen3 models in this setup. This provides some evidence of robustness within the Qwen3 family, but does not provide a transferable heuristic. The paper's future directions mention "developing new RLVR algorithms to better leverage high-entropy minority tokens," which might include adaptive or learned-threshold approaches, but these are not explored in the current work.

Hard Problems Are Not Addressed and May Be Fundamentally Out of Scope

The assumption or constraint. The paper's central mechanism — that RLVR works by amplifying exploration at high-entropy decision points inherited from pretraining — implies a hard boundary on what the approach can achieve. If a base model has near-zero entropy across all token positions for a class of problems (the model is uniformly confident, even when wrong), there are no high-entropy forking tokens to amplify, and RLVR cannot improve performance through exploration at decision points. The paper does not characterize what fraction of its training or evaluation data falls into this regime, nor does it analyze whether the approach differentially helps easy, medium, or hard problems.

The consequence. The approach may provide zero benefit — or be indistinguishable from vanilla DAPO — on problem distributions that are too far outside the base model's pretrained capabilities. This is directly analogous to the finding in the reference paper studied earlier, where test-time compute scaling provided no benefit on the hardest difficulty bin because the base model never generated correct solutions to refine. Similarly, here, if the base model has no uncertainty about its (incorrect) reasoning on hard problems, there are no forks to explore, and neither vanilla DAPO nor forking-token DAPO will improve performance. The paper's results on Llama-3.1-8B (Figure 11) hint at this: the base model collapsed to ~1% accuracy under direct RLVR, suggesting that for mathematical reasoning, Llama-3.1-8B's pretraining may not provide meaningful uncertainty at any decision points — the model is confidently wrong everywhere, leaving no forks for RLVR to leverage.

The paper's scaling trend (Figure 8) shows that the benefit of forking-token DAPO increases with model size. This is consistent with the interpretation that larger models inherit richer entropy landscapes from pretraining — they have more meaningful forks at more positions — and thus benefit more from selective gradient application. Smaller or less capable models may have flatter entropy distributions where the 20% threshold captures less meaningful variation, making the approach perform similarly to random token selection (which was never tested, but the 8B insensitivity to $\rho$ in Figure 7 is consistent with this).

What evidence exists in the paper. The paper does not report difficulty-stratified results. Table 2 provides aggregate benchmark scores but does not break down performance by problem difficulty within each benchmark. The failure of direct RLVR on Llama-3.1-8B (requiring cold-start SFT) is evidence that base-model capability matters, but the paper does not analyze this through the lens of entropy patterns — it does not show, for example, that Llama-3.1-8B's entropy distribution on math problems is less bimodal or has lower overall entropy than Qwen3-8B's.

Mitigation status. The paper does not address this limitation directly. The Limitations section mentions extending "the scope of our dataset... to encompass domains beyond mathematics," but does not discuss the capability-boundary issue within the math domain itself. The mechanism implies this boundary exists, but the paper does not attempt to characterize where it lies or how to detect it in practice.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a conceptual pivot in how the field understands RLVR for LLM reasoning: the locus of effective learning is not distributed uniformly across Chain-of-Thought trajectories but is concentrated at a small fraction of high-entropy "forking tokens" — and this concentration is not a bug to be corrected but a feature of how pretrained language models structure the action space available to RL. This reframes RLVR from a problem of credit assignment over homogeneous token sequences to a problem of identifying and amplifying exploration at pre-existing decision points inherited from pretraining.

Is this a paradigm shift or an incremental refinement? The paper is best understood as a diagnostic reframing with significant practical implications, rather than a paradigm shift. It does not propose a new RL algorithm — it uses DAPO as-is with a gradient mask — nor does it claim to discover a new training objective. Instead, it provides a new analytical lens (token-level entropy as a model-intrinsic signal for learning relevance) and uses this lens to explain, unify, and improve upon existing empirical findings. The conceptual shift is from asking "how should we assign credit across tokens based on outcomes?" (the traditional RL framing) to asking "which tokens does the model itself treat as decision points, and how can we focus learning there?" This reorients attention from advantage estimation (the output side) to the model's own uncertainty landscape (the input side), which is a genuinely different perspective on RLVR mechanics.

The magnitude of the practical implications is scale-dependent. For smaller models (8B), the approach is essentially neutral — it preserves performance without providing gains, making it a no-cost diagnostic rather than a performance booster. For larger models (32B), the gains are substantial (+11.04 on AIME'25, +7.71 on AIME'24), and the approach produces state-of-the-art results within its category. This scaling trend (Figure 8) suggests that the reframing becomes more consequential as models grow, which aligns with the broader trajectory of the field toward ever-larger models where understanding where learning happens becomes increasingly important for efficient training.

Resolving prior contradictions. The paper's token-entropy framework provides a mechanistic reconciliation of several previously puzzling findings:

• Gandhi et al. (2025) found that RLVR primarily learns reasoning behaviors (format, structure) rather than content. The paper's entropy-preservation finding (Table 1: 86.67% overlap in high-entropy positions between base and RLVR model) provides the mechanism: the structure of reasoning — which token positions are decision points — is fixed by pretraining. RLVR cannot create new forks; it can only adjust probabilities at existing ones. The learning is therefore constrained to refining how the model navigates a pre-existing decision structure, which manifests as improved reasoning format rather than novel content generation capabilities. This doesn't contradict Gandhi et al. — it explains why their finding holds, by identifying the structural constraint that channels RLVR's effects toward format learning.

• Li et al. (2025) showed that structure matters more than content in CoT demonstrations. The paper's word cloud analysis (Figure 2b, 2c) shows that high-entropy forking tokens are precisely the structural elements — logical connectors like "however," "therefore," "suppose" — while low-entropy tokens are content completions (word suffixes, mathematical notation, code fragments). If RLVR primarily operates on high-entropy tokens (as the gradient ablation demonstrates), then RLVR is structurally biased toward learning from structural tokens. This explains both why structure matters (it's where learning happens) and why content can be randomized without destroying learnability (the structural forks remain identifiable). The finding thus bridges Li et al.'s behavioral observation with a token-level mechanism.

• Chu et al. (2025) found that RL generalizes while SFT memorizes. The paper's Discussion 1 hypothesis — that SFT collapses entropy at forks while RLVR preserves or increases it — provides a testable mechanistic account. The temperature manipulation experiments (Figure 3) directly show that reducing entropy at forks (which SFT does by pushing toward one-hot distributions) destroys reasoning performance, while increasing it (which RLVR naturally does, per Figures 4 and 5) improves performance. This transforms the SFT-vs-RL generalization puzzle from an empirical observation into a prediction of the entropy framework: any training method that collapses forking-token entropy should impair generalization, regardless of whether it's called SFT or something else.

Research directions that become more attractive. The paper makes several lines of inquiry newly tractable or more compelling:

• Verifier-free token importance detection. The paper demonstrates that entropy alone — without any correctness signal — identifies learning-relevant tokens. This opens the possibility of importance-weighted training even in domains without verifiable rewards, using the model's own uncertainty as a proxy for where learning should focus. If entropy-based token selection works for RLVR, it might also improve SFT (by upweighting high-entropy tokens in the cross-entropy loss) or preference optimization (by focusing DPO/RLHF gradients on uncertain tokens).

• Entropy landscape analysis as a diagnostic for base model readiness. The paper implies that a base model's entropy distribution predicts how much it will benefit from RLVR. A model with a rich, bimodal entropy distribution (clear separation between low-entropy continuations and high-entropy forks) should benefit more than one with a flat entropy distribution. This suggests entropy profiling as a cheap pre-training diagnostic: before committing to an expensive RLVR run, sample a few hundred CoT responses, compute the entropy distribution, and check whether it exhibits the bimodal pattern. If it doesn't, the base model may need more pretraining or SFT initialization before RLVR can be effective.

• Selective entropy regularization as an alternative to uniform entropy bonuses. The paper argues (Discussion 3) that uniform entropy bonuses are suboptimal because they increase entropy at low-entropy continuations, degrading coherence. The forking-token approach achieves selective entropy increase through gradient masking rather than an explicit bonus term, but the principle suggests a direct algorithmic improvement: add an entropy bonus that is gated by token entropy, encouraging higher entropy only at already-high-entropy positions. This would be simpler to implement than per-batch thresholding and might achieve similar effects.

Research directions that become less attractive. The paper's findings also cast doubt on certain approaches:

• Uniform entropy bonuses for LLM reasoning (as in traditional RL). The paper provides direct evidence (Figure 3: increasing $T_{\text{low}}$ causes nonsensical outputs) that uniformly increasing entropy across all tokens is harmful. This suggests that RLVR implementations adding standard entropy bonus terms to PPO/GRPO objectives are likely working despite this term (because other mechanisms dominate) or are using bonus coefficients small enough to avoid catastrophic disruption of low-entropy continuations. The paper's frame implies that future work should abandon uniform entropy bonuses in favor of selective exploration mechanisms.

• Token-level credit assignment based solely on advantage magnitude. The success of entropy-based selection (without considering advantage magnitude) suggests that where the model is uncertain is more informative for learning than how much a particular token contributed to the outcome. This challenges approaches that use |Ât| as a criterion for which tokens to train on — the paper's results imply that low-advantage high-entropy tokens may be more valuable for exploration than high-advantage low-entropy tokens. This is a testable hypothesis the paper itself does not directly verify (since it never pits entropy against advantage as competing selection criteria), but it follows from the mechanism.

Follow-Up Research This Work Enables

1. Random-selection control to isolate whether entropy specifically matters or whether gradient sparsity alone provides the benefit. The single most important missing experiment in the current paper is training with a random 20% of tokens masked (keeping the retention fraction constant but selecting tokens uniformly rather than by entropy). This would cleanly separate two competing explanations for the paper's main result: (a) entropy identifies reasoning-critical decision points, as argued, versus (b) any form of gradient sparsity at ~20% density improves RLVR optimization dynamics (through implicit regularization, reduced gradient noise, or changed effective learning rate). A strong follow-up would train Qwen3-32B with random-20%, entropy-top-20%, and 100% token retention under identical DAPO hyperparameters, tracking AIME'24 scores and overall entropy throughout training. If random-20% performs similarly to entropy-top-20%, the "forking token" interpretation collapses — gradient sparsity, not entropy, would be the active ingredient. If random-20% performs similarly to or worse than 100% (vanilla DAPO), the entropy criterion is validated as specifically identifying learning-relevant tokens. If random-20% falls between entropy-top-20% and 100%, the interpretation is more nuanced: sparsity helps, but entropy-based selection helps more. This experiment would also clarify whether the 8B results (where forking-token DAPO is neutral) reflect a genuine null effect of entropy selection or a ceiling where even random sparsity would be neutral.

2. Entropy-landscape comparison across model families to identify what base-model properties predict RLVR benefit. The paper's core mechanism implies that the base model's entropy distribution determines how much RLVR can improve reasoning — models with cleanly bimodal entropy distributions (clear separation between low-entropy continuations and high-entropy forks) should benefit more from selective gradient application than models with flatter distributions. A systematic follow-up would collect CoT outputs from a diverse set of base models at comparable scales (e.g., Qwen3-14B, Llama-3.1-14B, Mistral-14B, Gemma-14B, DeepSeek-14B) on the same set of math problems (AIME'24 + AIME'25), compute per-token entropy distributions, and measure bimodality metrics (e.g., the dip statistic, variance ratio between high-entropy and low-entropy clusters, or simply the fraction of tokens with entropy < 10⁻² and > 0.5). These models would then each undergo vanilla DAPO training on DAPO-Math-17K, and the correlation between entropy bimodality and RLVR performance improvement would be computed. The prediction from the paper is that Qwen3 models have more bimodal entropy distributions than Llama-3.1, which would explain why Llama-3.1-8B required cold-start SFT and why Qwen3 models show strong RLVR gains. If the correlation holds, entropy profiling becomes a cheap pre-training diagnostic; if it doesn't, the mechanism is more complex than the paper's framework suggests.

3. Entropy-gated entropy bonus as a simpler alternative to gradient masking. The paper's gradient mask achieves selective entropy increase (only at forks) through a somewhat complex mechanism: compute all token entropies, sort, find the batch threshold, mask 80% of tokens in the loss. An alternative approach motivated by the paper's framework would be to add an entropy bonus term to the DAPO objective that is itself gated by token entropy: $\mathcal{L}_{\text{bonus}} = -\lambda \cdot \mathbb{I}[H_t > \tau] \cdot H_t$, which encourages higher entropy only at tokens that are already above the threshold. This is simpler to implement (no batch-level sorting, just a fixed threshold from the base model's entropy distribution) and directly operationalizes the paper's central insight. A follow-up experiment would compare forking-token DAPO (gradient masking) against entropy-gated-entropy-bonus DAPO on Qwen3-32B, tracking both AIME'24 scores and the evolution of per-percentile entropy (replicating Figure 5 for the bonus variant). Key questions: does the bonus approach achieve similar selective entropy increase? Does it produce the same training stability benefits (Figure 6a shows gradient masking stabilizes training)? Does the optimal gate threshold $\tau$ align with the 80th percentile of the base model's entropy distribution, as the paper's framework predicts?

4. Difficulty-stratified analysis to test the capability-boundary prediction. The paper's mechanism implies a hard boundary: if the base model has near-zero entropy at all positions for a class of problems (uniformly confident, even when wrong), there are no forking tokens to amplify, and RLVR cannot help regardless of gradient selection strategy. This can be tested by difficulty-stratifying the evaluation benchmarks. Using the same approach as the reference paper studied earlier (difficulty bins based on base-model pass@1 rate), a follow-up would bin AIME'24 and AIME'25 problems by how often Qwen3-32B-base generates correct answers, then evaluate forking-token DAPO and vanilla DAPO performance within each bin. The prediction is that the performance delta (forking minus vanilla) should be largest on medium-difficulty problems where the base model has meaningful uncertainty at decision points, and should approach zero on both very easy problems (where the model is already correct regardless of fork exploration) and very hard problems (where the model has no meaningful forks — it's uniformly confident in incorrect reasoning). If the delta is uniform across difficulty, the exploration-amplification mechanism is incomplete. If it follows the predicted inverted-U shape, the mechanism is validated and the approach can be targeted to problem distributions where it matters most.

5. SFT with entropy-preserving regularization to test the SFT-memorizes hypothesis. The paper's Discussion 1 proposes that SFT impairs generalization by collapsing entropy at forking tokens. This can be directly tested by modifying SFT to include an entropy-preservation term: during supervised fine-tuning on correct reasoning trajectories, add a regularization loss that penalizes entropy reduction at token positions that were high-entropy in the base model: $\mathcal{L}_{\text{reg}} = \lambda \cdot \sum_{t: H_t^{\text{base}} > \tau} \max(0, H_t^{\text{base}} - H_t)$, which penalizes the model for becoming more confident than the base model at positions the base model found uncertain. A follow-up would compare three training pipelines on Qwen3-14B: (a) standard SFT on correct reasoning trajectories, (b) SFT with entropy-preserving regularization at forking tokens, and (c) standard RLVR (DAPO). All three would be evaluated on both in-distribution (MATH500) and out-of-distribution (AIME'24, LiveCodeBench) benchmarks. The paper's framework predicts that entropy-preserving SFT should outperform standard SFT on out-of-distribution generalization (approaching RLVR's generalization performance while retaining SFT's training stability), and that the benefit should be concentrated on problems where the base model has clearly identifiable forking tokens. If entropy-preserving SFT matches standard SFT on OOD benchmarks, the SFT-memorizes hypothesis would need revision — entropy collapse at forks would not be the primary driver of SFT's generalization weakness. This experiment would also be a first step toward the paper's suggested future direction of "exploring how these insights can enhance not only RLVR but also other approaches, such as supervised fine-tuning (SFT)."

6. Token-importance criterion comparison: entropy vs. advantage magnitude vs. attention entropy. The paper argues that token entropy — a property of the model's output distribution — is the right signal for identifying learning-relevant tokens. But alternatives exist and the paper never compares against them. A systematic follow-up would compare five token selection criteria for gradient masking at $\rho = 20\%$ on Qwen3-32B with DAPO: (a) entropy (the paper's criterion), (b) absolute advantage $|\hat{A}_t|$ (tokens whose generation most influenced the outcome), (c) surprisal $-\log p(o_t | q, o_{<t})$ (how unexpected the actual token was), (d) attention entropy (entropy of the attention distribution from that token position, measuring how broadly it attends to context), and (e) random selection (the crucial control from direction #1 above). The evaluation would track AIME'24 scores, overall entropy, and response length throughout training. This comparison would establish whether entropy is uniquely informative or whether several correlated signals identify similar token subsets. If entropy and absolute advantage identify largely overlapping token sets (which is plausible — uncertain tokens might have more variable outcomes), the practical recommendation would be to use whichever is cheaper to compute. If entropy is uniquely effective, it validates the paper's conceptual framework that model-intrinsic uncertainty (not outcome-based importance) is the right signal. If attention entropy is equally effective, it suggests the mechanism operates through the model's contextual integration patterns rather than output uncertainty specifically.

Practical Applications and Downstream Use Cases

1. Cost-efficient RLVR training for large-scale reasoning models. For organizations training reasoning models at the 32B+ scale, the paper's approach offers a training method that achieves superior final performance without additional computational cost per training step. The forking-token DAPO variant uses identical batch sizes, forward/backward passes, and optimizer steps as vanilla DAPO — the only additional computation is the per-batch entropy calculation and thresholding, which is negligible compared to the transformer forward/backward passes. The benefit is a direct accuracy improvement: on Qwen3-32B, forking-token DAPO delivers +7.71 on AIME'24 and +11.04 on AIME'25 (Table 2) without any hyperparameter tuning beyond setting $\rho = 20\%$. For a production training run costing hundreds of thousands of GPU-hours, this improvement comes essentially for free. The training stability benefit (Figure 6a shows vanilla DAPO exhibits a mid-training performance degradation that forking-token DAPO avoids) further reduces the risk of wasted compute on runs that collapse and must be restarted. The primary caveat is that the approach has been validated only on Qwen3 models with mathematical reasoning data; organizations using other model families should first verify that their base model exhibits the bimodal entropy pattern (Figures 2, 12-17) before committing to a full training run.

2. Selective gradient application as a diagnostic for base model readiness. Before investing in a full RLVR training run, practitioners can cheaply assess whether their base model is likely to benefit from RLVR — and whether the forking-token approach specifically will help — by profiling the entropy distribution of CoT outputs. The procedure would be: generate ~100-200 CoT responses from the base model on a representative sample of training problems at $T=1.0$, compute per-token entropies, and check for the bimodal pattern (massive spike near zero entropy for continuations, sparse tail of high-entropy tokens). If the pattern is present, the base model has a well-structured uncertainty landscape and RLVR — especially with entropy-based gradient selection — should be effective. If the entropy distribution is flat (no clear separation) or uniformly low (base model is confident everywhere), the model may lack the necessary pretrained knowledge for RLVR to build on, and additional pretraining or SFT initialization (as the paper did for Llama-3.1-8B in Figure 11) would be a more efficient use of compute than direct RLVR. This diagnostic is essentially free (a few hundred forward passes) compared to the cost of a full RLVR training run, and can prevent expensive failures like the Llama-3.1-8B collapse to ~1% accuracy that the paper reports.

3. Inference-time decoding strategies for reasoning models based on forking-token identification. The temperature manipulation experiments in Figure 3 are conducted at inference time on a fixed model and demonstrate that selectively increasing temperature at forking tokens improves AIME scores (from ~71.04 at $T_{\text{high}}=1.0$ to ~72.22 at $T_{\text{high}}=3.0$) while increasing temperature at continuation tokens degrades performance. This suggests a practical decoding strategy for deployed reasoning models: during autoregressive generation, compute the entropy of the model's output distribution at each step using the unmodified logits ($T=1.0$), classify the token as forking or continuation based on whether its entropy exceeds the 80th percentile of the base model's entropy distribution, and apply a higher temperature ($T \approx 2.0-3.0$) to forking tokens while keeping continuation tokens at $T=1.0$. This requires no model modification — it's a purely decoding-time intervention — and the paper's Figure 3 provides the temperature sweep data to select the optimal $T_{\text{high}}$. The benefit is a small but measurable accuracy improvement (~1-2 AIME points in the paper's experiments) at zero training cost. The approach would be particularly valuable for applications where a deployed model must maximize accuracy on difficult reasoning problems within a fixed generation budget, and where the additional latency from entropy computation at each decoding step (a softmax + entropy calculation over the vocabulary) is acceptable.

4. Data filtering and curriculum design for RLVR training data. The paper's finding that only ~20% of tokens drive learning suggests that not all training examples are equally valuable — problems that elicit responses with many high-entropy forking tokens should produce richer learning signals than problems where the model is uniformly confident. A practical data filtering strategy would be: before full-scale RLVR training, generate a small number of responses for each problem in the training dataset, compute the average fraction of forking tokens per response, and filter or weight the training data to emphasize problems where the base model exhibits meaningful uncertainty at decision points. Problems where the base model is uniformly low-entropy (either because they're too easy and the model is correctly confident, or too hard and the model is incorrectly confident) could be downweighted or deferred to later training stages. This connects to curriculum learning: start RLVR on problems with intermediate forking-token density (where the model has clear decision points but isn't yet navigating them optimally), then gradually introduce harder problems as the model's entropy landscape evolves. The paper's Figure 5 provides the monitoring tool — tracking entropy percentile trajectories during training would reveal when the model has developed sufficient uncertainty at harder problems' decision points to benefit from training on them.

When to Prefer This Method

The paper itself articulates a clear tradeoff — the forking-token approach's benefit scales with model size and depends on the base model having a well-structured entropy landscape. The decision rule is not "always use forking-token DAPO" but rather conditional on several factors that the paper's experiments characterize:

• Prefer forking-token DAPO (top 20% entropy, $\rho = 20\%$) when:

  • Training larger models (14B+, especially 32B+) where the performance benefit is substantial (+4-11 AIME points) and training stability improvements (avoiding mid-training degradation visible in Figure 6a) reduce the risk of wasted compute.
  • The base model exhibits the bimodal entropy pattern — a clear spike at near-zero entropy with a sparse tail of high-entropy tokens — which can be verified by profiling ~100-200 CoT outputs before training.
  • Training on mathematical or logical reasoning tasks where the base model has meaningful pretrained knowledge (pass@1 is non-trivially above zero) and the CoT format naturally produces structural forking tokens (logical connectors, assumption introductions).
  • The deployment scenario values training stability — Figure 6 shows forking-token DAPO avoids the performance degradation around step 800 that vanilla DAPO experiences on 32B.

• Use vanilla DAPO (or either method) when:

  • Training smaller models (8B and below) where the performance benefit is minimal (+0.53 average across benchmarks, Table 2) and the approach adds implementation complexity without meaningful gains.
  • The base model's entropy distribution is flat or uniformly low — as the paper found with Llama-3.1-8B requiring cold-start SFT, models without clear entropy bimodality may not benefit from entropy-based selection.
  • The training domain lacks the structural forking tokens that mathematical CoT reasoning provides — for example, factual QA or summarization tasks where most tokens are content words rather than logical connectors, and the entropy distribution may not cleanly separate decision points from continuations.

• Consider entropy-gated temperature decoding (Figure 3) instead of full retraining when:

  • A reasoning model is already deployed and cannot be retrained, but accuracy improvements of 1-2 points on hard benchmarks would be valuable.
  • The cost of per-token entropy computation during decoding is acceptable for the deployment's latency requirements.
  • The model's base entropy distribution has been profiled and the 80th-percentile threshold $h_{\text{threshold}}$ has been calibrated (the paper used 0.672 for Qwen3-8B on AIME, but this should be recomputed for different models and tasks).