Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward

ArXiv: 2512.16912

🎯 Pitch

Giving LLMs completely random rewards during math training can actually improve their performance—and it works because PPO-style ratio clipping secretly acts as an entropy squeezer, not because the model learns anything from the noise. Stronger models gain more from this bizarre trick because their correct answers are less likely to be punished by false-negative mislabeling.

1. Executive Summary

This paper analyzes the interplay between clipping bias, policy entropy, and spurious (random) rewards in reinforcement learning with verifiable rewards (RLVR) for LLMs, conducting experiments across Qwen-Math, Llama, and QwQ model families on MATH500 and AIME benchmarks. The work demonstrates that clipping bias—the asymmetry in how PPO-style ratio clipping permits larger probability increases for already-high-probability tokens—provides no meaningful learning signal under random rewards, but instead functions as an implicit entropy minimization mechanism that drives the policy toward more deterministic, higher-confidence outputs. The authors further introduce a reward-misalignment model (formalizing correct-response advantage loss under binary mislabeling) showing that stronger models benefit more from spurious rewards because they incur fewer false-positive misallocations, establishing that performance gains under random rewards are robust and not confined to potentially contaminated Qwen-Math models—while entropy reduction alone proves insufficient for improvement, as evidenced by entropy decreases coinciding with performance collapse on harder datasets.

2. Context and Motivation

The Core Puzzle: An RLVR-Specific Paradox

Reinforcement learning with verifiable rewards (RLVR) has emerged as the dominant post-training paradigm for eliciting strong reasoning capabilities in LLMs. The recipe is conceptually straightforward: sample multiple completions from a model for a given prompt, compare each against a ground-truth answer (e.g., a math solution), assign binary rewards, compute relative advantages within the group, and update the policy using a clipped surrogate objective — typically via Group Relative Policy Optimization (GRPO), which has become popular for its computational simplicity and memory efficiency. This framework underpins models like DeepSeek-R1 and Kimi-K2, which achieve competitive performance on challenging benchmarks including International Mathematical Olympiad problems.

But RLVR, as practiced, harbors a deep puzzle. Two seemingly contradictory interventions both appear to improve reasoning performance, and understanding why — or whether — has remained elusive:

1. Spurious rewards improve performance. Shao et al. (2025) reported that replacing the verifiable ground-truth reward with random binary noise (independently sampled from Bernoulli(1/2), uncorrelated with answer correctness) yielded striking performance gains on MATH500 for the Qwen-Math model family. This is genuinely surprising: in classical RL, injecting random reward signals would be expected to degrade performance by encouraging suboptimal actions. Yet in RLVR, the opposite appears to happen.

2. Entropy minimization improves performance. Multiple recent works (Agarwal et al., 2025; Zhang et al., 2025; Zhao et al., 2025; Cui et al., 2025; Fu et al., 2025) have demonstrated that reducing policy entropy — making the model's output distribution more peaked, more deterministic, less exploratory — yields consistent gains in reasoning accuracy. Some even propose an explicit monotonic relationship where lower entropy strictly implies better performance (Cui et al., 2025). Gao et al. (2025) go further, showing that entropy minimization alone, without any verifiable feedback, can substantially improve reasoning.

The paradox is this: discouraging exploitation (via random rewards) and discouraging exploration (via entropy minimization) both help. In classical reinforcement learning, these would be opposing forces — spurious rewards should degrade performance by reinforcing incorrect actions, while entropy minimization should collapse exploration and risk premature convergence to suboptimal policies. That both are beneficial in RLVR signals a learning dynamic fundamentally different from what classical RL intuitions would predict.

This paradox is not merely an academic curiosity. It exposes a gap in our understanding of RLVR that has direct consequences for how we design training algorithms, allocate compute, and interpret results. The paper's opening paragraphs frame the problem precisely:

"This paper examines the exploration-exploitation trade-off in reinforcement learning with verifiable rewards (RLVR), a framework for improving the reasoning of Large Language Models (LLMs). Recent studies suggest that RLVR can elicit strong mathematical reasoning in LLMs through two seemingly paradoxical mechanisms: spurious rewards, which suppress exploitation by rewarding outcomes unrelated to the ground truth, and entropy minimization, which suppresses exploration by pushing the model toward more confident and deterministic outputs, highlighting a puzzling dynamic: both discouraging exploitation and discouraging exploration improve reasoning performance, yet the underlying principles that reconcile these effects remain poorly understood."

Why This Matters: Beyond the Qwen-Math Contamination Debate

The practical stakes of resolving this puzzle are high, and they crystallize around a specific controversy: do the reported gains from random rewards actually reflect improved reasoning, or are they artifacts of benchmark contamination?

The contamination hypothesis. Wu et al. (2025) found substantial contamination in Qwen-Math models on the MATH500 validation benchmark — meaning these models likely encountered MATH500 problems (or near-duplicates) during pretraining. Under contamination, the model already possesses the correct answers in its training distribution; random-reward fine-tuning might simply amplify these memorized trajectories rather than teaching new reasoning. If this hypothesis is correct, spurious-reward gains are a red herring — an evaluation artifact with no general significance for RLVR training.

Conflicting empirical reports. Oertell et al. (2025) further challenge the spurious-reward narrative, arguing that reported gains arise from algorithmic heuristics and evaluation artifacts rather than genuine improvement; in their experiments, random-reward fine-tuning does not consistently improve reasoning and can even degrade it. These conflicting findings — some papers showing benefits, others showing harm — create genuine confusion about whether random rewards are a useful training intervention or a dangerous source of overfitting to contaminated data.

What's at stake for RLVR design. If random-reward gains are purely a contamination artifact, the practical takeaway is narrow: be careful about data leakage, don't use random rewards. But if the gains reflect something deeper about RLVR learning dynamics — about how clipping interacts with policy entropy, about how model strength conditions the reliability of reward signals — then the implications are much broader. Understanding the mechanism would provide principles for:

• Entropy modulation during training: When should we compress the policy, and when should we preserve diversity?
• Clipping threshold selection: Is clipping's primary role to prevent gradient explosion, or does it also shape the policy's entropy trajectory in ways that matter for performance?
• Reward signal design: Could deliberately mixing true and spurious rewards, at controlled ratios, improve exploration beyond what pure verifiable signals achieve?
• Model-specific training recipes: If the benefits depend on baseline model capability (as the paper argues), then optimal training strategies might differ qualitatively between weak and strong models.

The paper's central motivation is to disentangle these mechanisms — clipping bias, policy entropy, and reward misalignment — to establish what is actually happening under random-reward training, and to determine whether the observed benefits generalize beyond potentially contaminated settings.

Where Prior Approaches Fall Short

The paper identifies four specific limitations in prior work that prevent a coherent understanding:

1. The upper-clipping bias explanation is incomplete and likely wrong.

The most prominent attempt to explain random-reward gains comes from Shao et al. (2025), who attribute improvements to PPO-style upper-clipping bias (Remark 2.5 in the paper). The logic is this: GRPO's clipped surrogate objective enforces r_t(θ) ≤ 1 + ε, meaning a token's probability cannot increase beyond (1 + ε) × π_old(token | context). Tokens that already have high probability under the old policy can receive larger absolute probability increases (ε × π_old) before hitting the cap, whereas low-probability tokens reach the clipping threshold much earlier with negligible absolute increases. Under random rewards, this asymmetry means high-prior responses get preferentially amplified — potentially surfacing latent knowledge the model already possesses (or has memorized).

The paper subjects this explanation to rigorous theoretical scrutiny and finds it wanting. Theorem 3.2 derives explicit bounds on the expected magnitude of the total upper-clipping correction term C_tot^+, and Theorem 3.4 compares this to the magnitude of the raw (unclipped) surrogate. Under the actual hyperparameters used in training (step size η = 5 × 10^(-7), clip ratio ε = 0.2, group size G = 16, observed clipping activation rate p^+ ≈ 0.001), Corollary 3.6 shows that the raw surrogate is more than 17 times larger in magnitude than the clipping correction. In the paper's words:

"the upper-clipping bias fails to provide meaningful learning signal towards the gradient, even under contaminated model and benchmark."

The clipping bias is simply too small to drive the observed performance changes. Something else must be at work.

2. Existing entropy analysis fails under random rewards.

Cui et al. (2025) provide a theoretical analysis of how policy entropy changes under GRPO updates, deriving the first-order approximation in Equation 5:

ℋ(π_new) - ℋ(π_old) ≈ -Cov_{y~π_old}(log π_old(y|x), A(x, y))

The intuition is clean: if the advantage A(x, y) is positively correlated with the log-probability of a response — meaning high-probability responses get positive advantages, low-probability responses get zero or negative advantages — then the policy concentrates further, reducing entropy. If the correlation is reversed, entropy increases.

But Remark 2.7 identifies a critical failure: under random rewards, this approximation predicts zero entropy change. Since A(x, y) is independent of π_old(y | x) and has zero mean (Lemma 2.4), the covariance term vanishes identically. The first-order analysis says nothing happens — yet the paper's experiments (Figures 1 and 2) show clear, systematic entropy changes under random-reward training. The discrepancy arises because Equation 5 (i) retains only first-order terms in the policy expansion, ignoring higher-order contributions, and (ii) assumes an unclipped formulation, whereas clipping — even when its bias is too small to drive learning — has a second-order effect on the policy's shape that systematically reduces entropy. This gap means prior theoretical tools are inadequate for understanding RLVR dynamics.

3. No framework for explaining when random rewards help.

Prior empirical work treats random-reward benefits as a binary phenomenon: either they help (Shao et al., 2025) or they don't (Oertell et al., 2025). Missing is any analysis of conditioning factors — model strength, dataset difficulty, clipping configuration, group size — that determine whether gains materialize. This absence matters because practitioners need to know: "Should I use random rewards as a regularization tool? Under what circumstances?" Without a model of when benefits occur, the empirical literature provides conflicting guidance with no resolution mechanism.

4. Contamination confounds the interpretation.

Wu et al. (2025) demonstrate substantial contamination in Qwen-Math models on MATH500, raising a legitimate challenge: observed improvements might reflect memorization reinforcement rather than genuine reasoning improvement. However, contamination alone cannot be the full explanation, because:

• If the model has memorized correct answers, why would random rewards help? Random rewards are just as likely to penalize correct answers as to reward them — they provide no signal about correctness.
• The paper reports (Section 5, Figure 4) that a stronger distilled Llama model — whose base and teacher models also exhibit contamination on MATH500 — shows comparable improvements to Qwen-Math. Meanwhile, a much weaker Qwen-Math-1.5B model (also potentially contaminated) fails to achieve similar gains. If contamination were the sole mechanism, model strength should not matter this much within the same contaminated family.

These observations motivate the paper's reward-misalignment model (Section 5) as an alternative explanation that doesn't depend on contamination — yet the prior literature had no such framework.

How This Paper Positions Itself

The paper frames its contribution not as proposing a new training algorithm or demonstrating a new empirical phenomenon, but rather as providing the theoretical and empirical clarity needed to reconcile conflicting findings and establish principles for RLVR design. Its positioning can be understood along three axes:

Axis 1: From "what" to "why." The paper acknowledges that the observations (random rewards help, entropy minimization helps) have been reported before. Its contribution is explaining why these observations occur, what mechanisms are responsible, and — crucially — what mechanisms are not responsible despite prior claims. This is primarily a paper of understanding, not of novel phenomena. The title itself — "Rethinking RLVR through Clipping, Entropy, and Spurious Reward" — signals this re-examination stance.

Axis 2: From model-specific to general. By conducting experiments across three model families (Qwen-Math, Llama, QwQ), multiple sizes (1.5B, 7B, 8B, 32B), and both base and distilled variants, the paper explicitly counters the narrative that random-reward benefits are a Qwen-Math peculiarity. The authors state:

"These results reconcile conflicting reports in the literature, demonstrating that performance improvements under spurious rewards are robust and not tied to any single model or dataset." (Contribution 2)

The experiments with Llama and QwQ families are particularly important because no contamination has been reported for these models on MATH500 in the community — yet they show qualitatively similar behavior to Qwen-Math. This strengthens the case that the underlying mechanism is general, not contamination-specific.

Axis 3: From isolated effects to interacting mechanisms. The paper's most distinctive intellectual move is treating clipping bias, policy entropy, and reward misalignment not as separate phenomena but as interacting components of a single system whose joint dynamics determine outcomes. The theoretical contributions are structured to trace causal chains:

• Clipping bias → (too small to drive learning) → but clipping → reduces entropy → changes policy structure
• Entropy reduction → (necessary but not sufficient for improvement) → performance gains require additional conditions (model strength, dataset difficulty)
• Random rewards → (through the lens of misalignment) → expected damage depends on baseline correct-rollout rate n_c → explains why stronger models benefit more

This systems-level framing distinguishes the paper from prior work that examined each component in isolation. It also provides the conceptual foundation for future work to design training procedures that deliberately modulate these interactions — for instance, using spurious-reward setups as entropy control mechanisms alongside explicit entropy regularization (Remark 4.4).

Departures from Classical RL Intuitions

The paper takes care (Section 1, paragraphs 2-3) to explain why RLVR for LLMs creates a fundamentally different exploration-exploitation regime than classical RL, and why classical intuitions break down. This context is essential for appreciating why the paper's findings are surprising:

1. Outcome-level, sparse rewards. In classical RL, per-step or shaped rewards provide continuous feedback about action quality. In RLVR, the reward is binary and available only at the end of a potentially long rollout. All intermediate token-level actions are reward-equivalent — there is no signal distinguishing a good intermediate step from a bad one until the final answer is checked against the ground truth.

2. Sequence-space exploration. Exploration in classical RL involves taking locally underexplored actions based on state-local counts or bonuses. In LLM RLVR, exploration unfolds in the exponentially vast space of token sequences, and is governed primarily by the decoding temperature — a global parameter, not a state-local bonus.

3. Ratio clipping with group-normalized advantages. Policy updates in GRPO use clipped importance ratios and group-relative advantage estimation (Equation 1). This makes updates sensitive to the relative ranking of responses within a group rather than their absolute reward values. A response can receive a positive advantage even if it's incorrect, as long as it's less incorrect (or more likely to be correct) than the group average.

Together, these properties mean that concepts like "exploitation" (reinforcing high-reward actions) and "exploration" (deliberately trying underexplored actions) don't map cleanly onto RLVR dynamics. A random reward — which in classical RL would unambiguously be harmful noise — can in RLVR reshape the policy's entropy landscape in ways that happen to concentrate mass on the right trajectories, if the model already places non-trivial probability on those trajectories. This is the insight the paper formalizes through the reward-misalignment model in Section 5.

Guiding Questions

The paper organizes its investigation around two fundamental questions (stated in the abstract and revisited in Section 6):

1. How does policy entropy relate to performance? Is the relationship causal (lower entropy → better reasoning), correlational (both driven by a third factor), or something more complex?

2. Can spurious rewards yield genuine gains? If so, through what mechanism — and under what conditions? Is clipping bias really the driver, or is something else going on?

These questions structure the paper's contributions: Section 3 addresses the clipping-bias mechanism (and finds it insufficient), Section 4 establishes the clipping-entropy link and examines entropy-performance relationships, and Section 5 develops the reward-misalignment model to explain when and why random rewards help. The concluding Section 6 revisits these questions with answers grounded in the theoretical and empirical results.

3. Technical Approach

3.1 Reader Orientation

This is primarily a theoretical and empirical analysis paper. It does not propose a new training algorithm or a novel RLVR variant. Instead, it builds a framework to understand how three interacting components — clipping bias, policy entropy, and reward misalignment under random rewards — jointly shape model performance during GRPO training for LLM reasoning. The core idea is that apparent paradoxes in the RLVR literature (spurious rewards help, entropy minimization helps) can be resolved by recognizing that clipping functions not as a learning signal but as an implicit entropy modulator, and that the benefits of random rewards are governed by a probabilistic misalignment model that depends on the model's baseline capability.

3.2 Big-Picture Architecture (Diagram in Words)

The paper's analytical system has four major components, layered in increasing abstraction from empirical measurement to theoretical modeling:

1. GRPO Training Pipeline — The standard RLVR setup: sample G completions per prompt from the current policy π_old, compute group-normalized advantages A_i from binary rewards (ground-truth or random), and update the policy using a clipped surrogate objective. This is the object of study, not a contribution. It is configured identically to Shao et al. (2025): batch size 128, group size G = 16, clipping ratio ε = 0.2, learning rate η = 5 × 10^(-7), KL coefficient 0, decoding temperature 1.0, on the DeepScaleR training dataset (Luo et al., 2025).

2. Clipping Bias Analysis — A theoretical decomposition of the clipped surrogate gradient into a raw (unclipped) term and a clipping-correction term (C_tot^+), with bounds derived to show that the correction is negligibly small (roughly 17× smaller) relative to the raw signal under practical hyperparameters. This component rules out clipping bias as a meaningful learning mechanism.

3. Policy Entropy Dynamics Analysis — A one-step entropy-shift formulation that isolates the effect of clipping on policy entropy. The analysis shows that, under random rewards, clipping reduces entropy by constraining the policy update, while unclipped training allows entropy to increase or decrease depending on initial policy skewness. This component establishes what clipping actually does: it regulates entropy, not learning.

4. Reward-Misalignment Model — A probabilistic model of correct-response advantage loss under random binary rewards, parameterized by the number of correct rollouts n_c in a group. This model predicts that stronger models (larger n_c) incur less misalignment damage and fewer false-positive misallocations, explaining when random rewards yield performance gains and why the effect is not contamination-specific.

Information flows through these components as follows: the GRPO training pipeline produces empirical observations (performance trajectories, entropy curves, clipping activation rates) → the clipping bias analysis takes observed activation rates and hyperparameters as inputs to bound the correction term → the entropy analysis uses the same hyperparameters and initial policy distribution to characterize entropy shifts → the reward-misalignment model uses the baseline correct-sampling rate n_c/G to predict expected advantage loss and its variance.

3.3 Roadmap for the Deep Dive

• First, the GRPO formulation and the softmax policy update, because all subsequent analysis — clipping bounds, entropy dynamics, advantage calculations — depends on understanding how the policy is parameterized and updated in token space.

• Second, the clipping bias analysis (Section 3 of the paper), decomposing the clipped surrogate into raw and correction terms, deriving bounds, and evaluating them numerically under practical hyperparameters to establish that clipping bias does not drive learning.

• Third, the one-step entropy shift formulation (Section 4 of the paper), deriving how policy entropy changes under both unclipped and clipped updates, and characterizing the deterministic link between clipping and entropy reduction.

• Fourth, the reward-misalignment model (Section 5 of the paper), formalizing correct-response advantage loss as a function of group composition, establishing expected damage and variance, and decomposing into false-positive and false-negative contributions to explain model-strength dependence.

3.4 Detailed, Sentence-Based Technical Breakdown

This is a theoretical and empirical analysis paper whose core idea is that clipping bias, policy entropy, and reward misalignment are interacting mechanisms in RLVR whose joint dynamics — not any single mechanism in isolation — determine performance outcomes under random rewards.

GRPO Formulation and the Softmax Policy Update

The paper builds its theoretical analysis on a specific formalization of GRPO, which it interprets as an instance of natural policy gradient (NPG) with a softmax tabular parameterization (Appendix C). This is not the exact GRPO implementation (which uses multiple micro-updates with recomputed importance ratios), but it is a justified reduction: the paper conditions on the empirically measured clipping activation rate and collapses multiple clipped micro-steps into a single NPG-style update that preserves the first-order effect of clipping, enabling tractable bounds.

The core objects are as follows. For a given prompt x drawn from distribution ρ, the current (old) policy π_old generates a group of G responses {y^(i)}_{i=1}^G, each a sequence of tokens. A verifier assigns a binary outcome reward r(x, y^(i)) = 1 if the final answer matches the ground truth and r(x, y^(i)) = 0 otherwise. The advantage for response i is computed via group normalization:

$A(x, y^{(i)}) = \frac{r(x, y^{(i)}) - \text{mean}(\{r(x, y^{(1)}), ..., r(x, y^{(G)})\})}{\text{std}(\{r(x, y^{(1)}), ..., r(x, y^{(G)})\})}$

where A_i is the group-normalized advantage, r(x, y^(i)) is the binary reward, and the mean and standard deviation are taken over the G responses in the group.

Why this form: Group normalization centers and scales advantages so that responses better than the group average receive positive advantage and responses worse than average receive negative advantage, regardless of the absolute reward scale. This makes the update sensitive to relative ranking within the group rather than to the absolute magnitude of rewards — a critical property for analyzing random rewards, where the absolute reward labels are uncorrelated with quality.

GRPO optimizes a clipped surrogate objective that, for each response i and each token position t, takes the minimum of the unclipped product r_t^(i)(θ) · A_i and the clipped product clip(r_t^(i)(θ), 1-ε, 1+ε) · A_i, where r_t^(i)(θ) = π_θ(y_t^(i) | h_t^(i)) / π_old(y_t^(i) | h_t^(i)) is the token-level importance ratio. Remark 2.1 notes a crucial simplification: under GRPO, the token-level advantage equals the response-level advantage A_i and is independent of token index t. This means all tokens in a response receive the same advantage signal.

The paper then derives the closed-form update rule for the unclipped surrogate with a forward KL penalty to π_old. Starting from the unclipped surrogate and applying mirror descent with KL divergence as the Bregman divergence, the solution for each history h is:

$\pi_{new}(a \mid h) = \frac{\pi_{old}(a \mid h) \exp(\eta \tilde{A}(h, a))}{\sum_{a' \in \mathcal{V}} \pi_{old}(a' \mid h) \exp(\eta \tilde{A}(h, a'))} \propto \pi_{old}(a \mid h) \exp(\eta \tilde{A}(h, a))$

where η > 0 is the step size (learning rate), V is the vocabulary, and Ã(h, a) is the empirical per-history advantage defined as:

$\tilde{A}(h, a) = \frac{1}{G} \sum_{i=1}^G \sum_{t=1}^{|y^{(i)}|} \left( \frac{\mathbf{1}\{h_t^{(i)} = h, y_t^{(i)} = a\}}{\pi_{old}(a \mid h)} \right) A_i$

Here, Ã(h, a) aggregates advantages across all response groups, weighted inversely by the old policy's probability of token a at history h. The indicator 1{h_t^(i) = h, y_t^(i) = a} selects token occurrences that match the specific (h, a) pair. Dividing by π_old(a | h) corrects for sampling bias — tokens that are rare under π_old receive larger weight when they do appear.

What it computes: For each possible next token a at history h, the update exponent of exp(η · Ã(h, a)) increases the probability of tokens that appeared in responses with high advantage, and decreases the probability of tokens that appeared in responses with low (negative) advantage. The denominator normalizes to maintain a valid probability distribution.

Why this form: The exponential update is the natural gradient under the softmax parameterization with KL divergence regularization. It has the property that the importance ratio r(h, a) = π_new(a | h) / π_old(a | h) depends on the advantage through r(h, a) ∝ exp(η Ã(h, a)), which connects directly to the clipping analysis: too-large advantages produce too-large ratios, which clipping constrains.

The paper assumes throughout that there exists π_min > 0 such that π_old(a | h) ≥ π_min for all (h, a), ensuring the importance ratios are well-defined and bounded. In practice, π_min = 10^(-7) is used for numerical evaluations.

Lemma 2.2 provides a critical Taylor expansion of this update:

$\log(r(h, a)) - \eta(\tilde{A}(h, a) - \mu(h)) + \frac{\eta^2}{2}\sigma^2(h) \leq C\eta^3$

where μ(h) = E_{a~π_old}[Ã(h, a)] is the mean advantage under the old policy at history h, σ²(h) = Var_{a~π_old}[Ã(h, a)] is the variance of advantages, and C = 1/(36√3 · (π_min)³) is a constant independent of η.

What it computes: The log-importance ratio log(r(h, a)) is approximately linear in the centered advantage Ã(h, a) - μ(h) with a quadratic correction for variance, plus a cubic remainder. Under the standardized setting where μ(h) = 0 and σ²(h) = 1 (which can be arranged by normalizing advantages), the approximation simplifies to log(r(h, a)) ≈ η Ã(h, a) - η²/2.

Why this form: This expansion is the foundation for analyzing both clipping bias and entropy dynamics. For clipping, it provides upper bounds on the maximum possible importance ratio (r ≤ exp(η/(2π_min))), which enters the clipping-correction bound. For entropy, the quadratic term η²/2 captures the variance-reduction effect that drives entropy change even when the first-order advantage is zero-mean (as under random rewards).

Clipping Bias Analysis: Decomposition and Bounds

The paper defines the upper-clipping correction formally in Definition 3.1. For a given rollout y of length L sampled from prompt x, with clip ratio ε ∈ (0, 1):

• r_t is the token-level importance ratio (abbreviated from r(h_t, y_t))
• r̄_t = clip(r_t, 1-ε, 1+ε) = max{min{r_t, 1+ε}, 1-ε} is the clipped ratio
• N_t = r_t · A(x, y) is the raw (unclipped) surrogate at token t
• N_t^clip = r̄_t · A(x, y) is the clipped surrogate at token t
• I_t^+ = 1{r_t > 1+ε} is the indicator that upper clipping is active at token t

The total upper clipping correction C_tot^+ aggregates the difference between clipped and raw surrogates over all tokens where upper clipping is active:

$C_{tot}^+ = \sum_{t=1}^L (N_t^{clip} - N_t) I_t^+ = \sum_{t=1}^L (\bar{r}_t - r_t) I_t^+ A(x, y)$

What it computes: For each token where the importance ratio exceeds 1+ε, clipping replaces r_t with 1+ε, reducing the product by (r_t - (1+ε)) · A(x, y) when the advantage A is positive. This is the "lost" update magnitude due to clipping. When A is negative and r_t > 1+ε, the clipped surrogate uses r_t · A directly (the min operator selects the unclipped term since it's more negative), so the correction is zero — this is why it's called upper clipping correction.

Why this form: Decomposing the surrogate this way separates the signal into a learnable component (raw contribution) and a regularization component (clipping correction). If the clipping correction were comparable in magnitude to the raw term, it could meaningfully bias the gradient and potentially drive the amplification effects hypothesized by Shao et al. (2025). The paper's goal is to show it is not.

Theorem 3.2 provides an upper bound on the expected magnitude of C_tot^+:

$\mathbb{E}[|C_{tot}^+|] \leq M\sqrt{2 p^+ L R_\eta^{max} \phi(R_\eta^{max})} + M L \Delta_\eta^+ \min\left\{p^+, \frac{\phi(R_\eta^{max})}{\phi(1+\varepsilon)}\right\}$

where M = √(G - 1) bounds the maximum absolute advantage (from Lemma 2.4), R_η^max = exp(η / π_min) is the maximum possible importance ratio (from Lemma 2.2's bound on |Ã(h, a)| ≤ 1/(2π_min)), φ(u) = u log u - u + 1 is the KL divergence between a point mass and a distribution giving probability 1/u to each of u equally likely alternatives, Δ_η^+ = (R_η^max - 1 - ε)_+ is the maximum possible excess above the clipping threshold, and p^+ = E[I_t^+] is the expected token-level clipping activation rate.

What it computes: An upper bound on the absolute magnitude of the clipping correction, expressed in terms of observable quantities: the activation rate p^+, the sequence length L, the clip ratio ε, the step size η, and the group size G. The first term captures the variance contribution (diagonal terms in the sum-of-squares expansion), and the second captures the cross-token correlation (off-diagonal terms).

Why this form: The bound is expressed in terms of p^+ rather than derived from first principles because the activation rate is model- and dataset-dependent and can only be measured empirically. This makes the bound practical: plug in your observed activation rate to assess whether clipping bias is meaningful for your training run. Remark 3.3 emphasizes this model-agnostic property: the bound applies to any architecture, with p^+ serving as the sufficient statistic.

For sufficiently small η, the bound simplifies to:

$\mathbb{E}[|C_{tot}^+|] \leq c_1 \eta L + \min\{c_2 \eta p^+ L, c_3 \eta^3 L\}$

where c_1 = M√(2e) / π_min, c_2 = M(e - 1) / π_min, and c_3 = M(e - 1) / (φ(1+ε) · π_min³).

Theorem 3.4 then compares the clipping correction to the raw surrogate. Define the raw surrogate sum N_raw = Σ_{t=1}^L r_t · A. The lower bound on its expected magnitude is:

$\mathbb{E}[|N_{raw}|] \geq L \cdot \mathbb{E}[|A|] \cdot e^{-C\eta^2} \geq L \cdot \mathbb{E}[|A|] \cdot (1 - C\eta^2)$

where C = 1/(8π_min²).

What it computes: The expected magnitude of the raw (unclipped) surrogate sum scales linearly with sequence length L and expected absolute advantage E[|A|], with a small downward correction from the exp(-Cη²) factor that accounts for the curvature of the exponential update.

Why this form: The proof uses a clever symmetry: under the NPG update, r_t(η) = exp(η Ã) / Z(η) and r_t(-η) = exp(-η Ã) / Z(-η). Averaging the two and applying Jensen's inequality yields a lower bound that depends only on cosh(η/(2π_min)), which is close to 1 for small η.

The ratio of the two yields the key comparison:

$\frac{\mathbb{E}[|N_{raw}|]}{\mathbb{E}[|C_{tot}^+|]} \geq \frac{\mathbb{E}[|A|] (1 - C\eta^2)}{L^{-1/2} \cdot \text{(bound from Theorem 3.2)}}$

Corollary 3.6 instantiates this for Qwen2.5-Math-7B under the training hyperparameters used in the experiments:

• η = 5 × 10^(-7) (learning rate)
• ε = 0.2 (clip ratio)
• p^+ = 0.001 (empirically observed clipping activation rate, per Remark 3.5: "the clipping activation ratio never exceeds 0.2%, with expected activation probability E[I_t] ≈ 0.001")
• G = 16 (group size)
• L = 4096 (sequence length)
• π_min = 10^(-7) (minimum token probability)
• M = 3.75 (from Lemma 2.4: M = √(G - 1))
• E[|A|] ≈ 0.967 (also from Lemma 2.4)
• R_η^max ≈ 1.649 (max importance ratio)
• φ(R_η^max) ≈ 0.176 (KL at max ratio)
• Δ_η^+ ≈ 0.449 (max excess above threshold)
• C = 1.25 × 10^11 (the constant from the e^{-Cη²} term)

Plugging in yields:

$\frac{\mathbb{E}[|N_{raw}|]}{\mathbb{E}[|C_{tot}^+|]} \approx 17.15$

What this means: The raw (unclipped) gradient signal is more than 17 times larger in expected magnitude than the clipping correction. Clipping bias, even if it systematically favored high-probability tokens, is simply too small relative to the dominant gradient signal to meaningfully change the update direction.

Why this matters: This directly refutes the "upper-clipping bias" hypothesis from Shao et al. (2025). The asymmetry described in Remark 2.5 — where high-probability tokens π_old(y_t | h_t) ≥ π_old(y_{t'} | h_{t'}) receive larger absolute increases under clipping — is mathematically present but numerically irrelevant because the clipping correction is dwarfed by the raw gradient. The experimental results in Figure 1 confirm this: enabling clipping can lead to a decline in validation performance, and disabling clipping often results in improvement. If clipping bias were the driver of random-reward gains, we would expect the opposite pattern — clipping should help, not hurt.

One-Step Policy Entropy Change: Unclipped Training

The paper models entropy dynamics by analyzing the one-step change ℋ(π_new) - ℋ(π_old) under the softmax update from Equation 2, working in the bandit (L = 1) setting for clarity (the extension to multi-token sequences follows by linearity if we assume independent updates per history).

Theorem 4.1 (presented in full detail in Appendix C.3, Lemma C.1 and its proof) establishes that under unclipped training with random rewards:

$\mathbb{E}[\mathcal{H}(\pi_{new}) - \mathcal{H}(\pi_{old})] = -c_G \Phi(\pi_{old}) \eta^2 + \mathbb{E}[R(\eta)]$

where c_G = (1 - 2^(1-G)) / (2G) is a constant depending only on group size, Φ(π_old) measures the skewness of the old policy distribution, and R(η) is a remainder term bounded by |E[R(η)]| ≤ C · η^4 where C depends on π_min, the vocabulary size |V|, and the fourth moment of advantages.

The skewness measure Φ(π) is defined as:

$\Phi(\pi) = |\mathcal{V}| - 1 + \sum_{a \in \mathcal{V}} \log(\pi(a)) - |\mathcal{V}| \left( \sum_{a \in \mathcal{V}} \pi(a) \log(\pi(a)) \right)$

where |V| is the vocabulary size, the sum of log-probabilities captures how evenly mass is distributed, and the weighted entropy term measures the policy's average uncertainty.

What it computes: Φ(π) is positive when the policy is relatively flat (probability mass is spread across many tokens), zero at some threshold of skewness, and negative when the policy is highly peaked on a small subset of tokens. Up to the O(η²) term, the expected entropy change is -c_G · Φ(π_old) · η²: negative (entropy decreases) when Φ > 0 (flat policy), positive (entropy increases) when Φ < 0 (skewed policy).

Why this form: The proof in Lemma C.1 derives the entropy change by Taylor-expanding ℋ(π_η) around η = 0. The first-order term vanishes because ℋ'(0) = 0 (the policy update is symmetric under η → -η when Ã is symmetrically distributed, which holds under random rewards by Lemma 2.4). The second-order term decomposes as:

$\mathcal{H}''(0) = -\mathbb{E}[\text{Var}_{\pi_{old}}(\tilde{A}) + \text{Cov}_{\pi_{old}}(\tilde{A}^2, \log \pi_{old})]$

The variance term Var_{π_old}(Ã) is always positive and contributes to entropy decrease — it captures the fact that tokens with higher (lower) advantages get increased (decreased) probability, concentrating the distribution. The covariance term Cov_{π_old}(ò, log π_old) can be positive or negative depending on whether high-variance tokens tend to have high or low log-probability. Under random rewards, Lemma C.3 computes these expectations explicitly to yield the Φ(π_old) expression.

Remark 4.2 provides a concrete two-armed example: for a policy π_old = (β, 1-β), we can compute Φ(π_old) = 1 + (1-2β) log(β/(1-β)). This quantity is non-negative when β ∈ [0.176, 0.824] (a less skewed policy) and negative when β > 0.824 or β < 0.176 (a more skewed policy). Thus, up to O(η²), entropy decreases for flatter policies and increases for more skewed policies under unclipped training with random rewards.

What this explains: Figure 2 (Left) shows that under unclipped training, Qwen2.5-Math-7B's policy entropy increases over training. This is consistent with Theorem 4.1 if the initial policy is sufficiently skewed (Φ < 0). Appendix B, Figure 8 confirms this: among 500 sampled questions from the DeepScaleR training set, 358 satisfy Φ(π(· | x_i)) < 0, so the model's policy is skewed on most prompts — random-reward training without clipping pushes it toward flatter, higher-entropy distributions. This is the opposite of entropy collapse.

One-Step Policy Entropy Change: Clipped Training

Theorem 4.3 extends the entropy analysis to the case with upper clipping. The clipped update solves a constrained optimization problem where the KL penalty is augmented with the upper clipping constraint. Lemma C.4 derives the KKT conditions for this constrained problem, showing that for tokens where the unclipped ratio would exceed 1+ε, the log-ratio is capped at a value determined by the negative-advantage tokens.

The key bound on the entropy change under clipping is:

$\mathbb{E}[\mathcal{H}(\pi_{new}^c) - \mathcal{H}(\pi_{old})] \leq -c_G \Phi(\pi_{old}) \eta^2 + \mathbb{E}[R(\eta)] + c(p) G \left(\rho \delta_{eff} - \frac{X_{max}}{2}(G-1)p\right)$

where the additional terms relative to the unclipped case capture the entropy-reducing effect of clipping. Here, C_i = {A_i > 0, r_u(y^(i)) > 1+ε} is the event that a response has positive advantage and its unclipped ratio would exceed the threshold, ρ = P(C_1) is its probability, δ = E[r_u(y^(1)) - (1+ε) | C_1] is the expected excess above the threshold conditional on C_1, X_max = exp(η/(2π_min)) - (1+ε) is the maximum possible excess, p ∈ (π_min, 1) is a free parameter controlling the threshold for "low-probability" tokens, and c(p) = -π_min (log(p · exp(η/(2π_min))))_- is a negative coefficient that ensures the clipping correction reduces entropy.

What it computes: The clipping correction term c(p) G (ρ δ_eff - (X_max/2)(G-1)p) is negative under practical parameter settings (verified in Remark C.7), meaning clipping adds a downward pressure on entropy beyond what the unclipped update would produce. The term δ_eff refines δ by subtracting out contributions from tokens with very low probability (which have a different bound M(p)), ensuring the correction only counts tokens where clipping actually constrains meaningful probability mass.

Why this form: The proof constructs a coupling between the clipped and unclipped update: ℋ(π_new^c) - ℋ(π_new^u) ≤ Σ_a (π_new^u(a) - π_new^c(a)) log(π_new^u(a)). On the event C_i, the unclipped policy gives more mass to token y^(i) than the clipped policy allows; the difference r_u(y^(i)) - (1+ε) is the "lost" log-ratio. Summing over these tokens and taking expectations yields the bound. The free parameter p controls the granularity of the analysis: by conditioning on whether π_old(y^(i)) ≤ p (low-probability tokens), the bound can separate out tokens where the ratio bound X_max is overly conservative.

Remark C.7 verifies that under the experimental hyperparameters (G = 16, η = 5×10^(-7), ε = 0.2, π_min = 10^(-7), ρ ≈ 0.001, δ ≈ 10), the clipping correction term evaluates to approximately -2.01 × 10^(-7), and the total expected entropy change is bounded by -1.49 × 10^(-7) < 0 — confirming that clipping produces a monotonic entropy decrease in expectation.

What this explains: Figure 2 (Middle) shows that under clipped training, Qwen2.5-Math-7B's policy entropy decreases monotonically. Theorem 4.3 explains this as the combination of (i) the -c_G Φ(π_old) η² term, which can be positive or negative depending on skewness, and (ii) the clipping correction, which is always negative and dominates in practice. Clipping acts as an entropy cap: by preventing the importance ratio from exceeding 1+ε, it limits how much probability mass can shift to tokens with positive advantage, thereby reducing the entropy increase that would otherwise occur for skewed policies under unclipped training.

Random Rewards and GRPO Advantages: Foundational Properties

Lemma 2.4 establishes the statistical properties of GRPO advantages under the random reward specified in Definition 2.3 (binary rewards r_i ~ Bernoulli(1/2) i.i.d., independent of response correctness). Fixing a group size G ≥ 2, define r̄ = (1/G) Σ_i r_i and S_r = (1/G) Σ_j (r_j - r̄)², so that the advantage is A_i = (r_i - r̄) / S_r.

The lemma proves three properties:

(i) Symmetry: A_i is symmetrically distributed around 0. This follows because flipping all rewards (replacing each r_j with 1 - r_j) negates all advantages while preserving the distribution of {r_j} (since Bernoulli(1/2) is invariant under complement). Consequently, E[A_i^(2k-1)] = 0 for all k ∈ N⁺ — all odd moments vanish.

(ii) Boundedness: |A_i| ≤ √((G-1)/G). This is a deterministic bound: the maximum absolute advantage occurs when exactly one response gets reward 1 and all others get 0 (or vice versa), making the group mean nearly 0 or 1, the standard deviation small, and the normalized deviation of the outlier large.

(iii) Lower bounds on absolute moments: For all integers k ≥ 2, E[|A_i|^k] ≥ 1 - 2^(1-G). This lower bound converges to 1 exponentially fast in G: for G = 16, it is approximately 0.99997, meaning advantages are almost certainly non-trivial in magnitude. The expected absolute advantage for G = 16 is approximately 0.967, as used in Corollary 3.6.

Why this matters: These properties are the foundation for all subsequent analysis. Symmetry (odd moments vanish) is used in Theorem 3.4 to relate r_t(η) and r_t(-η) and derive the lower bound on E[|N_raw|]. Boundedness (|A_i| ≤ M) provides the constant that appears in the clipping-correction bound. The lower bound on absolute moments (E[|A|] ≈ 1) ensures that the raw surrogate is not degenerate even under random rewards — the advantages are numerically substantial even though they carry no information about correctness.

Critically, Lemma 2.4 also explains why the first-order entropy approximation in Equation 5 fails under random rewards: because A_i is independent of π_old(y | x) and has zero mean (by symmetry), the covariance Cov(log π_old(y | x), A(x, y)) is identically zero. The approximation predicts ℋ(π_new) - ℋ(π_old) = 0 — no entropy change — which contradicts the empirical entropy trajectories in Figures 1 and 2. The actual entropy change comes from higher-order terms (the η² term in Theorem 4.1) and from the clipping correction (Theorem 4.3), both of which are invisible to the first-order analysis.

Reward-Misalignment Model: Correct-Response Advantage Loss

The reward-misalignment model in Section 5 provides a probabilistic framework for understanding when and why random rewards can yield performance improvements. The setup considers a single prompt x for which the current policy samples G rollouts. Among these, n_c are correct (their final answers match the ground truth) and n_i = G - n_c are incorrect. The verifier is replaced with i.i.d. Bernoulli(1/2) random rewards, independent of correctness.

Definition 5.1 formalizes the correct-response advantage loss Δ(f, g), parameterized by the number of false positives f (incorrect responses that happen to get random reward +1) and false negatives g (correct responses that happen to get random reward 0):

$Δ(f, g) = Σ_{\mathcal{C}}^{ideal} - Σ_{\mathcal{C}}(f, g)$

where Σ_C^ideal = n_c (1 - n_c/G) is the total advantage that would be assigned to correct responses under perfect (ground-truth) rewards, and Σ_C(f, g) = (n_c - g) - n_c · T/G is the actual advantage assigned under random rewards, with T = f + (n_c - g) being the total number of +1 random rewards in the group.

What it computes: Δ(f, g) measures how much of the ideal advantage mass for correct responses is lost (or, in rare cases, gained) due to random reward mislabeling. Under perfect rewards, correct responses receive positive advantage proportional to the fraction of incorrect responses in the group (1 - n_c/G); incorrect responses receive negative advantage. Under random rewards, correct responses that happen to get reward 0 (g of them) lose their advantage, and incorrect responses that happen to get reward 1 (f of them) receive spurious positive advantage. The net effect is captured by Δ.

Why this form: The advantage loss is the key quantity because GRPO updates are driven by advantages, not raw rewards. Even under random rewards, the policy update pushes toward responses with positive advantages and away from those with negative advantages. Δ quantifies how much of the "correct" gradient signal is corrupted: when Δ is large, correct responses receive less positive advantage than they should, reducing or reversing the gradient that would push the policy toward correct reasoning.

Proposition 5.2 derives the mean and variance of Δ under i.i.d. Bernoulli(1/2) rewards, with f ~ Binomial(n_i, 1/2) and g ~ Binomial(n_c, 1/2) independent:

$\mathbb{E}[Δ] = \frac{n_c (G - n_c)}{G}, \quad \text{Var}(Δ) = \frac{n_c (G - n_c)}{4G}$

What it computes: The expected damage (advantage loss) depends on the product n_c · n_i = n_c(G - n_c), which is maximized at n_c = G/2 (equal numbers of correct and incorrect) and decreases as n_c approaches either 0 or G. The variance also scales with this product, meaning it is largest when n_c ≈ n_i and shrinks as the group becomes homogeneous.

Why this explains the observed patterns: The expected damage decreases as the number of correct rollouts n_c increases (for n_c > G/2). Stronger models — those that already produce correct answers more frequently — experience less misalignment damage from random rewards because more of their rollouts are in the "correct" set, and the damage formula n_c(G - n_c)/G penalizes groups that are evenly split. When n_c is close to G, the damage is small because most responses are correct and random rewards only mislabel a minority. When n_c is small (weak model), the damage is again small, but for a different reason: there are few correct responses to lose advantage for. The worst case is n_c ≈ G/2, where mislabeling is most disruptive.

This directly explains why validation curves fluctuate less when baseline accuracy is high (Figure 1, Left vs. Middle): the variance of Δ shrinks with n_c, so each training step receives a more consistent signal. It also explains why stronger models are more likely to improve: they operate in a regime where n_c > G/2, so the expected damage is modest, and the false-positive contribution (incorrect responses competing for advantage) is relatively small compared to the false-negative contribution (correct responses losing some advantage). The policy can still find the correct trajectories because they receive positive advantage on average, just less than they would under perfect rewards.

Theorem 5.3 refines this by decomposing the expected damage into false-positive-dominated (f > g) and false-negative-dominated (g > f) regimes:

$\mathbb{E}[Δ · 1{f > g}] ≤ \mathbb{E}[Δ · 1{g > f}]$

for n_c > n_i. Moreover, as n_c increases on [G/2, G], the FP-dominated portion (E[Δ · 1{f > g}] / E[Δ]) constitutes a strictly smaller fraction of the total damage.

What this means: For stronger models (n_c > G/2), the damage from false negatives (correct responses incorrectly penalized) dominates the damage from false positives (incorrect responses incorrectly rewarded). And as n_c increases further, false positives become an even smaller share of the total misalignment. This is the key mechanism: stronger models are insulated from the most harmful form of misalignment (incorrect responses pulling the policy in wrong directions) because random rewards are more likely to penalize correct responses than to reward incorrect ones when correct responses are already the majority.

Why this matters for the contamination debate: If random-reward gains were purely contamination-driven (reinforcing memorized trajectories), model strength would not matter — even a weakly contaminated model should benefit if it has memorized some answers. The fact that stronger models benefit more, and that the benefit extends to non-contaminated model families (Llama, QwQ), supports the reward-misalignment mechanism as the explanation, not contamination. A strong model with non-trivial n_c on the training data receives a systematic, albeit noisy, pressure toward correct trajectories because of the asymmetry in damage composition.

The conditional variance analysis in Appendix C.4 provides further nuance: Var(Δ | f > g) < Var(Δ | g > f) when n_c > G/2. FP-dominated groups (where incorrect responses get more positive labels) have lower variance in their damage than FN-dominated groups. This means that when the model is strong, the most variable (and thus potentially most disruptive) training steps are those where correct responses are incorrectly penalized, not those where incorrect responses are incorrectly rewarded.

Numerical example to ground the intuition: Consider G = 16 with n_c = 12 (a strong model that gets 75% of rollouts correct). The expected advantage loss is E[Δ] = 12 × 4 / 16 = 3.0. Under perfect rewards, the total advantage assigned to correct responses would be Σ_C^ideal = 12 × (1 - 12/16) = 12 × 0.25 = 3.0. Under random rewards, the expected actual advantage is Σ_C(f, g) = Σ_C^ideal - E[Δ] = 3.0 - 3.0 = 0.0 — correct responses receive zero expected advantage on average. However, the variance is Var(Δ) = 3.0/4 = 0.75, so individual steps vary substantially. On steps where correct responses happen to get more +1 labels than average, they receive positive advantage and get reinforced; on steps where they get fewer, they receive negative advantage and get suppressed. The net effect over many steps depends on the balance of these fluctuations and on the model's initial concentration on correct vs. incorrect trajectories.

In contrast, for a weaker model with n_c = 4 (25% correct), E[Δ] = 4 × 12 / 16 = 3.0 (same expected damage magnitude, but now with a different composition). The FP-dominated damage (f > g) is more prevalent because incorrect responses outnumber correct ones. The model is pushed in conflicting directions, and with fewer correct trajectories to anchor the policy, training becomes less stable — hence the increased oscillation observed in Figure 4 (Left) for Qwen2.5-Math-1.5B and in Figure 3 (Middle-R) for Qwen2.5-Math-7B on the harder AIME dataset.

Summary of Design Choices and Their Justifications

The paper's analytical framework involves several non-obvious choices that shape its validity and scope:

Reduction to NPG with measured clipping activation rate. Rather than attempting to model the full µ-step GRPO inner loop (where importance ratios are recomputed at each micro-step and clipping applies at each gradient step), the paper collapses multiple clipped micro-steps into a single NPG update conditioned on the empirically measured token-level clipping activation rate p^+. This is justified in Appendix C (paragraph preceding Section C.1): conditioning on the observed activation rate preserves the first-order effect of clipping while making the update analytically tractable. The alternative — attempting to derive bounds that hold for all possible micro-step trajectories — would be either intractable or too loose to be informative. The cost is that the bounds are conditional on empirical measurements rather than being fully a priori.

Bandit (L = 1) simplification for entropy analysis. The entropy analysis in Section 4 and Appendix C.3 assumes L = 1 (single-token responses) for clarity. This is a significant simplification since actual LLM rollouts span thousands of tokens. The paper argues (implicitly) that if updates are applied independently per history h (which is approximately true given the softmax parameterization), the total entropy change decomposes as a sum over histories, so the per-history analysis carries over. However, the interaction between token-level entropy changes across positions in a sequence is not modeled, and the assumption that advantages are independent of token position (Remark 2.1: the token-level advantage equals the response-level advantage) may hide position-dependent effects.

Free parameter p in Theorem 4.3. The bound on entropy change under clipping involves a free parameter p ∈ (π_min, 1) that controls the threshold for separating "low-probability" tokens from the rest. The bound holds for any p, and the tightest bound can be obtained by optimizing over p. In Remark C.7, the paper chooses p = 2π_min for the numerical evaluation, which is a reasonable but not obviously optimal choice. The sensitivity of the bound to this choice is not explored.

Reward-misalignment model as a per-group analysis. The reward-misalignment model analyzes a single group of G rollouts in isolation, assuming the group composition (n_c, n_i) is fixed. In practice, n_c varies across prompts and across training steps, and the policy update averages over many groups. The paper's conclusions about model strength are based on the average n_c across the training distribution, but the variance across prompts could matter: a model that is very strong on some prompts and very weak on others might behave differently from one with uniform moderate strength, even if they have the same average n_c. This heterogeneity is not analyzed.

No model of how entropy changes translate to performance changes. The paper establishes that clipping reduces entropy and that entropy reduction is neither necessary nor sufficient for performance improvement (Section 4.3, Figure 3). But it does not provide a mechanistic model of when an entropy decrease leads to better or worse accuracy. The reward-misalignment model (Section 5) provides one piece of the puzzle — the expected advantage loss depends on n_c — but does not directly connect to entropy. The gap between "entropy changes" and "performance changes" is bridged empirically but not theoretically.

4. Key Insights and Innovations

Innovation 1: Clipping Bias as a Diagnostic Null Result — Refuting the Dominant Explanation for Spurious-Reward Gains

The most prominent explanation for why random rewards improve RLVR performance comes from Shao et al. (2025), who attribute the gains to upper-clipping bias: the asymmetry in PPO-style ratio clipping that permits larger absolute probability increases for already-high-probability tokens (Remark 2.5). Under this hypothesis, random-reward training amplifies the model's existing high-confidence responses, surfacing latent or memorized knowledge without teaching new reasoning. This explanation has shaped the community's interpretation of spurious-reward phenomena and, if correct, would imply that random-reward gains are fundamentally a form of memorization amplification — useful only when the model already possesses the answers.

The paper demonstrates that this explanation is mathematically untenable under realistic training hyperparameters. The innovation is not merely the negative result (clipping bias doesn't drive learning), but the diagnostic framework developed to reach that conclusion. The paper decomposes the clipped GRPO surrogate into a raw (unclipped) term N_raw and a clipping-correction term C_tot^+, derives explicit upper bounds on the correction using quantities measurable during training (the per-token clipping activation rate p^+, the maximum importance ratio bound R_η^max, the clip ratio ε), and compares their magnitudes numerically. Under the actual hyperparameters used in Qwen2.5-Math-7B training — learning rate η = 5 × 10^(-7), clip ratio ε = 0.2, observed activation rate p^+ ≈ 0.001, group size G = 16 — the raw gradient signal is more than 17× larger in expected magnitude than the clipping correction (Corollary 3.6).

This is a fundamental negative result, not an incremental refinement. What makes it intellectually distinctive is the way it shifts the burden of proof. Prior to this work, the clipping-bias hypothesis was plausible enough to serve as the field's default explanation. The paper's bound shows that the correction term is structurally small, not just accidentally small for one model — the dependence on η, p^+, and L means that for any reasonable hyperparameter configuration, the clipping correction will be dwarfed by the raw gradient. A researcher who still wants to defend the clipping-bias explanation must now explain why a signal contributing less than 6% of the gradient magnitude could account for the observed performance changes, or must demonstrate that the bound's assumptions (NPG reduction, p^+ measurement, π_min lower bound) fail in ways that qualitatively change the conclusion.

The experimental evidence in Figure 1 reinforces the theoretical case: if clipping bias were the driver of random-reward gains, disabling clipping should harm performance, yet the paper finds that unclipped training under random rewards often yields equal or better validation accuracy than clipped training. The direction of the effect is opposite to what the clipping-bias hypothesis predicts. This converging evidence — theoretical bounds showing the bias is negligible, plus empirical results showing clipping isn't necessary for improvement — makes the case unusually strong for a negative claim.

The broader significance is methodological. By expressing the clipping-correction bound in terms of the observable activation rate p^+ (Remark 3.3), the paper provides a model-agnostic diagnostic that any practitioner can apply to their own training runs: measure your clipping activation rate, plug it into Theorem 3.2, and determine whether clipping bias is meaningful for your setup. This transforms what could have been a model-specific empirical argument ("clipping bias doesn't matter for Qwen-Math-7B with these hyperparameters") into a general framework for assessing when clipping bias could matter ("it doesn't matter as long as p^+ is below ~1% and η is in the typical 1e-7 to 1e-6 range").

Innovation 2: Clipping as an Implicit Entropy Modulator — Reframing What Clipping Actually Does

If clipping bias doesn't provide a meaningful learning signal, what does clipping do? The paper's second major conceptual contribution is establishing that clipping functions primarily as an implicit entropy minimization mechanism — not as a gradient-shaping tool, but as a regularizer that constrains the policy's distributional shape independent of reward informativeness.

This reframing is significant because the field has understood clipping through the lens of PPO: it prevents excessively large policy updates by capping importance ratios, thereby maintaining a trust region around the old policy. Under that standard interpretation, clipping is a stabilizer — it prevents gradient explosion, reduces variance, and keeps training from diverging. The paper shows that this is incomplete. Clipping also has a directional effect on the policy's entropy that is independent of whether rewards are informative, and this effect is the primary mechanism through which clipping influences outcomes under random rewards.

The theoretical engine for this reframing is the one-step entropy-shift analysis (Theorem 4.1 for unclipped training, Theorem 4.3 for clipped training). Under unclipped training with random rewards, entropy can increase or decrease depending on the initial policy's skewness Φ(π_old): highly skewed policies (where probability mass is concentrated on a small set of tokens) become flatter and higher-entropy, while already-flat policies become more peaked and lower-entropy. Clipping overrides this dynamic: Theorem 4.3 establishes that the clipping correction term is always entropy-reducing under practical hyperparameters, driving a monotonic entropy decrease regardless of initial skewness.

Why this matters beyond the technical result: The paper is arguing that clipping and entropy are causally linked in RLVR under random rewards in a way that prior work missed. Cui et al. (2025) analyzed entropy dynamics but assumed unclipped training and used a first-order approximation that fails under random rewards (Remark 2.7: it predicts zero entropy change, which is empirically false). Agarwal et al. (2025) and Gao et al. (2025) showed that explicit entropy minimization improves performance, but treated entropy reduction as a deliberate objective, not as an implicit consequence of a stability mechanism. The paper bridges these literatures by showing that clipping achieves entropy minimization as a side effect — and that this side effect, not the clipping bias, is what matters for outcomes under spurious rewards.

The experimental signature of this mechanism is Figure 2 (Left vs. Middle): under unclipped training, entropy increases; under clipped training, entropy decreases. Both can coincide with performance improvements, which is another way of seeing that entropy change and performance are not monotonically linked (the paper's third major insight). Figure 2 (Right) provides the failure mode that explains why clipping is pragmatically necessary: for R1-Distill-Llama-8B, removing clipping causes gradient explosion around step 150, with accuracy collapsing from 76.6% to near-initial levels. Clipping prevents this catastrophic failure — but the paper's point is that this stabilization function operates through entropy control, not through gradient-signal shaping.

This reframing has direct implications for algorithm design. Remark 4.4 observes that existing approaches to entropy control in RLVR (Wang et al., 2025; Yao et al., 2025; Zheng et al., 2025; Cheng et al., 2025) use regularization techniques that only slow entropy decay. The paper's finding that unclipped training can increase entropy — while also improving performance — suggests a complementary strategy: deliberately use spurious-reward setups (with controlled clipping) to modulate entropy in both directions, rather than only fighting entropy collapse. This is a genuinely new design space that the paper opens up but doesn't fully explore.

Innovation 3: Entropy and Performance Are Not Causally Linked — Overturning a Proposed Monotonic Relationship

A significant thread in recent RLVR research has proposed that lower policy entropy directly and monotonically improves reasoning performance. Cui et al. (2025) fit an exponential decay relationship R = -a exp(ℋ) + b across extensive experiments, suggesting that performance increases as entropy decreases, plateauing only once entropy collapses too early. Agarwal et al. (2025) and Zhang et al. (2025) similarly report that entropy minimization alone, without verifiable feedback, produces reasoning gains. This represents a strong claim about RLVR learning dynamics: compress the policy, and reasoning improves.

The paper's third major contribution is demonstrating that this monotonic relationship does not hold in general and that entropy should be understood as a regularizer rather than a causal lever for performance. The evidence is carefully constructed to separate correlation from causation:

1. Entropy can increase while performance improves. Figure 7 (Right, Appendix B) shows a representative training run on the hard AIME dataset under unclipped training where policy entropy rises (the model becomes more exploratory) while MATH500 validation accuracy simultaneously increases. This directly contradicts the claim that lower entropy strictly implies better performance — here, the model gets better as it gets less deterministic.

2. Entropy can decrease while performance collapses. Figure 3 (Middle-R and Right) shows Qwen2.5-Math-7B trained on AIME under clipped training: clipping drives entropy down (as Theorem 4.3 predicts), but the validation accuracy trajectory resembles a random walk with no meaningful improvement. The policy is becoming more peaked, more confident, more deterministic — and none of that translates to better reasoning. The same entropy-reduction mechanism that benefits the model on the easier DeepScaleR dataset (Figure 2, Middle) fails entirely on harder data.

3. The direction of entropy change is orthogonal to performance outcomes under random rewards. Figure 1 (Left vs. Middle) shows that both unclipped training (entropy increases) and clipped training (entropy decreases) can yield performance improvements over the baseline. The direction of entropy change doesn't predict whether the model gets better; it predicts only whether the model becomes more or less exploratory, which helps or hurts depending on context.

What makes this intellectually distinctive is that the paper doesn't just show counterexamples to a proposed law — it provides a mechanistic explanation for why the law fails in predictable ways. The reward-misalignment model (Section 5) explains the context-dependence: entropy minimization concentrates probability mass on whatever trajectories currently receive high advantage. If those trajectories are predominantly correct (strong model on easy data), concentrating on them improves accuracy. If those trajectories are predominantly incorrect or mixed (weak model on hard data), concentrating on them degrades accuracy or produces random walks. The observed monotonic relationship in Cui et al. (2025) likely reflects the specific regime they tested (relatively strong models on moderately difficult data), not a universal law.

This has immediate practical consequences. Section 4.3 warns: "methods explicitly minimizing policy entropy should be applied with caution." The warning is grounded in a specific failure mode: entropy minimization is regime-dependent. On the AIME dataset, Qwen2.5-Math-7B's policy under random-reward training places most probability mass on incorrect trajectories. Entropy minimization in this regime reinforces those incorrect modes. On the easier DeepScaleR dataset, the same model's policy has more mass on correct trajectories, so entropy minimization helps. The key variable is not entropy per se, but what the policy is concentrating on when entropy decreases — a distinction that the monotonic-relationship literature blurred.

The conceptual contribution is a shift from "entropy is a knob that controls performance" to "entropy is a description of the policy's shape, and whether a shape change helps depends on what the policy already knows." This is a more nuanced, and ultimately more useful, framework for thinking about when and how to modulate entropy during RLVR training.

Innovation 4: A Probabilistic Model of Reward Misalignment That Explains Model-Strength Dependence Without Invoking Contamination

The observation that stronger models benefit more from random rewards had been noted (Shao et al., 2025) but explained either through contamination (Wu et al., 2025: Qwen-Math models have memorized MATH500 answers) or through clipping bias (Shao et al., 2025: stronger models have more high-probability trajectories to amplify). The paper introduces a third, fundamentally different explanation: the expected damage from random reward mislabeling depends on the model's baseline correct-sampling rate n_c/G, and this dependence naturally favors stronger models.

The reward-misalignment model (Definition 5.1, Proposition 5.2, Theorem 5.3) is a conceptual innovation, not a technical one — the mathematics is elementary binomial probability. What makes it distinctive is the framing: instead of asking "do random rewards help?", it asks "how much of the ideal gradient signal for correct responses is lost to mislabeling, and how does this loss depend on group composition?" This shifts the analysis from a binary outcome (improvement yes/no) to a continuous function of model capability, providing testable predictions about when improvements should occur.

The model's key predictions are:

1. Expected damage E[Δ] = n_c(G - n_c)/G is maximized at n_c = G/2 and decreases as n_c approaches G or 0. Stronger models (high n_c) and very weak models (low n_c) both experience less misalignment damage than models at intermediate capability. This means the worst regime for random-reward training is when the model gets roughly half its rollouts correct — exactly the regime of maximum uncertainty, where mislabeling is most disruptive.

2. Variance Var(Δ) = n_c(G - n_c)/(4G) also peaks at n_c = G/2, explaining why validation curves are more stable for strong models (Figure 1 vs. Figure 4 Left): each training step receives a more consistent signal, even though the expected signal magnitude is smaller.

3. For n_c > G/2, false-negative-dominated misalignment (g > f, correct responses incorrectly penalized) contributes a larger share of damage than false-positive-dominated misalignment (f > g, incorrect responses incorrectly rewarded), and the FP share decreases monotonically as n_c increases (Theorem 5.3). This is the mechanism that insulates stronger models: the most harmful form of misalignment (incorrect responses pulling the gradient toward wrong answers) becomes progressively rarer as the model produces more correct rollouts.

Why this constitutes an innovation rather than a minor formalization: prior to this work, the field had no framework for predicting which models would benefit from random rewards under which conditions. The Shao et al. (2025) result showed benefits for Qwen-Math; Oertell et al. (2025) showed no benefits under different configurations. These conflicting findings were unresolvable without a model of the conditioning variables. The reward-misalignment model provides such a framework, and its predictions align with the paper's empirical results across model families and difficulty levels:

• Stronger distilled Llama model (Figure 4, Middle): shows improvements comparable to Qwen-Math-7B, as predicted by higher n_c.
• Weaker Qwen-Math-1.5B (Figure 4, Left): fails to achieve similar gains, as predicted by lower n_c.
• Same Qwen-Math-7B on harder AIME data (Figure 3, Middle-R): performance becomes unstable and improvement stalls, as predicted by n_c dropping below the favorable regime on the harder dataset.
• Stronger QwQ-32B and R1-Distill-Llama-8B on AIME (Figure 3, Left and Middle-L): continue to benefit, as predicted by their higher capability maintaining n_c above the critical threshold even on harder data.

Critically, this model does not require contamination. The predictions hold for Llama and QwQ models with no reported MATH500 contamination. The paper can therefore argue (Section 5, Appendix A) that contamination is not the mechanism — or at least not the primary mechanism — behind random-reward gains. The mechanism is the probabilistic structure of misalignment, which favors models that already produce correct answers at non-trivial rates regardless of whether those answers come from reasoning or memorization.

The broader significance is that this model provides a principled way to think about reward signal quality in RLVR. Practitioners can estimate their model's n_c on the training distribution (by sampling rollouts and checking against ground truth on a subset) and predict whether random-reward regularization would help or hurt. More ambitiously, the model suggests that deliberately mixing true and random rewards at controlled ratios — where the random component acts as an entropy modulator while the true component provides correctness signal — could be a design pattern for RLVR training that balances exploration and exploitation in a principled way. The paper doesn't explore this, but the model provides the conceptual foundation for doing so.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The primary validation benchmark is MATH500 (Hendrycks et al., 2021), the 500-question test set from the MATH dataset of high-school competition-level math problems. Training uses the DeepScaleR dataset (Luo et al., 2025), consisting of selected questions from AMC, AIME, Omni-Math, and Still — importantly, none of these overlap with the datasets used to train Qwen2.5-Math (Appendix A, "Contamination" paragraph). Additional experiments on harder data use the AIME Past series directly as the training set (Section 4.3). The paper's empirical findings focus on MATH500 validation accuracy throughout.

• Base model(s). Experiments span three model families and four scales to test generalization. The primary model is Qwen2.5-Math-7B (chosen for moderate parameter count and 4K context window, which stabilizes training and avoids gradient explosion issues that would confound the clipping analysis). For cross-family comparison: Qwen2.5-Math-1.5B (weaker, potentially contaminated variant), R1-Distill-Llama-8B (stronger distilled variant with reported MATH500 contamination from teacher, Appendix A), and QwQ-32B (strong model with no reported contamination). The diversity of families and sizes is deliberate: it tests whether random-reward benefits are specific to Qwen-Math (as contamination arguments would suggest) or generalize across model families.

• Metrics. The primary metric is MATH500 validation accuracy (%), computed as the fraction of 500 test questions for which the model's final boxed answer matches the ground truth, using the verl framework's default answer extraction (which instructs the model to enclose its final answer in \boxed{}). Policy entropy ℋ(π) is tracked as a secondary metric throughout training, defined as the negative expected log-probability of sampled responses (Definition 2.6). The clipping activation rate p^+ = E[I_t^+] — the expected fraction of tokens where the importance ratio exceeds 1+ε — is monitored to parameterize the theoretical bounds (Remark 3.5, Figure 1 Right).

• Baselines. The paper's experimental structure is primarily comparative across configurations rather than against external baselines. The key comparisons are: (a) clipped vs. unclipped training under random rewards (matching all hyperparameters except ε = 0.2 vs. ε = ∞); (b) random-reward vs. ground-truth reward training (same architecture, same hyperparameters, different reward signal); (c) model families and sizes under identical random-reward training configuration. The ground-truth reward baseline is implicit in the reported starting accuracies (e.g., Qwen2.5-Math-7B achieves a certain MATH500 accuracy before random-reward training begins) and in comparisons to prior work like Shao et al. (2025). Since the experiments train from the base model with random rather than verifiable rewards, the baseline is effectively the base model's performance before any RLVR fine-tuning.

• Generation budget / compute accounting. The paper uses a fixed training configuration matching Shao et al. (2025): batch size of 128 prompts, group size G = 16 responses per prompt, decoding temperature 1.0, clipping ratio ε = 0.2 (when enabled), learning rate η = 5 × 10^(-7), and KL coefficient 0 within the GRPO objective. Training steps serve as the compute axis — all comparisons are at matched step counts, not matched FLOPs. Rollout length is capped at 4096 tokens for Qwen-Math models (matching their context window) and 8192 tokens for the Llama and QwQ experiments (Section 4.3, Figure 3 Left and Middle-Left). The paper does not perform a FLOPs-matched comparison across configurations, focusing instead on how performance, entropy, and clipping activation co-evolve over training steps under different conditions.

• Cross-validation / statistical protocol. The paper runs multiple independent trials (typically six) for each experimental configuration, reporting individual-run trajectories (faint lines) alongside smoothed averages (bold lines) in all figures. Error bars or confidence intervals are not reported, but run-to-run variance is visually apparent and discussed qualitatively (e.g., Figure 1: "some runs fail to improve, as predicted by our probabilistic reward-misalignment framework"). Ablation experiments for clipping threshold ε ∈ {0.1, 0.15, 0.2, ∞} and group size G ∈ {8, 16} use six independent runs each (Figures 5 and 6, Appendix B). No train/validation split is needed since training uses DeepScaleR (not MATH500), so MATH500 serves as a held-out validation set throughout. The policy skewness measurement Φ(π) uses the first 500 examples from DeepScaleR, not the MATH500 test set (Appendix B, "Policy skewness").

Main Quantitative Results

The paper's experimental narrative is organized around testing the consequences of its theoretical framework rather than benchmarking methods. The results are structured to answer a sequence of causal questions: (1) Does clipping bias drive random-reward gains? (2) What does clipping actually do? (3) Does entropy reduction cause performance improvement? (4) When and why do random rewards help?

Clipping Does Not Drive Random-Reward Gains (Section 3.2, Figures 1 and 5)

The paper's first empirical claim is that clipping bias — hypothesized by Shao et al. (2025) as the mechanism for random-reward improvements — does not account for observed performance changes. The evidence comes from comparing clipped and unclipped training trajectories under identical random-reward configurations.

Figure 1 (Left vs. Middle) shows independent training runs of Qwen2.5-Math-7B on the DeepScaleR dataset under random rewards drawn from Bernoulli(1/2). Under unclipped training (Left), individual runs show substantial variance but the smoothed trajectory exhibits a clear upward trend, with several runs reaching approximately 70% MATH500 accuracy. Under clipped training (Middle), the smoothed trajectory is flatter, and some individual runs show declining performance relative to their starting point. The paper's interpretation: "enabling clipping can lead to a decline in validation performance, whereas disabling clipping often results in improvement."

Why this is evidence against clipping bias: If the upper-clipping bias mechanism were responsible for gains — amplifying high-probability responses preferentially — then removing clipping should weaken or eliminate those gains. Instead, the opposite occurs: unclipped training produces equal or better accuracy. The direction of the effect contradicts the clipping-bias hypothesis.

Clipping activation rates (Figure 1, Right) confirm a key input to the theoretical bounds: for Qwen2.5-Math-7B, the clipping activation ratio never exceeds 0.2%, with expected activation probability E[I_t] ≈ 0.001 (Remark 3.5). This low activation rate enters Corollary 3.6 to produce the 17.15× ratio between raw and clipping-correction magnitudes.

Ablation over clipping thresholds (Appendix B, Figure 5): Varying ε ∈ {0.1, 0.15, 0.2} (tighter vs. looser clipping) produces "behavior consistent with Figure 1: (i) some runs fail to improve… and (ii) successful runs converge to roughly 70% validation accuracy regardless of the clipping strength." Among improving trials, stricter clipping tends to reduce variance across seeds, reflecting a more deterministic policy. The key finding is that the presence or tightness of clipping does not determine whether improvement occurs — only how much run-to-run variance there is.

Ablation over group size (Appendix B, Figure 6): Reducing group size from G = 16 to G = 8 allows most runs to improve but produces "higher variance and less stable learning dynamics throughout training." From the reward-misalignment perspective, smaller groups increase the likelihood of extreme misalignment events (e.g., entire groups receiving reward 0 despite containing correct rollouts), which amplifies instability. This is consistent with Proposition 5.2: the variance of advantage loss Var(Δ) = n_c(G - n_c)/(4G) increases as G decreases, since the denominator shrinks.

Clipping Systematically Reduces Policy Entropy Under Random Rewards (Section 4.2, Figure 2)

The paper's second major empirical claim is that clipping functions as an entropy control mechanism, not a learning signal. The evidence comes from tracking policy entropy throughout training under clipped and unclipped configurations.

Figure 2 (Left): Unclipped training entropy increases. Under unclipped random-reward training of Qwen2.5-Math-7B, policy entropy rises monotonically over training steps. This is consistent with Theorem 4.1: if the initial policy is sufficiently skewed (Φ(π_old) < 0), the -c_G Φ(π_old) η² term is positive, and without clipping to counteract it, entropy grows. The paper verifies this diagnosis in Appendix B (Figure 8): among 500 sampled prompts from DeepScaleR, 358 exhibit Φ(π(· | x_i)) < 0 — the majority of the policy's per-prompt distributions are skewed enough to produce entropy increases under unclipped updates.

Figure 2 (Middle): Clipped training entropy decreases. Under clipped random-reward training (ε = 0.2), policy entropy falls monotonically. This is consistent with Theorem 4.3: the clipping correction term is negative under practical hyperparameters (Remark C.7 evaluates it at -2.01 × 10^(-7)), and it dominates the entropy dynamics, producing a net entropy decrease regardless of initial skewness. The paper's interpretation: clipping "functions primarily as a form of regularization: by capping per-token likelihood ratios, it effectively reduces the update step size and prevents the policy from drifting too far from its previous distribution."

Figure 2 (Right): Unclipped training can cause gradient explosion for stronger models with longer rollouts. Training R1-Distill-Llama-8B without clipping initially raises MATH500 validation accuracy from 65.6% to 76.6% within 100 steps, but around step 150 the gradients explode, causing a sharp performance collapse. This demonstrates that clipping's original purpose — preventing destabilizing updates — remains essential for some model/context-length combinations, even as the paper argues this stabilization operates through entropy control.

Correspondence between entropy direction and performance is non-monotonic. Comparing Figure 2 (Left) and (Middle): both increasing entropy (unclipped) and decreasing entropy (clipped) can coincide with accuracy improvements over the base model. This is preliminary evidence for the paper's claim that entropy and performance are not causally linked — expanded in Section 4.3.

Entropy Reduction Is Neither Necessary Nor Sufficient for Performance Gain (Section 4.3, Figures 3 and 7)

The paper challenges the claim (from Cui et al., 2025; Agarwal et al., 2025) that lower entropy directly causes better reasoning performance. The evidence comes from showing that entropy can decrease while performance collapses, and entropy can increase while performance improves, depending on model strength and dataset difficulty.

Figure 3 (Middle-R): Entropy decreases, performance stagnates on hard data. Qwen2.5-Math-7B trained on the harder AIME dataset under clipped random rewards shows the same entropy decrease as on DeepScaleR (by Theorem 4.3), but the validation trajectory "resembles a random walk with little meaningful improvement in validation accuracy." After 20 training epochs, there is no systematic upward trend. The paper's explanation: on harder data, the policy places most probability mass on incorrect trajectories — entropy minimization concentrates mass on those incorrect modes, which does not improve accuracy.

Figure 3 (Left and Middle-L): Stronger models continue to benefit on hard data. QwQ-32B and R1-Distill-Llama-8B (rollout length 8192, all other settings identical to the 7B configuration) trained on AIME exhibit "steady early-epoch gains" under clipped random rewards. The paper's explanation: these models have higher baseline n_c (correct-rollout rate) on AIME, so the reward-misalignment damage is smaller (Proposition 5.2) and the trajectories being concentrated onto are predominantly correct.

Figure 3 (Right): A specific example of entropy minimization producing suboptimal outcomes. Under a "noisier and more difficult training environment" (details not explicitly specified in the figure caption beyond the AIME context), entropy decreases while performance degrades relative to the starting point.

Figure 7 (Appendix B): Unclipped training on AIME — entropy increases, some runs improve, others degrade. Under unclipped random-reward training on AIME, Qwen2.5-Math-7B shows increasing entropy (per Theorem 4.1) and a mix of outcomes: some independent runs improve in MATH500 accuracy, others degrade. Figure 7 (Right) provides a representative case where "performance improves even as entropy increases." This directly contradicts any monotonic entropy-performance relationship: higher entropy coincides with better performance in this run.

The regime-dependence summary (Section 4.3, concluding paragraph): "the effectiveness of entropy minimization is regime-dependent: for strong models on easier data, it can further concentrate mass on correct trajectories, whereas for weaker models or harder data, it may reinforce incorrect modes and stall, or degrade performance."

Stronger Models Benefit More from Random Rewards — Across Families (Section 5, Figure 4)

The paper's final empirical claim is that reward-misalignment dynamics, not contamination, explain when random rewards help. The evidence compares models of varying strength across families under identical random-reward training.

Figure 4 (Left): Weaker Qwen2.5-Math-1.5B fails to benefit. Under clipped random-reward training on DeepScaleR, Qwen2.5-Math-1.5B shows no systematic improvement — the validation trajectory is flat or slightly declining. This model has both lower baseline capability (so n_c is likely below G/2 on many prompts, maximizing misalignment damage per Proposition 5.2) and is potentially contaminated (from Qwen-Math training data) — yet contamination does not rescue it.

Figure 4 (Middle): Stronger R1-Distill-Llama-8B benefits comparably to Qwen-Math-7B. Under clipped random-reward training with rollout length 8192 tokens and all other hyperparameters matching the Qwen-Math configuration, R1-Distill-Llama-8B shows improvements "comparable to those in Figure 2." This model and its teacher both exhibit contamination on MATH500 (Appendix A), yet the direction of the effect is the same as for Qwen-Math-7B — the model improves. The paper's argument: if contamination were the mechanism, the same contamination would not produce opposite effects (Qwen-1.5B fails, Llama-8B succeeds). The differentiating factor is model strength, which the reward-misalignment model predicts.

Figure 4 (Right): Percentage improvement summary across models. Averaged over six independent runs under the same training and validation setup, the bar chart quantifies improvement relative to each model's starting accuracy. Qwen2.5-Math-7B and R1-Distill-Llama-8B show comparable positive improvements; Qwen2.5-Math-1.5B shows near-zero or negative change.

Why this matters for the contamination debate: The paper explicitly addresses contamination in Appendix A, noting that DeepScaleR — the training dataset used in all experiments — consists of selected questions from AMC, AIME, Omni-Math, and Still. "None of these datasets appear in the training sources listed for Qwen2.5-Math." Therefore, the training data itself is uncontaminated, even though the MATH500 validation set may have leaked into Qwen-Math's pretraining. For Llama and QwQ models, no contamination has been reported on MATH500 at all. The fact that all three families show qualitatively similar random-reward behavior (stronger variants benefit, weaker variants don't) when trained on uncontaminated data supports the reward-misalignment explanation over the contamination explanation.

Ablation Studies and Robustness Checks

• Clipping threshold ε: Varying ε ∈ {0.1, 0.15, 0.2, ∞} (unclipped) (Figure 5, Appendix B) shows that successful runs converge to roughly 70% MATH500 accuracy regardless of clipping strength. Among improving trials, stricter clipping reduces variance across seeds. Some runs fail to improve under all ε values, consistent with the probabilistic reward-misalignment model. The key result: the presence of clipping is not necessary for improvement, but stricter clipping reduces run-to-run variance — consistent with clipping functioning as a regularizer, not a learning driver.

• Group size G: Reducing from G = 16 to G = 8 (Figure 6, Appendix B) increases run-to-run variance and produces less stable learning dynamics while still allowing most runs to improve. This matches the reward-misalignment prediction: smaller groups amplify misalignment variance (Var(Δ) = n_c(G - n_c)/(4G), which is inversely proportional to G for fixed n_c/G), making training noisier.

• Reward signal type (ground-truth vs. random): The paper's main results compare random-reward training against the model's own baseline accuracy before training, not against ground-truth reward training at matched steps. This is a deliberate design choice — the question is "can random rewards improve performance?" not "are random rewards better than verifiable rewards?" The ground-truth baseline is implicit: the base model's accuracy sets the floor, and any systematic improvement over that floor under random rewards is evidence of a genuine effect. The paper does not report head-to-head comparisons of random-reward vs. verifiable-reward RLVR at matched compute.

• Model family and scale: Results are replicated across Qwen-Math (1.5B, 7B), Llama (8B), and QwQ (32B), with both base and distilled variants. The pattern — stronger models benefit more, entropy decreases under clipping regardless of model — holds across families, supporting generalizability. The QwQ-32B result (Figure 3, Left) is particularly important because no contamination has been reported for this model.

• Policy skewness measurement: Appendix B reports Φ(π) evaluation on 500 prompts from DeepScaleR (Figure 8). Among 500 sampled questions, 358 satisfy Φ(π(· | x_i)) < 0, confirming that Qwen-Math-7B's initial policy is sufficiently skewed on most prompts to produce entropy increases under unclipped training (per Theorem 4.1). This measurement validates the theoretical prediction against empirical policy distributions.

• Training on harder data (AIME): Replacing DeepScaleR with AIME (Figure 3, Figure 7) shows that dataset difficulty modulates the effectiveness of random-reward training. For Qwen2.5-Math-7B, the shift to AIME reduces the baseline correct-sampling rate n_c, moving into a regime where misalignment damage is higher (per Proposition 5.2) and entropy minimization concentrates mass on incorrect trajectories, stalling progress. Stronger models (QwQ-32B, R1-Distill-Llama-8B) continue to benefit, consistent with their higher n_c on AIME.

• Prompt formatting sensitivity: The paper notes (Appendix A, "Spurious reward in RLVR") that Shao et al. (2025) omit the standard Qwen-style boxed-answer prompt formatting, while the current experiments use the default verl prompt that explicitly instructs the model to place final answers in \boxed{}. The paper argues this makes results not directly comparable to Shao et al. (2025), but all experiments within this paper use consistent formatting.

Critical Assessment

Can random rewards improve model performance, and under what conditions?

The experiments demonstrate that random-reward training can improve MATH500 accuracy, but with substantial qualifications:

• Not every run improves. Figures 1 and 5 clearly show that individual training runs diverge significantly — some reach ~70% accuracy and sustain it, others show flat or declining performance. The probabilistic nature of improvement is consistent with the reward-misalignment model (randomness in f and g per group means some training trajectories are luckier than others), but the paper does not provide a systematic analysis of what distinguishes successful from unsuccessful runs beyond the aggregate model-strength effect.

• The improvement ceiling appears bounded. Successful runs across models converge to roughly 70% MATH500 accuracy (Figures 1, 2, 4). The paper does not discuss whether this represents a fundamental limit — perhaps the accuracy achievable when all correct trajectories the model can reach have been concentrated onto — or whether longer training would push higher. Comparative numbers for ground-truth reward RLVR on the same models would contextualize this ceiling but are not reported.

• The conditions are: model must be strong enough relative to dataset difficulty. The paper's clearest empirical finding is the strength-dependence: Qwen2.5-Math-7B improves on DeepScaleR but not on AIME; Qwen2.5-Math-1.5B does not improve on DeepScaleR; QwQ-32B and R1-Distill-Llama-8B improve on AIME. This is consistent with the reward-misalignment model's prediction that n_c must be sufficiently high for gains to materialize, but the threshold n_c is not quantified — the paper provides no estimate of the minimum correct-sampling rate needed for improvement, nor does it measure n_c for any model-dataset pair. The model's predictions are tested qualitatively (higher n_c → better outcomes) but not quantitatively (predicting how much improvement given a measured n_c).

• What about truly from-scratch reasoning? The paper does not test whether random rewards can elicit new reasoning capabilities the model doesn't already possess. If a problem requires a reasoning pattern the model has zero probability of producing under its current policy, no amount of entropy modulation will surface it — the correct trajectory simply has zero mass in the proposal distribution. The experiments don't distinguish between "random rewards help the model find solutions it already was sampling at low probability" and "random rewards teach the model to produce entirely novel solutions." The reward-misalignment model suggests the former: n_c must be non-zero for improvement. The latter would require a different mechanism entirely.

Does clipping bias provide a meaningful learning signal?

The theoretical case is strong: Corollary 3.6 establishes a >17× ratio between raw and clipping-correction magnitudes under practical hyperparameters. The empirical case, while consistent (unclipped training doesn't harm performance), has a limitation: the paper does not directly test whether clipping bias could matter under different hyperparameters. The bound in Theorem 3.2 depends on η, p^+, L, and G — so one could ask: is there a hyperparameter regime where the clipping correction is comparable to the raw signal? The paper doesn't explore this. A broader learning rate sweep (beyond the single η = 5 × 10^(-7)) or measurements on models with significantly different clipping activation rates would test generality. The activation rate p^+ ≈ 0.001 for Qwen-Math-7B is quite low; models with higher activation rates (perhaps those trained from different initializations or with different architectures) might show different behavior. The bound's value as a diagnostic is that any practitioner can check their own p^+, but the paper only provides one data point.

Is there a direct causal relationship between policy entropy and policy performance?

The paper's strongest empirical contribution may be its negative result here: entropy and performance are not causally linked. The evidence — entropy decreases while performance collapses on AIME (Figure 3, Middle-R), entropy increases while performance improves on some AIME runs (Figure 7, Right) — is compelling. However, there are gaps:

• No intervention study on entropy. The paper observes co-occurrence of entropy changes and performance changes, and argues against causation by showing that the relationship breaks in specific regimes. But it does not perform a controlled experiment where entropy is manipulated independently (e.g., by adding an explicit entropy bonus) to test whether the same entropy change causes the same performance change. The paper's entropy changes come from clipping vs. no-clipping under random rewards — both variables change simultaneously. This is correlation evidence against a proposed monotonic law, not proof of absence of causation.

• The role of entropy in the positive cases is not explained. When entropy decreases and performance improves (e.g., Qwen-Math-7B on DeepScaleR under clipping, Figure 2 Middle), is the entropy decrease the mechanism of improvement, or an epiphenomenon of the policy settling onto correct trajectories? The paper argues the latter: clipping reduces entropy, and whether that reduction helps depends on what trajectories are being concentrated onto. But this implies entropy is not causal — it's a side effect of the policy moving toward something, and whether the "something" is good depends on the model's initial distribution over correct vs. incorrect solutions. The paper doesn't provide a model of when moving toward lower entropy coincides with moving toward correct solutions, beyond the qualitative observation that it depends on model strength and dataset difficulty.

Omitted experiments that would strengthen the claims

• Comparison to ground-truth reward RLVR. The paper never reports what accuracy the same models achieve with verifiable (ground-truth) rewards under the same GRPO configuration. This makes it impossible to assess the cost of using random rewards — how much performance is left on the table relative to using the real reward signal? The paper's framing is "random rewards can help," not "random rewards are as good as real rewards," but the magnitude of the gap matters for practical recommendations.

• Quantitative measurement of n_c during training. The reward-misalignment model makes predictions in terms of n_c (the number of correct rollouts per group), but n_c is never directly measured in the experiments. The paper infers n_c qualitatively from model strength and dataset difficulty. A direct measurement — sample rollouts, check correctness, estimate n_c distribution — would allow quantitative testing of Proposition 5.2's predictions about expected damage and variance.

• Longer training horizons. Most figures show training up to ~20 epochs or a few hundred steps. Is the 70% accuracy ceiling a genuine asymptote or an artifact of insufficient training? Does the gap between clipped and unclipped training widen or narrow with more steps? Do the "unsuccessful" runs eventually recover or permanently diverge?

• Multiple random seeds for the strong-model AIME results. Figure 3 (Left, Middle-L) shows QwQ-32B and R1-Distill-Llama-8B on AIME, but does not indicate how many independent runs were conducted. Given the high run-to-run variance observed in other configurations (Figures 1, 5, 6), single-run results on these expensive models would be insufficient to establish reliability.

• Evaluation beyond MATH500. All validation results use MATH500. The paper acknowledges extensive contamination analysis for Qwen-Math on this benchmark (Appendix A) but evaluates all models — including those with no reported contamination — only on MATH500. Testing on an uncontaminated benchmark (e.g., a recent competition set not in any training data) would strengthen the claim that random-reward gains are not contamination-driven.

• Direct manipulation of entropy via explicit regularization. To disentangle clipping's effect on entropy from its effect on gradient stability, an experiment that varies entropy directly (e.g., adding an entropy bonus term with tunable coefficient, under both clipped and unclipped training) would isolate the entropy-performance relationship from the clipping mechanism.

Where the claims hold conditionally

The paper's three main empirical claims are conditional in the following ways:

1. "Random rewards can improve model performance" — holds when the model's baseline correct-sampling rate n_c is sufficiently high relative to group size and dataset difficulty. Fails when n_c is low (weak model or hard data) or when training is unstable (gradient explosion without clipping). The threshold for "sufficiently high" is not quantified.

2. "Clipping bias does not provide a meaningful learning signal" — holds under the specific hyperparameters tested (η = 5 × 10^(-7), ε = 0.2, p^+ ≈ 0.001, G = 16, L = 4096). The theoretical bound suggests it holds for any practically small η and p^+, but the paper does not verify this across a wide range of η or on models with higher activation rates.

3. "Policy entropy and performance do not exhibit a deterministic causal relationship" — demonstrated for the specific case of Qwen2.5-Math-7B on DeepScaleR vs. AIME, where the same entropy decrease helps in one context and fails in another. Whether this non-causal relationship generalizes to other models, other datasets, and other entropy-modulation mechanisms (explicit regularization, decoding temperature changes) is not tested.

6. Limitations and Trade-offs

Assumption: Difficulty Estimation Cost Is Unaccounted For

The paper introduces a probabilistic reward-misalignment model (Section 5) whose key explanatory variable is n_c — the number of correct rollouts the model produces in a group of G responses. The model's predictions (expected damage, variance, FP-vs-FN composition) all depend on n_c, and the paper uses n_c to explain why stronger models benefit more from random rewards. However, n_c is never directly measured in any experiment. The paper's entire case for the reward-misalignment model rests on qualitative inference — "stronger models have higher n_c" — without any quantitative verification of what n_c actually is on the training distribution for any model-dataset pair.

The consequence is that Proposition 5.2 and Theorem 5.3 cannot be tested quantitatively against the experimental data. The paper predicts that expected damage E[Δ] = n_c(G - n_c)/G is maximized at n_c = G/2 and decreases as n_c approaches G — but without measuring n_c, we cannot verify whether models that improve are indeed operating in the regime n_c > G/2 and models that fail are operating near n_c ≈ G/2 or below. The model provides a qualitative framework consistent with observed patterns (stronger models benefit more, weaker models don't), but the central quantitative prediction — that damage depends on the product n_c × n_i — is never validated. This leaves open the possibility that the observed patterns have a different explanation, and the reward-misalignment model is merely post-hoc rationalization rather than a verified mechanism.

What evidence exists in the paper: The paper reports no measurement of n_c for any model-dataset pair. The closest proxy is the baseline MATH500 validation accuracy (e.g., Qwen2.5-Math-7B starts around 50-60%, QwQ-32B is stronger), but validation accuracy on MATH500 does not directly measure n_c on the training distribution (DeepScaleR or AIME). The paper acknowledges in Appendix A that training and validation datasets are distinct (DeepScaleR does not overlap with MATH500), so the model's sampling behavior on training prompts is unobserved.

Mitigation status: None. The paper does not discuss the absence of n_c measurement, does not propose a method for estimating it (which would require either ground-truth labels — defeating the purpose of random-reward training — or a separate evaluation pipeline), and does not qualify the strength of its claims in light of this gap. Future work suggested by the paper (e.g., "using spurious-reward setups to more effectively preserve and modulate entropy," Remark 4.4) would benefit from direct n_c tracking to test the model's predictions.

Single Benchmark, Single Task Family: All Validation on MATH500

All experimental results in the paper — every figure, every quantitative claim about accuracy improvement — uses MATH500 as the validation benchmark. The training data varies (DeepScaleR for most experiments, AIME in Section 4.3), but the evaluation of outcomes is exclusively on a single 500-question math competition dataset. The paper acknowledges this narrowness implicitly (the abstract references "mathematical reasoning" and the experiments use math datasets exclusively) but never discusses whether the findings generalize beyond mathematics.

The consequence is that we have no evidence about whether the central claims — random rewards can improve performance, clipping reduces entropy independently of reward informativeness, entropy and performance are not causally linked — hold for other reasoning domains (code generation, logical deduction, scientific QA) or for tasks without clean verifiable answers (open-ended generation, summarization, dialogue). This matters because the RLVR framework itself is math-centric: it requires deterministic ground-truth answers that can be verified with exact matching. The paper's findings about random rewards — where even the verifiability requirement is dropped — might seem to suggest broader applicability. But if the mechanisms depend on properties specific to math reasoning (structured output formats, discrete answer spaces, token-level patterns that correlate with correctness), they may not transfer. The entropy dynamics in particular might behave differently in domains where "correctness" is less well-defined and the model's initial policy distribution has different skewness properties.

What evidence exists: None beyond MATH500. The paper includes models from multiple families (Qwen-Math, Llama, QwQ) and sizes (1.5B to 32B), which addresses the model-specificity concern, but all evaluations are on the same benchmark type. The paper does not run even a single experiment on a non-math dataset or discuss domain generalization as a limitation.

Mitigation status: None. The paper's claims are implicitly scoped to mathematical reasoning (the abstract says the framework is "for improving the reasoning of Large Language Models" in the context of RLVR, which is primarily applied to math), but the paper never explicitly states that its conclusions are limited to math or discusses what would need to change for other domains. A practitioner working on code generation or scientific reasoning cannot infer from this paper whether random-reward training would help, hurt, or do nothing in their domain.

No Comparison to Ground-Truth (Verifiable) Reward RLVR

The paper's central empirical claim is that random rewards "can improve model performance" (Section 6, revisiting guiding question (i)). The evidence demonstrates that MATH500 accuracy increases relative to the base model's starting point under random-reward GRPO training. However, the paper never reports what accuracy the same models achieve with standard verifiable (ground-truth) reward GRPO under identical hyperparameters and compute budgets. There is no head-to-head comparison between random-reward training and the RLVR method it is meant to illuminate.

The consequence is that we cannot assess the cost of using random rewards. The paper frames random rewards as a lens for understanding RLVR dynamics, not as a practical training method — but the implications section (Section 6) draws practical conclusions: "random rewards can improve model performance," "stronger models are more likely to realize gains," "clipping bias does not supply a useful signal." Without knowing the gap between random-reward and verifiable-reward outcomes, a practitioner cannot evaluate tradeoffs. If ground-truth reward GRPO achieves 85% MATH500 accuracy on Qwen2.5-Math-7B while random-reward training plateaus at 70%, the 15-point gap represents the price of using uninformative rewards — a price the paper never quantifies. Conversely, if the gap is small, that would be a striking finding in itself (random noise nearly matches verifiable feedback), but the paper provides no evidence either way.

This omission also weakens the paper's argument about mechanisms. The paper claims that "clipping bias does not provide a meaningful learning signal" (Section 3.2) and that clipping functions primarily as an entropy modulator (Section 4.2). These claims are about what doesn't matter for learning under random rewards. But if we don't know how much learning actually occurs under random rewards relative to verifiable rewards, we can't assess whether the mechanisms the paper rules out (clipping bias) or identifies (entropy modulation, reward misalignment) are the right ones to focus on. If random-reward training achieves only a small fraction of the improvement that verifiable-reward training achieves, then the mechanisms the paper studies may explain only a minor component of RLVR dynamics, with the dominant mechanisms still unknown.

What evidence exists: None. The paper reports base model accuracy as the implicit baseline (performance before any RLVR training), and shows improvement over that baseline under random rewards. Verifiable-reward GRPO results are not reported for any model, any dataset, or any hyperparameter configuration. The paper does not discuss this omission.

Mitigation status: None. The paper cites Shao et al. (2025) for the original observation that random rewards improve Qwen-Math performance, and Shao et al. (2025) presumably reports verifiable-reward baselines, but the current paper does not replicate or reference those numbers in its own experimental context. A reader must consult external work to assess the magnitude of the gap between random and verifiable rewards.

Entropy-Performance Relationship Established Through Correlation, Not Intervention

The paper's third major claim — that "policy entropy and performance do not exhibit a deterministic causal relationship" (Section 6) — is supported by experiments showing that entropy can decrease while performance collapses (Figure 3, Middle-R: Qwen2.5-Math-7B on AIME under clipping) and entropy can increase while performance improves (Figure 7, Right: unclipped training on AIME). These are correlational observations: entropy and performance co-vary in ways that contradict a monotonic law. However, the paper never performs a controlled intervention that manipulates entropy independently of other training variables. In every experiment, entropy changes are produced by toggling clipping (which also affects gradient stability, update magnitudes, and the trust region) or by changing the training dataset (which also affects the reward signal distribution and the model's baseline n_c).

The consequence is that we cannot rule out the possibility that a third factor — not entropy — drives both entropy changes and performance changes, producing a spurious correlation. For instance, on easy data with a strong model, both entropy decreases (because the policy concentrates on high-advantage trajectories) and accuracy increases (because those trajectories are correct). On hard data with a weak model, the same clipping mechanism reduces entropy, but the trajectories being concentrated on are incorrect, so accuracy does not improve. The paper interprets this as evidence that entropy reduction is not causal. But an alternative interpretation is that entropy reduction is causal, but its effect depends on what the policy is concentrating on — which is determined by the reward signal and baseline policy, not by entropy per se. The paper's data cannot distinguish these interpretations because entropy is never manipulated in isolation.

To establish non-causation, one would need experiments where entropy is varied while holding the reward signal and baseline policy constant — for example, by adding an explicit entropy bonus with tunable coefficient, or by varying decoding temperature at inference time while keeping the training objective fixed. If increasing entropy (via a negative entropy bonus) consistently failed to harm performance across datasets and models, that would support non-causation. If decreasing entropy (via a positive entropy bonus) sometimes helped and sometimes hurt, that would support the paper's regime-dependence claim but would still be consistent with entropy having a causal effect whose sign depends on context. The paper's current evidence only shows that the sign of the entropy-performance correlation is not fixed — which is a weaker claim than "no causal relationship."

What evidence exists: Correlational. Figure 3 (Middle-R) shows clipped training on AIME producing entropy decrease + performance stagnation. Figure 7 (Right) shows unclipped training on AIME producing entropy increase + performance improvement. Figure 2 (Left and Middle) show both increasing entropy (unclipped) and decreasing entropy (clipped) coinciding with accuracy gains on DeepScaleR. These establish that entropy direction does not determine performance direction — a valid and important finding. But they do not establish that entropy is not causal, only that it is not deterministic in sign.

Mitigation status: None. The paper does not discuss the correlational nature of its evidence, does not propose an intervention experiment, and does not qualify its causal claims. Section 4.3 presents the non-causal relationship as a conclusion ("methods explicitly minimizing policy entropy should be applied with caution") without acknowledging the inferential limitation. This is a significant gap because the paper's practical recommendation — that entropy minimization should not be pursued as a standalone objective — is based on this causal claim.

The Reward-Misalignment Model Analyzes Per-Group Statics, Not Training Dynamics

The reward-misalignment model (Section 5, Proposition 5.2, Theorem 5.3) analyzes a single group of G rollouts with fixed composition (n_c correct, n_i incorrect) under i.i.d. Bernoulli(1/2) rewards. It computes expected advantage loss, variance, and FP-vs-FN composition as functions of n_c. This is a static, per-step analysis — it describes what happens to the advantage signal in a single GRPO update, given the current policy's sampling distribution.

However, RLVR training is a dynamic process: the policy changes at each step, which changes n_c for subsequent steps, which changes the misalignment damage, which further changes the policy. The model provides no analysis of this feedback loop. It cannot predict whether n_c will increase or decrease over training, whether the policy will converge to a high-n_c or low-n_c regime, or under what conditions random-reward training leads to a virtuous cycle (policy improves → n_c increases → damage decreases → policy improves further) versus a vicious cycle (policy degrades → n_c decreases → damage increases → policy degrades further).

The consequence is that the model is descriptive of step-level damage but not predictive of training outcomes. It explains why a strong model at a given step experiences less misalignment than a weak model, but it does not explain why some training runs improve and others degrade from the same starting point (as observed in Figures 1, 5, 6). The model predicts that runs with different initial n_c (different models or datasets) will have different expected damage, which matches the aggregate pattern. But within a single model-dataset configuration, all runs start with the same n_c distribution, yet outcomes diverge substantially — some Qwen2.5-Math-7B runs on DeepScaleR reach ~70% accuracy, others stagnate (Figure 1). The static model cannot explain this divergence because it treats n_c as given rather than as evolving.

This also means the model provides no guidance on how long to train with random rewards, whether there is a point of diminishing returns or a risk of eventual collapse, or how training hyperparameters (learning rate, group size, number of steps) affect the dynamics of n_c. The paper's experiments use fixed training horizons (typically 20 epochs or a few hundred steps) and report final accuracy, but the model has nothing to say about whether these are appropriate stopping points.

What evidence exists: The paper acknowledges a related gap in Section 3.2 regarding difficulty estimation cost but does not discuss the static-vs-dynamic limitation of the reward-misalignment model specifically. The model's predictions are tested only against aggregate outcomes (stronger models benefit more, Figure 4 Right) and against qualitative patterns (more oscillation for weaker models, Figures 1 and 4 Left). The run-to-run variance that the model cannot explain is visible in every multi-run figure but is not analyzed through the model's lens.

Mitigation status: None. The paper does not discuss extending the model to a dynamic setting, does not track n_c over training, and does not analyze whether the step-level predictions aggregate to explain trajectory-level outcomes. This is a fundamental gap between the model's scope (static per-step damage) and the phenomena it is used to explain (multi-step training dynamics with divergent outcomes).

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not introduce a new training algorithm, a new benchmark, or a new model. Its contribution is conceptual reorientation: it takes two puzzling empirical phenomena in RLVR — spurious rewards improve performance, entropy minimization improves performance — and provides a mechanistic framework that resolves their apparent contradiction while overturning the field's dominant explanation (that clipping bias drives random-reward gains). The shift is from treating these phenomena as independent curiosities to understanding them as interacting components of a single system governed by clipping's role as an entropy modulator and by the probabilistic structure of reward misalignment.

Magnitude: a diagnostic reframing, not a paradigm shift. The paper does not claim to change how RLVR should be done in practice — it does not propose a new loss function, a new clipping schedule, or a new reward design. Rather, it changes how researchers should think about what is happening during RLVR training. Prior to this work, the field had two competing narratives: (1) random-reward gains are a contamination artifact specific to Qwen-Math (Wu et al., 2025), and (2) random-reward gains arise from clipping bias amplifying high-probability responses (Shao et al., 2025). The paper demonstrates that neither narrative is correct. Contamination cannot explain why comparable gains appear in non-contaminated Llama and QwQ models (Figure 4). Clipping bias cannot explain anything because its magnitude is structurally negligible — more than 17× smaller than the raw gradient signal under practical hyperparameters (Corollary 3.6).

This resolves a specific, active controversy in the RLVR literature. The question "do random rewards actually help?" now has a conditional answer grounded in mechanism rather than model family: they help when the model's baseline correct-sampling rate n_c is sufficiently high relative to dataset difficulty, and the benefit is mediated through entropy modulation, not through biased gradient signals. The fact that the paper demonstrates this across three model families (Qwen-Math, Llama, QwQ) and multiple scales (1.5B to 32B) gives the finding generalizability that prior Qwen-Math-specific results lacked.

What becomes more attractive as a research direction:

• Understanding entropy dynamics in RLVR as a first-class object of study. The paper establishes that clipping functions as an implicit entropy modulator, that the direction of entropy change depends on initial policy skewness, and that entropy change is not monotonically linked to performance. This makes entropy dynamics a legitimate target for theoretical analysis and algorithmic design, not merely a diagnostic metric. Prior work that treated entropy as a nuisance to be prevented from collapsing (DAPO-style entropy bonuses) or as a direct lever for improvement (explicit entropy minimization objectives) now must contend with the paper's finding that entropy reduction is context-dependent in its effects.

• Designing reward signals as entropy control mechanisms. Remark 4.4 explicitly suggests "using spurious-reward setups to more effectively preserve and modulate entropy" and proposes "combining true and spurious rewards to better balance exploration and exploitation." This reframes random rewards from a puzzling anomaly into a potential design tool — if you understand how they interact with clipping and initial policy distribution, you can deliberately inject controlled amounts of reward noise to shape the policy's entropy trajectory during training. This is a genuinely new design space that the paper opens but does not explore.

What becomes less attractive:

• Chasing ever-lower entropy as a universal objective. The paper provides clear counterexamples to the proposed monotonic entropy-performance relationship from Cui et al. (2025): entropy decreases can coincide with performance collapse (Qwen2.5-Math-7B on AIME under clipping, Figure 3 Middle-R), and entropy increases can coincide with performance improvement (unclipped training on AIME, Figure 7 Right). A research program that treats entropy minimization as an unqualified good — adding entropy penalties, designing algorithms to accelerate entropy collapse — is directly challenged by this evidence. The paper's reframing is that entropy shape matters only insofar as it concentrates mass on the right trajectories, which depends on the model's initial capability and the dataset's difficulty relative to that capability.

• Appealing to clipping bias as an explanation for RLVR phenomena. The paper's bound in Theorem 3.2 and its numerical instantiation in Corollary 3.6 provide a general diagnostic: if your clipping activation rate p^+ is below ~1% and your learning rate η is in the typical 1e-7 to 1e-6 range, the clipping correction is mathematically negligible relative to the raw gradient. Future work that invokes clipping bias as a mechanism — for explaining spurious-reward effects, for motivating clipping threshold choices, for analyzing policy behavior — must either work in a hyperparameter regime where the bound does not apply (very large η or very high p^+) or provide evidence that the bound's assumptions fail in a qualitatively important way.

Reconciling prior contradictions. The paper provides a unified framework for understanding why Huang et al. (2023) found that "LLMs cannot self-correct reasoning" while Shao et al. (2025) found that random rewards improve Qwen-Math, while Oertell et al. (2025) found that random rewards don't consistently help. The resolution is that these studies tested different models on different datasets with different configurations, placing them in different regimes of the n_c vs. dataset-difficulty space. A model with high n_c on its training data (strong model, easy data) benefits from random-reward training because misalignment damage is small and the policy concentrates on predominantly correct trajectories. A model with low n_c (weak model, hard data, or aggressive hyperparameters that produce more incorrect rollouts) experiences larger misalignment damage and may see no benefit or even degradation. The paper does not explicitly map prior studies onto this framework, but the conceptual machinery is there: n_c is the sufficient statistic that determines outcome direction.

Follow-Up Research This Work Enables

1. What is the quantitative relationship between n_c and random-reward training outcomes? The reward-misalignment model (Proposition 5.2, Theorem 5.3) makes specific quantitative predictions about expected advantage loss, variance, and FP-vs-FN damage composition as functions of n_c. However, n_c is never directly measured in the paper's experiments — the model is tested only qualitatively (stronger models benefit more). A direct test would: (a) for a given model and training dataset, sample rollouts and compute the empirical n_c distribution (how many groups of G = 16 have 0, 1, 2, ..., 16 correct responses); (b) run random-reward GRPO and track how n_c evolves over training steps; (c) test whether the predicted damage E[Δ] = n_c(G - n_c)/G correlates with per-step gradient variance and with aggregate performance trajectories. This would elevate the model from qualitative explanation to quantitative prediction, and would identify the critical n_c threshold above which random-reward training reliably improves performance. The paper's DeepScaleR training setup with Qwen2.5-Math-7B provides a natural starting point, since the model's accuracy on MATH500 (~50-60% at initialization) suggests moderate n_c on the uncontaminated training data.

2. Can deliberate mixing of true and spurious rewards improve exploration beyond pure verifiable rewards? Remark 4.4 speculates that "using spurious-reward setups to more effectively preserve and modulate entropy" could complement true reward signals. The paper establishes that (a) random rewards modulate entropy in predictable ways (increasing it without clipping for skewed policies, decreasing it with clipping), and (b) explicit entropy minimization works in some regimes but not others. A natural experiment: for a fixed compute budget, compare (i) pure verifiable-reward GRPO, (ii) pure random-reward GRPO, and (iii) GRPO where each group receives a mixture — e.g., 80% of advantages computed from verifiable rewards and 20% from random rewards, or interleaved steps of verifiable and random rewards. The hypothesis is that the random component prevents premature entropy collapse while the verifiable component provides correctness signal, yielding better final accuracy than either pure approach. The paper's finding that entropy collapse is a known failure mode in RLVR (DAPO, Yu et al., 2025; multiple entropy-control papers cited in Appendix A) makes this a practically motivated extension. The Qwen2.5-Math-7B on DeepScaleR configuration, where both pure random-reward and pure verifiable-reward baselines are straightforward to run, provides a clean testbed.

3. Does the clipping-entropy relationship hold across learning rates, and can clipping thresholds be tuned for entropy targets? Theorem 4.3's bound on entropy change under clipping depends on η, ε, G, and the policy's initial distribution. The paper tests a single learning rate (η = 5 × 10^(-7)) and a narrow range of ε values ({0.1, 0.15, 0.2, ∞}). The theoretical framework predicts that the entropy-reducing effect of clipping should scale with η (through X_max, δ_eff, and the other bound components) and with ε (tighter clipping should produce larger entropy reduction). A systematic sweep: vary η across {1e-7, 5e-7, 1e-6, 5e-6} and ε across {0.05, 0.1, 0.2, 0.4, ∞}, measure the resulting entropy trajectory and the clipping activation rate p^+, and test whether the theoretical bound's predictions match empirical entropy changes. This would validate (or refine) the paper's claim that clipping "functions primarily as a form of regularization" and would provide practitioners with a principled way to select ε based on desired entropy trajectory rather than on PPO defaults. The paper's existing verl-based experimental infrastructure (Sheng et al., 2025) makes this sweep straightforward.

4. Is entropy modulation the mechanism, or is there a confounded variable? The paper establishes correlation between clipping, entropy, and performance but acknowledges (implicitly, through the limitations of its experimental design) that it does not perform controlled entropy interventions. To isolate entropy: run GRPO with a tunable entropy bonus coefficient (positive to decrease entropy, negative to increase it) while holding all other hyperparameters fixed, under both verifiable and random rewards. If adding a positive entropy bonus (forcing entropy lower) on AIME with Qwen2.5-Math-7B degrades performance — matching the clipped-training result in Figure 3 Middle-R — that would directly support the paper's claim that entropy reduction per se is harmful in that regime. If adding a negative entropy bonus (forcing entropy higher) on DeepScaleR with the same model degrades performance — even though unclipped training (which increases entropy, Figure 2 Left) improves it — that would suggest clipping does something beyond entropy modulation (e.g., gradient stabilization, trust-region enforcement) that matters for outcomes. This experiment would transform the paper's correlational evidence into causal evidence and would clarify whether the practical recommendations ("entropy minimization should be applied with caution") are well-founded.

5. Cross-domain replication: do random-reward gains generalize to code generation? The paper's entire evaluation is on MATH500, a math competition benchmark. Code generation shares key structural properties with math reasoning (deterministic verifiable rewards via unit tests, structured output format, multi-step reasoning) and has clean ground-truth signals for computing n_c and verification accuracy. Running the same experimental protocol — random-reward vs. ground-truth reward GRPO, clipped vs. unclipped, across model families — on HumanEval or MBPP would test whether the paper's mechanisms are specific to math or generalize to other verifiable-reward domains. Specific predictions from the paper's framework: (a) stronger code models (higher pass@1) should benefit more from random rewards than weaker ones; (b) clipping should reduce entropy regardless of domain; (c) entropy reduction should help when the model's initial policy is concentrated on correct code solutions and hurt when it is concentrated on buggy patterns. A negative result — no random-reward improvement for code regardless of model strength — would suggest that math-specific properties (e.g., the token-level structure of chain-of-thought reasoning, the nature of mathematical errors) are necessary for the effect. A positive result would substantially strengthen the paper's claim of generality.

6. What happens at the per-token level during entropy increase vs. decrease? The paper's entropy analysis operates at the response level (bandit case, L = 1) and treats entropy as a scalar summary of the policy's distribution over full responses. But the GRPO update operates at the token level, and the paper's own Remark 2.1 notes that all tokens in a response receive the same advantage A_i. When entropy increases under unclipped training (Theorem 4.1, Figure 2 Left), which tokens are becoming more probable? Are they tokens associated with correct reasoning patterns, incorrect patterns, or format tokens (e.g., \boxed{} delimiters)? A fine-grained analysis tracking per-token probability changes — separately for tokens in correct vs. incorrect rollouts, early vs. late positions, content vs. formatting tokens — would reveal whether the entropy increase reflects genuine exploratory broadening of the reasoning distribution or merely noisier formatting. This matters because if entropy increases are concentrated on format tokens, the paper's claim that "higher entropy reflects stronger exploration: the policy is flatter and thus more capable of discovering new trajectories" (Section 4.3) may overstate the mechanism. The verl framework's logging capabilities should support per-token probability tracking with modest instrumentation.

Practical Applications and Downstream Use Cases

1. Diagnosing whether random-reward training will help a specific model-dataset pair. The paper's reward-misalignment model provides a directly actionable diagnostic: before committing to a full random-reward training run (or an entropy-minimization strategy), sample G rollouts per prompt for a representative subset of the training data, count how many are correct (n_c), and compute the expected advantage loss E[Δ] = n_c(G - n_c)/G. If n_c is consistently above G/2 (the model gets more than half its rollouts correct on average), the expected damage is in the favorable regime where false negatives dominate and the total damage is modest — random-reward training is predicted to help. If n_c ≈ G/2, the model is in the worst-case regime for misalignment, and training will likely be unstable with high run-to-run variance. If n_c is very low, the model lacks the baseline capability for random rewards to concentrate mass on correct trajectories — verifiable rewards or further pretraining are needed. This diagnostic costs one forward pass and one correctness check per prompt in the sample, which is negligible relative to a full training run.

2. Selecting clipping thresholds based on desired entropy trajectory rather than PPO defaults. The paper's finding that clipping "functions primarily as a form of regularization: by capping per-token likelihood ratios, it effectively reduces the update step size and prevents the policy from drifting too far from its previous distribution" (Section 4.2) implies that the clipping threshold ε can be tuned based on how much entropy modulation is desired, not based on PPO's default of ε = 0.2. For a strong model on easy data where early entropy collapse is a risk (policy concentrates prematurely on a subset of correct trajectories, limiting exploration of alternative solutions), a looser clip (ε = 0.3 or 0.4) or even unclipped training may preserve more entropy and enable broader exploration — as suggested by Figure 2 (Left), where unclipped training increases entropy while improving accuracy. For a weaker model on harder data where entropy increases risk gradient instability, tighter clipping (ε = 0.1) provides stronger entropy control and more stable training — as shown in Figure 5 (Appendix B), where stricter clipping reduces run-to-run variance. The paper's monitoring of clipping activation rate p^+ (Figure 1 Right) provides a real-time signal for whether the chosen ε is actively constraining the policy or is effectively irrelevant.

3. Debugging RLVR training runs that show entropy collapse without performance improvement. A common failure mode in RLVR is "entropy collapse": the policy's output distribution becomes highly peaked early in training, and subsequent accuracy plateaus or declines. The standard response is to add an entropy bonus (DAPO, Yu et al., 2025) or adaptive entropy control (Shen et al., 2025). The paper's framework suggests a different diagnosis: entropy collapse accompanied by performance stagnation may indicate that the policy is concentrating on incorrect trajectories — which happens when n_c is low (the model's baseline accuracy on the training data is poor). In this regime, adding an entropy bonus fights the symptom (low entropy) without addressing the cause (the policy doesn't know which trajectories are correct). The paper's finding that unclipped training can increase entropy while improving performance (Figure 2 Left) suggests a concrete intervention: if you observe entropy collapse with stagnant accuracy, try disabling clipping temporarily to see whether the policy naturally diversifies — and whether that diversification leads to discovery of correct trajectories that were previously sampled at low probability. If it does, the problem was premature concentration on incorrect modes; if it doesn't, the model simply lacks the capability to produce correct solutions on this data, and no amount of entropy modulation will help.

4. Designing compute budgets for self-improvement pipelines with mixed reward quality. Self-improvement loops (STaR, ReST^EM, rejection sampling fine-tuning) generate training data by sampling from the current model and filtering or weighting by correctness. In practice, correctness labels may be noisy (inexact verifiers, ambiguous answers, partial credit), creating a regime analogous to the paper's random-reward setting. The reward-misalignment model provides guidance on how much noise is tolerable: if the model's baseline n_c is high, substantial label noise (even approaching 50% random) still leaves a usable signal because false negatives dominate false positives and the expected damage is modest. If n_c is low, even small amounts of label noise can be catastrophic because false positives (incorrect responses incorrectly labeled as correct) are more prevalent and pull the gradient in wrong directions. This has direct implications for how much verification budget to allocate: for strong models on well-defined tasks, cheap approximate verifiers may suffice; for weak models or ambiguous tasks, investing in high-quality verification (human labels, multiple verifiers, consensus mechanisms) is essential because the damage from mislabeling scales nonlinearly with n_c.