ArXiv: 2509.21880
🎯 Pitch
Standard RL for LLM reasoning throws away nearly all computation on prompts where every sampled answer is either all right or all wrong—up to 99% of a batch—yet this paper shows that directly rewarding correctness without requiring contrasting rollouts, scaled by token-level entropy, boosts math accuracy by up to 8.6 points over GRPO and even beats methods that filter these prompts and use 3–5× more rollouts.
1. Executive Summary
This paper introduces Reinforcement Learning with Zero-Variance Prompts (RL-ZVP), a novel algorithm that extracts useful learning signals from prompts where all sampled responses receive identical rewards—so-called zero-variance prompts—which existing methods like GRPO discard by assigning them zero advantage. Across six math reasoning benchmarks using Qwen3-1.7B-Base and Qwen3-8B-Base, RL-ZVP employs an entropy-guided advantage shaping formula that directly rewards correct responses and penalizes incorrect ones without requiring contrasting answers within the same group (e.g., scaling token-level updates by entropy magnitude so that high-entropy reasoning tokens receive larger gradient updates than trivial tokens on correct responses, while being penalized less severely on incorrect ones). RL-ZVP achieves improvements of up to 8.61 points in accuracy and 7.77 points in pass rate over GRPO, and consistently outperforms prompt-filtering baselines even when those baselines are given 3×–5× more rollouts—establishing that zero-variance prompts contain meaningful learning signals for policy optimization, not just wasted computation.
2. Context and Motivation
The Core Problem: Zero-Variance Prompts Are a Massive Source of Wasted Computation in LLM Reinforcement Learning
The fundamental problem this paper addresses is deceptively simple: when training LLMs with reinforcement learning on verifiable rewards (RLVR), a large fraction of the training prompts produce groups of responses that are either all correct or all incorrect — and current methods throw away all learning signal from these cases. The paper calls these zero-variance prompts: prompts where the rewards across a group of sampled responses have zero variance. In the standard GRPO formulation (Section 2), this causes the advantage values to collapse to zero, which zeroes out the gradient update (Remark 1), making the entire computational investment in generating those rollouts worthless for policy optimization.
This is not a niche edge case. The paper demonstrates in Figure 3 that zero-variance prompts account for roughly 30% to 99% of each training batch, depending on the model scale, training dataset difficulty, and training stage. Specifically:
- At the beginning of training on challenging datasets (e.g., Qwen3-8B-Base on DAPO-Math-17k), the model solves fewer than 5% of problems per batch, meaning the vast majority of prompts are all-incorrect zero-variance cases.
- Near convergence on easier datasets (e.g., Qwen3-1.7B-Base on MATH), the model becomes strong enough to solve many questions entirely across all sampled responses, producing all-correct zero-variance cases.
The scale of this waste is substantial because rollout generation is the dominant bottleneck in RL training. The paper's preliminary experiments (Figure 2) show that response sampling accounts for approximately 50% of the total time per training step. If 30–99% of that sampling effort produces zero learning signal, then the majority of training time is being spent on computation that contributes nothing to policy improvement.
This matters for several practical reasons the paper highlights (Section 1, Section 3.1):
- Training cost and throughput: Every zero-variance prompt represents GPU-hours burned on generating, scoring, and processing rollouts that produce no gradient update. In large-scale training pipelines where thousands of prompts are processed per step, this inefficiency directly translates to slower convergence, higher training costs, and longer experimental iteration cycles.
- Data efficiency: If zero-variance prompts are discarded, the effective number of informative training examples per batch is much lower than the batch size suggests — in early training on hard datasets, the effective batch size might be only 1–5% of the nominal batch size.
- Learning stability: The paper shows (Section 4.2, Figure 5) that GRPO suffers from sharp performance drops during early training on larger-scale experiments, which the authors attribute to sparse and noisy feedback when the majority of prompts are zero-variance. RL-ZVP's ability to extract signal from all-incorrect prompts provides denser, more stable gradients.
Prior Approaches: Filtering Out Zero-Variance Prompts (And Why This Falls Short)
The paper identifies two broad categories of existing responses to the zero-variance prompt problem, neither of which attempts to actually extract learning signal from these prompts.
Post-rollout filtering via dynamic sampling (GRPO-DS / DAPO). DAPO (Yu et al., 2025) addresses zero-variance prompts by repeatedly sampling questions, generating groups of responses, discarding zero-variance cases, and constructing batches containing only non-zero-variance prompts. This ensures that every prompt in the batch contributes a non-zero gradient signal. The paper's evaluation (Section 4.2) reveals that this approach has a severe hidden cost: when the policy model initially solves only ~2% of problems per batch (as in the large-scale experiments with DAPO-Math-17k), the dynamic sampling procedure must over-sample approximately 50× to fill a single non-zero-variance batch. As shown in Table 1, GRPO-DS-r (same rollout budget) achieves only 32 gradient steps compared to 1,056 for GRPO, because so many rollouts are wasted on discarded zero-variance prompts. Even GRPO-DS-g — given 5.29× more rollouts to match the gradient step count — still underperforms RL-ZVP on the majority of benchmarks (Table 2).
This reveals a fundamental tension in dynamic sampling: when the model is weak, zero-variance prompts are most prevalent, but the oversampling required to avoid them is most expensive. The approach essentially trades computational waste (discarded rollouts) for gradient sparsity (fewer gradient updates), and the paper shows this is not a favorable tradeoff.
Pre-rollout filtering via difficulty prediction (GRESO). GRESO (Zheng et al., 2025) attempts to avoid the cost of generating doomed rollouts by predicting, before sampling, whether a prompt is likely to be zero-variance. It uses a probabilistic filtering strategy based on historical training dynamics — if a prompt has been consistently all-correct or all-incorrect in recent training iterations, it is downweighted or skipped. This saves the computational cost of generating rollouts for predicted zero-variance prompts. However, the paper's results (Table 2, Figure 4) show that GRESO-r (same rollout budget) and GRESO-g (same gradient steps, 1.58×–3.99× more rollouts) both consistently underperform RL-ZVP across all six benchmarks and both model scales. More critically, GRESO-r on the large scale suffers from the same gradient sparsity problem as GRPO-DS-r: the policy solves so few problems initially that even though GRESO avoids wasting rollouts, it also avoids taking gradient steps.
Implicit discarding in standard GRPO. The base GRPO algorithm (Shao et al., 2024) does not explicitly filter zero-variance prompts, but its advantage normalization formula (Equation 1) implicitly discards them: when , the advantage is undefined or collapses to zero, and the objective becomes (Remark 1, Section 3.1). This is the default behavior in most RLVR implementations and represents the status quo the paper challenges.
The Conceptual Gap: Why No One Tried to Use Zero-Variance Prompts
The paper argues that the field's response to zero-variance prompts has been shaped by a specific conceptual limitation: the assumption that reinforcement requires contrast. In GRPO, a correct response is only rewarded if there exists at least one incorrect response in the same group to contrast against; an incorrect response is only penalized if there is at least one correct response. This is built into the advantage formula (Equation 1): the mean-normalization ensures that the sign of the advantage for any response depends on its reward relative to the group average, not its absolute correctness.
From this perspective, zero-variance prompts truly seem useless: without contrast within the group, how can the model know what to reinforce or suppress? The implicit answer in GRPO and its derivatives is: it can't. The paper's central conceptual contribution is to challenge this premise directly: the policy model should be rewarded for correct answers and penalized for incorrect ones based solely on their correctness, even when no contrasting answers exist in the same group.
This reframing connects to a broader intuition about learning from feedback: if a student solves a problem correctly, the teacher should reinforce that behavior regardless of whether other students also solved it correctly. If a student makes a mistake, the teacher should provide corrective feedback regardless of whether other students also made mistakes. Contrast provides additional information (how much better/worse this response is than alternatives), but correctness alone — independent of contrast — still provides useful directional signal.
How This Paper Positions Itself
The paper positions RL-ZVP not as a replacement for GRPO, but as a generalization of it (Section 3.3, Remark 3). Specifically, GRPO's objective can be expressed in a branching form (Equation 7) where it evaluates to zero for zero-variance prompts. RL-ZVP replaces that zero with a non-trivial advantage formulation while preserving identical behavior on non-zero-variance prompts. This is framed explicitly: "RL-ZVP behaves identically to GRPO on non-zero-variance prompts, but adopts a different strategy for zero-variance ones, thereby generalizing GRPO" (Section 3.2).
This positioning is strategic because it means RL-ZVP introduces no new loss terms or hyperparameters into the GRPO framework — it only changes what happens in the special case where advantage would otherwise vanish. This makes it conceptually simple to integrate into existing training pipelines (the paper implements it in the verl framework alongside GRPO) and ensures that any benefits come purely from the zero-variance prompt handling, not from unrelated algorithmic changes.
The paper also positions itself within a broader line of work on fine-grained reward shaping (Section 5). Recent methods have explored token-level adjustments to advantage based on token roles, sentiment, or other signals (Cui et al., 2025; Agarwal et al., 2025; Wang et al., 2025), motivated by the observation that GRPO's sample-level advantage broadcasting assigns the same weight to every token regardless of its informativeness. These methods, however, still suffer from advantage vanishing on zero-variance prompts — if the sample-level advantage is zero, multiplying it by token-level factors still yields zero. RL-ZVP addresses this limitation directly by providing a non-zero base advantage for zero-variance prompts, which is then modulated by token-level entropy. In this sense, RL-ZVP is compatible with and complementary to existing fine-grained reward shaping approaches: it provides the base signal that those methods can then further refine.
The Missing Piece: An Appropriate Strategy for Extracting Signal
The paper's key insight — and what distinguishes it from a naive "just reward correct and penalize incorrect" approach — is that the magnitude of reward or penalty must be appropriately scaled, not just its direction. The ablation in Table 3 confirms this: removing the entropy scaling factor (using only sample-level +1/-1 as advantage for zero-variance prompts) causes the most severe performance degradation, even compared to removing positive or negative prompts entirely. This means that simply assigning uniform positive/negative advantages to zero-variance prompts is insufficient; the quality of the learning signal depends critically on how the advantage magnitude is modulated.
The paper's choice of token-level entropy as the modulation mechanism is motivated by two complementary intuitions (Section 3.2):
-
For correct responses: High-entropy tokens — those where the model was uncertain across many possible next tokens — are typically crucial for reasoning (connectors, branching points, strategic decisions). These deserve larger reinforcement to encourage sophisticated reasoning behaviors like reflection and verification. Low-entropy tokens — predictable text completions or trivial continuations — receive smaller updates to avoid over-exploitation of surface patterns.
-
For incorrect responses: High-entropy tokens that initiated promising but ultimately failed reasoning paths should be penalized less severely, preserving flexibility for the model to revisit those paths in future iterations. Low-entropy tokens — repetitive mistakes or confidently wrong predictions — should be penalized more heavily to discourage their recurrence.
This asymmetric design (scaling up for correct, scaling inversely for incorrect) is intentional and empirically validated: the ablation in Appendix C.2 shows that a symmetric variant (using identical entropy scaling for both positive and negative prompts) performs worse than both the asymmetric RL-ZVP and even the GRPO baseline.
The paper thus positions entropy not merely as a curiosity or diagnostic metric, but as a first-class signal for advantage shaping that captures the informativeness and strategic importance of individual tokens in the reasoning process. This connects to broader observations in the LLM reasoning literature about the emergence of certain token patterns (like self-reflection, verification, backtracking) being associated with successful reasoning trajectories (Guo et al., 2025; Jin et al., 2024). By amplifying the gradient signal at these tokens, RL-ZVP explicitly encourages the policy model to develop and strengthen these beneficial reasoning behaviors.
The Scope of the Contribution
The paper is careful about the scope of its claims. It states explicitly that this is "the first work to demonstrate that zero-variance prompts can be exploited rather than discarded" (Section 6). The contribution is not a fundamentally new RL algorithm, but rather a perspective shift: zero-variance prompts are not useless computational waste; they are an untapped source of learning signal that, with the right advantage formulation, can meaningfully improve policy optimization.
This perspective shift has practical and conceptual implications:
- Practically, it means training pipelines can achieve better performance with the same compute budget by extracting signal from cases that were previously discarded.
- Conceptually, it means that contrast (relative reward within a group) is a sufficient condition for effective advantage, but not a necessary one — absolute reward, appropriately modulated, can also drive effective learning.
3. Technical Approach
3.1 Reader Orientation
This paper introduces RL-ZVP, a modified policy optimization algorithm that extends GRPO to extract learning signal from prompts where all sampled responses within a group receive identical rewards — cases where standard GRPO produces zero gradient update. The system solves the advantage vanishing problem by substituting GRPO's zero advantage for zero-variance prompts with an entropy-guided advantage shaping formula that assigns non-zero, token-level advantages based on response correctness and per-token prediction uncertainty, thereby converting previously wasted computation into meaningful training signal.
3.2 Big-Picture Architecture (Diagram in Words)
The RL-ZVP training pipeline has five major components operating within a standard RLVR loop:
-
Policy Model (
$\pi_\theta$) — the LLM being trained (e.g., Qwen3-1.7B-Base or Qwen3-8B-Base). It generates groups of responses to prompts from a training dataset at each iteration. -
Reward Verifier — a deterministic function (typically string-matching against ground-truth answers for math problems) that assigns binary rewards (
$R_i \in \{0, 1\}$) to each generated response based on correctness. -
Zero-Variance Detector — a simple statistical check that computes the standard deviation of rewards within a group; if
$\text{std}(\{R_i\}_{i=1}^G) = 0$, the prompt is flagged as zero-variance and the group is further classified as positive (all$R_i > 0$) or negative (all$R_i = 0$). -
Entropy Computer — for each token in each response, computes the entropy of the policy model's predictive distribution over the vocabulary
$V$at that position:$H_{i,t} = -\sum_{j=1}^{|V|} \pi_\theta(v_j | q, o_{i,<t}) \log \pi_\theta(v_j | q, o_{i,<t})$. The entropy values are detached from the computational graph (they serve as scalar scaling factors, not differentiable parameters). -
Advantage Shaper — for zero-variance prompts, constructs token-level advantages
$\hat{A}_{\text{ZVP}}(o_{i,t})$using the formula in Equation (5) that combines correctness direction (positive/negative) with entropy magnitude. For non-zero-variance prompts, it falls back to standard GRPO advantage normalization (Equation 1). The resulting advantages then flow into the clipped policy gradient objective (Equation 6).
Information flows as follows: at each iteration, the policy model generates $G$ rollouts per prompt → the reward verifier scores each rollout → the zero-variance detector tests whether $\text{std}(\{R_i\}) = 0$ → if yes, the entropy computer computes per-token entropies and the advantage shaper produces $\hat{A}_{\text{ZVP}}$; if no, standard GRPO advantages are computed → both advantage types feed into the same GRPO-style clipped objective → gradient updates are applied to the policy model.
3.3 Roadmap for the Deep Dive
- First, the GRPO objective and its breakdown for zero-variance prompts — establishing the mathematical baseline and why advantage vanishing occurs.
- Second, the entropy computation (
$H_{i,t}$) — how it is calculated, why entropy captures token informativeness, and what it measures in the policy model's behavior. - Third, the RL-ZVP advantage formulation (Equation 5) — how direction (sign) and magnitude (entropy-scaled value) are combined for positive and negative zero-variance prompts, including the asymmetric treatment.
- Fourth, the scaling factor
$\alpha$and its role — why unscaled entropy advantages lead to exploding gradients, and how the hyperparameter is tuned. - Fifth, the full RL-ZVP objective (Equation 6) — how it generalizes GRPO by branching on zero-variance detection rather than treating it as a separate loss.
- Sixth, the training algorithm (Algorithm 1) — the outer loop, the mini-batch structure, and where RL-ZVP diverges from standard GRPO training.
3.4 Detailed, Sentence-Based Technical Breakdown
This is primarily an algorithm design paper whose core idea is that the standard GRPO advantage normalization formula implicitly zeroes out learning signal for zero-variance prompts, and that by replacing this zero with an entropy-guided advantage formulation, the policy model can extract useful gradient information from previously wasted computation — improving both training efficiency (faster convergence, better use of rollout budget) and final reasoning performance (more sophisticated reasoning behaviors, longer chain-of-thought).
GRPO Objective Decomposition and the Advantage Vanishing Problem
The paper begins its technical development by examining the GRPO objective to establish precisely why zero-variance prompts produce no gradient update, and to identify the specific mathematical locus where intervention is possible.
GRPO's advantage normalization. For a single prompt $q$ with $G$ sampled responses $\{o_i\}_{i=1}^G$ and their corresponding binary rewards $\{R_i\}_{i=1}^G$ (where $R_i = 1$ if $o_i$ matches the ground-truth answer and $R_i = 0$ otherwise), GRPO computes the advantage for each response as:
where is the arithmetic average of the group rewards, and is the group standard deviation.
What it computes: the advantage is a z-score normalized reward — it measures how many standard deviations above or below the group mean each response's reward falls. If a response is correct and the group contains both correct and incorrect responses, its reward is above the mean and . If incorrect and the group is mixed, . This normalized advantage is then broadcast to all tokens in the response: .
Why z-score normalization: it serves two purposes. First, it eliminates the need for a learned value function (unlike PPO), since the group mean acts as an empirical baseline. Second, it provides a natural contrastive signal: responses better than average are reinforced, worse than average are suppressed. However, this normalization fundamentally requires within-group variance — the denominator must be non-zero, and the mean must differ from at least some rewards for any advantage to be non-zero.
The zero-variance case. When all $G$ responses receive the same reward (all 0 or all 1), the standard deviation is zero. Formally:
- If all responses are incorrect: for all , so , , and (or NaN, handled as zero in practice).
- If all responses are correct: for all , so , , and .
In both cases, the advantage collapses to zero for every token in every response.
The GRPO objective with zero-variance prompts. The standard GRPO objective for a single prompt is:
where is the importance sampling ratio comparing the current policy's probability of token to the old policy's probability (the policy that generated the rollouts), and is the clipping threshold (set to 0.2 with an asymmetric clip-higher scheme using and ).
What it computes: for each token in each response, the clipped surrogate objective multiplies the importance sampling ratio by the advantage. The
$\min$and$\text{clip}$ensure the policy does not move too far from the old policy in a single update step (standard PPO-style trust region). The objective is averaged over all tokens in all responses.Why this form with broadcasting: GRPO assigns the same advantage to every token in a response because the reward signal is at the response level (the verifier only checks the final answer). This means every token in a correct response is reinforced equally (proportional to the advantage), and every token in an incorrect response is penalised equally. The paper notes that this coarse-grained approach "motivated research into fine-grained reward shaping" — techniques that adjust token-level advantages based on per-token characteristics rather than simply broadcasting the sample-level advantage.
When advantage vanishes, the objective vanishes. Substituting for all into the GRPO objective yields:
and consequently the gradient — no parameter update occurs for that prompt. The paper expresses this as a branching formulation (Equation 7), showing that GRPO's behavior on zero-variance prompts is functionally equivalent to a branch that returns zero:
The gradient interpretation. The paper takes the gradient of the GRPO objective (omitting the clipping factor for clarity) to show the role of advantage in parameter updates:
where is the gradient of the log-probability of token under the current policy.
What this gradient expression means in operational terms: the update to the policy parameters is a weighted sum of per-token log-probability gradients. Each token's gradient points in the direction that would increase the probability of that token under the policy. The weight is the product of the importance sampling ratio and the advantage . When , the token's probability is increased (reinforcement); when , it is decreased (suppression). The magnitude of the advantage determines the step size in that direction.
Why this decomposition matters: it reveals that for zero-variance prompts, the gradient is identically zero regardless of , because zeroes out every term. The importance sampling ratio, which could theoretically be non-trivial (since the old policy that generated the rollout differs from the current policy after previous updates), cannot rescue the update — anything multiplied by zero is zero. The paper calls this out explicitly (Section 3.1): "if we ignore the importance-sampling ratio, each token in a response receives the same weight, regardless of its role or informativeness."
This establishes the baseline behavior that RL-ZVP aims to change: for non-zero-variance prompts, GRPO works as intended (contrastive signal); for zero-variance prompts, it produces zero update. RL-ZVP preserves the former case and replaces the zero-update case with a non-trivial advantage formulation.
Entropy Computation as a Token-Level Informativeness Measure
The central mechanism in RL-ZVP's advantage formulation is token-level entropy from the policy model's output distribution. The paper motivates this choice through connections to the reasoning behavior of LLMs and prior work on fine-grained reward shaping.
Definition. For a token at position $t$ in response $o_i$, given the prompt $q$ and preceding tokens $o_{i,<t}$, the entropy $H_{i,t}$ is:
where $V = \{v_1, v_2, \ldots, v_{|V|}\}$ is the vocabulary (the set of all possible tokens the model can generate), and $\pi_\theta(v_j \mid q, o_{i,<t})$ is the probability the policy model assigns to token $v_j$ at position $t$, conditioned on the prompt and preceding tokens.
What it computes: for each token position, the Shannon entropy of the model's predictive distribution over the entire vocabulary. If the model is highly certain (probability mass concentrated on one or a few tokens), entropy is low. If the model is uncertain (probability spread across many plausible next tokens), entropy is high. The computation sums over all
$|V|$vocabulary items, which is computationally intensive but feasible because modern LLM frameworks compute$\log \pi_\theta(v \mid \cdot)$for all$v \in V$simultaneously as part of the forward pass (via the softmax output layer).Why entropy rather than probability of the chosen token: the probability of the specific token
$o_{i,t}$that was actually sampled only captures how confident the model was in that particular choice. Entropy captures whether other plausible alternatives existed. A token could have high probability (e.g., 0.7) but low entropy if only one other token is remotely plausible; or it could have the same probability but high entropy if probability is spread across many alternatives. The paper argues that for reasoning, the existence of plausible alternatives — branching points in the chain-of-thought — is more informative than the absolute probability of the chosen path.
Implementation detail: gradient detachment. The paper explicitly notes that "we detach every occurrence of $H_{i,t}$ from the computational graph so that it acts as a scalar factor rather than a differentiable parameter" (Section 3.2, footnote 3). This means:
- The entropy values are computed using the current policy's probabilities, but the computation is wrapped in a stop-gradient operation (e.g.,
.detach()in PyTorch ortf.stop_gradient()in TensorFlow). - When the advantage
$\hat{A}_{\text{ZVP}}(o_{i,t})$is multiplied by the importance sampling ratio$r_{i,t}(\theta)$in the objective, gradients flow only through$r_{i,t}(\theta)$and the log-probability gradient$\nabla_\theta \log \pi_\theta(o_{i,t})$, not through the entropy calculation. - Why detachment matters: if gradients were allowed to flow through the entropy computation, the policy model could learn to artificially manipulate its own entropy values to maximize the advantage (an form of reward hacking where the model increases its entropy at tokens it "wants" to be reinforced). Detaching prevents this feedback loop and ensures the entropy serves purely as a scaling coefficient determined by the model's current state.
Intuition: why high-entropy tokens matter for reasoning. The paper connects token entropy to reasoning behavior through several observations from the LLM post-training literature (Section 3.2, Section 4.2):
-
Connectors and strategic words — tokens like "therefore," "however," "alternatively," "wait," "let's double-check" — are typically high-entropy because many plausible continuations could follow. These tokens mark transitions in reasoning, strategic shifts, or metacognitive behaviors (reflection, verification, backtracking). Wang et al. (2025) specifically identify "high-entropy minority tokens" as critical drivers of effective RL for reasoning.
-
Branching points — when the model must decide between multiple solution strategies (e.g., "we can solve this by factoring" vs. "alternatively, use the quadratic formula"), the tokens that initiate these branches tend to have high entropy. Reinforcing these branching tokens encourages the model to explore diverse reasoning paths.
-
Trivial completions — tokens in formulaic phrases ("the answer is," "therefore," in boilerplate contexts) or predictable numeric computations tend to have low entropy because the model is certain about what follows. Reinforcing these tokens heavily would overfit to surface patterns rather than reasoning quality.
For incorrect responses, the intuition reverses:
- High-entropy tokens that initiated promising-but-failed reasoning paths (e.g., a strategic choice that led to a dead end) should be penalized less severely to preserve flexibility — the model should be willing to try that strategy again in future iterations, but adjust the subsequent steps.
- Low-entropy tokens that represent confident, repetitive mistakes (e.g., consistently misapplying a formula with high confidence) should be penalized more heavily to discourage their recurrence.
Entropy ranges and scaling considerations. The paper provides partial insight into entropy magnitudes through the $\alpha$ hyperparameter tuning (Section 4.3, Table 4): "at the beginning of training, the top 20% of tokens with the highest entropy often take values between 2 and 10." This suggests that token entropies in the policy model's vocabulary space can vary widely — from near-zero (deterministic choices) to values around 10 nats (very uncertain), depending on vocabulary size and model calibration. This wide range motivates the need for a scaling factor $\alpha$ to prevent advantage values from becoming unreasonably large.
RL-ZVP Advantage Formulation
The core technical contribution of the paper is the advantage formula for zero-variance prompts (Equation 5), which replaces the zero advantage of GRPO with a token-level, entropy-scaled value. This formulation has two independent design axes: direction (sign of the advantage, determined by response correctness) and magnitude (absolute value, determined by token entropy with a scaling hyperparameter).
The full formula:
where $\alpha$ is the scaling factor hyperparameter (set to 0.10 in all main experiments), $H_{i,t}$ is the per-token entropy defined above, and $\max_{k=1}^{|o_i|}(H_{i,k})$ is the maximum entropy value across all tokens in response $o_i$.
What it computes for positive prompts (all correct): for each token in a correct response, the advantage is
$\alpha H_{i,t}$— a positive value proportional to the token's entropy. High-entropy tokens get larger positive advantages (stronger reinforcement), low-entropy tokens get smaller positive advantages (weaker reinforcement). All tokens are reinforced (probability increased), but the degree of reinforcement varies by token informativeness.What it computes for negative prompts (all incorrect): for each token in an incorrect response, the advantage is
$-\alpha(\max_k(H_{i,k}) - H_{i,t})$— a negative value whose magnitude is inversely proportional to the token's entropy. High-entropy tokens receive smaller-magnitude negative advantages (weaker penalty), low-entropy tokens receive larger-magnitude negative advantages (stronger penalty). The inversion via$\max_k(H_{i,k}) - H_{i,t}$ensures that the highest-entropy token in the response receives the smallest penalty magnitude (close to zero if$H_{i,t}$is near the maximum), while the lowest-entropy token receives the largest penalty magnitude.Why this form rather than uniform
$\pm\alpha$: the ablation in Table 3 demonstrates that "removing the entropy scaling factor (e.g., using only sample-level$+1/-1$as the advantage) causes the most severe degradation" — performance drops from an average Acc@8 of 49.90 to 46.88, and this variant even underperforms GRPO on some benchmarks. This confirms that the direction alone (reinforce correct, penalize incorrect) is insufficient; the magnitude must be modulated appropriately for the learning signal to be effective.
The asymmetric treatment of positive vs. negative prompts. The paper's design is deliberately asymmetric: correct responses use $+ \alpha H_{i,t}$ (entropy linearly scales the positive advantage), while incorrect responses use $-\alpha(\max_k(H_{i,k}) - H_{i,t})$ (entropy inversely scales the negative advantage magnitude). This asymmetry is motivated by different desiderata for reinforcing good behavior versus suppressing bad behavior:
-
For reinforcement (positive): the goal is to strengthen the model's confidence in good reasoning patterns. High-entropy tokens — the uncertain, decision-critical ones — deserve the strongest reinforcement because they represent the non-trivial reasoning choices the model made correctly. Low-entropy tokens — the predictable parts — need only modest reinforcement to maintain their probability.
-
For suppression (negative): the goal is to discourage mistakes while preserving the model's ability to explore similar reasoning paths in the future. High-entropy tokens that initiated promising but failed strategies should be penalized lightly (small magnitude advantage) so the model remains willing to try those branching points again. Low-entropy tokens representing confident but wrong choices should be penalized more heavily (large magnitude) to strongly discourage their repetition.
The paper validates this asymmetry through an ablation in Appendix C.2 (Table 10), which tests a "symmetric" variant where negative prompts use $-\alpha H_{i,t}$ (the exact negative of the positive prompt formula). The symmetric variant achieves an average Acc@8 of 46.64 — worse than both asymmetric RL-ZVP (49.90) and even the GRPO baseline (46.79). This empirically confirms that "a symmetric penalty is counterproductive in this setting" and justifies the inverse-entropy design.
The max-normalization in negative prompts. The term $\max_{k=1}^{|o_i|}(H_{i,k})$ — the maximum entropy across all tokens in the response — serves as a normalization constant within each response. By subtracting the token's entropy from this maximum, the formula ensures that the most uncertain token in the response receives the smallest-magnitude penalty (close to zero), while more certain tokens receive progressively larger penalties.
Why max-normalization rather than mean-normalization or no normalization: the maximum provides a natural upper bound for inversion — it guarantees that the penalty magnitude is always non-negative (since
$\max_k(H_{i,k}) - H_{i,t} \geq 0$for all tokens) and that at least one token (the maximum-entropy one) receives near-zero penalty. Mean-normalization would center the penalty around the average entropy, which doesn't provide the same interpretable property of "the most uncertain token is penalized least." Without normalization,$-\alpha H_{i,t}$would penalize high-entropy tokens more heavily (the opposite of the intended design), as confirmed by the symmetric ablation's poor performance.
The per-response computation of $\max_k(H_{i,k})$. The maximum is computed independently for each response within the group, not globally across all responses. This means two incorrect responses with different entropy profiles will have different maximum values, and the penalty scaling is response-specific. This is consistent with the paper's view that different incorrect responses may exhibit different reasoning patterns — some may be "almost correct" with high entropy at strategic junctures, while others may be "confidently wrong" with uniformly low entropy.
Relationship to GRPO's advantage normalization. The paper emphasizes (Section 3.2) that RL-ZVP's advantage "behaves identically to GRPO on non-zero-variance prompts." The zero-variance formula is only invoked when $\text{std}(\{R_i\}_{i=1}^G) = 0$, which is precisely the condition under which GRPO's advantage would be zero (or undefined). This conditional branching means:
- For prompts with mixed rewards (some correct, some incorrect responses in the group), the standard GRPO advantage
$\hat{A}_i = \frac{R_i - \text{mean}}{\text{std}}$is computed and used directly, with its contrastive properties intact. - For prompts with uniform rewards, RL-ZVP's entropy-guided formula substitutes for the zero advantage that GRPO would produce.
The paper explicitly frames this as a generalization of GRPO (Remark 3): "our formulation serves as a generalization of GRPO — reducing exactly to GRPO when prompts are non-zero-variance, while resulting in a non-trivial update for zero-variance prompts rather than collapsing to zero."
Advantage broadcasting (or not). A key distinction from GRPO is that RL-ZVP's advantage for zero-variance prompts is token-level: $\hat{A}_{\text{ZVP}}(o_{i,t})$ varies across tokens within the same response based on their entropy. In GRPO, the advantage is response-level and broadcast uniformly to all tokens. In RL-ZVP's non-zero-variance branch, the standard GRPO broadcasting still applies. In the zero-variance branch, each token receives a potentially different advantage value, introducing fine-grained per-token signal.
This token-level variation is what enables the differentiated treatment of reasoning-critical vs. trivial tokens: the reinforcement magnitude for a correct response is not uniform — the model receives stronger gradient updates at the tokens where it was uncertain, directly encouraging the development of sophisticated reasoning behaviors at those decision points.
The Scaling Factor $\alpha$ and Gradient Stability
The entropy values for tokens in a language model's vocabulary space can range widely — the paper reports that high-entropy tokens "often take values between 2 and 10" at the start of training (Section 4.3). If used directly as advantages, these values would produce gradient updates 2–10× larger than typical GRPO advantages (which are typically in the range $[-1, 1]$ after z-score normalization, depending on group size and reward distribution). This would cause instability and potential training collapse.
The role of $\alpha$. The scaling factor $\alpha$ is multiplied into the entropy-derived advantage to bring its magnitude into a range comparable to GRPO's normalized advantages. The paper sweeps values of $\alpha \in \{0.05, 0.10, 0.20, 0.30\}$ on the small-scale setup and also tests a wider range $\alpha \in \{0, 0.001, 0.01, 0.10, 1.0, 10.0\}$ in Appendix C.1 (Table 9).
What
$\alpha$controls: it linearly scales the magnitude of the advantage signal from zero-variance prompts. When$\alpha = 0$, RL-ZVP reduces exactly to GRPO (the zero-variance branch produces zero advantage). When$\alpha$is very small (0.001, 0.01), the signal from zero-variance prompts is negligible, and performance is only marginally better than GRPO. When$\alpha$is too large (1.0, 10.0), the advantages dominate the gradient updates and destabilize training — Table 9 shows Acc@8 dropping from 49.90 ($\alpha = 0.1$) to 44.32 ($\alpha = 1.0$) and 39.96 ($\alpha = 10.0$).Why
$\alpha = 0.10$is the empirical optimum: the paper finds that 0.10 and 0.20 both work well (49.90 and 48.15 average Acc@8 respectively), with 0.10 yielding "the most consistent and best performance" (Table 4). This value places the typical advantage from a high-entropy token (e.g.,$H = 5$) at$\alpha H = 0.5$, which is comparable to the magnitude of GRPO advantages in mixed groups (where z-scores of$\pm 1-2$are common). The range of$\alpha H$values thus roughly matches the scale of the existing GRPO advantage landscape, allowing zero-variance prompts to contribute gradient signal without overwhelming the contrastive signal from non-zero-variance prompts.
Gradient explosion risk without $\alpha$. The paper notes (Section 4.3) that "at the beginning of training, the top 20% of tokens with the highest entropy often take values between 2 and 10, which can result in very large advantages if left unscaled." If these raw entropy values were used as advantages, the gradient update for those tokens would be 2–10× larger than typical, potentially causing:
- Loss spikes: the clipped objective would clip the importance sampling ratio, but the advantage magnitude itself is outside the clipping mechanism — large advantages directly produce large gradients, potentially causing parameter updates that violate the trust region intent of PPO-style clipping.
- Policy collapse: the model could rapidly shift probability mass toward or away from certain tokens, disrupting learned behaviors and potentially leading to degenerate outputs (repetition, mode collapse).
- Unstable training dynamics: the paper's large-scale results (Figure 5d) show GRPO suffering a sharp accuracy drop in early training, which the authors attribute to sparse/noisy feedback. Overly large advantages would exacerbate such instability.
The $\alpha$ hyperparameter thus serves as a stability knob: it controls how aggressively RL-ZVP exploits zero-variance prompts, with larger values extracting more signal at the risk of destabilization.
Tuning strategy. The paper empirically tunes $\alpha$ via grid search on the small-scale setup (Qwen3-1.7B-Base on MATH), then applies the best-found value ($\alpha = 0.10$) to all experiments including the large-scale setup (Qwen3-8B-Base on DAPO-Math-17k) and the alternative training data experiments (NuminaMath-CoT, Appendix B.1). This suggests that $\alpha$ is relatively robust to model scale and dataset choice, though the paper does not systematically verify this via a full sweep on the large-scale setting (likely due to computational constraints, as noted in Section 6).
The RL-ZVP Objective and Its Relationship to GRPO
The final objective function (Equation 6) combines the standard GRPO formulation for non-zero-variance prompts with the entropy-guided advantage for zero-variance prompts through a conditional branching:
where $\mathcal{D}_{\text{ZVP}}$ is the set of zero-variance prompts encountered during training (those where $\text{std}(\{R_i\}_{i=1}^G) = 0$).
What it computes: for each prompt in a training batch, the objective checks whether the group of responses has zero reward variance. If it does, RL-ZVP's entropy-guided advantage
$\hat{A}_{\text{ZVP}}(o_{i,t})$replaces the zero advantage that GRPO would use. The remainder of the objective — the importance sampling ratio, the clipping mechanism, the averaging over responses and tokens — is identical to GRPO. If the prompt is non-zero-variance, the standard GRPO objective is computed unchanged.Why this conditional form rather than a separate loss term: the paper explicitly states (Remark 3) that "RL-ZVP does not introduce any new loss term beyond the GRPO objective. It simply reformulates the special case where the prompt is zero-variance." This design choice is deliberate: it ensures that RL-ZVP's behavior is identical to GRPO in all cases where GRPO already works well (non-zero-variance prompts), and only changes behavior in the degenerate case. This makes the algorithm easy to integrate — it is a drop-in replacement for the advantage computation step in existing GRPO implementations, with no changes to the optimizer, learning rate schedule, or other hyperparameters.
The clipping mechanism remains in effect. The $\min$ and $\text{clip}$ operations in the objective serve the same trust-region purpose as in GRPO: they prevent the policy from changing too much in a single update step relative to the policy that generated the rollouts. For RL-ZVP's zero-variance branch:
-
If
$\hat{A}_{\text{ZVP}}(o_{i,t}) > 0$(positive prompt, correct response): the objective encourages increasing the probability of token$o_{i,t}$. The clipping limits how much the probability can increase: if$r_{i,t}(\theta) > 1 + \epsilon$(the current policy already assigns much higher probability than the old policy), the clipped term$\text{clip}(r_{i,t}(\theta), 1-\epsilon, 1+\epsilon) \hat{A}_{\text{ZVP}} = (1+\epsilon) \hat{A}_{\text{ZVP}}$is used, capping the reinforcement. -
If
$\hat{A}_{\text{ZVP}}(o_{i,t}) < 0$(negative prompt, incorrect response): the objective encourages decreasing the probability of token$o_{i,t}$. The clipping limits how much the probability can decrease: if$r_{i,t}(\theta) < 1 - \epsilon$(the current policy already assigns much lower probability than the old policy), the clipped term$(1-\epsilon) \hat{A}_{\text{ZVP}}$is used, capping the suppression.
The paper uses an asymmetric clipping scheme: $\epsilon_{\text{low}} = 0.2$ and $\epsilon_{\text{high}} = 0.28$ (Appendix A, Table 5). This "clip-higher" scheme, adapted from DAPO (Yu et al., 2025), allows slightly larger increases in probability (for beneficial tokens) than decreases (for harmful tokens), providing an asymmetric trust region that favors exploration over suppression.
The KL penalty is omitted. The paper's implementation follows recent RLVR work in excluding the KL divergence penalty term from the objective, unlike the original PPO formulation which includes a KL penalty to prevent the policy from deviating too far from a reference model. The clipping mechanism alone is relied upon for trust-region enforcement. This is consistent with GRPO implementations in the verl framework and aligned with the experimental setups of DAPO and GRESO.
Full-dataset objective. The per-prompt objective is averaged over the training dataset to form the overall training objective:
This expectation is approximated by Monte Carlo sampling: at each training iteration, a batch of prompts is sampled from the dataset, rollouts are generated from the old policy, and the per-prompt objectives are averaged to form the batch loss.
The importance of the old policy $\pi_{\theta_{\text{old}}}.** Like GRPO, RL-ZVP uses the policy from the previous training iteration as the reference policy for importance sampling. After each iteration (which consists of multiple gradient steps on mini-batches), the old policy is updated: $\pi_{\theta_{\text{old}}} \leftarrow \pi_\theta$. This means that the importance sampling ratio $r_{i,t}(\theta)$ starts at 1.0 for all tokens at the beginning of each iteration and deviates as gradient updates are applied. This periodic re-anchoring to the current policy is standard in online RL for LLMs and distinguishes it from offline RL methods that use a fixed reference model.
The Training Algorithm
The paper presents the RL-ZVP training procedure in Algorithm 1, which follows the structure of standard GRPO training with the zero-variance advantage formulation inserted at the appropriate point.
Algorithm walkthrough:
-
Initialization. The algorithm takes as input an initial policy model
$\pi_\theta$(a pretrained base model like Qwen3-1.7B-Base or Qwen3-8B-Base) and a training dataset$\mathcal{D}$(MATH, DAPO-Math-17k, or NuminaMath-CoT). -
Outer loop (iterations). For
$M$iterations (672 for small scale, 1056 for large scale):a. Update old policy. The previous policy is set to the current policy:
$\pi_{\theta_{\text{old}}} \leftarrow \pi_\theta$. This is the reference against which importance sampling ratios will be computed.b. Sample prompt batch. A batch
$\mathcal{B} \subset \mathcal{D}$of prompts is sampled without replacement from the dataset. The batch size is 512 prompts (Table 5).c. Generate rollouts. For each prompt
$(q, a)$in the batch,$G$responses are sampled from the old policy:$\{o_i\}_{i=1}^G \sim \pi_{\theta_{\text{old}}}(\cdot \mid q)$. For the small-scale setup,$G = 8$; for the large-scale setup,$G = 16$. Rollouts are generated at temperature 1.0 with top-p = 1.0 (no nucleus filtering), encouraging diverse exploration.d. Compute rewards. Each response
$o_i$is verified against the ground-truth answer$a$to produce a binary reward$R_i$(using the MATH grading function from Lightman et al., 2024, which handles equivalent mathematical expressions).e. Inner loop (gradient steps). For
$\mu$gradient steps per iteration (16 gradient steps, since the batch of 512 is split into mini-batches of 32):i. Sample mini-batch. A mini-batch
$\mathcal{B}_{\text{mini}} \subset \mathcal{B}$of prompts is sampled without replacement (32 prompts per mini-batch).ii. Compute advantages. For each prompt in the mini-batch: - If
$\text{std}(\{R_i\}_{i=1}^G) > 0$(non-zero-variance): compute advantages$\{\hat{A}_{i,t}\}$using the standard GRPO formula (Equation 1) with z-score normalization. - If$\text{std}(\{R_i\}_{i=1}^G) = 0$(zero-variance): compute advantages$\{\hat{A}_{\text{ZVP}}(o_{i,t})\}$using the entropy-guided formula (Equation 5).iii. Update policy. Maximize the RL-ZVP objective (Equation 6) by taking a gradient step with the AdamW optimizer.
-
Output. After
$M$iterations, the final policy model$\pi_\theta$is returned.
Key implementation details from Appendix A (Table 5):
- Optimizer: AdamW with learning rate
$3 \times 10^{-6}$(small scale) or$1 \times 10^{-6}$(large scale), with 10 warmup steps and either a cosine decay schedule (small scale) or constant schedule (large scale). - Number of epochs: 3 epochs over the training data for small scale, 2 epochs for large scale.
- Max response length: 4096 tokens for small scale (Qwen3-1.7B-Base on MATH), 8192 tokens for large scale (Qwen3-8B-Base on DAPO-Math-17k).
- Validation temperature: 1.0 with top-p = 0.7 (different from training top-p of 1.0, providing a more focused sampling for evaluation).
- Training temperature: 1.0 with top-p = 1.0 (maximum diversity for exploration).
Where RL-ZVP diverges from standard GRPO. The only algorithmic change relative to a standard GRPO training loop is step 2(e)ii: the conditional branching that checks for zero-variance prompts and, if detected, substitutes the entropy-guided advantage formula for the z-score normalization that would otherwise produce zero. All other components — the outer loop structure, the generation process, the reward computation, the clipping mechanism, the mini-batch SGD, the optimizer, and the learning rate schedule — remain identical.
Computational overhead of RL-ZVP vs. GRPO. The paper does not explicitly quantify the additional computation cost of the entropy calculation, but it can be inferred:
- The entropy
$H_{i,t}$requires computing the full softmax distribution over the vocabulary$V$at each token position. This is already computed during the forward pass of the policy model (it is a byproduct of generating log-probabilities for the importance sampling ratio), so the additional cost is primarily the summation$\sum_{j=1}^{|V|} \pi_\theta(v_j) \log \pi_\theta(v_j)$. - For a vocabulary size of ~150,000 tokens (typical for modern LLMs), this summation is a vectorized operation that adds negligible overhead relative to the forward/backward passes through the transformer.
- The zero-variance detection (
$\text{std}(\{R_i\})$) is a trivial computation over$G$binary values. - The paper's Table 1 shows that RL-ZVP uses exactly the same number of rollouts and gradient steps as GRPO (172,032 rollouts and 672 gradient steps at small scale; 540,672 rollouts and 1,056 gradient steps at large scale). This confirms no additional sampling cost compared to GRPO — all rollouts generated are used for training, whether zero-variance or not.
In contrast, the prompt-filtering baselines (GRPO-DS and GRESO) require more rollouts to achieve the same number of gradient steps (Table 1: GRPO-DS-g uses 2.45×–5.29× more rollouts than GRPO/RL-ZVP), because many generated rollouts are discarded when they form zero-variance groups. RL-ZVP's computational advantage over these baselines comes from eliminating this wasted sampling — not from being faster per-gradient-step, but from achieving more effective gradient steps per rollout generated.
Design choice: why not separate zero-variance and non-zero-variance training objectives? The paper's branching formulation (Equation 6) handles both cases within a single objective function. An alternative design would be to treat zero-variance prompts with a separate loss term (e.g., a supervised fine-tuning loss on correct responses or a behavioral cloning loss on incorrect ones). The paper does not explore this design space, sticking instead to the principle that RL-ZVP should be a minimal modification to GRPO — changing only what happens in the degenerate zero-advantage case while preserving the standard GRPO behavior elsewhere. This principle is stated explicitly in Remark 3 and ensures compatibility with existing RLVR infrastructure (the verl framework, standard hyperparameters, etc.).
4. Key Insights and Innovations
Innovation 1: Reframing Zero-Variance Prompts from Computational Waste to Untapped Learning Signal
The paper's most fundamental contribution is not an algorithmic tweak but a conceptual reframing of what zero-variance prompts represent in RLVR training. Prior to this work, the field's implicit consensus — encoded directly in the GRPO advantage formula and made explicit in filtering methods like DAPO and GRESO — was that prompts producing uniform rewards within a group are information-free dead ends. The reasoning was simple: without within-group contrast, there is no relative ranking, and without ranking, there is no reinforcement signal. This assumption is so baked into GRPO's design that the algorithm literally produces zero gradient update for these cases (Section 3.1, Remark 1).
The paper challenges this consensus at its root by asking: does reinforcement truly require contrast, or is the absolute correctness of a response sufficient for a useful directional signal? This is a conceptual move, not a technical one — it questions the necessity of a premise that the field had treated as self-evident. The answer the paper provides is that absolute reward is sufficient, but only if the magnitude of the resulting advantage is appropriately shaped. This distinguishes RL-ZVP from a naive "just use ±1 as advantage" approach, which the ablation in Table 3 shows performs worse than GRPO. The insight is that contrast-free signal can work, but it must be modulated by something that captures within-response informativeness variation — which is why the entropy mechanism (Section 3) matters, even though the core conceptual move stands independent of it.
The significance of this reframing extends beyond the specific algorithm. It opens a new axis for thinking about sample efficiency in RLVR: rather than asking "how do we avoid generating zero-variance prompts?" (the filtering paradigm), the new question becomes "how do we extract maximal signal from whatever prompts we generate?" This shift from avoidance to exploitation mirrors a broader pattern in machine learning — moving from discarding "hard" or "noisy" examples to finding ways to use them productively (e.g., curriculum learning vs. intelligent data augmentation). In the RLVR context, where Figure 2 shows that rollout generation dominates training time, maximizing the return on sampling investment is a higher-leverage strategy than minimizing sampling waste, because waste reduction is fundamentally capped (you can't get below zero waste) while signal extraction from previously zeroed-out cases has no theoretical ceiling.
Evidence for the power of this reinterpretation is the consistent outperformance of RL-ZVP over filtering baselines even when those baselines are given 3×–5× more rollouts (Table 2, Figure 4). If zero-variance prompts truly contained no useful signal, then GRPO-DS-g — which matches RL-ZVP's gradient step count but filters out all zero-variance cases — would perform equally well or better, since it sees only "informative" prompts. The fact that RL-ZVP wins (by an average 5.51 Acc@8 points on the large scale) demonstrates that the discarded prompts contained signal the filtering approach threw away.
This innovation is fundamental rather than incremental: it challenges a premise the field had treated as axiomatic, not just a hyperparameter or implementation detail of existing methods.
Innovation 2: Entropy as a First-Class Signal for Advantage Magnitude in LLM Fine-Tuning
While token-level entropy has been studied as a diagnostic metric in LLM training (tracking model uncertainty, detecting overfitting, monitoring exploration), and prior work on fine-grained reward shaping has used token-level features to modulate advantages (Cui et al., 2025; Wang et al., 2025), RL-ZVP's deployment of entropy is distinctive in two ways: it uses entropy as the sole basis for advantage magnitude in a regime where the sample-level advantage has vanished (rather than as an adjustment to an existing non-zero advantage), and it applies entropy asymmetrically between positive and negative prompts in a principled way grounded in the different desiderata of reinforcement vs. suppression.
The conceptual insight here is that entropy captures something beyond model uncertainty — it captures token-level strategic informativeness in reasoning trajectories. When a model is uncertain at a particular token position (high entropy), it typically means that token represents a genuine decision point — a branching moment where multiple plausible continuations exist. Reinforcing correct decisions at those moments (positive prompts) strengthens reasoning capability more than reinforcing predictable completions ("the answer is 42"). Conversely, penalizing incorrect decisions at high-entropy tokens too harshly (negative prompts) would discourage the model from exploring those branching points again, even though the branching point itself may have been reasonable — only the subsequent path was wrong.
This is not just a "use entropy as a feature" contribution. The paper provides empirical evidence that differentiating token-level treatment matters critically, and that doing it wrong (symmetrically) is worse than not doing it at all. The symmetric ablation in Appendix C.2 (Table 10) — where negative prompts use the same entropy scaling as positive prompts — underperforms even the GRPO baseline that discards zero-variance prompts entirely. This is a striking negative result: it shows that extracting the "wrong" signal from zero-variance prompts is actively harmful, which explains why the naive approach of "just reward correct and penalize incorrect uniformly" (the ablation in Table 3, "- w/o entropy scaling") also underperforms. The signal must be appropriately shaped to be useful; poorly shaped signal is worse than no signal.
This finding has implications for the broader fine-grained reward shaping literature. It suggests that the effectiveness of token-level modulation depends not just on the informativeness of the feature used, but on the alignment between the modulation strategy and the learning objective at that token type. For positive prompts, the objective is to increase probability (reinforcement), and higher entropy tokens naturally benefit more because they represent non-trivial decisions. For negative prompts, the objective is to decrease probability (suppression), but high-entropy tokens benefit less from suppression because they represent exploratory decisions that should remain available. The asymmetry in RL-ZVP's design is therefore not an arbitrary choice but a reflection of this alignment principle.
This innovation is fundamental in its diagnostic value (the symmetric ablation reveals a property of the learning dynamics that was not previously characterized) but incremental in its mechanism (it builds on existing fine-grained reward shaping ideas and applies them to a new regime). The combination — a new diagnostic finding plus a mechanism that capitalizes on it — makes it more than the sum of its parts.
Innovation 3: Empirical Characterization of the Quantity and Dynamics of Zero-Variance Prompts Across Training
The paper provides the first systematic quantification of how zero-variance prompts manifest across different model scales, dataset difficulties, and training stages (Figure 3, Figure 8, and Appendix E). While prior work had identified zero-variance prompts as a problem (Zheng et al., 2025; Yu et al., 2025), it had not characterized the scale or dynamics of the phenomenon — how many prompts are affected, when in training they appear, and how this varies with experimental setup.
The paper's empirical findings here are illuminating and counterintuitive:
-
On challenging datasets (DAPO-Math-17k, Figure 3), zero-variance prompts constitute ~99% of the batch at the start of training. This means that for the first hundred+ gradient steps, GRPO is effectively training on only 1% of its batch — the other 99% of generated rollouts are producing zero gradient. This explains the training instability observed in Figure 5d for GRPO on the large scale: with such sparse feedback, the policy receives noisy, high-variance gradient updates that cause the sharp accuracy drop.
-
The proportion of zero-variance prompts evolves over training in opposite directions depending on dataset difficulty. On hard datasets (DAPO-Math-17k), the ratio starts near 99% and declines as the model improves, but remains non-negligible throughout. On easier datasets (MATH, Figure 3), the ratio starts lower (~30%) but increases as the model converges and begins solving many questions entirely — a phase where GRPO would discard the signal from the model's best behavior.
-
The dataset difficulty dramatically affects zero-variance prevalence: training the same Qwen3-8B-Base on DAPO-Math-17k vs. NuminaMath-CoT (Figure 8) produces wildly different zero-variance ratios (~99% vs. ~30% at training start), even though the model architecture and training setup are identical. This means the effectiveness of filtering approaches like GRPO-DS is highly sensitive to the training data — they are least efficient on the hardest datasets where zero-variance prompts are most prevalent.
This characterization is significant beyond documenting a known issue because it explains disparate empirical findings in the RLVR literature and provides a diagnostic framework for practitioners. The extreme sparsity of signal in early training on hard datasets explains why some groups have reported instability and poor convergence with GRPO, while others (using easier data or larger models with higher initial solve rates) have not. It also provides a predictive tool: knowing a dataset's difficulty relative to the base model's capability allows estimating the zero-variance ratio and anticipating whether FILTERING or signal-extraction approaches are more appropriate.
This innovation is primarily diagnostic and empirical — it doesn't propose a new method but provides the characterization that makes method design principled rather than ad-hoc. It is the RLVR analog of the scaling law analyses that characterized pretraining compute inefficiency before Chinchilla — it quantifies a previously qualitative observation and turns it into a measurable, actionable training dynamic.
Innovation 4: Demonstrating That Learning from Negative (All-Incorrect) Prompts Is Crucial for Training Stability on Hard Datasets
The paper's training dynamics analysis (Figure 5) reveals a previously undocumented phenomenon: on the large-scale setup with DAPO-Math-17k, GRPO suffers a sharp accuracy drop during the first ~200 gradient steps, while RL-ZVP exhibits a smooth, monotonically improving curve. The authors attribute this to the fact that RL-ZVP can extract learning signal from all-incorrect zero-variance prompts, while GRPO receives essentially no signal during this phase (since ~95–99% of prompts are all-incorrect and zeroed out).
This finding is conceptually important because it inverts the intuitive priority for zero-variance prompt exploitation. One might naturally assume that all-correct prompts (where the model has mastered the problem) are the most valuable to exploit — they represent the model's best behavior, and reinforcing them would seem to strengthen good patterns. However, the paper's evidence suggests that all-incorrect prompts are actually more critical for training progress, at least in the early stages on hard datasets. The reason: they provide the dense, consistent negative signal that prevents the policy from drifting into degenerate behaviors when positive examples are scarce.
This connects to a broader principle in reinforcement learning: the value of negative feedback (what not to do) can exceed the value of positive feedback (what to do) when the action space is large and the reward signal is sparse. In the early training of a weak model on hard math problems, the policy's natural tendency is to produce a wide variety of incorrect reasoning patterns (guessing, incomplete chains, nonsensical derivations). Without consistent negative feedback suppressing these patterns, the model can drift randomly or collapse into repetitive failure modes. RL-ZVP's ability to penalize all response tokens in all-incorrect prompts — with entropy-guided magnitude — provides this stabilizing negative signal, while GRPO receives nothing until the model occasionally produces a mixed group.
The ablation in Table 3 supports this interpretation: removing negative prompts from RL-ZVP ("- w/o negative prompts") causes a larger average performance drop (from 49.90 to 47.75 Acc@8) than removing positive prompts (to 47.25), suggesting that negative signal contributes slightly more to the overall gain. More importantly, the training curve comparison in Figure 5 shows that the stability benefit manifests specifically in the early phase where negative prompts dominate.
This innovation is fundamental in its reframing of priorities (negative signal matters more than previously recognized for stability) but incremental in its mechanism (it's a consequence of the advantage formulation, not a separate technique). The contribution is the empirical demonstration and the interpretive framework, not a new algorithmic component.
5. Experimental Analysis
Evaluation Methodology
-
Dataset. The primary training datasets are MATH (Hendrycks et al., 2021; Lightman et al., 2024) — 7,500 problems for the small-scale setup — and DAPO-Math-17k (Yu et al., 2025) — 17,000 more challenging problems for the large-scale setup. An alternative training dataset, a random 20k-prompt subset of NuminaMath-CoT (LI et al., 2024), is used in Appendix B.1 for additional validation. Evaluation is performed on six standard math reasoning benchmarks: Minerva, AMC23, MATH500 (the MATH test set), AIME24, AIME25, and OlympiadBench (He et al., 2024). No details are provided about the exact sizes of each evaluation benchmark, though the validation set used for intermediate evaluation during training is described as containing 1,534 problems (Appendix A).
-
Base model(s). All main experiments use Qwen3-1.7B-Base and Qwen3-8B-Base (Yang et al., 2025; Liu et al., 2026) to assess performance across model scales. The small-scale setup pairs Qwen3-1.7B-Base with MATH, while the large-scale setup pairs Qwen3-8B-Base with DAPO-Math-17k. An additional experiment in Appendix B.3 uses Llama-3.2-3B-Instruct (Grattafiori et al., 2024) to test generalization across model families. The choice of Qwen3 models is not explicitly justified beyond being contemporary base models that allow studying RLVR training dynamics at two parameter scales.
-
Metrics. Two metrics are reported: Acc@8 and Pass@8. For each evaluation question, eight responses are generated from the trained policy model, and the metrics capture whether at least one of the eight responses is correct. The paper does not provide a precise definition distinguishing these two metrics, though the naming convention suggests Acc@8 might measure exact-match accuracy (the fraction of questions where the majority or best answer is correct) while Pass@8 might measure the fraction of questions where at least one of the eight generated responses is correct. The MATH grading function from Lightman et al. (2024) is used to handle equivalent mathematical expressions when verifying correctness against ground-truth answers (Appendix G). Both metrics are reported as percentages.
-
Baselines. Four baselines are compared against RL-ZVP: (1) GRPO (Shao et al., 2024) — the standard Group Relative Policy Optimization with z-score normalized advantages that implicitly zeroes out zero-variance prompts; (2) GRPO-DS (GRPO with Dynamic Sampling, from DAPO; Yu et al., 2025) — repeatedly samples prompts and discards zero-variance groups until a batch containing only non-zero-variance prompts is formed, evaluated under two settings: GRPO-DS-r (same number of rollouts as RL-ZVP) and GRPO-DS-g (same number of gradient steps as RL-ZVP, requiring 2.45×–5.29× more rollouts); (3) GRESO (Zheng et al., 2025) — uses a probabilistic pre-rollout filtering strategy based on historical training dynamics to skip prompts likely to be zero-variance, also evaluated under two settings: GRESO-r (same rollouts) and GRESO-g (same gradient steps, requiring 1.51×–3.99× more rollouts). The base model (Qwen3-1.7B-Base or Qwen3-8B-Base before any RL training) is also evaluated for out-of-domain experiments (Table 7).
-
Generation budget / compute accounting. The primary unit of compute measurement is number of rollouts (sampled responses). Table 1 provides exact counts: for the small scale, GRPO and RL-ZVP each use 172,032 rollouts over 672 gradient steps; GRPO-DS-r uses the same 172,032 rollouts but achieves only 320 gradient steps due to discarded zero-variance prompts; GRPO-DS-g uses 421,888 rollouts (2.45× more) to match the 672 gradient steps; GRESO-r uses 173,600 rollouts for 320 gradient steps; GRESO-g uses 272,030 rollouts (1.58× more) for 672 gradient steps. For the large scale, GRPO and RL-ZVP use 540,672 rollouts over 1,056 gradient steps; GRPO-DS-r uses 827,392 rollouts (1.53×) for only 32 gradient steps; GRPO-DS-g uses 2,859,008 rollouts (5.29×) for 1,056 gradient steps. The number of gradient steps is the alternative comparison axis: RL-ZVP and GRPO are compared at identical rollout budgets and gradient step counts, while the "-g" variants of filtering baselines require substantially more rollouts to achieve the same gradient step count, and the "-r" variants use the same rollout budget but achieve far fewer gradient steps.
-
Cross-validation / statistical protocol. Evaluation is performed periodically during training: on the small scale, Minerva, AMC23, and MATH are evaluated after every training iteration (16 gradient steps) and the best result is reported, while AIME24, AIME25, and OlympiadBench use only the final checkpoint; on the large scale, Minerva, AMC23, MATH, AIME24, and AIME25 are evaluated after every iteration with the best result reported, while OlympiadBench uses only the final checkpoint (Appendix A). The paper does not employ k-fold cross-validation or report confidence intervals or standard deviations for any results. There is no multiple-seed analysis; the paper implicitly reports results from a single training run per configuration, which means the observed differences between methods may be confounded with random seed variation — an issue that is not discussed.
Main Quantitative Results
Aggregate Performance Comparison Across Six Benchmarks
The headline result appears in Table 2: RL-ZVP consistently outperforms GRPO and all filtering baselines across all six math reasoning benchmarks and both model scales. On Qwen3-1.7B-Base trained with MATH, RL-ZVP achieves an average Acc@8 of 49.90 and average Pass@8 of 69.77, compared to GRPO's 46.79 and 66.41 — improvements of 3.11 and 3.36 points respectively when averaged across the three benchmarks evaluated per-iteration (Minerva, AMC23, MATH500). Individual benchmark gains range from +0.87 Acc@8 on Minerva to +6.56 Acc@8 on AMC23, and from +1.57 Pass@8 on Minerva to +6.75 Pass@8 on AIME24.
On Qwen3-8B-Base trained with DAPO-Math-17k, RL-ZVP achieves an average Acc@8 of 55.09 and average Pass@8 of 69.12 (computed across all six benchmarks), compared to GRPO's 49.94 and 65.29 — average improvements of 5.15 Acc@8 and 3.83 Pass@8. The largest gains are on OlympiadBench (+8.61 Acc@8, +6.08 Pass@8) and MATH500 (+6.73 Acc@8, +3.73 Pass@8). On AMC23, RL-ZVP achieves 78.75 Acc@8 vs. GRPO's 73.75 (+5.00), and on AIME24, 33.33 vs. 28.33 (+5.00). Every single benchmark shows a positive delta for RL-ZVP over GRPO on both model scales.
A critical observation from Table 2: on the large scale, the "-r" variants of filtering baselines (GRPO-DS-r and GRESO-r) perform dramatically worse than GRPO, despite receiving the same or more rollouts. GRPO-DS-r achieves only 29.96 Acc@8 on Minerva (vs. GRPO's 44.99), 47.19 on AMC23 (vs. 73.75), and 68.20 on MATH500 (vs. 83.00). This catastrophic underperformance is explained by Table 1: these baselines achieve only 32 gradient steps compared to GRPO's 1,056, because the vast majority of rollouts are discarded as zero-variance when the policy model initially solves only ~2% of problems per batch. The 1.53× more rollouts are insufficient to compensate for the 33× fewer gradient updates. This is not a flaw in the filtering approaches per se — it is a fundamental consequence of the interaction between low initial model capability, high dataset difficulty, and the filtering strategy. The paper explicitly notes this in Table 1's caption: "This leads to extremely poor data efficiency, requiring over-sampling of ~50× to form a full non-zero-variance batch for each gradient step."
Even when filtering baselines are given 3×–5× more rollouts (the "-g" variants), RL-ZVP still outperforms them on the majority of benchmarks. On the large scale, GRPO-DS-g (5.29× more rollouts) achieves 44.76 Acc@8 on Minerva vs. RL-ZVP's 45.96 (+1.20), 76.25 on AMC23 vs. 78.75 (+2.50), 27.08 on AIME24 vs. 33.33 (+6.25), and 19.58 on AIME25 vs. 24.58 (+5.00). GRESO-g (3.99× more rollouts) similarly underperforms RL-ZVP on most benchmarks. The paper reports average improvements of 0.99 Acc@8 and 1.19 Pass@8 on the small scale, and 5.51 Acc@8 and 5.32 Pass@8 on the large scale, over the best filtering baseline. Figure 4 visualizes these average results as bar charts.
Training Dynamics: Stability and Consistency
Figure 5 presents validation accuracy, entropy, and response length as training progresses, comparing RL-ZVP and GRPO at both model scales. The key findings are:
-
On the small scale (Figure 5a-c): RL-ZVP and GRPO perform similarly for the first ~500 gradient steps, after which RL-ZVP pulls ahead significantly. RL-ZVP's entropy (Figure 5b) steadily decreases from roughly 0.75 to 0.40 over training, while GRPO's entropy drops initially but then oscillates without clear trend. RL-ZVP's response length (Figure 5c) increases from roughly 750 to 1,250 tokens over training, while GRPO's response length remains flat near 750, showing essentially no growth in reasoning verbosity.
-
On the large scale (Figure 5d-f): The difference is stark. GRPO suffers a sharp accuracy drop during the first ~200 gradient steps, falling from roughly 45% to below 35%, before recovering to around 45% by step 1,000. RL-ZVP exhibits a smooth, monotonically increasing accuracy curve from roughly 38% to 50%+ with no early-stage collapse. Entropy (Figure 5e) drops sharply for both methods in the first 200 steps but stabilizes for RL-ZVP around 0.30–0.35, while GRPO's entropy continues to drift downward toward 0.20 by step 1,000 — approaching entropy collapse. Response length (Figure 5f) grows steadily for RL-ZVP from ~1,200 to ~2,800 tokens, while GRPO's response length grows more slowly and plateaus around ~2,200.
The paper attributes GRPO's early-stage instability to "sparse and noisy feedback when the majority of prompts are zero-variance" (Section 4.2), noting that on the large scale, the policy initially solves fewer than 5% of problems per batch, meaning GRPO receives gradient signal from only ~5% of its prompts. RL-ZVP's ability to extract signal from the ~95% of all-incorrect prompts provides "denser and thereby resulting in greater training stability in the early stages" (Section 4.2). This explanation is consistent with the zero-variance ratio dynamics shown in Figure 3: the ratio starts near 99% on DAPO-Math-17k and decreases as training progresses, but remains non-negligible throughout.
The entropy and response length trends are interpreted as evidence of reasoning quality improvement. The paper argues that longer responses coupled with low but non-collapsing entropy indicate "enhanced capabilities due to the emergence of long-form reasoning behaviors — the so-called aha-moment — and greater confidence in exploratory decisions" (Section 4.2). For the small scale specifically, the paper notes that "GRPO-trained models exhibit minimal growth in response length, aligning with their bounded validation performance and potentially their inability to generate multi-step, well-structured solutions" (Section 4.2). For the large scale, the entropy stabilization around 0.30–0.35 rather than collapsing to near-zero is highlighted as evidence that RL-ZVP "rules out entropy collapse and emphasizes RL-ZVP's ability to promote detailed and confident reasoning" (Section 4.2).
Comparison of Reasoning Quality: Qualitative Evidence
The paper provides qualitative examples in Appendix F comparing GRPO-trained and RL-ZVP-trained model outputs on four MATH problems (Minerva, AIME24, AIME25, MATH500). The analysis in Section 4.2 summarizes these examples:
-
On the small scale: "GRPO-trained models produce simple and direct solutions, jumping straight into problem-solving with little structure. Even when they show glimpses of more complex behaviors such as reflection, these behaviors are shallow and disorganized." In contrast, "models trained with RL-ZVP display much more sophisticated and systematic reasoning strategies: they restate and reflect on the key hypotheses, carefully interpret the clues, plan solutions step by step, organize their reasoning into major steps with detailed sub-solutions, and reasonably attempt alternative strategies when encountering dead ends."
-
Exploration patterns differ qualitatively: "GRPO-trained models often rely on shallow trial-and-error heuristics (e.g., 'let's try another set of values', 'let's try a different method'), which resemble second-guessing and lack of direction." RL-ZVP-trained models "adopt more principled and comprehensive strategies (e.g., 'let's double check', 'assume symmetry to simplify the problem', 'try to factor directly', 'let's try to express the sides using the Law of Cosines'), demonstrating focused and systematic exploration."
-
On the large scale: "both GRPO and RL-ZVP produce systematic solutions with clear structure. However, GRPO outputs tend to be repetitive and rigid in tone, while RL-ZVP exhibits a more flexible and exploratory style of reasoning, reflecting a deeper ability to search for and evaluate alternative solution paths."
These qualitative claims are supported by the four example pairs in Appendix F, though the paper does not provide a systematic quantitative analysis of reasoning strategy frequency (e.g., counting occurrences of reflection tokens, measuring structural organization metrics, or conducting blinded human evaluation of solution quality). The examples are illustrative but selected — the paper does not state how they were chosen or whether they are representative of average behavior rather than best-case examples.
Out-of-Domain Generalization
Table 7 in Appendix B.2 evaluates the Qwen3-1.7B-Base model trained on MATH on four out-of-domain benchmarks: LiveCodeBench (coding), MMLU-Pro (general knowledge), GPQA-Diamond (scientific reasoning), and BIG-Bench Hard (logical reasoning). The base model achieves an average Acc@8 of 14.50 across these four benchmarks. GRPO training improves this to 27.84, demonstrating that math-focused RLVR training transfers to other reasoning domains. RL-ZVP further improves average Acc@8 to 30.19, a gain of +2.35 over GRPO. The largest domain-specific gains are on BIG-Bench Hard (+5.90 Acc@8, +1.07 Pass@8) and GPQA-Diamond (+2.41 Acc@8, +6.44 Pass@8). LiveCodeBench shows the smallest gains (+0.58 Acc@8, +0.29 Pass@8). RL-ZVP also outperforms all filtering baselines on out-of-domain benchmarks, with the best filtering baseline (GRESO-g) achieving 28.48 average Acc@8 vs. RL-ZVP's 30.19.
Alternative Training Data
Table 6 in Appendix B.1 replicates the main experiment using a different training dataset (20k subset of NuminaMath-CoT) rather than MATH or DAPO-Math-17k. On Qwen3-1.7B-Base, RL-ZVP achieves average Acc@8 improvements of +1.58 over GRPO across the six benchmarks, with the largest gains on MATH500 (+3.17 Acc@8, +3.66 Pass@8) and OlympiadBench (+2.33 Acc@8, +4.27 Pass@8). On Qwen3-8B-Base, the average improvements are larger: +3.20 Acc@8 across benchmarks, with AMC23 (+6.25 Acc@8, +3.81 Pass@8) and MATH500 (+3.60 Acc@8, +1.88 Pass@8) showing the largest deltas. The corresponding training dynamics (Figure 6) replicate the patterns from the main experiments: RL-ZVP exhibits lower entropy, longer response lengths, and consistently higher validation accuracy throughout training compared to GRPO.
Generalization Across Model Families
Table 8 in Appendix B.3 evaluates RL-ZVP on Llama-3.2-3B-Instruct trained with MATH, representing a different model family from the main Qwen3 experiments. RL-ZVP achieves average Acc@8 of 24.16 across benchmarks vs. GRPO's 22.95, an improvement of +1.21. Individual benchmark gains are modest: +1.20 on Minerva, +2.50 on AMC23, +1.03 on MATH500, +2.50 on AIME24, with near-zero deltas on AIME25 (0.00) and OlympiadBench (+0.04). The smaller magnitude of improvements relative to the Qwen3 results is not discussed; possible explanations include the instruction-tuned nature of the Llama base model (which may have different initial entropy characteristics or reasoning patterns) or the smaller model scale (3B vs. the 1.7B and 8B tested on Qwen3). The paper notes simply that "RL-ZVP continues to deliver consistent and meaningful improvements over GRPO and other baselines across most benchmarks."
Ablation Studies and Robustness Checks
Sub-component removal (Table 3, Section 4.3): Removing any single component from RL-ZVP degrades performance, with the entropy scaling factor being the most critical. Full RL-ZVP achieves an average Acc@8 of 49.90 on the small-scale benchmarks. Removing negative prompts ("- w/o negative prompts") reduces this to 47.75; removing positive prompts ("- w/o positive prompts") reduces to 47.25; removing entropy scaling ("- w/o entropy scaling," i.e., using uniform ±1 as advantage for zero-variance prompts) causes the largest drop to 46.88 — only marginally better than the GRPO baseline of 46.79. This ordering holds consistently across individual benchmarks: on AMC23, for example, Acc@8 drops from 48.75 with full RL-ZVP to 44.69 without negative prompts, 41.56 without positive prompts, and 42.50 without entropy scaling. The negative-result finding — that uniform ±1 advantages are essentially equivalent to GRPO — directly validates the paper's central claim that advantage magnitude modulation (not just direction) is necessary for extracting useful signal from zero-variance prompts.
Symmetric vs. asymmetric advantage (Table 10, Appendix C.2): A variant of RL-ZVP where negative prompts use the symmetric advantage $-\alpha H_{i,t}$ (the exact negative of the positive prompt formula, rather than the inverse-entropy formula $-\alpha(\max_k(H_{i,k}) - H_{i,t})$) achieves an average Acc@8 of 46.64 — worse than both asymmetric RL-ZVP (49.90) and the GRPO baseline (46.79). On individual benchmarks, the symmetric variant underperforms GRPO on AMC23 (41.88 vs. 42.19) and MATH500 (66.44 vs. 69.09). This is the paper's most striking negative result: not only does extracting signal from zero-variance prompts require appropriate shaping, but shaping it incorrectly (symmetrically) is actively worse than providing no signal at all. The paper interprets this as confirming that "a symmetric penalty is counterproductive in this setting" and validating the asymmetric design where high-entropy tokens in incorrect responses are penalized less, not more.
Scaling factor α (Table 4, Section 4.3; Table 9, Appendix C.1): The scaling factor exhibits a clear optimal range. On the small scale, α = 0.10 achieves the highest average Acc@8 of 49.90, with α = 0.20 a close second at 48.15. Setting α = 0.05 reduces performance to 46.78 (nearly equivalent to GRPO's 46.79), and α = 0.30 degrades to 45.81 (worse than GRPO). The wider sweep in Table 9 confirms this pattern extends across orders of magnitude: α = 0.001 achieves 47.83 (marginal improvement over GRPO at 46.79), α = 0.01 achieves 47.43, α = 1.0 collapses to 44.32, and α = 10.0 collapses further to 39.96 — severely worse than GRPO. The α = 0 case (RL-ZVP identity with GRPO) is not shown in Table 4 but appears in Table 9 as the α = 0 row, reproducing GRPO's 46.79 Acc@8 exactly. The paper interprets this U-shaped relationship as evidence that "overly small values fail to sufficiently utilize learning signal from zero-variance prompts, while overly large values may cause unstable training with large gradient updates."
Alternative training data robustness (Tables 6–8, Appendix B): RL-ZVP's improvements transfer to the NuminaMath-CoT dataset (Table 6), where average gains over GRPO are 1.58 Acc@8 (1.7B) and 3.20 Acc@8 (8B). They also transfer across model families (Table 8, Llama-3.2-3B-Instruct), though with smaller magnitude (+1.21 average Acc@8). No further hyperparameter tuning was performed for these alternative setups — the same α = 0.10 was used — which makes these robustness checks rather than optimized configurations.
Training dynamics comparison with filtering baselines (Figure 7, Appendix D): The extended training dynamics plot adds GRPO-DS and GRESO to the comparison. On the small scale, filtering baselines achieve accuracy between GRPO and RL-ZVP throughout training, but "their entropy rises sharply, suggesting a form of entropy explosion" while RL-ZVP maintains stable and decreasing entropy. On the large scale, filtering baselines "lag significantly behind RL-ZVP" and show "limited growth in response length relative to RL-ZVP." The paper notes that Figure 7 "inherently favors GRPO-DS and GRESO, because under the same number of gradient steps, these methods perform many more rollouts than GRPO and RL-ZVP" (Appendix D). Despite this advantage in total rollouts, the filtering baselines still underperform RL-ZVP.
Critical Assessment
Claim 1: RL-ZVP extracts useful learning signals from zero-variance prompts that directly improve policy optimization. The evidence for this claim is strong and multi-faceted. The most direct evidence is the sub-component ablation (Table 3): full RL-ZVP (49.90 Acc@8) substantially outperforms the no-entropy-scaling variant (46.88), which essentially tests the proposition "is absolute reward without contrast sufficient?" The answer is yes, but only with appropriate magnitude modulation — uniform ±1 advantages are nearly identical to GRPO. The symmetric ablation (Table 10) further refines this: the advantage must be shaped correctly for negative prompts (inverse entropy, not direct entropy) or the signal is actively harmful. The consistent outperformance of RL-ZVP over GRPO across all benchmarks and scales (Table 2) with identical rollout budgets and gradient step counts — combined with the training dynamics showing more stable learning (Figure 5) — provides converging evidence that the signal extracted from zero-variance prompts is not just real but beneficial.
One weakness in this evidence chain is the absence of an ablation that tests whether entropy per se is the optimal modulation mechanism, or whether alternative token-level features would work equally well or better. The paper motivates entropy through intuition and prior work, but does not compare against alternatives like token probability, token position in the response, syntactic role, or learned importance weights. It is possible that any reasonable token-level variation of advantage magnitude (as opposed to uniform broadcasting) would produce similar benefits, and entropy happens to work because it is a convenient, readily available signal rather than because it captures something unique about reasoning quality. The symmetric ablation partially addresses this by showing that the shape of the modulation matters, but does not isolate whether the feature (entropy vs. alternatives) matters.
Claim 2: RL-ZVP consistently outperforms prompt-filtering baselines that discard zero-variance prompts. The evidence is mixed and requires careful interpretation. Under the fair comparison setting (same rollout budget, "-r" variants), RL-ZVP massively outperforms filtering baselines on the large scale — but this is largely because the "-r" baselines are effectively crippled, achieving only 32 gradient steps vs. RL-ZVP's 1,056 (Table 1). This is a real-world limitation of filtering approaches on hard datasets, but it is not a fair comparison of algorithmic effectiveness; it is a comparison of practical feasibility under a fixed rollout budget constraint. The filtering baselines are not given enough computation to succeed.
Under the alternative comparison setting (same gradient steps, "-g" variants), the filtering baselines receive 3×–5× more rollouts and achieve some, but not all, of the performance gains of RL-ZVP. On the large scale, RL-ZVP outperforms the best filtering baseline by 5.51 Acc@8 on average, which is substantial. However, the "-g" comparison is not entirely clean either: the filtering baselines use 3×–5× more compute, but also see 3×–5× more unique prompts (since they over-sample to filter out zero-variance cases). This means they may be benefiting from greater prompt diversity, not just more rollouts — a confound the paper does not discuss.
A cleaner comparison — giving all methods the same total computation budget but allowing the filtering baselines to dynamically allocate it between rollouts and gradient steps — is not performed. This would require sweeping over the dynamic sampling ratio to find the optimal tradeoff for each method, which is computationally prohibitive. The paper's choice of two comparison axes (same rollouts and same gradient steps) is reasonable given computational constraints, but the results should be interpreted as upper and lower bounds on the relative performance, not as precise efficiency ratios.
Claim 3: RL-ZVP promotes more stable training and prevents the early-stage performance collapse observed with GRPO on hard datasets. This claim is supported by Figure 5d, which shows GRPO dropping sharply from ~45% to <35% in the first 200 gradient steps on the large scale, while RL-ZVP improves smoothly. The paper attributes this to RL-ZVP extracting signal from all-incorrect prompts during the phase where the policy solves <5% of problems. This explanation is plausible and consistent with the zero-variance ratio data (Figure 3), but it is not experimentally verified. A direct test would be to run GRPO with an artificially reduced zero-variance ratio (e.g., by pre-filtering the training data to remove the hardest problems, or by warm-starting from a partially trained policy that already has >5% solve rate) and observe whether the performance collapse disappears. Without such a test, the causal attribution to zero-variance prompts is inferential rather than demonstrated.
Additionally, the paper does not report whether the early-stage GRPO collapse on the large scale is reproducible across multiple random seeds or is a single-run artifact. Given that the sharp drop occurs during the first 200 steps — when the model receives gradient signal from only ~1% of its batch and individual gradient updates have unusually high variance — run-to-run variability could be substantial. Multiple-seed experiments would substantially strengthen confidence in this finding.
Claim 4: RL-ZVP elicits more sophisticated reasoning behaviors, including longer chain-of-thought and more structured exploration strategies. The evidence is suggestive but preliminary. The response length data (Figures 5c and 5f) clearly show RL-ZVP generating longer responses than GRPO on both scales, with the gap widening over training. The entropy data (Figures 5b and 5e) show RL-ZVP maintaining lower but non-collapsed entropy relative to GRPO. The qualitative examples in Appendix F are compelling illustrations of the claimed behavioral differences. However, the paper does not provide systematic quantitative metrics of reasoning quality — no measurement of solution step count, reflection token frequency, backtracking rate, answer verification rate, or any other behavioral marker. The qualitative examples are selected rather than randomly sampled, and no inter-annotator agreement or blinded evaluation is reported. The claim that RL-ZVP elicits "more sophisticated and systematic reasoning strategies" is therefore supported primarily by length and entropy trends (which are indirect proxies) and illustrative examples (which may not be representative). A more rigorous demonstration would include automated behavioral metrics (e.g., frequency of self-correction patterns, structural organization scores) or human evaluation of solution quality, but neither is provided.
Missing experiments that would strengthen the paper:
-
Multiple random seeds. All results appear to be from single training runs. Given the well-documented variability of RL training for LLMs (sensitive to initialization, data ordering, and sampling stochasticity), confidence in the reported deltas would be substantially higher with error bars across 3–5 seeds. This is particularly important for the early-stage stability claim on the large scale, where single-run artifacts are most likely.
-
Comparison against other token-level reward shaping methods. The paper positions itself within the fine-grained reward shaping literature but does not compare against any specific method from that literature (e.g., Cui et al., 2025; Wang et al., 2025) either as standalone baselines or in combination with RL-ZVP. Such comparisons would contextualize the contribution — is the gain from entropy-guided advantage shaping on zero-variance prompts larger or smaller than the gain from alternative token-level shaping on all prompts?
-
Scaling to larger models. The paper explicitly acknowledges this as a limitation (Section 6), noting that experiments are limited to models up to 8B parameters. Whether RL-ZVP's benefits persist, diminish, or amplify at 14B, 32B, or 70B scales is unknown. The zero-variance ratio might change at larger scales (larger models have higher initial solve rates, potentially reducing zero-variance prevalence), which could alter the relative benefit of signal extraction vs. filtering.
-
Ablation on group size G. All experiments use G = 8 (small scale) or G = 16 (large scale). Group size directly affects GRPO's advantage normalization and the prevalence of zero-variance prompts (larger groups are less likely to have uniform rewards by chance). RL-ZVP's benefit might vary systematically with G — for very large groups, zero-variance prompts become rarer, potentially reducing the benefit of handling them. Conversely, for very small groups, zero-variance prompts become more common, potentially amplifying the benefit. No sweep over G is reported.
-
Verifier robustness analysis. All experiments use binary rewards from exact-match verification. The paper does not test sensitivity to reward noise (e.g., misgraded answers) or explore whether RL-ZVP is more or less robust to verifier errors than GRPO. This matters for practical deployment where verifiers may be imperfect.
Conditional scope of the claims. The paper's claims are demonstrated for:
- Two model families (Qwen3 and Llama-3.2), though the Llama results are weaker in magnitude.
- Two model scales within Qwen3 (1.7B and 8B).
- Three training datasets (MATH, DAPO-Math-17k, NuminaMath-CoT subset).
- Six math reasoning evaluation benchmarks plus four out-of-domain benchmarks.
- Binary reward signals from exact-match verification.
The claims have not been tested for: non-binary rewards (partial credit, graded feedback), non-math reasoning domains (beyond the four out-of-domain benchmarks tested), models larger than 8B, or training regimes substantially longer than ~1,000 gradient steps. The paper is appropriately circumspect about these limitations in Section 6. The practical takeaway — that zero-variance prompts contain usable signal and that entropy-guided advantage shaping provides a functional mechanism for extracting it — is well-supported within the tested regime. Whether the 4–8 point accuracy gains over GRPO represent an upper bound or a lower bound for what signal extraction can achieve remains an open question.
6. Limitations and Trade-offs
The Difficulty Estimation Cost in RL-ZVP Is Not Accounted for in the Headline Efficiency Numbers
RL-ZVP solves the zero-variance prompt problem by extracting signal from prompts that would otherwise be discarded, eliminating the wasteful oversampling required by filtering baselines. However, this framing implicitly assumes that the zero-variance detection mechanism itself is free — that checking whether std({R_i}) = 0 and computing per-token entropy H_{i,t} adds negligible overhead relative to the cost of generating and discarding additional rollouts.
The paper provides only partial accounting of this overhead. The computational cost of the zero-variance detection step is trivial: computing the standard deviation of G binary rewards (where G = 8 or G = 16) is effectively free. The cost of computing per-token entropy is more substantial. The entropy calculation requires the full softmax distribution over the vocabulary V at every token position: H_{i,t} = -\sum_{j=1}^{|V|} \pi_\theta(v_j | q, o_{i,<t}) \log \pi_\theta(v_j | q, o_{i,<t}). For a modern LLM with vocabulary size |V| ≈ 150,000 tokens, this is a vector summation over 150,000 elements per token position, per response. For a batch of 512 prompts with G = 16 responses each averaging, say, 2,000 tokens on the large scale, this means approximately 512 × 16 × 2000 = 16.4 million entropy summations over 150,000-element vectors per training iteration — operations that must run on GPU before the advantage can be computed and the gradient step can proceed.
The paper does not measure this computational cost, does not compare the wall-clock time per training step of RL-ZVP vs. GRPO, and does not include any latency numbers in its analysis. Table 1 reports identical rollout counts and gradient step counts for RL-ZVP and GRPO, establishing that the methods are matched on the dominant cost (rollout generation), but the per-step overhead of the entropy computation — which GRPO does not perform at all — is never quantified. The paper's Figure 2 shows that rollout generation accounts for ~50% of total training time per step, but does not break down the remaining 50% to show whether entropy computation meaningfully increases the "other" category.
The practical consequence is that the true cost of RL-ZVP relative to GRPO is unknown. If the entropy computation adds, say, 5% overhead per training step, this is negligible and well worth the accuracy gains. If it adds 30%, the comparison against filtering baselines (which avoid both the waste of discarding rollouts and the cost of entropy computation, at the price of oversampling) becomes more nuanced — a practitioner might prefer GRESO-g with 4× more rollouts and identical wall-clock time if the accuracy gap narrows. The paper's claim that RL-ZVP "mitigates useless rollout costs" (Section 5) is therefore partially substantiated for the dominant cost category (rollout generation) but unverified for the secondary cost (per-step computation).
The mitigation status is that the paper does not address this limitation at all. There is no measurement of wall-clock time, no profiling of the entropy computation within the training loop, and no discussion of whether the entropy values could be approximated (e.g., using only a subset of the vocabulary, or computing entropy from the top-k tokens rather than the full distribution) to reduce overhead. A practitioner implementing RL-ZVP would need to profile this cost independently.
All Results Are on a Single Training Run; Statistical Significance Is Not Assessed
The paper reports results from what appears to be a single training run per configuration (model scale × training dataset × method). There is no mention of multiple random seeds, no error bars or confidence intervals on any reported metric, and no discussion of run-to-run variability. In reinforcement learning for LLMs — where training dynamics are known to be sensitive to random initialization, data ordering, and sampling stochasticity — single-run results can be misleading, particularly for claims about training stability.
The consequence is that the paper's headline findings — including the critical claim that RL-ZVP prevents GRPO's early-stage performance collapse on the large scale (Figure 5d) — may be confounded with random seed effects. The sharp accuracy drop GRPO experiences during the first 200 gradient steps on the large scale could be a stochastic artifact of a particular unlucky data ordering or sampling trajectory; without multiple seeds, we cannot distinguish a systematic failure mode from a single-run anomaly. Similarly, the exact magnitude of RL-ZVP's improvements over GRPO (4–8 Acc@8 points, Table 2) could vary substantially across seeds, and the difference between RL-ZVP and the filtering baselines on the large scale might not be statistically significant in some pairwise comparisons given the small test sets involved.
The paper does not report the sizes of the evaluation benchmarks beyond the validation set containing 1,534 problems (Appendix A). If individual benchmarks like AIME24 or AIME25 contain only tens of questions (AIME competitions typically have 15–30 problems each), then differences of a few percentage points in accuracy could represent a difference of a single correctly solved question — easily within the range of sampling noise. The per-benchmark results in Table 2 show AIME24 Acc@8 improving from 28.33 (GRPO) to 33.33 (RL-ZVP) on the large scale, a +5.00 point gain. On a 30-question test set, this represents roughly 1.5 more correct answers. Without confidence intervals, we cannot assess whether this difference is reliably above noise.
The mitigation status is that the paper does not acknowledge this limitation. The absence of multiple-seed analysis is not mentioned in Section 6 (Limitations & Future Directions), which focuses on model scale, binary rewards, and extending beyond zero-variance prompts. In practice, reproducing these experiments with even 3 seeds per configuration would multiply the already-substantial computational cost (Table 1 reports 172,032 to 2,859,008 rollouts per configuration, generated on 4× RTX 3090 or 8× A100 GPUs). This cost likely explains the omission, but it does not excuse the absence of any discussion of the statistical reliability of the findings. A practitioner deciding whether to adopt RL-ZVP would want to know whether the reported gains are robust to the inherent randomness of RL training.
The Scaling Behavior of RL-ZVP Beyond 8B Parameters Is Completely Unknown
The paper's experiments are limited to models of 1.7B and 8B parameters (Qwen3-1.7B-Base and Qwen3-8B-Base), with a single additional experiment on Llama-3.2-3B-Instruct. The authors explicitly acknowledge this in Section 6: "Due to computational constraints, our experiments are limited to models up to 8B parameters; future work should investigate how RL-ZVP scales to larger models (e.g., 14B or 32B)."
The consequence of this limitation is that we cannot predict whether RL-ZVP's benefits persist, diminish, or amplify at the model scales where RLVR post-training is most commonly deployed. Frontier reasoning models (DeepSeek-R1, Kimi k1.5, OpenAI o1) are in the hundreds of billions of parameters. At these scales, several factors that affect RL-ZVP's behavior may shift qualitatively:
-
Zero-variance prevalence: Larger base models typically have higher initial math reasoning capability. Qwen3-8B-Base solved fewer than 5% of DAPO-Math-17k problems at the start of training (Figure 3). A 70B model might solve 20–30%, which would substantially reduce the fraction of all-incorrect zero-variance prompts that Figure 5 shows are critical for early-stage stability. RL-ZVP's benefit might therefore be largest at smaller scales (where zero-variance prompts are most prevalent) and diminish at larger scales (where GRPO already receives denser signal).
-
Entropy calibration: The distribution of token entropies may differ across model scales. Larger models are typically better calibrated (their predictive probabilities more accurately reflect uncertainty), which could change the distribution of
H_{i,t}values and affect the optimalαscaling factor. The paper findsα = 0.10optimal for both 1.7B and 8B models (Table 4, Table 5), but this does not guarantee the same value would work at 32B or 70B. -
Response length effects: RL-ZVP encourages longer responses (Figure 5c, 5f), with the 8B model's responses growing from ~1,200 to ~2,800 tokens. At much larger scales, models already produce long chain-of-thought by default, and further incentivizing length could push against context window limits or produce diminishing reasoning returns.
The paper provides no evidence on any of these scaling questions. The only cross-scale comparison available is between the 1.7B and 8B Qwen3 results, which show that RL-ZVP's relative gains over GRPO are larger at 8B (+5.15 Acc@8) than at 1.7B (+2.84 Acc@8) on the main benchmarks. However, the training datasets differ between these scales (MATH for 1.7B, DAPO-Math-17k for 8B), confounding the comparison. The Llama-3.2-3B experiment (Table 8) shows smaller gains (+1.21 average Acc@8), but this could reflect model family differences or instruction-tuning effects rather than a scaling trend.
The mitigation status is that the paper acknowledges this as a clear limitation and flags it for future work, but makes no attempt to extrapolate or bound the expected behavior at larger scales. A practitioner training at 70B scale cannot infer from this paper whether RL-ZVP would provide similar, larger, or smaller benefits than GRPO.
The Qualitative Claims About Reasoning Quality Are Not Systematically Validated
Section 4.2 makes specific claims about the reasoning behavior of RL-ZVP-trained models: they "display much more sophisticated and systematic reasoning strategies," "adopt more principled and comprehensive strategies," and exhibit "a more flexible and exploratory style of reasoning, reflecting a deeper ability to search for and evaluate alternative solution paths." These claims are supported by four illustrative examples in Appendix F and by aggregate trends in response length and entropy (Figure 5).
The limitation is that these claims about reasoning quality — which go beyond simple accuracy metrics to characterize how the model reasons — are not validated through any systematic quantitative analysis. The paper does not report:
-
Behavioral marker frequencies: How often do RL-ZVP-trained models engage in specific reasoning behaviors (self-verification, backtracking, alternative strategy exploration, assumption checking) compared to GRPO-trained models? One could count occurrences of key phrases ("let's double check," "alternatively," "wait," "this approach seems") to quantify these patterns.
-
Structural metrics: How organized are the solutions? One could measure the number of explicit steps, the use of section headers, the depth of nested reasoning, or the presence of summary/conclusion sections.
-
Human evaluation: Do human raters — blinded to the training method — rate RL-ZVP solutions as more coherent, better-structured, or more trustworthy than GRPO solutions?
The four examples in Appendix F are illustrative but selected by the authors. The paper does not state how they were chosen (randomly? curated to show maximum contrast? cherry-picked from among many candidates?). The danger is that these examples represent best-case behavior that is not representative of typical outputs. It is entirely possible that RL-ZVP-trained models occasionally produce beautifully structured solutions with explicit self-verification (as shown in the examples) while their median output is only marginally more structured than GRPO's median output.
The consequence is that the paper's narrative about RL-ZVP eliciting fundamentally superior reasoning behaviors is not rigorously established. The accuracy improvements (Table 2) are well-measured and convincing; the claim that these improvements arise specifically from "more sophisticated and systematic reasoning strategies" is plausible but unverified. The response length and entropy trends (Figure 5) are consistent with this claim but do not constitute direct evidence — longer responses could reflect verbosity rather than better reasoning, and lower entropy could reflect overconfidence rather than genuine certainty.
The mitigation status is that the paper does not address this limitation. The qualitative claims are presented as observations rather than hypotheses, and no future work is suggested for systematically validating reasoning quality. A practitioner evaluating RL-ZVP might care whether the accuracy gains come with genuinely better reasoning (which would transfer to novel problems) or merely with longer, more confident outputs that happen to be correct more often on the test distribution (which might not transfer).
The Method Is Only Validated on Math Reasoning with Binary Verifiable Rewards
The paper's entire experimental evaluation — training and testing — uses mathematical reasoning problems with ground-truth answers that can be verified via exact string matching (with the MATH grading function handling equivalent expressions). The reward signal is binary: R_i = 1 if the extracted final answer matches the ground truth, and R_i = 0 otherwise. The paper acknowledges this scope limitation in Section 6: "we only validate RL-ZVP on verifiable tasks with binary rewards; extending it to settings with graded or ambiguous feedback remains an open challenge."
The consequence is that RL-ZVP's core mechanism — assigning positive vs. negative advantage direction based on reward sign in Equation 5 — has no demonstrated applicability to domains where rewards are continuous, graded, multi-dimensional, or ambiguous. Binary rewards provide a clean separation: all-correct groups are unambiguously positive, all-incorrect groups are unambiguously negative. In domains with continuous rewards (e.g., a 0–10 quality score for open-ended generation), an all-positive-reward group (all scores above some threshold) would be classified as a "positive prompt" under RL-ZVP's current formulation, but the magnitude of the positive advantage would be determined purely by token entropy — not by how good the responses are. A response scoring 9/10 and one scoring 6/10 would receive identical entropy-scaled advantages, discarding the reward magnitude information that could indicate that the 9/10 response should be reinforced more strongly than the 6/10.
Conversely, in domains with graded feedback where no clear positive/negative binary exists, the zero-variance detection std({R_i}) = 0 becomes less meaningful. If all responses in a group receive rewards of 4.2, 4.3, and 4.1 (low variance but not identical), the standard deviation is non-zero and GRPO's z-score normalization would produce small but non-zero advantages — RL-ZVP's zero-variance branch would not activate, even though the within-group contrast is very weak and the situation is analogous to the zero-variance case.
The out-of-domain evaluation in Appendix B.2 (Table 7) tests the trained math models on coding, general knowledge, scientific reasoning, and logical reasoning benchmarks — but the training is still math-only with binary rewards. RL-ZVP is evaluated on out-of-domain tasks, but not trained with out-of-domain reward structures. This means the results show that math RLVR training transfers to other reasoning domains (which is unsurprising and also true of GRPO), but they do not demonstrate that RL-ZVP works when the training signal itself comes from non-math, non-binary rewards.
The mitigation status is that the paper transparently acknowledges this limitation and flags it as future work, but makes no attempt to test RL-ZVP on a non-math domain (e.g., code generation with unit test pass/fail rewards, which is also binary but in a different reasoning domain) or a non-binary reward setting (e.g., partial-credit grading on math problems). The practical applicability of RL-ZVP is currently demonstrated only for math reasoning with exact-match verification — a narrow but important domain. A practitioner working on RLVR for code generation, dialogue, or summarization would need to extrapolate from the paper's results with no direct evidence.
RL-ZVP Has Not Been Tested in Combination with Other Fine-Grained Reward Shaping Methods
The paper positions RL-ZVP within the broader literature on fine-grained reward shaping (Section 5), noting that methods like those of Cui et al. (2025), Agarwal et al. (2025), and Wang et al. (2025) "leverage token-level signals to augment verifiable rewards" but "still suffer from advantage vanishing when faced with zero-variance prompts." The logical extension — which the paper explicitly suggests as future work in Section 6 — is to combine RL-ZVP's zero-variance advantage formulation with these token-level reward shaping methods on non-zero-variance prompts.
The paper does not perform this combination. No experiment tests RL-ZVP with any token-level reward shaping method applied to non-zero-variance prompts. The ablation in Table 3 removes RL-ZVP's own entropy scaling and shows performance degrades to near-GRPO levels, but this tests removing the primary token-level modulation mechanism from RL-ZVP rather than adding a complementary one.
The consequence is that the paper cannot quantify how much of RL-ZVP's benefit comes specifically from handling zero-variance prompts vs. from the token-level entropy modulation per se. It is possible — though the symmetric ablation (Table 10) argues against this — that simply applying entropy-guided advantage shaping to all prompts (zero-variance and non-zero-variance) would outperform RL-ZVP's branching formulation, and that the zero-variance handling is incidental rather than central. The paper's framing emphasizes zero-variance prompts as the key innovation, but without testing RL-ZVP's entropy modulation against alternative token-level shaping on non-zero-variance prompts, we cannot isolate the contribution of the zero-variance-specific component.
More practically, a practitioner implementing RL-ZVP might also want to apply, for example, Wang et al.'s high-entropy minority token modulation to non-zero-variance prompts. The paper provides no guidance on whether these approaches are complementary (additive gains) or redundant (overlapping mechanisms). The ablation removing entropy scaling suggests that token-level modulation is critical for RL-ZVP's performance on zero-variance prompts, but does not tell us whether different modulation strategies on non-zero-variance prompts would further improve results, or whether RL-ZVP's entropy scaling on zero-variance prompts would conflict with alternative scaling on non-zero-variance prompts.
The mitigation status is that the paper flags this direction explicitly: "a promising direction is to generalize our advantage formulation beyond zero-variance prompts or to combine it with existing token-level reward shaping methods to further improve GRPO" (Section 6). This is a recognition of the limitation, but the paper provides no empirical evidence or theoretical analysis to guide such combinations. A practitioner attempting to integrate RL-ZVP into a training pipeline that already uses fine-grained reward shaping would need to perform this analysis from scratch.
7. Implications and Future Directions
How This Work Changes the Landscape
This paper shifts the RLVR field's stance on zero-variance prompts from avoidance to exploitation. Prior to this work, the dominant paradigm — encoded in GRPO's advantage formula and made operational in filtering methods like DAPO (Yu et al., 2025) and GRESO (Zheng et al., 2025) — was that prompts producing uniform rewards within a group are information vacuums best eliminated from the training pipeline. The reasoning was structural: GRPO's z-score normalization requires within-group variance to produce non-zero advantages, and without contrast, there is no relative ranking to drive reinforcement. Filtering approaches accepted this premise and optimized around it — either by oversampling to construct variance-rich batches (post-rollout) or by predicting and skipping likely zero-variance prompts (pre-rollout).
RL-ZVP demonstrates that this premise is false. The paper provides clear empirical evidence that absolute reward signal — correctness independent of within-group contrast — can drive effective policy optimization, provided the advantage magnitude is appropriately shaped. The ablation removing entropy scaling (Table 3) is the critical negative control: uniform ±1 advantages for zero-variance prompts perform nearly identically to GRPO's zero advantage, confirming that the naive "just use absolute reward" approach that the field had implicitly rejected is indeed insufficient. But entropy-guided magnitude modulation makes the difference, converting an apparently useless signal into one that yields 4–8 point accuracy gains over GRPO and consistently outperforms filtering baselines even when those baselines receive 3×–5× more rollouts (Table 2).
This reframing matters because it addresses a genuine scale problem the paper quantifies. Figure 3 shows zero-variance prompts constituting 30–99% of training batches depending on model scale, dataset difficulty, and training stage. On the hard end (Qwen3-8B-Base on DAPO-Math-17k at training start), 99% of generated rollouts produce zero gradient under GRPO — and Figure 2 shows rollout generation accounts for ~50% of total training time. This is not a minor inefficiency; it is a dominant source of wasted computation in RLVR training. Filtering approaches can reduce this waste (by not generating rollouts for predicted zero-variance prompts) but cannot eliminate it — and on hard datasets, the oversampling required to form non-zero-variance batches can be catastrophically expensive (Table 1: GRPO-DS-r requires 1.53× more rollouts but achieves only 32 gradient steps vs. 1,056 for GRPO). RL-ZVP's exploitation approach bypasses this tension entirely: every generated rollout contributes gradient information, regardless of within-group reward variance.
The paper also resolves a specific empirical puzzle in the literature. Prior work reported conflicting experiences with GRPO training stability — some groups observed smooth convergence, others reported sharp early-stage performance collapse. The paper's zero-variance ratio characterization (Figure 3) and training dynamics analysis (Figure 5) provide a unifying explanation: the collapse occurs when the base model's initial capability is mismatched to the dataset difficulty, producing near-100% zero-variance rates that leave GRPO with extremely sparse feedback. This explains why groups training strong base models on easier data (low zero-variance ratio) saw stability, while groups training weaker models on hard data (high zero-variance ratio) encountered collapse. RL-ZVP's ability to extract signal from all-incorrect prompts provides a practical fix for this failure mode — Figure 5d shows smooth, monotonic improvement where GRPO suffers a sharp drop.
In terms of conceptual magnitude, this is more a reframing than a paradigm shift. The paper does not introduce a new RL algorithm class or challenge the fundamental GRPO formulation for non-zero-variance prompts. It identifies a specific degenerate case (zero-variance advantage vanishing), provides a principled replacement, and demonstrates that doing so produces substantial practical gains. The contribution is analogous to identifying that a widely-used estimator produces zero-information output in a common edge case and showing that a carefully designed fallback estimator recovers useful signal — significant in practical impact but bounded in theoretical novelty.
The reframing does, however, redirect research attention in a concrete way. Before this paper, the natural follow-up to "zero-variance prompts waste computation" was "how can we more accurately predict and avoid them?" — the direction pursued by GRESO and related work. After this paper, the more productive question becomes "what other forms of apparently useless training signal can we extract value from, and what modulation mechanisms make that extraction effective?" This shifts the field from a filtering mindset (select which data to use) toward a signal-extraction mindset (extract maximal value from all available data) — a higher-ceiling approach since filtering can at best recover to 100% efficiency while extraction can potentially exceed the value of the original signal.
Follow-Up Research This Work Enables
Testing whether RL-ZVP's gains come from the zero-variance handling specifically, or from the entropy-guided token-level advantage modulation more generally. The paper's branching formulation (Equation 6) applies entropy scaling only to zero-variance prompts, leaving non-zero-variance prompts with GRPO's uniform advantage broadcasting. A strong follow-up would test two alternative designs: (1) applying RL-ZVP's entropy-guided advantage to all prompts (zero-variance and non-zero-variance), and (2) applying GRPO's uniform broadcasting to zero-variance prompts (i.e., using ±α without entropy scaling). Experiment (1) tests whether entropy modulation is beneficial everywhere, not just in the zero-variance edge case. Experiment (2) is partially covered by the "- w/o entropy scaling" ablation in Table 3, but extending it to non-zero-variance prompts would clarify whether the entropy mechanism's value is specific to the zero-variance regime or generalizes. The paper's symmetric ablation (Table 10) provides partial evidence that the shape of the modulation matters, but doesn't isolate whether the regime (zero-variance vs. all prompts) matters. This experiment would use the same Qwen3-1.7B-Base + MATH setup as the paper's small-scale experiments, measuring Acc@8 and Pass@8 on the six standard benchmarks plus training dynamics (entropy, response length, zero-variance ratio curves).
Characterizing whether RL-ZVP's stability benefit on hard datasets is robust across random seeds and training durations. The paper's most striking finding — that RL-ZVP prevents GRPO's early-stage performance collapse on Qwen3-8B-Base + DAPO-Math-17k (Figure 5d) — is based on a single training run per method. RL training for LLMs is known to be sensitive to random initialization, data ordering, and sampling stochasticity. A rigorous follow-up would run both GRPO and RL-ZVP on the large-scale setup with 5–10 random seeds each, reporting mean and standard deviation of Acc@8 at each evaluation checkpoint, and testing whether the early-stage GRPO collapse is systematic (occurs in a majority of seeds) or a single-run artifact. This experiment would also extend training to 2,000–3,000 gradient steps (vs. the paper's 1,056) to determine whether GRPO eventually recovers to RL-ZVP's performance level or whether the gap persists or widens. The paper's Figure 5d shows GRPO recovering to ~45% Acc@8 by step 1,000, still below RL-ZVP's ~50% — longer training would reveal whether this is a permanent capability gap or a convergence rate difference. The computational cost is substantial (each run requires 540,672 rollouts on 8× A100 GPUs per Table 1, and 5 seeds × 2 methods = 10 runs), but the result would determine whether RL-ZVP's stability claim is a reliable finding or a seed-dependent observation.
Evaluating whether RL-ZVP's entropy-guided advantage formulation transfers to non-math reasoning domains with binary verifiable rewards. The paper validates RL-ZVP exclusively on math reasoning with exact-match verification. A natural extension is code generation, where unit test pass/fail provides a similarly binary reward signal. The experiment would train Qwen3-1.7B-Base or Qwen3-8B-Base on a code dataset (e.g., the training split of LiveCodeBench or APPS) using GRPO vs. RL-ZVP, evaluating on LiveCodeBench, HumanEval, and MBPP. The key question is whether the zero-variance phenomenon manifests similarly in code (are there prompts where the model consistently passes or consistently fails all unit tests?) and whether entropy-guided advantage scaling provides the same benefits when the token-level uncertainty reflects code structure (e.g., variable names, API calls, algorithmic choices) rather than mathematical reasoning steps. The paper's out-of-domain evaluation in Table 7 shows that math-trained RL-ZVP marginally outperforms math-trained GRPO on LiveCodeBench (+0.58 Acc@8), but this tests transfer, not code-specific training. A direct code-domain experiment would clarify whether RL-ZVP is a math-specific technique or a general RLVR improvement.
Investigating alternative token-level modulation signals beyond entropy. The paper's entropy-guided advantage formulation (Equation 5) uses exactly one token-level feature: the Shannon entropy of the policy's predictive distribution. The ablation in Table 3 shows that removing entropy scaling (using uniform ±1 advantages) degrades performance to near-GRPO levels, confirming that some form of token-level modulation is essential — but it does not establish that entropy is the optimal modulation signal. A comparative study would replace H_{i,t} in Equation 5 with alternative per-token features while keeping the branching structure and asymmetric design intact: (a) the policy's probability of the chosen token π_θ(o_{i,t} | q, o_{i,<t}) (inverse of entropy, measuring certainty rather than uncertainty), (b) the token's position in the response (normalized index t / |o_i|, testing whether earlier reasoning steps benefit more from reinforcement), (c) the token's syntactic role (testing whether certain parts of speech — e.g., connectors, verbs, numbers — are more informative to modulate), or (d) a learned scalar importance weight from a small auxiliary model trained to predict which tokens in correct responses are most predictive of final answer correctness. This experiment would use the paper's small-scale setup (Qwen3-1.7B-Base + MATH) for tractability, and would determine whether RL-ZVP's gains are attributable to the general principle of token-level modulation or specifically to entropy as a feature. A negative result (all alternatives underperform entropy) would strengthen the paper's entropy-based motivation; a positive result (some alternative outperforms entropy) would redirect the fine-grained reward shaping literature toward more effective token-level signals.
Combining RL-ZVP with existing fine-grained reward shaping methods to test for additive gains. The paper explicitly positions RL-ZVP as complementary to token-level reward shaping methods like those of Cui et al. (2025) and Wang et al. (2025), noting that these methods "still suffer from advantage vanishing when faced with zero-variance prompts." The natural follow-up is a combination experiment: apply an existing token-level reward shaping method (e.g., Wang et al.'s high-entropy minority token upweighting) to non-zero-variance prompts (replacing GRPO's uniform advantage broadcasting), while using RL-ZVP's entropy-guided formula for zero-variance prompts. The experiment would compare four conditions: GRPO baseline, RL-ZVP alone, the existing token-level method alone, and the combination. This would quantify: (a) how much of RL-ZVP's gain over GRPO is attributable to the zero-variance handling vs. the token-level modulation (by comparing RL-ZVP to the existing method, which modulates tokens but still zeroes out zero-variance prompts), and (b) whether the benefits are additive or subadditive (by comparing the combination to the better of the two individual methods). The paper's small-scale setup would be used for tractability. A finding of superadditive gains would strongly motivate integrating RL-ZVP into all fine-grained reward shaping pipelines; a finding of subadditive gains would suggest the mechanisms overlap and that practitioners should choose one approach based on their zero-variance ratio.
Stress-testing RL-ZVP under reward noise and verifier imperfection. All the paper's experiments use clean binary rewards from exact-match verification with the MATH grading function. In practical RLVR deployments, verifiers may be imperfect — grading errors, ambiguous answers, or partially correct solutions that receive binary reward when they deserve partial credit. A stress-test would introduce controlled noise into the reward signal: with probability p, a correct answer is flipped to incorrect (false negative) or an incorrect answer is flipped to correct (false positive). The experiment would sweep p ∈ {0.01, 0.05, 0.10, 0.20} and measure how GRPO vs. RL-ZVP performance degrades. The key question is whether RL-ZVP's reliance on absolute reward direction (positive vs. negative prompt classification in Equation 5) makes it more or less robust to reward noise than GRPO's contrastive approach. A false positive on an all-incorrect group would flip it from negative to positive under RL-ZVP's classification, causing the model to reinforce incorrect reasoning patterns — potentially more damaging than GRPO's zeroing out of that group. Conversely, a false negative on an all-correct group would flip it from positive to negative, penalizing correct reasoning. This experiment would clarify the robustness-accuracy tradeoff and determine whether RL-ZVP should be deployed only when verifier accuracy is above some threshold.
Practical Applications and Downstream Use Cases
Cost-efficient RLVR training on difficult datasets where the base model initially has low solve rates. The paper's Table 1 and Figure 3 characterize a common practical scenario: when training a modestly-sized base model (e.g., 8B parameters) on a challenging reasoning dataset (e.g., competition math), the model initially solves fewer than 5% of problems per batch. In this regime, GRPO wastes 95%+ of its generated rollouts on all-incorrect zero-variance prompts that produce zero gradient, while filtering approaches like DAPO require catastrophic oversampling (50× in the paper's measurements) to construct informative batches. RL-ZVP extracts gradient signal from every rollout — including the all-incorrect ones that dominate early training — providing denser feedback that Figure 5d shows prevents the sharp accuracy collapse GRPO experiences. The practical benefit is a 4–8 point accuracy improvement on standard math benchmarks at identical rollout budgets (Table 2), with smoother training dynamics that reduce the risk of wasting compute on runs that collapse and must be restarted. For a team training reasoning models on datasets like AIME or Olympiad problems where base model solve rates are inherently low, this translates directly to better final models and more reliable training runs.
Reducing total computation for a target accuracy level by eliminating rollout waste. The paper's comparison against filtering baselines (Table 2, Figure 4) shows that RL-ZVP with 540,672 rollouts (the standard GRPO budget) outperforms GRPO-DS-g with 2,859,008 rollouts (5.29× more) on the large scale. If a practitioner's goal is to reach, say, 50% average Acc@8 on the paper's benchmark suite, GRPO-DS-g requires over 5× more generated rollouts to achieve that target than RL-ZVP requires. Since the paper's Figure 2 shows rollout generation dominating training time, this translates to roughly 5× less wall-clock training time (ignoring the unquantified overhead of RL-ZVP's entropy computation). For organizations with limited GPU budgets or those iterating rapidly on training recipes, this efficiency gain directly converts to faster experimentation cycles and lower cloud compute costs. The specific deployment scenario is a training pipeline where the base model is known to be weak on the target dataset (zero-variance ratio >50%) — in this regime, RL-ZVP provides the largest relative efficiency gains over both standard GRPO and filtering approaches.
Stabilizing open-source RLVR training at the 1B–8B parameter scale. The paper's training dynamics (Figure 5) demonstrate a practical problem with GRPO on smaller models: Qwen3-8B-Base trained with GRPO on DAPO-Math-17k suffers a sharp accuracy drop from ~45% to <35% in the first 200 gradient steps, while RL-ZVP improves smoothly. For open-source practitioners training reasoning models at modest scales (where zero-variance ratios are highest due to limited base model capability), this instability can cause training runs to be abandoned prematurely or require careful tuning of learning rate schedules and warmup to avoid collapse. RL-ZVP provides an off-the-shelf solution — the paper used standard GRPO hyperparameters with no special stability modifications (Appendix A, Table 5), and the only change was the advantage formula for zero-variance prompts. The practical benefit is reduced need for hyperparameter tuning and training restarts, which is particularly valuable in resource-constrained settings where each training run represents a significant time investment. The specific deployment scenario is a researcher or small team training a 1.7B–8B reasoning model from a base checkpoint; adopting RL-ZVP over GRPO (a one-line change in the advantage computation in the verl framework, since the paper's implementation is a drop-in replacement for the standard GRPO objective) provides immediate stability improvements with no additional hyperparameter tuning required.
