Quantile Advantage Estimation: Stabilizing RLVR for LLM Reasoning

ArXiv: 2509.22611

🎯 Pitch

Reinforcement learning for LLM reasoning doesn’t just suffer from entropy collapse—preventing it can trigger entropy explosion that silently plateaus performance. This paper shows that simply swapping the mean baseline in GRPO or DAPO for a group-wise quantile baseline stabilizes both extremes, converts the advantage estimator into a hard query-gate that reinforces only rare successes on hard problems, and delivers consistent pass@1 gains across multiple model scales and math benchmarks.

---

1. Executive Summary

This paper introduces Quantile Advantage Estimation (QAE), a minimal modification to value-free RLVR methods that replaces the conventional mean reward baseline with a group-wise K-quantile baseline, implementing a response-level two-regime gate that routes updates to rare successes on hard queries and to remaining failures on easy ones. Evaluated on Qwen3-8B-Base, Qwen3-14B-Base, and Qwen3-30B-A3B-Base across AIME'24, AIME'25, and AMC'23, QAE stabilizes policy entropy — curbing both the entropy explosion that standard mean-baseline methods induce through negative-advantage samples and the entropy collapse that prior token-level controls target — while sparsifying credit assignment so that roughly 80% of sampled responses receive zero advantage. The one-line swap yields consistent pass@1 gains (e.g., +21.5% on AIME'24 for Qwen3-8B-Base over DAPO's Clip-Higher, and +8.6% when layered atop GSPO on the 30B model) while maintaining comparable pass@16 performance, establishing that baseline design — rather than token-level hyperparameter tuning — is the primary mechanism for scaling RLVR, and that principled entropy regulation requires two-sided control since mitigating collapse alone can inadvertently induce explosion.

2. Context and Motivation

The Core Problem: RLVR Training Oscillates Between Two Failure Modes

The fundamental problem this paper addresses is that Reinforcement Learning with Verifiable Rewards (RLVR) for LLM reasoning is unstable. Training runs predictably hit one of two failure regimes: entropy collapse, where the policy's output distribution sharpens prematurely and traps the model in narrow, suboptimal reasoning patterns, or entropy explosion, where the policy becomes excessively stochastic, degrading the quality of credit assignment and stalling performance.

This is not a minor numerical instability. As Figure 1 (left) demonstrates on a representative Qwen3-8B-Base run with DAPO's Clip-Higher, the training dynamics follow a characteristic pattern: after Clip-Higher successfully prevents early collapse, the policy entropy spikes between steps 10–80, and while this doesn't immediately destroy performance, it creates long-term consequences. After step 100, entropy remains high and volatile, while pass@1 plateaus. The model becomes increasingly stochastic without converting that stochasticity into better reasoning outcomes.

The paper's central claim is that this entropy dilemma — collapse and explosion as two poles of the same underlying problem — has been systematically misunderstood. Prior work treats entropy collapse as the sole threat and designs interventions (token-level clipping, KL penalties, uplifting low-probability tokens) to prevent it. But preventing collapse often pushes training into explosion, and this "success" is then mistaken for good performance because the immediate accuracy doesn't drop — it simply stops improving. The paper argues this plateau is the silent signature of explosion-induced degradation, and that failing to recognize it as a problem has caused the field to accept a performance ceiling that is not fundamental but rather an artifact of poor baseline design.

Why This Problem Matters: Practical and Scientific Significance

The practical stakes are high. RLVR has become central to eliciting reasoning capabilities from large language models. OpenAI's o1 (ope), DeepSeek R1 (DeepSeek-AI et al., 2025), Kimi K1.5 (Team et al., 2025), and numerous open-source systems (Yue et al., 2025; Hu et al., 2025) all rely on value-free RL algorithms — primarily GRPO (Shao et al., 2024) or its derivative DAPO (Yu et al., 2025) — as the core fine-tuning step after pretraining and supervised fine-tuning. These methods are the difference between a base model that generates plausible-sounding but incorrect solutions and a reasoning model that can engage in structured, multi-step deduction.

If these algorithms are fundamentally bottlenecked by training instability — and specifically, by instability that current approaches only address on one side — then the entire RLVR pipeline is leaving performance on the table. The observation in Figure 2 that Clip-Higher's pass@1 plateaus while entropy remains elevated (Observation 1: token-level control does not guarantee sustained reasoning gains) suggests exactly this: the current generation of RLVR methods has hit an artificial ceiling, not a capability ceiling. The QAE modification directly addresses this ceiling, yielding substantial pass@1 improvements (e.g., +21.5% on AIME'24 for the 8B model, Table 2) without changing the model architecture, training data, or decoding strategy.

The scientific significance is equally important. The paper makes a strong case that the entropy dilemma is a baseline-design problem, not a token-level hyperparameter problem. This is a conceptual reframing. Prior work overwhelmingly focuses on how much to update (clipping ratios, adaptive learning rates, gradient penalties) and where to update (token-level importance weighting, raising log-probabilities of low-probability tokens). The paper argues that the more fundamental question is who gets updated — which samples in a group receive non-zero advantage. The mean baseline distributes advantage broadly and symmetrically (Figure 4, middle/right, shows GRPO/DAPO apply weight proportional to $\sqrt{p(1-p)}$, hitting all queries with intermediate success rates the hardest), while the quantile baseline gates advantage asymmetrically based on query difficulty. This is a fundamentally different mechanism that the paper formally analyzes through an entropy-covariance identity, proving two-sided entropy safety guarantees that no token-level method can provide.

Prior Approaches and Where They Fall Short

The Dominant Paradigm: Value-Free RL with Mean Baselines

The workhorse algorithms for RLVR are Group Relative Policy Optimization (GRPO) and Dynamic Sampling Policy Optimization (DAPO). Both are value-free: unlike Proximal Policy Optimization (PPO, Schulman et al., 2017), which requires a separately trained value network to estimate advantages, GRPO and DAPO estimate advantages entirely within each batch by comparing individual sample rewards to the group's statistics:

$\hat{A}_i = \frac{R_i - \text{mean}(\{R_k\}_{k=1}^G)}{\text{std}(\{R_k\}_{k=1}^G)}$

where each response receives a binary reward $R_i \in \{0, 1\}$ based on whether its final answer matches the ground truth. This is computationally elegant — no value network to train, no reward model to maintain — and has driven impressive results.

GRPO (Section 2, Eq. 2) adds a KL penalty to prevent the policy from drifting too far from its initialization during training. The intuition is that constraining the KL divergence preserves general capabilities while the reward signal shapes the reasoning distribution. GRPO also uses symmetric PPO-style clipping on both the low and high ends.

DAPO (Section 2) refines GRPO along several axes. It removes the KL penalty entirely (finding it unnecessary with proper clipping), introduces asymmetric clipping ($\epsilon_{\text{low}}$ vs $\epsilon_{\text{high}}$) to allow larger updates for advantageous actions, normalizes the objective at the token level (dividing by the total number of tokens in the group), and crucially implements a dynamic sampling constraint:

$0 < |\{o_i \mid \text{is equivalent}(a, o_i)\}| < G$

This ensures every training batch contains both correct and incorrect responses — a guard against degenerate batches where all samples are correct (producing zero advantage for everyone) or all incorrect (also producing zero advantage because the mean equals every value). DAPO's Clip-Higher mechanism ($\epsilon_{\text{high}}$ specifically) is the primary prior attempt to control entropy, allowing positive-advantage samples to receive larger updates by using a wider upper clipping bound.

Three Categories of Prior Entropy Interventions

The paper groups existing approaches to entropy control in RLVR into three strands (Section 7):

1. Mechanistic analyses of where exploration concentrates. Wang et al. (2025b) identify high-entropy "forking" tokens — specific positions in the generation where the model's uncertainty branches — and show that 20% of tokens drive 80% of exploration. Qian et al. (2025) study "thinking tokens" as information peaks. Cui et al. (2025) analyze sequence-level entropy dynamics but focus primarily on collapse. These works are descriptive: they characterize the phenomenon but don't propose structural solutions.

2. Objective-level regulation. Zhang et al. (2025) demonstrate that maximum entropy objectives can mislead policy optimization in LLM reasoning. Zhu et al. (2025) show that learning primarily from negative samples preserves diversity better than balanced learning. Agarwal et al. (2025) find that entropy minimization itself can be unreasonably effective. These works modify what the objective optimizes — e.g., changing the positive-negative sample ratio or adding entropy terms — but they operate at the level of the loss function rather than the baseline.

3. Recipe/system-level heuristics. Cheng et al. (2025) shape the advantage function to inject exploration. Wang et al. (2025a) modulate gradients based on uncertainty estimates. Song et al. (2025) propose outcome-based exploration. These are pragmatic engineering solutions that work but don't address the structural properties of the mean baseline.

Where All Prior Approaches Fall Short: Asymmetric Treatment and Incomplete Guarantees

The paper identifies three specific failure modes in this landscape:

Failure 1: One-sidedness breeds the opposite pathology. Every collapse-oriented intervention — Clip-Higher, uplifting low-probability tokens, learning from negative samples — pushes the policy away from determinism. When this succeeds in preventing collapse, it can push too far, triggering explosion. The paper's Observation 2 (Figure 3) shows that Clip-Higher initially promotes diverse anthropomorphic tokens (wait, perhaps) but by step 80-200 the distribution homogenizes around rigid templates (so, let, find), and accuracy plateaus. The intervention worked to prevent collapse but failed to sustain productive exploration. This is because token-level clipping is fundamentally one-sided: it can prevent the policy from becoming too deterministic but cannot prevent it from becoming too random. As the paper argues in Section 4.3 (Proposition 4.2): "Token-level mechanisms only rescale steps and do not change the response-level baseline, so they cannot realize [two-sided] guarantees."

Failure 2: The mean baseline is structurally flawed under reward outliers. The key insight in Section 3.2 (Observation 3 and Figure 4 left) is that entropy explosion is disproportionately driven by negative-advantage samples. Positive-advantage samples maintain relatively stable entropy; it's the negative-advantage samples that show the steepest growth and largest share of entropy in early training. Why? Because the mean baseline $b = p(q)$ is sensitive to the empirical success rate. On a hard query where only one out of $G$ samples is correct ($p \approx 1/G$), that single success has a moderately positive advantage, but the $G-1$ failures each get a slightly negative advantage. The GRPO/DAPO objective then pushes the policy away from all $G-1$ negative samples simultaneously, creating a strong dispersion force that inflates entropy. On an easy query with $p \approx 1$, the same thing happens in reverse: the few failures get strongly negative advantages, and the policy disperses away from those failure modes. In both extremes, the mean baseline drives entropy growth. Only at intermediate difficulty ($p \approx 0.5$) does the symmetric weight $\sqrt{p(1-p)}$ balance positive and negative forces.

The paper's quantile baseline directly addresses this: it nullifies the disadvantage on one side per regime, so updates only push the policy toward rare successes (hard queries) or away from residual failures (easy queries), never both simultaneously. This structural difference is what produces the two-sided entropy safety guarantee.

Failure 3: No framework unifies exploration and exploitation through a single controllable parameter. Prior methods for adjusting exploration (positive-negative ratios, entropy bonuses, adaptive clipping) each introduce their own hyperparameters with non-linear interactions. The paper's framing — $K$ as a single knob that shifts the update focus between exploration and exploitation by changing only the quantile threshold — provides a conceptual unification that simplifies both intuition and implementation. The operational rule-of-thumb from Section 6 (choose $K=0.6$ when entropy is low, $K=0.4$ when entropy is high) gives practitioners a direct, observable diagnostic rather than a blind hyperparameter sweep.

How This Paper Positions Itself

The paper's positioning is explicit and sharp: it argues that baseline design is the primary mechanism for entropy regulation in RLVR, and that the field's overwhelming focus on token-level heuristics has been a category error. The evidence for this claim is multifaceted:

Theoretical grounding (Section 4.3). The paper proves that under first-order softmax updates (the entropy-covariance identity adapted from Cui et al., 2025), the entropy change $\Delta H(q; b)$ is strictly increasing in the baseline $b$. This is a clean, monotonic relationship: higher baselines → larger entropy increases; lower baselines → smaller entropy increases. The K-quantile baseline achieves the extremal values: $b_K = 0$ (minimum entropy increase) when $p(q) \leq 1-K$ (hard queries) and $b_K = 1$ (maximum entropy increase) when $p(q) > 1-K$ (easy queries). No token-level method can produce this guarantee because token-level clipping rescales the magnitude of updates but does not change the response-level baseline $b(q)$ that determines the sign pattern across samples in a group.

Empirical decomposition (Section 3.2). The paper traces the entropy explosion phenomenon to its root cause: negative-advantage samples under the mean baseline (Figure 4, left). Observation 4 (Table 1) shows that tuning the token-level hyperparameter $\epsilon_{\text{high}}$ from 0.20 to 0.28 — the primary knob in DAPO for controlling update aggressiveness — produces only marginal effects (±a few percent) and does not resolve the late-stage plateau. The issue is not "how much to update per token" but "which responses to update at all."

Discriminative reformulation (Section 4.2, Proposition 4.1). By recasting the GRPO objective in the discriminative framework of DisCO (Li et al., 2025), the paper shows that the mean baseline produces a symmetric, bell-shaped query weight $\sqrt{p(q)(1-p(q))}$ that most heavily updates moderately-difficult queries while down-weighting both very easy and very hard ones. The quantile baseline replaces this with a gated, asymmetric weighting: only positive examples contribute on hard queries (weight $\sqrt{p/(1-p)}$, which amplifies rare successes), and only negative examples contribute on easy queries (weight $\sqrt{(1-p)/p}$, which amplifies remaining failures). The two regimes are mutually exclusive — a query is either "hard" ($p \leq 1-K$, exploitation-focused) or "easy" ($p > 1-K$, exploration-focused) — and the binary gate eliminates precisely the symmetric dispersion force that drives entropy explosion.

Compositional compatibility (Table 2). The paper emphasizes that QAE is orthogonal to existing techniques. It improves Clip-Higher, Clip-Cov, KL-Cov, and GSPO — four different token-level and sequence-level methods — without changing their hyperparameters. This demonstrates that baseline design operates at a different level of the optimization stack than token-level controls, and that the two can be composed additively. Figure 6 (b-c) provides the most instructive ablation: when Clip-Higher uses weak clipping ($\epsilon_{\text{high}} = 0.28$), entropy explosion dominates and masking negatives (NEG-MASK) nearly matches full QAE; with strong clipping ($\epsilon_{\text{high}} = 0.20$), collapse pressure dominates and masking positives (POS-MASK) is more important. The mean baseline forces a one-size-fits-all solution; QAE adapts through a single parameter $K$.

The response-level 80/20 rule (Figure 5, right). A striking empirical finding: with tuned $K$, roughly 80% of responses receive zero advantage. This sparsity is not engineered through explicit regularization — it emerges naturally from the quantile baseline gating. The paper interprets this as evidence that the mean baseline is deeply redundant: most responses in a group provide no useful gradient signal, and the quantile baseline filters them out, concentrating the computational budget on the informative minority. This connects conceptually to Wang et al. (2025b)'s finding that 20% of tokens drive 80% of exploration, but operates at the response level rather than the token level.

In positioning itself, the paper also explicitly connects to Arnal et al. (2025), which analyzes tunable baselines in REINFORCE. The quantile baseline is a data-adaptive, group-level instantiation of that idea, adapted for the specific structure of RLVR with binary rewards and group sampling. Unlike a fixed constant baseline (which would be statistically inefficient), the K-quantile adapts to the empirical success rate $p(q)$ of each query, producing the two-regime gating behavior without requiring per-query manual tuning.

3. Technical Approach

3.1 Reader Orientation

This is primarily an algorithmic analysis and lightweight modification paper whose core idea is that the mean reward baseline used in value-free RLVR algorithms (GRPO, DAPO) is the root cause of both entropy collapse and entropy explosion, and that replacing it with a group-wise K-quantile baseline provides a single-parameter mechanism to control the exploration–exploitation balance through two-sided entropy safety guarantees — no token-level hyperparameter tuning required.

The system being modified is a standard RLVR fine-tuning pipeline: a pretrained base LLM generates groups of candidate solutions per query, receives binary correctness rewards, computes per-sample advantages by normalizing rewards against group statistics, and updates its policy via a clipped surrogate objective. QAE changes exactly one component — the baseline used in advantage estimation — and leaves everything else (clipping, dynamic sampling, token-level loss normalization, training hyperparameters) untouched. The "shape" of the solution is therefore a drop-in replacement: swap mean({R_i}) for Quantile_K({R_i}) in the advantage formula, and the system's entropy dynamics shift from unconstrained (prone to both collapse and explosion) to bounded (two-sided safety guarantee with a single tunable knob $K$).

3.2 Big-Picture Architecture (Diagram in Words)

The QAE system has four conceptual layers stacked on top of a standard value-free RLVR pipeline:

1. Base Policy ($\pi_{\theta_{\text{old}}}$) — the pretrained LLM (e.g., Qwen3-8B-Base) that generates groups of $G$ complete reasoning trajectories per query via autoregressive sampling. This is the standard proposal distribution in GRPO/DAPO, unchanged by QAE.

2. Reward Assigner — a verifier that checks each generated response's final answer against a ground-truth answer and assigns a binary reward $R_i \in \{0, 1\}$. This is domain-specific (math benchmarks use exact-answer matching with grading functions) and is unchanged by QAE.

3. Quantile Advantage Estimator (the QAE contribution) — for each group of $G$ responses to a query $q$, this module computes the empirical success rate $p(q) = \frac{1}{G} \sum R_i$, determines the K-quantile baseline $b_K(q)$ (equal to 0 if $p(q) \leq 1-K$, equal to 1 if $p(q) > 1-K$), and produces standardized advantages $\hat{A}_i = (R_i - b_K(q)) / (\text{std}(\{R_j\}) + \epsilon)$. This replaces the mean baseline $\hat{A}_i = (R_i - \text{mean}(\{R_j\})) / \text{std}(\{R_j\})$ in GRPO/DAPO.

4. Policy Update Module — the standard clipped surrogate objective (DAPO's token-level normalized loss with asymmetric clipping $[\epsilon_{\text{low}}, \epsilon_{\text{high}}]$) that takes the per-token probability ratios and the advantages from layer 3 and produces a gradient update to $\theta$. This module is identical to DAPO; QAE only changes the advantages fed into it.

Information flow: A query $q$ enters → the base policy samples $G$ responses → the reward assigner produces binary rewards → the quantile advantage estimator computes $p(q)$, determines the regime (hard vs. easy) via the threshold $1-K$, and produces sparse advantages (zero for most samples) → the policy update module applies the clipped surrogate loss per token, normalized by total group tokens, updating $\theta$ → the updated policy serves as $\pi_{\theta_{\text{old}}}$ for the next iteration.

3.3 Roadmap for the Deep Dive

• First, the formal definition of the K-quantile baseline and its reduction to a threshold decision rule on the empirical success rate $p(q)$. This is the mathematical core — understanding why a quantile produces exactly two regimes for binary rewards.

• Second, the discriminative reformulation (Proposition 4.1) that recasts the GRPO objective under the quantile baseline, showing how it differs from the standard mean-baseline objective in query weighting, gating, and discriminative term structure. This connects the mathematical form to the update behavior.

• Third, the gradient analysis and query-level weight comparison with the standard GRPO/DAPO objective. This explains why the quantile baseline produces different entropy dynamics: the symmetric $\sqrt{p(1-p)}$ weight is replaced by asymmetric monotonic factors and one side is nullified per query.

• Fourth, the entropy-covariance identity and two-sided safety guarantee (Proposition 4.2). This is the theoretical justification — the proof that under first-order softmax updates, the K-quantile baseline achieves extremal entropy change bounds that no token-level method can realize.

• Fifth, the response-level sparsity mechanism that emerges from the two-regime gate: why roughly 80% of responses receive zero advantage, why this is beneficial rather than wasteful, and how it connects to the observed training stability.

3.4 Detailed, Sentence-Based Technical Breakdown

This section provides a complete technical walkthrough of Quantile Advantage Estimation, from the mathematical definition through the discriminative reformulation to the entropy safety guarantees. The treatment is self-contained and assumes only familiarity with the GRPO/DAPO framework established in the prior sections.

---

The K-Quantile Baseline: Definition and Binary Reward Reduction

The core mathematical innovation of QAE is replacing the group mean in the GRPO/DAPO advantage formula with a group-wise empirical K-quantile. For a query $q$ and a group of $G$ sampled responses $\{o_i\}_{i=1}^G$ with binary rewards $R_i \in \{0, 1\}$, the empirical cumulative distribution function (CDF) of rewards is defined as:

$\hat{F}_q(x) := \frac{1}{G} \sum_{j=1}^G \mathbb{1}\{R_j \leq x\}$

where $\mathbb{1}\{\cdot\}$ is the indicator function (1 if the condition holds, 0 otherwise), and $x \in \{0, 1\}$ is the reward value being evaluated.

What it computes: $\hat{F}_q(x)$ is the fraction of responses in the group whose reward is less than or equal to $x$. For binary rewards, $\hat{F}_q(0)$ equals the fraction of incorrect responses ($1 - p(q)$), and $\hat{F}_q(1)$ equals 1 (all responses have reward ≤ 1). This is a standard empirical CDF — no approximation, just counting.

The right-continuous K-quantile baseline is then defined as:

$b_K(q) := Q_K(\{R_j\}_{j=1}^G) = \inf\{x : \hat{F}_q(x) \geq K\}, \quad K \in (0, 1)$

where $\inf$ denotes the infimum (greatest lower bound), $K$ is the quantile parameter (a number between 0 and 1, exclusive), and $\hat{F}_q$ is the empirical CDF defined above.

What it computes: $b_K(q)$ is the smallest reward value $x$ such that at least a fraction $K$ of the group's rewards are $\leq x$. In plain language: it finds the reward threshold below which at least $K \times 100\%$ of the samples fall. For binary $\{0, 1\}$ rewards, this calculation has exactly two possible outcomes: if the fraction of zeros ($1 - p(q)$) is at least $K$, then the baseline is 0; otherwise, it must be 1.

Why this form: The right-continuous quantile definition ensures the baseline is well-defined for all $K \in (0,1)$ and produces a deterministic threshold. The infimum construction handles edge cases where the empirical CDF jumps (e.g., when all rewards are identical) gracefully — it picks the smallest $x$ satisfying the quantile condition rather than requiring an interpolation that doesn't exist for a discrete distribution.

For the specific case of binary rewards with empirical success rate $p(q) = \frac{1}{G} \sum R_i$, this definition reduces to a clean threshold rule:

$b_K(q) = \begin{cases} 0, & p(q) \leq 1 - K \\ 1, & p(q) > 1 - K \end{cases}$

where $p(q)$ is the fraction of correct responses in the group, and $1-K$ is the difficulty threshold.

What this reduction means operationally: The quantile baseline is a binary switch. If the group success rate is low — $p(q) \leq 1-K$, meaning the query is "hard" for the current policy — then the baseline is 0. Correct responses ($R_i = 1$) get positive advantage ($1 - 0 = 1$, before standardization), and incorrect responses ($R_i = 0$) get zero advantage ($0 - 0 = 0$). If the group success rate is high — $p(q) > 1-K$, meaning the query is "easy" — then the baseline is 1. Correct responses get zero advantage ($1 - 1 = 0$), and incorrect responses get negative advantage ($0 - 1 = -1$, before standardization).

Why this threshold form matters: The parameter $K$ directly controls the boundary between "hard" and "easy." A small $K$ (e.g., $K = 0.2$) makes $1-K = 0.8$, so most queries — those with $p(q) \leq 0.8$ — are classified as hard and receive the $b_K = 0$ (exploitation) regime. A large $K$ (e.g., $K = 0.8$) makes $1-K = 0.2$, so most queries — those with $p(q) > 0.2$ — are classified as easy and receive the $b_K = 1$ (exploration) regime. This is the core control mechanism: $K$ acts as a lever that shifts how many queries are updated to reinforce rare successes versus suppress residual failures.

The standardized advantage with the K-quantile baseline is then:

$\hat{A}_i = \frac{R_i - b_K(q)}{\text{std}(\{R_j\}_{j=1}^G) + \varepsilon}, \quad \varepsilon > 0$

where $\text{std}(\{R_j\})$ is the empirical standard deviation of rewards in the group, and $\varepsilon$ is a small constant (typically $10^{-8}$ or similar) preventing division by zero when all rewards are identical ($p \in \{0, 1\}$).

What standardization does: For a group with success rate $p$, the binary reward variance is $p(1-p)$, so the standard deviation is $\sqrt{p(1-p)}$. Standardization divides the centered reward by this quantity, producing advantages that are scale-invariant with respect to the group's difficulty. When $b_K = 0$ and $R_i = 1$ on a hard query, the advantage is approximately $1/\sqrt{p(1-p)}$, which is large when $p$ is small — amplifying the signal from rare successes. When $b_K = 1$ and $R_i = 0$ on an easy query, the advantage is approximately $-1/\sqrt{p(1-p)}$, which is large in magnitude when $1-p$ is small — amplifying the signal from rare failures.

Why not use the mean: The mean baseline $b = p(q)$ produces symmetric advantages: correct responses get $(1-p)/\sqrt{p(1-p)} = \sqrt{(1-p)/p}$ and incorrect responses get $-p/\sqrt{p(1-p)} = -\sqrt{p/(1-p)}$. Both signs are always present (unless $p = 0$ or $p = 1$, in which case the dynamic sampling constraint in DAPO already prevents such batches), which means the policy is simultaneously pushed toward correct responses and away from incorrect responses on every query. The quantile baseline breaks this symmetry: on hard queries, only the "pull toward correctness" is active; on easy queries, only the "push away from errors" is active. This asymmetric gating is the structural property that produces two-sided entropy control.

---

Discriminative Reformulation: From Advantage Estimation to Query-Level Weighting

To understand why the quantile baseline produces different training dynamics, the paper recasts the GRPO objective in a discriminative framework, following the analysis of DisCO (Li et al., 2025). This reformulation separates the objective into a query-level weight (how much each query contributes to the gradient) and a discriminative term (how the policy distinguishes correct from incorrect responses).

Let $\pi^+_{\text{old}}(\cdot \mid q)$ denote the conditional distribution of responses that received reward 1 (correct responses), and $\pi^-_{\text{old}}(\cdot \mid q)$ denote the conditional distribution of responses that received reward 0 (incorrect responses). These are empirical distributions: for a group with $pG$ correct responses, $\pi^+_{\text{old}}$ is uniform over those correct responses, and $\pi^-_{\text{old}}$ is uniform over the $(1-p)G$ incorrect responses.

Let $s^+_\theta(o, q)$ be the token-normalized score function for positive examples — essentially the average (over tokens in response $o$) of the policy gradient term when the advantage is positive, measuring how much the policy's log-probability of generating $o$ needs to increase. Similarly, let $s^-_\theta(o, q)$ be the token-normalized score function for negative examples — the average policy gradient term when the advantage is negative, measuring how much the log-probability needs to decrease.

Under these definitions, Li et al. (2025) showed that the standard GRPO objective (using the mean baseline) can be written as:

$J_{\text{GRPO}}(\theta) = \mathbb{E}_q \left[ \underbrace{\sqrt{p(q)(1 - p(q))}}_{\text{query weight}} \cdot \underbrace{\mathbb{E}_{o \sim \pi^+_{\text{old}}, o' \sim \pi^-_{\text{old}}} \left[ s^+_\theta(o, q) - s^-_\theta(o', q) \right]}_{\text{discriminative term}} \right]$

where $p(q)$ is the empirical success rate for query $q$, and $\mathbb{E}_q$ denotes expectation over the query distribution.

What each component means in operational terms:

• The query weight $\sqrt{p(q)(1-p(q))}$ is a symmetric, bell-shaped function of the success rate. It is maximal when $p = 0.5$ (the query is at exactly intermediate difficulty — half the samples are correct) and goes to zero as $p \to 0$ or $p \to 1$. This means GRPO/DAPO with the mean baseline focuses gradient effort on moderately difficult queries and largely ignores queries that are trivially easy (already solved) or impossibly hard (no successes).

• The discriminative term $\mathbb{E}_{o \sim \pi^+, o' \sim \pi^-} [s^+_\theta(o, q) - s^-_\theta(o', q)]$ measures the gap between the policy's score on correct responses and incorrect responses. Maximizing this gap means increasing the probability of generating correct responses and decreasing the probability of generating incorrect responses. This term is always active — both correct and incorrect responses contribute, regardless of query difficulty — because the mean baseline assigns non-zero advantage to both types.

Why this form emerges from the mean baseline: When the advantage for correct responses is $\sqrt{(1-p)/p}$ and for incorrect responses is $-\sqrt{p/(1-p)}$, the magnitude of each term scales with the inverse square root of the opposite class's frequency. The product of these scaling factors with the class probabilities ($p$ for correct, $1-p$ for incorrect) in the expectation produces the common factor $\sqrt{p(1-p)}$. The symmetric weight penalizes queries at both extremes equally, which the paper argues is suboptimal: easy queries don't need to be pushed away from errors (they're mostly correct already — the residual errors are the only informative signal), and hard queries don't need to be pulled toward correctness and pushed from errors simultaneously (the few successes are the only informative signal).

---

The Quantile-Regulated Objective (Proposition 4.1)

Substituting the K-quantile baseline into a GRPO-style surrogate objective (using the same token-normalized loss, homogeneous scaling of the clipped surrogate, and binary reward structure) produces a fundamentally different decomposition. The paper proves this in Proposition 4.1, stated here in full operational detail.

$J_{\text{Quantile}}(\theta) = \mathbb{E}_q \left[ \mathbb{1}\{p(q) \leq 1-K\} \sqrt{\frac{p(q)}{1-p(q)}} \; \mathbb{E}_{o \sim \pi^+_{\text{old}}(\cdot|q)} s^+_\theta(o, q) \;-\; \mathbb{1}\{p(q) > 1-K\} \sqrt{\frac{1-p(q)}{p(q)}} \; \mathbb{E}_{o' \sim \pi^-_{\text{old}}(\cdot|q)} s^-_\theta(o', q) \right]$

where $\mathbb{1}\{\cdot\}$ is the indicator function (1 when the condition is true, 0 otherwise), $p(q)$ is the empirical success rate for query $q$, $K$ is the quantile parameter, $\pi^+_{\text{old}}$ is the empirical distribution over correct responses in the group, $\pi^-_{\text{old}}$ is the empirical distribution over incorrect responses, $s^+_\theta(o, q)$ is the token-normalized score function for positive examples (measuring how much to increase the log-probability of response $o$), and $s^-_\theta(o', q)$ is the token-normalized score function for negative examples (measuring how much to decrease the log-probability of response $o'$).

What this equation computes, piece by piece:

The objective has two mutually exclusive terms, selected by the indicator functions:

• Hard-query term (first line, active when $p(q) \leq 1-K$): The indicator $\mathbb{1}\{p(q) \leq 1-K\}$ gates this term on for queries where the success rate is below the threshold. The weight is $\sqrt{p(q)/(1-p(q))}$, which is small when $p$ is near 0 (amplifying the rare positive signal proportionally to the square root of the odds ratio), and grows as $p$ increases toward $1-K$. The expectation $\mathbb{E}_{o \sim \pi^+_{\text{old}}} s^+_\theta(o, q)$ averages the token-normalized score function only over the correct responses — incorrect responses are completely masked (their score function does not appear). This means the gradient update on hard queries only pushes the policy toward generating responses similar to the few successes, with no countervailing push away from the many failures.

• Easy-query term (second line, active when $p(q) > 1-K$): The indicator $\mathbb{1}\{p(q) > 1-K\}$ gates this term on for queries where the success rate is above the threshold. The weight is $\sqrt{(1-p(q))/p(q)}$, which is large when $p$ is near 1 (amplifying the rare negative signal proportionally to the square root of the inverse odds ratio), and shrinks as $p$ approaches $1-K$ from above. The expectation $\mathbb{E}_{o' \sim \pi^-_{\text{old}}} s^-_\theta(o', q)$ averages the token-normalized score function only over incorrect responses — correct responses are completely masked. This means the gradient update on easy queries only pushes the policy away from generating responses similar to the few failures, with no reinforcing push on the many successes.

Note the minus sign between the two terms: the hard-query term adds to the objective (pushing toward correct responses), while the easy-query term subtracts (pushing away from incorrect responses). Both increase $J_{\text{Quantile}}$ when the policy improves.

Why this form is structurally different from GRPO:

1. Gating replaces symmetry. The GRPO objective always includes both $s^+_\theta$ and $s^-_\theta$ terms with symmetric weights. The quantile objective gate-selects exactly one term per query based on the success rate. This eliminates the simultaneous push-pull dynamic that drives entropy growth in the mean baseline — on hard queries, the policy is not simultaneously pushed away from the $G-1$ failure modes, which is what causes the dispersion force identified in Observation 3 (Figure 4, left).

2. Weights are monotonic, not bell-shaped. The GRPO weight $\sqrt{p(1-p)}$ is symmetric and peaks at $p=0.5$, down-weighting both easy and hard queries. The quantile weights are monotonic: $\sqrt{p/(1-p)}$ increases with $p$ (hard queries with $p$ near 0 get very small weight because successes are rare and the signal is noisy; as $p$ approaches $1-K$, the weight increases because there are more successes to learn from), while $\sqrt{(1-p)/p}$ decreases with $p$ (easy queries with $p$ near 1 get very small weight; as $p$ approaches $1-K$ from above, the weight increases because there are more failures to correct). This focuses gradient effort on queries near the difficulty boundary $1-K$, where the signal-to-noise ratio is highest.

3. Sparsity is built in, not imposed. Because exactly one of the two indicator functions fires for each query (they partition the space of $p$ values, with $p > 1-K$ and $p \leq 1-K$ being complementary), exactly one class of responses (correct or incorrect) contributes to the gradient per query. The other class receives literally zero advantage — not a small number, not a clipped value, but exactly zero. This is the mechanism behind the 80% sparsity observation (Figure 5, right): for a tuned $K$, most responses in most groups fall into the masked category.

---

Gradient Analysis: How the Quantile Baseline Reshapes the Update Distribution

To understand the practical effect of the quantile-regulated objective, compare it to GRPO on a per-query basis at different difficulty levels.

On a very hard query ($p \approx 0.1$, one correct response out of 10):

• GRPO: The query weight is $\sqrt{0.1 \times 0.9} \approx 0.3$. Both correct and incorrect responses contribute to the discriminative term. The policy is pulled toward the single correct response (with advantage $\sqrt{0.9/0.1} \approx 3.0$) and pushed away from the 9 incorrect responses (each with advantage $-\sqrt{0.1/0.9} \approx -0.33$). The dispersion away from 9 responses creates a much larger total gradient norm than the attraction toward 1 response, driving entropy growth. This is the "negative-advantage-dominated entropy explosion" observed in Figure 4 (left).

• QAE (with $K=0.4$, so $1-K=0.6$, and $p=0.1 \leq 0.6$): Only the hard-query term activates. The weight is $\sqrt{0.1/0.9} \approx 0.33$. Only the single correct response contributes — the 9 incorrect responses have zero advantage. The gradient update is a pure attraction toward the successful trajectory, with no dispersion away from failures. Entropy remains controlled because the policy is not forced to simultaneously unlearn 9 different failure modes.

On a very easy query ($p \approx 0.9$, nine correct responses out of 10):

• GRPO: The query weight is $\sqrt{0.9 \times 0.1} \approx 0.3$. Both correct and incorrect responses contribute. The policy is pulled toward 9 correct responses (each with advantage $\sqrt{0.1/0.9} \approx 0.33$) and pushed away from the single incorrect response (with advantage $-\sqrt{0.9/0.1} \approx -3.0$). The strong push away from one rare failure can cause the policy to over-disperse, losing the structure that made the query easy in the first place.

• QAE (with $K=0.4$, so $1-K=0.6$, and $p=0.9 > 0.6$): Only the easy-query term activates. The weight is $\sqrt{0.1/0.9} \approx 0.33$. Only the single incorrect response contributes — the 9 correct responses have zero advantage. The gradient update is a pure repulsion from the failure mode, with no reinforcing attraction toward the successes (which are already sufficiently probable). This is "targeted refinement" on easy queries: fix what's still broken without disturbing what works.

On a moderately difficult query ($p \approx 0.5$):

• GRPO: The query weight is $\sqrt{0.5 \times 0.5} = 0.5$, the maximum possible. Both correct and incorrect responses contribute with symmetric advantages (magnitude $\sqrt{0.5/0.5} = 1.0$ each). The gradient balances attraction and repulsion equally.

• QAE (with $K=0.4$): $p = 0.5 \leq 0.6$, so the hard-query term activates. Only correct responses contribute. The weight is $\sqrt{0.5/0.5} = 1.0$. This is where QAE's behavior is most different from GRPO: on queries with a moderate success rate, QAE ignores the failure modes entirely and focuses exclusively on reinforcing what worked, while GRPO splits its attention. Whether this is beneficial depends on whether the failures on moderate queries are informative — the paper's empirical results suggest that for the math reasoning benchmarks tested, ignoring failures on queries with $p \leq 0.6$ is more effective than trying to learn from them.

---

The Entropy–Covariance Identity and Proof of Two-Sided Entropy Safety (Proposition 4.2)

The paper provides a theoretical guarantee that the quantile baseline achieves two-sided entropy control — preventing both collapse and explosion — under a simplified but instructive model of policy updates. The analysis uses a bandit reduction and a first-order approximation of softmax policy gradients.

Setup: Bandit reduction. Instead of the full token-by-token autoregressive generation, consider a simplified setting where producing a complete response $y$ to query $q$ is treated as a single action (a "bandit arm"). The policy $\pi(\cdot \mid q)$ is a softmax distribution over possible responses, and the entropy $H(q)$ is the length-normalized (token-averaged) policy entropy. This reduction abstracts away the token-level structure to focus on the response-level effect of the baseline. While a simplification, it captures the core mechanism: the baseline determines which responses in a group receive non-zero advantage, and the policy gradient moves probability mass toward or away from those responses.

First-order logit update. Under a softmax policy parameterized by logits, a gradient step with learning rate $\eta$ on the policy gradient objective produces (to first order in $\eta$) a change in the logits proportional to the advantage-weighted policy gradient. The entropy change induced by this logit update can be expressed through an entropy–covariance identity (adapted from Cui et al., 2025):

$\Delta H(q) \approx -\eta \; \text{Cov}_{y \sim \pi(\cdot|q)} \left( \log \pi(y \mid q), \; \pi(y \mid q) \; \hat{A}(y, q) \right), \quad \eta > 0$

where $\Delta H(q)$ is the change in entropy for query $q$ after one gradient step, $\eta$ is the learning rate (step size), $\text{Cov}_{y \sim \pi}$ denotes the covariance over responses $y$ sampled from the current policy $\pi$, $\log \pi(y \mid q)$ is the log-probability of response $y$ under the policy, $\pi(y \mid q)$ is the probability itself (acting as a weight in the covariance), and $\hat{A}(y, q) = r(y, q) - b(q)$ is the response-level advantage with baseline $b(q)$ and binary reward $r(y, q) \in \{0, 1\}$.

What this identity says in plain language: The entropy change is proportional to the negative covariance between how surprising a response is ($\log \pi$, with more negative values meaning higher probability) and its probability-weighted advantage. If high-advantage responses tend to be high-probability (which makes $\log\pi$ less negative), the covariance is positive and the negative sign in front means entropy decreases — the policy concentrates. If high-advantage responses tend to be low-probability, the covariance is negative and entropy increases — the policy disperses. The baseline $b(q)$ shifts the sign pattern of advantages, which changes this covariance, which changes the entropy dynamics.

Baseline as a linear knob. For a binary reward $r \in \{0, 1\}$ and any baseline $b \in [0, 1]$, define:

$F_q(b) := \text{Cov}_{\pi}\left( \log \pi, \; \pi (r - b) \right)$

By linearity of covariance (the covariance of a sum is the sum of covariances, and scaling one variable scales the covariance):

$F_q(b) = \text{Cov}_{\pi}(\log \pi, \pi r) - b \cdot \text{Cov}_{\pi}(\log \pi, \pi) =: F_q(0) - b \cdot C_q$

where $C_q := \text{Cov}_{\pi}(\log \pi, \pi)$ is the covariance between the log-probability and the probability itself under the policy.

Why $C_q > 0$ always (for non-uniform policies): The function $u \mapsto \log u$ and the function $u \mapsto u$ are both strictly increasing on $(0, 1]$. By Chebyshev's sum inequality (a rearrangement inequality; see Hardy et al., 1952), the covariance of two co-monotone functions of the same random variable is non-negative, and strictly positive when the variable is non-constant. Since $\pi(y \mid q)$ is non-constant whenever the policy is non-uniform (i.e., it doesn't assign equal probability to all responses), we have $C_q > 0$. This is the key property: the covariance between $\log\pi$ and $\pi$ is always positive, which means $F_q(b)$ is strictly decreasing in $b$. As the baseline increases, $F_q(b)$ decreases linearly with slope $-C_q$.

Translating to entropy change. Since $\Delta H(q; b) \approx -\eta F_q(b)$, and $F_q(b)$ decreases with $b$, the entropy change $\Delta H(q; b)$ is strictly increasing in $b$. A higher baseline → larger $F_q(b)$ reduction → less negative $\Delta H$ → more entropy growth (or less contraction). A lower baseline → smaller $F_q(b)$ reduction → more negative $\Delta H$ → more entropy contraction (or less growth). This is a monotonic, one-dimensional relationship: the baseline $b$ directly controls the direction and magnitude of entropy change.

The K-quantile baseline achieves the extremal values. For a query with success rate $p(q)$:

• The K-quantile baseline $b_K(q)$ is either 0 or 1, which are the minimum and maximum possible values for a baseline over binary rewards in $[0, 1]$. No baseline can be less than 0 or greater than 1 (since rewards are in $\{0, 1\}$, any reasonable baseline is a convex combination, hence in $[0, 1]$).

• When $p(q) \leq 1-K$ (hard query), $b_K = 0 = \min[0, 1]$. Since $\Delta H$ increases with $b$, this choice produces the minimum possible entropy increase among all baselines in $[0, 1]$ — including the mean baseline $b = p(q)$, which would be somewhere in $(0, 1-K]$. This is the explosion-proof property: on hard queries, the quantile baseline maximally suppresses entropy growth, preventing the negative-advantage-driven dispersion.

• When $p(q) > 1-K$ (easy query), $b_K = 1 = \max[0, 1]$. This choice produces the maximum possible entropy increase among all baselines in $[0, 1]$. This is the collapse-proof property: on easy queries, the quantile baseline maximally promotes entropy growth, preventing the policy from prematurely converging to a narrow mode and allowing continued exploration of the residual failure modes.

The two regimes together constitute the two-sided entropy safety guarantee stated in Proposition 4.2:

1. Low-success (explosion-proof): If $p(q) \leq 1-K$ so $b_K(q) = 0$, then for any baseline $b \in [0, 1]$ (including the mean $b = p(q)$ or token-level clipping/KL that keep $b$ unchanged), $\Delta H(q; b_K) \leq \Delta H(q; b)$.

2. High-success (collapse-proof): If $p(q) > 1-K$ so $b_K(q) = 1$, then for any baseline $b \in [0, 1]$, $\Delta H(q; b_K) \geq \Delta H(q; b)$.

Why token-level methods cannot achieve this: Token-level controls (Clip-Higher, KL penalties, uplifting low-probability tokens, entropy bonuses) rescale the step size or modify per-token gradient magnitudes, but they do not change the response-level baseline $b(q)$. The baseline $b(q)$ in GRPO/DAPO is the mean of the group rewards, computed once per query and applied uniformly to all tokens in all responses. Token-level clipping modifies $\text{clip}(r_t(\theta), 1-\epsilon_{\text{low}}, 1+\epsilon_{\text{high}})$, which affects how much the probability ratio can change, but the sign of the advantage — and therefore whether a token's probability is increased or decreased — is determined by $b(q)$. The quantile baseline changes the sign pattern across responses, which is a fundamentally different mechanism from changing the magnitude of per-token updates.

---

Response-Level Sparsity: The 80/20 Rule and Its Mechanism

A striking empirical observation in Figure 5 (right) is that with a tuned $K$ (default $K = 0.4$ in all main experiments), approximately 80% of sampled responses receive zero advantage throughout training. This sparsity is not an explicit design goal — it emerges naturally from the quantile baseline gating.

Why 80% sparsity occurs: For a typical training batch with $G$ responses per query and binary rewards, the quantile baseline partitions responses into three categories:

• Zero-advantage responses (the masked side): On a hard query ($p \leq 1-K$), all incorrect responses have $R_i = 0$ and baseline $b_K = 0$, so their unstandardized advantage is $0 - 0 = 0$. On an easy query ($p > 1-K$), all correct responses have $R_i = 1$ and baseline $b_K = 1$, so their unstandardized advantage is $1 - 1 = 0$. These responses contribute nothing to the gradient.

• Non-zero-advantage responses (the active side): On hard queries, correct responses have advantage $> 0$; on easy queries, incorrect responses have advantage $< 0$. These are the only responses that produce gradient updates.

The fraction of zero-advantage responses depends on $K$ and the distribution of $p(q)$ across queries. With $K = 0.4$, the threshold is $1-K = 0.6$. Queries with $p \leq 0.6$ mask the (often numerous) incorrect responses; queries with $p > 0.6$ mask the (often numerous) correct responses. Across a typical training distribution — where many queries are either quite hard ($p$ near 0) or quite easy ($p$ near 1), with fewer at intermediate difficulty — the masked side dominates the total count, producing roughly 80% sparsity.

Why this sparsity is beneficial rather than wasteful:

1. Redundancy reduction. On a hard query with one success and nine failures, GRPO/DAPO computes gradients for all ten responses, but nine of those gradients are pushing away from failure modes that may or may not be informative. The quantile baseline computes gradients only for the one success, concentrating the computational signal on the most informative sample. The nine failures are not "ignored" in a harmful sense — they are redundant because pushing away from one failure mode when the policy is already bad at the task doesn't help it find the correct mode.

2. Noise reduction. The masked responses, if updated, would produce gradients with random directions (since failures on hard queries are diverse and not necessarily related to each other in a way that one gradient step can usefully avoid). Zeroing them out reduces gradient variance, which stabilizes training and allows a larger effective learning rate on the informative samples.

3. Entropy stabilization. As established in Proposition 4.2, masking the negative-advantage samples on hard queries prevents the dispersion force that drives entropy explosion. Masking the positive-advantage samples on easy queries prevents the concentration force that drives entropy collapse. The sparsity is the mechanism by which two-sided control is achieved.

The sparsity fraction as a diagnostic for $K$ selection: The paper's operational rule-of-thumb in Section 6 uses policy entropy — not sparsity directly — to choose $K$. When the baseline policy has low entropy (risk of mode collapse), choose a larger $K$ (e.g., $K = 0.6$), which makes $1-K = 0.4$, classifying more queries as easy and applying the $b_K = 1$ (collapse-proof, maximum entropy increase) regime. When the baseline policy has high entropy (risk of instability), choose a smaller $K$ (e.g., $K = 0.4$, the default), which makes $1-K = 0.6$, classifying more queries as hard and applying the $b_K = 0$ (explosion-proof, minimum entropy increase) regime. The sparsity fraction follows: larger $K$ means fewer queries are hard, so more correct responses are masked (they're on easy queries), and the sparsity increases; smaller $K$ means more queries are hard, so more incorrect responses are masked, and the sparsity increases from the other direction. The default $K = 0.4$ strikes a balance where approximately 80% of all responses — from both directions — are masked, maintaining stable entropy and productive learning.

---

Mask Ablation: Disentangling Positive and Negative Gating

To verify that the two-regime structure is the active mechanism (not just the sparsity itself), the paper constructs one-sided masked objectives in Section 5.3. These ablate each side of the quantile gate independently.

POS-MASK (Eq. 7): Masks only the positive-advantage samples on easy queries, leaving negative-advantage samples active on all queries. Formally, the indicator $\mathbb{1}\{p(q) \leq 1-K\}$ is removed from the positive term, so correct responses always contribute (not just on hard queries), while the masking of incorrect responses on hard queries remains:

$J_{\text{POS-MASK}}(\theta) = \mathbb{E}_q \left[ \sqrt{\frac{p(q)}{1-p(q)}} \; \mathbb{E}_{o \sim \pi^+_{\text{old}}} s^+_\theta(o, q) \;-\; \mathbb{1}\{p(q) > 1-K\} \sqrt{\frac{1-p(q)}{p(q)}} \; \mathbb{E}_{o' \sim \pi^-_{\text{old}}} s^-_\theta(o', q) \right]$

NEG-MASK (Eq. 8): Masks only the negative-advantage samples on hard queries. Formally, the indicator $\mathbb{1}\{p(q) > 1-K\}$ is removed from the negative term, so incorrect responses always contribute (not just on easy queries), while the masking of correct responses on easy queries remains:

$J_{\text{NEG-MASK}}(\theta) = \mathbb{E}_q \left[ \mathbb{1}\{p(q) \leq 1-K\} \sqrt{\frac{p(q)}{1-p(q)}} \; \mathbb{E}_{o \sim \pi^+_{\text{old}}} s^+_\theta(o, q) \;-\; \sqrt{\frac{1-p(q)}{p(q)}} \; \mathbb{E}_{o' \sim \pi^-_{\text{old}}} s^-_\theta(o', q) \right]$

What these ablations test: POS-MASK prevents entropy collapse (by keeping the collapse-proof masking of positives on easy queries) but does NOT prevent entropy explosion (because negatives on hard queries are not masked, so the simultaneous push-away from many failure modes remains active). NEG-MASK prevents entropy explosion (by keeping the explosion-proof masking of negatives on hard queries) but does NOT prevent entropy collapse (because positives on easy queries are not masked). Full QAE includes both masks, providing both guarantees simultaneously.

Results (Figure 6, b-c): The relative importance of the two masks depends on the dominant failure mode, which is determined by the token-level clipping strength $\epsilon_{\text{high}}$:

• Weak clipping ($\epsilon_{\text{high}} = 0.28$, Figure 6b): The dominant failure mode is entropy explosion (the policy is insufficiently constrained, and negative-advantage-driven dispersion dominates). NEG-MASK nearly matches full QAE performance, while POS-MASK is noticeably worse. This confirms that under weak token-level constraints, the explosion-proof property of masking hard-query negatives is the critical mechanism.

• Strong clipping ($\epsilon_{\text{high}} = 0.20$, Figure 6c): The dominant failure mode flips to entropy collapse (the policy is overly constrained, and the lack of exploration causes stagnation). POS-MASK outperforms NEG-MASK, confirming that under strong token-level constraints, the collapse-proof property of masking easy-query positives becomes the critical mechanism.

Why this matters: This ablation demonstrates that the two regimes are not redundant — each addresses a distinct failure mode, and both are necessary when the token-level clipping is tuned for overall performance (as in the default $\epsilon_{\text{high}} = 0.28$ used in all main experiments). The quantile baseline provides both masks simultaneously through a single parameter $K$, while the mean baseline provides neither. This is the structural advantage: the mean baseline forces a single, symmetric update regime that cannot adapt to whether the dominant threat is collapse or explosion on a per-query basis.

---

Design Choices and Their Justifications (Summary)

• K-quantile rather than median or fixed constant baseline: The median ($K = 0.5$) would produce a single fixed threshold ($1-K = 0.5$), while the general K-quantile allows tuning the threshold to match the policy's entropy state. A fixed constant baseline (e.g., always use $b = 0.3$) would not adapt to per-query difficulty — it would apply the same regime regardless of $p(q)$, losing the two-regime gating that is essential for two-sided control. The quantile is data-adaptive (changes with the group's empirical success rate) while being tunable (through $K$).

• Right-continuous quantile definition: Ensures well-defined behavior for discrete binary rewards where the empirical CDF has jumps. Without right-continuity, the infimum might be ambiguous at jump points. With right-continuity, $b_K(q) = 0$ exactly when the fraction of zeros equals or exceeds $K$, which for binary rewards means $b_K(q) = 0$ when $1-p(q) \geq K$, i.e., $p(q) \leq 1-K$ — the clean threshold rule in Eq. 4.

• Standardization by group standard deviation (not omitted): Retained from GRPO/DAPO to keep advantages scale-invariant with respect to the group's difficulty. Without standardization, the magnitude of non-zero advantages would depend on whether the baseline is 0 or 1 and on the absolute number of correct/incorrect responses, which would couple the learning rate to the batch composition. Standardization normalizes this away, making the effective learning rate independent of $p(q)$.

• Default $K = 0.4$ rather than 0.5: The median ($K = 0.5$) would treat queries with $p > 0.5$ as easy and $p \leq 0.5$ as hard. The default $K = 0.4$ shifts the threshold to $1-K = 0.6$, classifying more queries as hard (exploitation-focused). This choice is motivated by the observation (Section 6) that with Clip-Higher, the baseline policy tends toward high entropy (explosion risk), so the lower $K$ applies the explosion-proof regime ($b_K = 0$) to a wider range of queries. When entropy is low, $K = 0.6$ is recommended instead. The choice is entropy-diagnostic, not accuracy-diagnostic: practitioners inspect the training entropy curve, not the evaluation metric, to select $K$.

• No change to token-level loss, clipping, dynamic sampling, or KL penalty: All these components are inherited from the baseline method (DAPO or GRPO). QAE is strictly an advantage estimation modification — it changes the $\hat{A}_i$ values fed into the existing loss, nothing else. This is what makes it a "drop-in" replacement compatible with any value-free method that uses group-wise reward normalization.

4. Key Insights and Innovations

Innovation 1: Reframing Entropy Regulation as a Baseline-Design Problem

The paper's most fundamental intellectual move is a category shift — relocating the entropy control problem from the how and where of gradient updates (token-level clipping, weighting, bonuses) to the who of advantage assignment (which responses in a group receive non-zero advantage signals). This reframing is subtle but profound because it implies that an entire class of prior work has been optimizing the wrong variable.

What the field did before: Prior approaches to entropy control in RLVR overwhelmingly operate at the token level. Clip-Higher (Yu et al., 2025) adjusts the upper clipping bound to allow larger updates for advantageous actions. Entropy-modulated policy gradients (Wang et al., 2025a) scale token-level updates by uncertainty estimates. KL penalties (Shao et al., 2024) constrain the per-token divergence from a reference policy. Cui et al. (2025) analyze entropy dynamics at the sequence level but propose token-level objectives. In every case, the response-level baseline — the $b$ in $\hat{A}_i = (R_i - b) / \sigma$ — is treated as a fixed constant: the group mean. No prior method considers that changing $b$ might be the primary entropy control mechanism, or that the mean baseline itself might be structurally flawed.

What makes QAE's framing distinctive: The paper identifies that under a softmax policy, the one-step entropy change $\Delta H(q; b)$ is a strictly increasing function of the baseline $b$. This is a clean, monotonic, one-dimensional relationship: the baseline is a linear knob that directly controls entropy direction and magnitude. By establishing this through the entropy–covariance identity (Proposition 4.2), the paper shows that all token-level methods share a fundamental limitation: they rescale step sizes but leave $b$ unchanged, so they cannot produce two-sided entropy guarantees. The K-quantile baseline achieves what no token-level method can — it sets $b$ to the extremal values of 0 or 1 depending on query difficulty — and thereby realizes both lower and upper bounds on entropy change simultaneously.

Why the reframing matters beyond performance gains: This is not merely a new method; it is a diagnostic insight that reinterprets prior empirical contradictions. The field's inconsistent results with entropy interventions — some methods prevent collapse but induce explosion, some are stable but under-explore — are explained by the fact that all prior methods operate on a common, flawed baseline. The paper's Observation 3 (Figure 4, left) that entropy explosion is "disproportionately driven by negative-advantage samples" makes no sense under a token-level framing (why would negative tokens be different from positive tokens in their entropy contribution?) but becomes obvious under the baseline-design framing: the mean baseline simultaneously pushes the policy away from all $G-1$ failures on hard queries, creating a dispersion force that token-level clipping cannot counteract because it doesn't change the sign pattern.

Evidence anchoring: The two-sided entropy safety guarantee is proved analytically in Proposition 4.2. Empirically, Figure 5 (middle) shows DAPO's entropy growth concentrated in negative-advantage samples, which QAE suppresses. The mask ablation (Figure 6, b–c) demonstrates that the two regimes are complementary: NEG-MASK dominates when explosion is the threat (weak clipping), POS-MASK dominates when collapse is the threat (strong clipping), and full QAE provides both. This is evidence that the baseline-design reframing identifies a real structural property, not a post-hoc rationalization.

Assessment: fundamental shift, not incremental refinement. The paper changes what variable practitioners should think about when debugging RLVR instability. The operational rule-of-thumb in Section 6 — "choose $K$ by inspecting training entropy, not evaluation accuracy" — is a direct consequence of the reframing and would be unintuitive under a token-level perspective. This is a genuinely new conceptual axis for RLVR algorithm design.

---

Innovation 2: Identifying the Mean Baseline as the Root Cause of Both Collapse and Explosion

Prior work treats entropy collapse and entropy explosion as separate phenomena requiring separate fixes — collapse is addressed by uplift (Yu et al., 2025), explosion (to the extent it's even recognized) is treated as a hyperparameter tuning issue. The paper's second major insight is that both pathologies share a single root cause: the mean baseline's symmetric, unbounded advantage assignment.

What was previously assumed: The default view in the GRPO/DAPO literature is that the mean baseline is a natural, unbiased estimator of the expected reward, and that any instability must therefore arise from elsewhere — the clipping bounds, the learning rate, the KL penalty, or the sampling distribution. DAPO's dynamic sampling constraint (ensuring $0 < |\{o_i \mid \text{correct}\}| < G$) is designed to prevent degenerate batches that would make the mean baseline produce zero advantage for everyone. But the paper shows that even with non-degenerate batches, the mean baseline is problematic because it never produces zero advantage for any response (except in the exact $p = 1$ or $p = 0$ case that dynamic sampling prevents). Every response always contributes to the gradient, creating a simultaneous push-pull dynamic that drives entropy growth.

The diagnostic move: The paper traces this to a specific mechanism. On a hard query with one correct and nine incorrect responses, the mean baseline $b = 0.1$ gives the single success a positive advantage of $+0.9$ (before standardization) and each of the nine failures a negative advantage of $-0.1$. The total gradient norm pushing away from failures is $9 \times 0.1 = 0.9$, matching the pull toward the single success. This symmetry means the policy receives equal force to disperse (away from nine distinct failure modes) and to concentrate (toward one success mode), which on net drives entropy up because dispersion into nine directions dominates concentration into one. The problem is not that any individual advantage is miscalibrated — it's that all samples are always active, creating a structural over-dispersion bias on hard queries and a symmetrical over-concentration bias on easy queries.

Why this is a fundamental insight rather than an obvious fix: The community has spent significant effort designing token-level entropy controls without questioning the mean baseline itself. The paper's Observation 4 (Table 1) shows that sweeping $\epsilon_{\text{high}}$ — the primary token-level knob in DAPO — from 0.20 to 0.28 produces only marginal accuracy changes (a few percentage points) and does nothing to the late-stage plateau. This is direct evidence that the bottleneck is not token-level update magnitudes but the response-level advantage sign pattern. The quantile baseline fixes this by selectively nullifying one sign per query, producing exactly the sparsity (80% zero-advantage responses, Figure 5 right) that the mean baseline structurally cannot achieve.

Evidence anchoring: The discriminative reformulation (Proposition 4.1) makes the structural difference between mean and quantile baselines explicit. GRPO/DAPO's objective has symmetric query weight $\sqrt{p(1-p)}$ and always-active discriminative term; QAE replaces this with gated, asymmetric terms that are mutually exclusive per query. Figure 4 (middle, right) visualizes this: the mean baseline applies continuous, bell-shaped weighting to all queries, while the quantile baseline applies a hard threshold with monotonic weights on only one side per query. The empirical consequence is the stabilization in Figure 5 (left): DAPO's pass@1 plateaus after step ~100 while QAE continues to improve, and the entropy dynamics (Figure 5, middle) shift from divergent (negative-driven surge) to bounded.

Assessment: fundamental diagnostic contribution. The paper doesn't just propose a better baseline — it provides a causal explanation for why the mean baseline fails that was not previously articulated. The claim that both collapse and explosion stem from the same structural property (symmetric, full-coverage advantage assignment) is a unified explanation for RLVR instability that was missing from the literature.

---

Innovation 3: The K-Quantile as a Single-Parameter Entropy Knob with Provable Two-Sided Guarantees

While the baseline-as-entropy-knob concept is novel in itself (Innovation 1), the specific choice of K-quantile as the baseline function is a distinct intellectual contribution. It is not the only possible alternative to the mean — one could use a fixed constant, a median, a trimmed mean, or a learned baseline. The paper's selection of the K-quantile is motivated by a combination of theoretical elegance (it naturally produces a two-regime gate for binary rewards), practical simplicity (one parameter $K$ with a clear operational interpretation), and provable guarantees (two-sided entropy safety that no other baseline form achieves).

What other baselines could have been chosen and why they're inferior: A fixed constant baseline (e.g., $b = 0.3$ always) would not adapt to per-query difficulty — it would always be "hard-query mode" for queries with $p < 0.3$ and "easy-query mode" for queries with $p > 0.3$, but with no ability to tune the boundary. The median ($K = 0.5$) is a special case of the K-quantile that fixes the boundary at $p = 0.5$, providing no knob for the entropy state. A learned baseline (a small network predicting $b(q)$) would introduce training overhead and potential co-adaptation issues. The K-quantile wins by being simultaneously data-adaptive (the boundary shifts with the group's empirical $p(q)$), tunable (through $K$), and parameter-free beyond $K$.

What makes the two-sided guarantee conceptually significant: The paper proves (Proposition 4.2) that the K-quantile baseline attains the extremal one-step entropy change among all baselines in $[0, 1]$. This is a stronger statement than "QAE is better than the mean baseline" — it says that no possible baseline choice (including all of the alternatives listed above) can provide a tighter entropy bound than the K-quantile, because $b_K = 0$ and $b_K = 1$ are the absolute minimum and maximum values a baseline can take for binary rewards. The guarantee is provable, not empirical: it follows directly from the entropy–covariance identity and the monotonicity of $\Delta H$ in $b$, which holds for any non-uniform softmax policy under first-order updates. This is a rare example of a simple algorithmic modification to RLVR that comes with non-trivial theoretical backing.

The operational rule-of-thumb as a design insight: The paper's recommendation to select $K$ by inspecting training entropy rather than evaluation accuracy (Section 6) is a direct translation of the theory into practice. If entropy is low (collapse risk), increase $K$ to make more queries "easy" and apply the entropy-increasing $b_K = 1$ regime. If entropy is high (explosion risk), decrease $K$ to make more queries "hard" and apply the entropy-suppressing $b_K = 0$ regime. This makes $K$ selection a diagnostic feedback loop rather than a blind hyperparameter sweep. The default $K = 0.4$ is not arbitrary — it's chosen because Clip-Higher pushes the baseline toward high entropy, so the lower $K$ (making $1-K = 0.6$) applies the explosion-proof regime to a wider slice of queries.

Evidence anchoring: The sensitivity analysis in Figure 9 (Appendix B.3) shows the $K$-dependence clearly: $K = 0.2$ produces low-entropy, over-regularized training with limited accuracy gains; $K = 0.8$ produces high-entropy, volatile training with an early plateau; $K = 0.4$ strikes the balance. The mask ablation (Figure 6, b–c) confirms that the two regimes are both active and complementary. The theoretical guarantee is proved in Proposition 4.2 and is not contingent on hyperparameter choices — it follows from the structure of the K-quantile and the linearity of covariance.

Assessment: a genuine theoretical contribution with practical bite. The two-sided entropy safety guarantee is not a loose bound or an asymptotic result — it's an exact extremal property that directly motivates the practical use of QAE. This is a rare combination in the RLVR literature, where most innovations are either purely empirical (recipe-level heuristics) or purely theoretical (with unclear practical translation). The K-quantile baseline sits at the intersection: the theory predicts exactly what the experiments show (entropy stabilization, complementary masking regimes), and the practical knob $K$ is grounded in the theoretical monotonicity of $\Delta H$ in $b$.

---

Innovation 4: The Response-Level 80/20 Rule as Emergent Credit Sparsification

A surprising empirical finding — not designed, not optimized for, but emergent — is that QAE with $K = 0.4$ causes approximately 80% of sampled responses to receive zero advantage throughout training (Figure 5, right). This "response-level 80/20 rule" is conceptually important because it reveals a hidden redundancy in the mean-baseline approach and connects to a broader theme in LLM optimization: sparse credit assignment is more effective than dense credit assignment.

What makes this distinctive as an insight: The paper does not set out to create a sparse update mechanism. The sparsity emerges from the two-regime gate: on hard queries, all incorrect responses (the majority) are masked; on easy queries, all correct responses (the majority) are masked. Across the training distribution, this naturally zeros out roughly 80% of advantages. The fact that this sparsity correlates with — and plausibly causes — improved stability and sustained gains is an empirical discovery, not a design target. The paper interprets this as evidence that most responses in a standard RLVR batch are uninformative redundancies, and that the mean baseline's dense credit assignment is not just unnecessary but actively harmful because it introduces noise and drives entropy.

Connection to prior sparsity findings: Wang et al. (2025b) observed that 20% of tokens drive 80% of exploration in RLVR — a token-level 80/20 rule. QAE's response-level sparsity operates at a coarser granularity (whole responses, not individual tokens) but points to the same underlying phenomenon: effective learning concentrates on a small fraction of informative samples, and forcing the model to learn from everything degrades performance. The quantile baseline achieves this concentration automatically, without explicit sparsity regularization, because the difficulty-threshold gate naturally separates informative from uninformative responses.

Why this reframes the RLVR design space: If the 80/20 rule is general — and the paper's results across three model sizes and three benchmarks are suggestive but not definitive — then future RLVR methods should prioritize selection (deciding which samples to update) over magnitude (deciding how much to update each sample). This is a direct challenge to the dominant paradigm of clipping, adaptive learning rates, and per-token importance weighting, all of which modify update magnitudes but keep all samples active. QAE demonstrates that simply zeroing out the uninformative 80% produces better results than carefully tuning the update magnitudes for 100%.

Evidence anchoring: Figure 5 (right) tracks the fraction of advantages equal to +1.0, 0.0, and −1.0 (before standardization) across training. The zero-advantage fraction is consistently around 80% from early to late training for the default $K = 0.4$, showing that sparsity is not a transient phase but a persistent property of the quantile-baseline dynamics. The mask ablation (Figure 6, b–c) shows that each mask (POS-MASK, NEG-MASK) independently captures a subset of QAE's benefits, consistent with the interpretation that each mask zeroes out a different redundant subset of responses.

Assessment: an empirically-grounded conceptual contribution with practical implications. The 80/20 rule is not proven theoretically — it depends on the query difficulty distribution and the $K$ parameter — but it is a robust empirical observation that changes how one thinks about RLVR batch construction. If confirmed across other domains and model families, it would suggest that RLVR training is fundamentally bottlenecked by credit assignment sparsity, not by update stability per se. QAE provides a minimal mechanism to achieve that sparsity (a one-line baseline swap), making it a strong baseline for future work on sparse credit assignment in RLVR.

5. Experimental Analysis

Evaluation Methodology

• Dataset. All experiments use three standard math reasoning benchmarks: AIME'24, AIME'25, and AMC'23. These are competition-level mathematics problem sets drawn from the American Invitational Mathematics Examination (AIME) and the American Mathematics Competitions (AMC). The paper does not report exact dataset sizes for each benchmark, but AIME typically contains 30 problems per year and AMC contains 25 problems per contest. The evaluation protocol uses all problems in each benchmark (zero-shot, no few-shot exemplars).

• Base model(s). Experiments span three model scales from the Qwen3 family (Yang et al., 2025a): Qwen3-8B-Base (8 billion parameters), Qwen3-14B-Base (14 billion parameters), and Qwen3-30B-A3B-Base (a Mixture-of-Experts model with 30 billion total parameters and 3 billion active parameters per token). These are chosen to test the method across a ~4× parameter scaling range and across dense (8B, 14B) and MoE (30B-A3B) architectures. All models are base models (not instruction-tuned), consistent with standard RLVR practice where reasoning capabilities are elicited from pretrained checkpoints through reinforcement learning.

• Metrics. The primary metric is pass@1: for each query, the model generates one complete solution (temperature T=0.7) and passes if the final answer matches the ground truth, with accuracy averaged over all queries. The secondary metric is pass@16: for each query, the model generates 16 independent completions, and the query is considered passed if at least one of the 16 answers matches the ground truth — this measures the model's best-case performance and its ability to produce correct solutions somewhere in its output distribution. The paper also tracks average tokens per response (response length) and policy entropy (computed over the token distribution) as diagnostics during training.

• Baselines. The primary baseline is DAPO with Clip-Higher (Yu et al., 2025), configured with the recommended hyperparameters: ϵ_high = 0.28, ϵ_low = 0.20, global batch size 512, 16 gradient accumulation steps with mini-batch size 32, learning rate 10⁻⁶ with no warmup or decay, maximum response length 20,480 tokens with a 4,096-token cache for reward shaping, and no KL divergence or entropy loss terms. This configuration includes dynamic sampling (ensuring each batch contains both correct and incorrect responses), token-level policy gradient loss, and overlong reward shaping as described in the DAPO paper.

  Additional baselines tested for compositionality include Clip-Cov and KL-Cov (Cui et al., 2025) — token-level entropy control methods that modify the clipping objective and add KL-based covariance penalties, respectively — and GSPO (Group Sequence Policy Optimization; Zheng et al., 2025), a sequence-level optimization method. For GSPO on the 30B-A3B model, a cold-start training run is conducted with the official VERL implementation, using clipping ranges of 3×10⁻⁴ (left) and 4×10⁻⁴ (right) and four gradient accumulation steps per batch.

  All baselines use the mean reward baseline in their advantage estimation (Equation 2): Â_i = (R_i − mean({R_k})) / std({R_k}). QAE replaces exactly this term with the K-quantile baseline (Equation 3-4, Section 4.1); all other hyperparameters, loss functions, and training configurations are held identical to enable clean comparison.

• Generation budget / compute accounting. The paper does not use a separate "generation budget" metric in the sense of fixed inference FLOPs comparisons. Instead, fairness is maintained by identical training configurations: all methods (baseline and QAE) use the same number of training steps, same batch size (512), same group size G per query (determined by the DAPO dynamic sampling constraint), and same number of generated tokens. The compute cost of QAE is effectively identical to the baseline since computing a K-quantile from G binary rewards requires only a sort or a count of successes, which is negligible compared to the forward and backward passes through the LLM. No additional forward passes, reward model evaluations, or auxiliary networks are required.

• Cross-validation / statistical protocol. The paper does not describe a cross-validation or statistical significance testing protocol. Results are reported as single-run final pass@1 and pass@16 scores on each benchmark (Table 2). Training dynamics are shown as curves over training steps (Figures 1, 2, 5, 9, 10, 11) with individual data points plotted, which implicitly provides information about trajectory variance, but no confidence intervals, error bars, or multi-seed averages are reported. This is a limitation: the paper's claims about sustained gains and entropy stabilization are based on single training runs, and the robustness of these trajectories across random seeds is not assessed. The mask ablation (Figure 6) and quantile parameter sweep (Figure 9) also present single-run outcomes.

---

Main Quantitative Results

Overall Performance Across Benchmarks, Models, and Baselines

Table 2 is the central results table, reporting pass@1 and pass@16 for Qwen3-8B-Base and Qwen3-30B-A3B-Base on AIME'25, AIME'24, and AMC'23. The headline finding is that QAE yields consistent pass@1 gains across all tested configurations while maintaining or improving pass@16.

Qwen3-8B-Base with DAPO (Clip-Higher): Adding QAE to the Clip-Higher baseline produces:

• AIME'25: pass@1 improves from 32.71 to 34.90 (+6.7%); pass@16 improves from 56.66 to 57.92 (+2.2%).
• AIME'24: pass@1 improves from 39.69 to 48.23 (+21.5%); pass@16 marginally improves from 71.23 to 71.63 (+0.6%).
• AMC'23: pass@1 improves from 92.11 to 92.97 (+0.9%); pass@16 is identical at 97.50.

The large relative gain on AIME'24 pass@1 (+21.5%) is the most striking single result, but the near-zero gain on AMC'23 pass@1 (+0.9%) suggests that QAE's benefits are most pronounced on benchmarks where there is substantial room for improvement — AMC'23 at 92.11% is close to saturation, so the small gain is unsurprising and not a negative signal.

Qwen3-8B-Base with Clip-Cov: Adding QAE produces:

• AIME'25: pass@1 from 33.02 to 37.40 (+13.3%); pass@16 from 52.27 to 56.29 (+7.7%).
• AIME'24: pass@1 from 42.40 to 46.04 (+8.6%); pass@16 from 68.58 to 73.16 (+6.7%).
• AMC'23: pass@1 from 87.42 to 90.23 (+3.2%); pass@16 unchanged at 96.25.

Notably, QAE + Clip-Cov achieves the highest AIME'25 pass@1 (37.40) among all 8B configurations, exceeding QAE + Clip-Higher (34.90) and QAE + KL-Cov (33.44). This demonstrates compositionality: QAE's baseline-level gating is orthogonal to Clip-Cov's token-level covariance penalty, and the two mechanisms combine additively.

Qwen3-8B-Base with KL-Cov: Adding QAE produces:

• AIME'25: pass@1 from 33.33 to 33.44 (+0.3%); pass@16 from 45.86 to 51.62 (+12.6%).
• AIME'24: pass@1 from 44.90 to 44.69 (−0.5%); pass@16 from 73.00 to 77.08 (+5.6%).
• AMC'23: pass@1 from 86.02 to 87.97 (+2.3%); pass@16 from 95.00 to 96.25 (+1.3%).

This is the only configuration where QAE shows a slight pass@1 regression (−0.5% on AIME'24, which is within noise for a single run and may not be statistically significant), but the pass@16 improvement is substantial (+5.6%, the second-largest relative gain on AIME'24 pass@16 across all 8B configurations). The large pass@16 gain despite flat pass@1 suggests that KL-Cov + QAE is broadening the model's output distribution — more correct answers exist in the top-16 samples even if the most likely answer doesn't improve. This is consistent with the entropy-control theory: KL-Cov applies a strong regularization force that may cause some under-exploration on its own, and QAE's collapse-proof gating (masking positives on easy queries) counteracts this by maintaining diversity.

Qwen3-30B-A3B-Base with GSPO: Adding QAE produces:

• AIME'25: pass@1 from 31.15 to 32.50 (+4.3%); pass@16 from 46.59 to 48.01 (+3.0%).
• AIME'24: pass@1 from 43.75 to 47.50 (+8.6%); pass@16 from 67.91 to 71.72 (+5.6%).
• AMC'23: pass@1 from 90.00 to 89.38 (−0.7%); pass@16 from 99.39 to 97.21 (−2.2%).

This configuration shows the only consistent pass@16 regressions (−2.2% on AMC'23, −0.7% pass@1 on AMC'23). This is the most challenging test case: GSPO is a sequence-level method that already optimizes at a different granularity than token-level DAPO, and the 30B-A3B MoE architecture may have different entropy dynamics than the dense 8B and 14B models. The paper does not ablate $K$ for this configuration (all main results use the default $K=0.4$), so the regressions may reflect suboptimal $K$ selection rather than a fundamental incompatibility.

Cross-method summary (rows of Table 2): Across the 8 configurations where QAE is added (4 baselines × 2 models, though GSPO only on 30B), QAE improves pass@1 in 10 out of 12 comparisons (12 comparisons = 3 benchmarks × 4 configurations with pass@1 reported; the 2 regressions are KL-Cov on AIME'24 at −0.5% and GSPO on AMC'23 at −0.7%). Pass@16 improves or stays equal in 10 out of 12 comparisons (slight regressions on GSPO + AMC'23 at −2.2% and no change on two Clip-Higher and Clip-Cov configurations). The consistency of pass@1 gains — across model scales (8B, 30B), architectures (dense, MoE), and baseline methods (Clip-Higher, Clip-Cov, KL-Cov, GSPO) — is the strongest evidence for QAE's generality.

---

Training Dynamics: Entropy Stabilization and Sustained Performance

Figure 5 presents three panels tracking QAE versus DAPO (Clip-Higher) on Qwen3-8B-Base across 600 training steps for AIME'24.

Panel (a): Pass@1 and pass@16 trajectories. The left panel plots pass@1 and pass@16 over training. DAPO's pass@1 rises rapidly in the first ~100 steps, reaching approximately 0.32 by step 100, then plateaus and oscillates between 0.32 and 0.38 through step 600. QAE's pass@1 rises more slowly in early training (consistent with the reduced gradient signal from sparser updates) but continues to climb through step 600, reaching approximately 0.46–0.48 — a sustained improvement that does not plateau within the training horizon shown. QAE's pass@1 exceeds DAPO's at approximately step 200–250 and the gap widens monotonically thereafter.

Pass@16 trajectories show a smaller but consistent gap: QAE reaches approximately 0.71 versus DAPO's approximately 0.68–0.69. The pass@16 gap is narrower than the pass@1 gap, consistent with Table 2 where QAE's pass@16 improvements (0.6–7.7%) are generally smaller than pass@1 improvements (0.3–21.5%). This pattern is consistent with QAE's mechanism: by stabilizing entropy and sparsifying updates, QAE improves the policy's most likely output (pass@1) without necessarily expanding the tail of the distribution (pass@16). The model gets better at producing the right answer on its first try rather than occasionally producing it in the top-16.

Panel (b): Entropy dynamics by advantage sign. The middle panel decomposes policy entropy into components from positive-advantage and negative-advantage samples. Under DAPO (dashed lines), negative-advantage entropy rises sharply from roughly 0.6 at step 20 to approximately 1.2–1.3 by step 100, and remains elevated and volatile (0.9–1.4) through step 300. Positive-advantage entropy is relatively stable around 0.4–0.5 throughout. This panel directly visualizes Observation 3 from Section 3.2: entropy explosion is disproportionately driven by negative-advantage samples.

Under QAE (solid lines), negative-advantage entropy is suppressed — it rises modestly from approximately 0.6 to 0.8 by step 150 and remains bounded below 1.0, substantially below the DAPO trajectory. Positive-advantage entropy is comparable to DAPO. Mean entropy (presumably the unweighted average, shown as a separate line) shows QAE maintaining approximately 0.7–0.8 compared to DAPO's 1.0–1.3. This panel is the most direct empirical evidence for the explosion-proof mechanism: by masking negative-advantage samples on hard queries (where they would otherwise drive entropy growth), QAE keeps the policy's entropy within a productive, bounded range.

Panel (c): Advantage value sparsity (the 80/20 rule). The right panel shows the proportion of responses receiving advantage = +1.0, 0.0, −1.0, and "Others" (presumably non-standardized or edge-case values) over training. The zero-advantage fraction (center band, uncolored in the figure) is consistently approximately 80% throughout training — rising from ~75% in early steps to ~80–82% in later steps. The positive-advantage fraction (top band) is approximately 8–12%, and the negative-advantage fraction (bottom band) is approximately 8–10%. This sparsity is persistent, not a transient phase, confirming that QAE's gating mechanism zeros out the majority of responses at every step.

---

Scaling Across Model Sizes: 8B and 14B Training Curves

Figures 10 and 11 (Appendix B.4) show training curves for DAPO versus DAPO + QAE on Qwen3-8B-Base and Qwen3-14B-Base, with three panels each: entropy, AIME'24 pass@1, and response length.

Qwen3-8B (Figure 10): DAPO entropy rises from approximately 0.3 at step 0 to a peak of approximately 0.95 at step 80, then declines to 0.7–0.8 but remains volatile (oscillations of 0.2–0.3 between steps 100–300). QAE entropy rises more gradually to approximately 0.55 by step 80, maintains 0.45–0.55 through step 200, and shows smaller oscillations (0.1–0.15). The entropy gap is largest at the peak (0.95 vs. 0.55) and narrows but persists through late training.

DAPO accuracy rises to approximately 0.32 by step 100 and plateaus at 0.32–0.38 through step 300. QAE accuracy rises more slowly to step 100 (approximately 0.28) but continues climbing, reaching approximately 0.44–0.48 by step 300. The crossover occurs around step 180–200.

DAPO response length rises from approximately 2,000 tokens at step 0 to a volatile range of 5,000–9,000 by step 80, with large oscillations (spikes to 8,500, dips to 4,000) through step 300. QAE response length stabilizes between 4,000–5,500, with much smaller oscillations. The correlation between entropy and response length under DAPO — both show large oscillations — is consistent with the interpretation that entropy explosion causes the model to generate longer, more meandering responses that don't improve accuracy.

Qwen3-14B (Figure 11): The same patterns hold at larger scale. DAPO entropy rises to approximately 0.42 by step 50 and remains elevated (0.35–0.45) with volatility through step 300. QAE entropy stabilizes lower, at approximately 0.25–0.32, with smaller oscillations. DAPO accuracy rises to approximately 0.42 by step 120 and plateaus at 0.42–0.48 through step 300. QAE accuracy starts lower but crosses DAPO around step 180, reaching approximately 0.52 by step 300. DAPO response length spikes to approximately 9,000 tokens around step 180 (accompanied by an entropy spike), while QAE response length stabilizes at 5,000–7,000.

Takeaway from scaling: The entropy stabilization effect of QAE is scale-robust: it manifests at both 8B and 14B, with the same qualitative pattern (lower entropy, smaller oscillations, delayed but sustained accuracy gains, stable response lengths). The magnitude of the entropy spike is somewhat smaller at 14B (DAPO peak entropy ~0.45 vs. ~0.95 at 8B), but QAE still provides a clear benefit. This suggests that entropy explosion is not solely a small-model phenomenon — it persists at larger scales, though potentially with reduced severity, and QAE's mechanism generalizes across the tested scale range.

---

Difficulty-Dependent Behavior via the K Parameter (Figure 9)

Figure 9 (Appendix B.3) sweeps K ∈ {0.2, 0.4, 0.6, 0.8} on Qwen3-8B-Base, showing entropy, AIME'24 pass@1, and response length trajectories.

K = 0.2: Entropy is the lowest (approximately 0.2–0.3), with minimal oscillations. Accuracy rises slowly and plateaus early, around step 100, at approximately 0.35 — the lowest final accuracy among all K values. Response length is the shortest and most stable (2,000–3,000 tokens). This is the over-regularized, exploration-poor regime: with K = 0.2, the threshold 1−K = 0.8, so only queries with p > 0.8 are classified as easy and get the entropy-increasing b_K = 1 baseline. Most queries (p ≤ 0.8) are treated as hard and get the entropy-suppressing b_K = 0 baseline. The result is too little entropy growth — the policy never explores enough to discover better reasoning modes — and pass@1 stagnates.

K = 0.8: Entropy is the highest (rising to approximately 2.5–3.0 by step 80), with extreme volatility (oscillations of 1.0–2.0). Accuracy rises but then plateaus early, around step 60, at approximately 0.36–0.40, and remains volatile. Response length explodes to approximately 12,000–14,000 tokens with wild oscillations. This is the entropy explosion regime: with K = 0.8, the threshold 1−K = 0.2, so most queries (p > 0.2) are classified as easy and get the entropy-increasing b_K = 1 baseline. Only very hard queries (p ≤ 0.2) get the entropy-suppressing b_K = 0 baseline. The result is unchecked entropy growth across most of the query distribution, causing the same degradation observed under the mean baseline — but worse, because the quantile baseline makes the explosion even more extreme by setting b_K = 1 (maximum entropy increase) for ~80% of queries instead of the mean baseline's intermediate values.

K = 0.4 (default): Entropy is moderate (0.4–0.6), with small oscillations. Accuracy continues to improve through step 250, reaching approximately 0.45, the highest among all K values. Response length is stable at 4,000–5,000. This is the balanced regime where the threshold 1−K = 0.6 splits the query distribution roughly evenly: hard queries get explosion-proof gating (b_K = 0), easy queries get collapse-proof gating (b_K = 1), and neither extreme dominates.

K = 0.6: Entropy is elevated (approximately 0.8–1.2) compared to K = 0.4, with moderate oscillations. Accuracy is intermediate — better than K = 0.2 or 0.8 but worse than K = 0.4. Response length is elevated (6,000–8,000) but more stable than K = 0.8. This configuration would be appropriate when the baseline policy has very low entropy (collapse risk) — the higher K makes more queries "easy" and applies the entropy-increasing b_K = 1 to a larger fraction, which the paper recommends for low-entropy scenarios (Section 6).

Interpretation of the K-sweep: The U-shaped relationship between K and final accuracy, with K = 0.4 as the optimum for this model-baseline combination, is consistent with the two-sided entropy safety theory. Too little entropy (K too low) causes stagnation from insufficient exploration. Too much entropy (K too high) causes degradation from noise-dominated gradients. The optimum balances explosion-proof and collapse-proof gating. The paper does not report whether K = 0.4 generalizes as the optimum across model scales or baselines — the single sweep is on Qwen3-8B with Clip-Higher — so the default K = 0.4 should be understood as a robust choice for the Clip-Higher configuration specifically, not as a universal constant.

---

Mask Ablation: Disentangling the Two Regimes (Figure 6)

Figure 6 presents three sets of results:

Panel (a): QAE on Qwen3-14B-Base. Adding QAE to DAPO on the 14B model improves AIME'25 pass@1 from 45.21 to 46.88 and AIME'24 pass@1 from 56.56 to 58.96. These are modest but consistent gains (~3–4% relative improvement) that confirm the 8B results scale to 14B. The smaller relative gains compared to the 8B model (+6.7% and +21.5% on AIME'25 and AIME'24, respectively) may reflect the 14B model's lower baseline entropy (as seen in Figure 11, the 14B DAPO entropy spike is ~0.45 vs. ~0.95 for 8B), meaning there is less explosion to correct.

Panel (b): Mask ablation with weak clipping (ϵ_high = 0.28). Three methods are compared: POS-MASK (masks positives on easy queries only, Eq. 7), NEG-MASK (masks negatives on hard queries only, Eq. 8), and full QAE (both masks). On AIME'25, NEG-MASK (approximately 47.5) nearly matches QAE (approximately 48.0), while POS-MASK is noticeably lower (approximately 45.5). On AIME'24, the same ordering holds: NEG-MASK ≈ QAE > POS-MASK. This confirms that under weak clipping, the dominant failure mode is entropy explosion, and the NEG-MASK (explosion-proof gating on hard queries) is the critical mechanism. POS-MASK alone is insufficient because it doesn't prevent the negative-advantage-driven entropy surge.

Panel (c): Mask ablation with strong clipping (ϵ_high = 0.20). The ordering flips: POS-MASK now outperforms NEG-MASK on both AIME'25 and AIME'24. On AIME'25, POS-MASK achieves approximately 30.0 versus NEG-MASK's 24.5; on AIME'24, POS-MASK achieves approximately 40.0 versus NEG-MASK's 34.5. Full QAE achieves the highest in both cases (approximately 32.0 on AIME'25, 44.0 on AIME'24). This confirms that under strong clipping, the dominant failure mode shifts to entropy collapse, and the POS-MASK (collapse-proof gating on easy queries) becomes the critical mechanism. The strong clipping (ϵ_high = 0.20, compared to the default 0.28) heavily constrains update magnitudes, making it harder for the policy to explore, so the collapse-proof gating (removing the positive-advantage concentration force on easy queries) is essential to maintain diversity.

What this ablation establishes: The two masking regimes are complementary and non-redundant — each addresses a distinct failure mode, and their relative importance depends on the token-level clipping strength. Full QAE provides both, making it robust to both collapse and explosion regardless of the clipping configuration. This directly supports the paper's claim that two-sided entropy control is necessary and that the quantile baseline provides it through a single mechanism (the two-regime gate induced by the K-quantile).

---

Anthropomorphic Token Dynamics Under QAE (Figure 7)

Figure 7 (Appendix B.2) replicates the analysis from Figure 2 but for QAE instead of DAPO (with Clip-Higher). The anthropomorphic high-entropy token count (green bars) and overall pass@1 (orange line) are tracked over training steps.

Under DAPO (Figure 2), the two quantities show a "correlated growth" phase (steps 0–150) followed by a "decoupling & plateau" phase (steps 150–300), where token counts decline while pass@1 plateaus. Under QAE (Figure 7), the pattern is different: token counts and pass@1 show sustained co-growth. Both metrics rise together from step 0 to approximately step 200, and while token counts stabilize after step 200, pass@1 continues to improve. The key difference is that under QAE, the plateau in token counts does not coincide with a plateau in accuracy — the model continues converting exploration into better reasoning even after token-level diversity stabilizes.

Figure 8 provides a finer-grained snapshot at steps 20, 80, and 200, showing the probability mass over top high-entropy tokens. At step 20, anthropomorphic markers (wait, perhaps) are sparse. By step 80, they separate more distinctly from reasoning tokens (so, let), and by step 200, they stabilize while reasoning tokens continue to diversify. This contrasts with DAPO (Figure 3), where step 200 shows homogenization around rigid reasoning templates. Under QAE, the token distribution maintains a balance between anthropomorphic and reasoning tokens, consistent with the interpretation that QAE sustains productive exploration rather than allowing it to either collapse (homogenization) or explode (random noise).

---

Ablation Studies and Robustness Checks

• K sensitivity (Figure 9, Appendix B.3): Sweeping K ∈ {0.2, 0.4, 0.6, 0.8} on Qwen3-8B-Base reveals a U-shaped accuracy profile with optimum at K=0.4. K=0.2 produces lowest entropy but early accuracy plateau (over-regularized, exploration-poor). K=0.8 produces highest entropy with extreme volatility and accuracy stagnation (entropy explosion). K=0.6 is intermediate — elevated entropy but better than K=0.2 or 0.8. The default K=0.4 balances explosion-proof and collapse-proof gating. The paper's operational rule-of-thumb (Section 6) formalizes this: choose K=0.4 when baseline entropy is high, K=0.6 when baseline entropy is low. The sensitivity range is consistent with the theoretical monotonicity: K directly controls the fraction of queries receiving the entropy-increasing (b_K=1) vs. entropy-suppressing (b_K=0) regime.

• Mask ablation (Figure 6, b–c, Section 5.3): POS-MASK and NEG-MASK are one-sided versions of QAE that ablate each regime independently. Under weak clipping (ϵ_high=0.28, Figure 6b), NEG-MASK (explosion-proof) nearly matches full QAE, while POS-MASK is worse — confirming explosion is the dominant failure mode. Under strong clipping (ϵ_high=0.20, Figure 6c), POS-MASK (collapse-proof) outperforms NEG-MASK — confirming the dominant failure mode flips to collapse. Full QAE outperforms both one-sided masks in both clipping regimes. This is the key evidence that the two regimes are complementary and that the quantile baseline's two-sided gating (not just sparsity or asymmetry) is the active mechanism.

• Composition with token-level methods (Table 2, Section 5.1): QAE improves Clip-Cov, KL-Cov, and GSPO without changing their hyperparameters. The gains are consistent (10/12 pass@1 improvements, 10/12 pass@16 improvements or ties), demonstrating that QAE operates at a different level (baseline design) than token-level (Clip-Cov penalty, KL-Cov regularization) or sequence-level (GSPO) methods. The one configuration with slight pass@1 regression (KL-Cov + AIME'24, −0.5%) shows a large pass@16 gain (+5.6%), suggesting that QAE broadens the output distribution under strong KL regularization — a net positive for best-of-16 but not for greedy decoding.

• Model scale robustness (Figures 10, 11, Appendix B.4): QAE stabilizes entropy and sustains pass@1 gains on both Qwen3-8B (Figure 10) and Qwen3-14B (Figure 11). The qualitative pattern — reduced entropy volatility, delayed but sustained accuracy crossover, stabilized response length — is consistent across scales. The magnitude of the effect is somewhat smaller at 14B (the DAPO entropy spike is less severe at 14B), but QAE still provides clear benefits. The 14B model under DAPO peaks at ~0.45 entropy (vs. ~0.95 for 8B), suggesting entropy explosion is partially mitigated by scale but not eliminated, and QAE further reduces it.

• Architecture robustness (Table 2): QAE works on both dense models (8B, 14B) and Mixture-of-Experts models (Qwen3-30B-A3B). The GSPO + QAE results on 30B-A3B are mixed: pass@1 gains on AIME'25 (+4.3%) and AIME'24 (+8.6%), but regressions on AMC'23 pass@1 (−0.7%) and pass@16 (−2.2%). This is the only configuration with pass@16 regressions. Possible explanations: (a) the default K=0.4 is suboptimal for the 30B MoE architecture or for GSPO's optimization dynamics; (b) the 30B model's different entropy characteristics (possibly lower baseline entropy due to the MoE's implicit sparsity) require a different K; (c) GSPO's sequence-level loss interacts differently with the quantile baseline than DAPO's token-level loss. The paper does not ablate K for this configuration, which is a notable gap.

• Anthropomorphic token diagnostics (Figures 7, 8, Appendix B.2): Under QAE, anthropomorphic token counts and pass@1 exhibit sustained co-growth rather than the decoupling observed under DAPO. Token-level snapshots at steps 20/80/200 show that QAE maintains a balance between anthropomorphic and reasoning tokens, unlike DAPO's homogenization toward rigid templates. This is a qualitative diagnostic that supports the entropy-safety narrative but is not a controlled ablation — it describes the outcome rather than testing a specific mechanism.

• No explicit ablation of the standard deviation normalization: The paper retains the standard deviation denominator in the advantage formula (Eq. 3) without abating whether it is necessary. Since the quantile baseline already zeroes out advantages for one class per query, the standardization's role may be different than in GRPO/DAPO. An ablation comparing standardized vs. unstandardized quantile advantages would clarify whether the normalization is load-bearing or merely inherited.

• No ablation of dynamic sampling constraint: DAPO's dynamic sampling constraint (ensuring 0 < |{correct}| < G) is retained but not ablated. It's possible that this constraint interacts with the quantile baseline — for instance, if the constraint prevents the p = 0 and p = 1 edge cases where the standard deviation is zero (necessitating the ε in Eq. 3). The paper doesn't test QAE without dynamic sampling, so whether QAE would work with pure random batching is unknown.

---

Critical Assessment

Does QAE genuinely stabilize entropy and sustain pass@1 gains?

The evidence strongly supports this claim. Figure 5 (middle) shows DAPO's entropy growth concentrated in negative-advantage samples, which QAE suppresses. Figure 5 (left) shows QAE's pass@1 continuing to improve after DAPO plateaus (~step 100). This pattern replicates at both 8B (Figure 10) and 14B (Figure 11). The K-sweep (Figure 9) demonstrates that the entropy level is controllable through K and that the default K=0.4 balances the extremes. The mask ablation (Figure 6, b–c) shows that both explosion-proof and collapse-proof gating contribute, with their relative importance depending on clipping strength.

Caveat: All training curves are from single runs. Without multiple seeds, it's impossible to assess whether the observed pass@1 advantage at step 300 (approximately 0.48 vs. 0.35 for DAPO on 8B, Figure 10) is statistically reliable or within the noise of training stochasticity. The curves for DAPO in Figure 5 (left) show pass@1 oscillations of ~0.05 from step to step, while QAE's oscillations are ~0.02–0.03. This reduced variance is itself evidence of stabilization, but the mean difference should be confirmed with multi-seed averages and confidence intervals. The paper's single-run reporting is standard in the RLVR literature (DAPO, GRPO, and related work typically report single runs due to compute cost), but it limits the strength of statistical conclusions.

Does the K-quantile baseline genuinely cause the entropy stabilization, or is it merely the sparsity (fewer active samples) that matters?

The mask ablation (Figure 6, b–c) partially addresses this. If the mechanism were purely sparsity (fewer non-zero advantages), then POS-MASK and NEG-MASK should perform similarly regardless of clipping strength, since both reduce the number of active samples by roughly the same amount. Instead, their relative ordering flips with clipping strength: NEG-MASK dominates under weak clipping (explosion regime), POS-MASK dominates under strong clipping (collapse regime). This interaction with clipping strength is inconsistent with a pure sparsity explanation and consistent with the two-regime entropy-safety theory — each mask addresses a different entropy pathology, and the dominant pathology depends on clipping.

However, a gap remains: the paper does not compare QAE against a simpler sparsity-inducing baseline, such as randomly zeroing out 80% of advantages or using a fixed sparsity mask. If random sparsity also stabilized entropy and improved pass@1, then the quantile mechanism (and the two-sided guarantee) would be less distinctive. This ablation is not present.

Does QAE work because of the quantile specifically, or would any baseline that produces the two-regime gating work?

The threshold rule (Eq. 4) — b_K = 0 if p(q) ≤ 1−K, b_K = 1 if p(q) > 1−K — is the essential ingredient, and it emerges from the quantile definition applied to binary rewards. Other constructions could produce the same threshold rule (e.g., a percentile-based rule, a fixed constant comparison), but the quantile provides a natural, distribution-aware formulation. The paper argues (Section 6, Related Work connection to Arnal et al., 2025) that the quantile is the appropriate data-adaptive, group-level baseline, but it doesn't empirically compare against, say, a fixed baseline b = 1−K that doesn't adapt to p(q). This comparison would test whether the data-adaptive threshold (quantile) matters or just the two-regime structure.

Does the two-sided entropy safety guarantee (Proposition 4.2) hold in practice, or is it a simplified-model result?

The guarantee is proven for a bandit reduction with first-order softmax updates (Section 4.3). The actual RLVR training involves token-by-token autoregressive generation, multi-step gradient accumulation, clipping, and dynamic sampling — all of which violate the assumptions of the proof. The empirical results (Figure 5, Figures 10–11) are consistent with the guarantee (entropy is stabilized, explosion is suppressed), but the guarantee itself is a qualitative insight rather than a quantitative bound. The paper appropriately frames it as demonstrating the structural property of the quantile baseline (monotonic relationship between baseline and entropy change) rather than as a tight prediction of training dynamics.

Gaps and missing experiments:

1. No multi-seed statistics. All training curves and final results are single-run. This is the most significant methodological limitation. With 8B models and a 17K training dataset, 2–3 seeds per configuration would be feasible at moderate compute cost and would substantially strengthen the reliability of the pass@1 comparisons.

2. No K-tuning for non-Clip-Higher baselines. The default K=0.4 is justified for Clip-Higher by the entropy diagnostic (baseline entropy is high, so use lower K). For KL-Cov, Clip-Cov, and GSPO, the baseline entropy characteristics may differ, and the optimal K may not be 0.4. The paper doesn't report K-sweeps for these baselines, which could explain the weaker or negative results in some configurations (e.g., GSPO + AMC'23 regressions).

3. No random-sparsity baseline. As noted above, comparing QAE against a random 80% sparsity mask or a fixed-value baseline would help isolate whether the mechanism is the quantile gating or just the reduced number of active samples.

4. No comparison with learned baselines. A small value network predicting b(q) from query features (or from the group's response embeddings) would be a natural competing approach to the K-quantile. While this would add training overhead, it would test whether the quantile's simple, purely-statistical baseline is competitive with a learned, query-aware alternative.

5. No analysis of the interaction between K and group size G. The quantile baseline's behavior depends on G (the number of samples per group) because p(q) = (number of correct)/G has granularity 1/G. For small G, the threshold 1−K may fall between achievable p(q) values, making the gating coarser. The paper's experiments use the default DAPO group size (dynamic, likely G=8 or G=16 based on typical configurations), but no G-sweep is reported.

6. Single benchmark domain (math only). All three benchmarks (AIME'24, AIME'25, AMC'23) are competition-level math problems. The paper doesn't test on code generation (HumanEval, MBPP), logical reasoning (ARC, FOLIO), or general knowledge QA, which limits the generality of the entropy stabilization claim. Math reasoning may have particular entropy characteristics (long reasoning chains, clear correctness signals) that interact favorably with QAE; whether QAE would help on domains with shorter responses or fuzzier reward signals is unknown.

7. No long-training-horizon results. Figures 5 and 10 show training up to 300–600 steps. Whether QAE's accuracy advantage continues to grow, plateaus at a higher level, or eventually converges with DAPO at much longer horizons (e.g., 2,000+ steps) is unknown. The sustained upward trajectory of QAE's pass@1 in Figure 5 (left) suggests continued improvement, but without longer runs, the asymptotic gap cannot be assessed.

8. No test-time compute scaling analysis. The paper focuses on training stability and pass@1. Since QAE changes which responses are updated during training, it may also change the model's test-time scaling behavior (e.g., how much pass@k improves with k). The pass@16 results (Table 2) provide a point estimate, but a full pass@k curve for k=1,2,4,8,16,32 would reveal whether QAE models have broader or narrower output distributions, which would connect directly to the entropy stabilization mechanism.

Bottom line: The experimental evidence establishes that QAE stabilizes training entropy and improves pass@1 on math reasoning benchmarks for Qwen3 models when added to DAPO and related baselines. The effect is robust across model scales (8B, 14B) and architectures (dense, MoE), and composes with existing token-level and sequence-level methods. The two-regime mechanism is supported by the mask ablation, which shows complementary contributions from positive and negative gating depending on clipping strength. However, the single-run reporting, single-domain evaluation, and absence of several informative baselines (random sparsity, fixed baseline, learned baseline, long-horizon runs) mean the claims should be considered well-evidenced but not exhaustively validated. The paper's primary contribution — the baseline-design reframing and the K-quantile as a principled entropy knob — is supported by the experiments, but the degree to which QAE generalizes beyond the tested configurations (math reasoning, Qwen3 family, DAPO training recipe) remains an open question.

6. Limitations and Trade-offs

6.1 Single-Run Reporting Without Statistical Confidence

The assumption or constraint. All training curves (Figures 1, 2, 5, 9, 10, 11), final benchmark scores (Table 2), and ablation results (Figure 6) are based on single training runs — one random seed per configuration. The paper does not report confidence intervals, standard errors, or multi-seed averages for any result. There is no mention of statistical significance testing, no cross-validation protocol for final evaluation (the two-fold cross-validation mentioned in Section 3.2 of the prior analysis is not discussed in the main paper), and no assessment of how much the observed pass@1 differences (ranging from −0.7% to +21.5%) vary across runs.

The consequence. RLVR training is known to be sensitive to random seed through batch composition, sampling stochasticity, and initialization. Without multi-seed statistics, it is impossible to determine whether a reported pass@1 improvement of +0.3% (KL-Cov on AIME'25, Table 2) or a regression of −0.5% (KL-Cov on AIME'24) is a genuine effect or within the noise floor of training variance. The paper's central claim — that QAE stabilizes entropy and sustains pass@1 gains — is supported by visually consistent trajectory shapes, but the magnitude of the gains and the reliability of the crossover points (e.g., QAE overtaking DAPO around step 180–200 in Figure 10) cannot be assessed. For a practitioner deciding whether to adopt QAE, the expected gain per training run (and its variance) matters: if QAE improves the best-case run by 20% but sometimes underperforms DAPO due to seed sensitivity, the adoption decision changes. The reduced entropy volatility in QAE trajectories (oscillations of ~0.02–0.03 vs. ~0.05 for DAPO in Figure 5, left) is suggestive of reduced variance, but this is post-hoc visual inspection, not a statistical comparison.

What evidence exists in the paper. None. The paper provides no seed-level data. All figures plot single trajectories. Table 2 reports point estimates with no uncertainty quantification. The paper does not discuss this limitation or acknowledge it as a constraint on the strength of its conclusions.

Mitigation status. The paper does not attempt to address this limitation and does not discuss it in Section 8 (Limitations and Future Work). This is a significant methodological gap, particularly for a paper that claims to "identify baseline design — rather than token-level heuristics — as the primary mechanism for scaling RLVR" (Abstract) and that reports percentage gains as specific as +21.5%. The authors' compute constraints are understandable (training 8B and 14B models on math reasoning datasets is expensive), but even 2–3 seeds on a single core configuration (e.g., DAPO + Clip-Higher ± QAE on Qwen3-8B) would provide a baseline estimate of run-to-run variance and substantially strengthen the reliability of the comparisons.

---

6.2 Single-Domain Evaluation Limited to Math Reasoning

The assumption or constraint. All experiments are conducted exclusively on three competition-level mathematics benchmarks: AIME'24, AIME'25, and AMC'23 (Section 5). The paper provides no results on code generation (e.g., HumanEval, MBPP, LiveCodeBench), logical or scientific reasoning (e.g., ARC, GPQA, FOLIO), general knowledge QA, or any domain where correctness verification is less clean than exact-answer matching for math problems. The Qwen3 models are pretrained general-purpose LLMs, but the RLVR training uses only the DAPO-Math-17K dataset (Appendix B.1), and evaluation is limited to math.

The consequence. Math reasoning has several structural properties that may interact favorably with QAE's mechanism in ways that do not generalize: (1) rewards are binary and determined by exact-answer matching with a deterministic grading function, providing clean, noise-free correctness signals; (2) problems have unambiguous ground-truth answers, so the baseline's hard/easy classification (based on empirical success rate $p(q)$) operates on a well-defined signal; (3) reasoning chains are typically long (thousands of tokens, as shown in Figures 10–11), giving entropy dynamics substantial room to manifest; and (4) the task is purely deductive — there is no factual recall or open-ended generation component where "correctness" is ambiguous or multi-dimensional.

For a practitioner applying RLVR to code generation, the reward signal may be noisier (unit tests can pass for functionally incorrect code, or fail for trivially incorrect implementations that happen to pass tests). For reasoning tasks with multiple valid solution paths (e.g., debate, essay writing, creative problem-solving), binary rewards collapse a rich signal space. The paper's entropy-safety analysis (Proposition 4.2) assumes binary rewards explicitly — the proof uses $r \in \{0, 1\}$ and the threshold rule in Equation 4 depends on the empirical success rate $p(q)$ being well-defined from binary outcomes. Extending QAE to continuous or structured rewards would require reformulating the quantile baseline and re-proving the entropy guarantees. The paper does not discuss what happens to QAE when rewards are non-binary, noisy, or multi-objective.

What evidence exists in the paper. None beyond math benchmarks. The paper does not claim generality to other domains, but it also does not explicitly restrict its claims to math reasoning. The title ("Stabilizing RLVR for LLM Reasoning"), abstract ("sustained pass@1 gains on AIME'24/'25 and AMC'23"), and all experiments are math-specific, but the theoretical framework (Sections 4.2–4.3) and the baseline-design reframing (Section 3.2) are presented domain-agnostically. The reader is left to infer whether QAE is a math-reasoning-specific technique or a general RLVR stabilization method — and if the latter, the absence of non-math evaluation is a notable gap.

Mitigation status. The paper does not acknowledge this as a limitation in Section 8, which focuses on dynamic/automatic K-tuning and PPO integration as future work. The authors do not suggest evaluating QAE on other domains. This is a significant omission for a paper whose primary contribution is positioned as a general solution to RLVR instability ("reframes entropy regulation as a baseline-design problem rather than a token-level tuning problem," Section 1), not a math-specific recipe.

---

6.3 K Selection Requires Training-Phase Entropy Monitoring and Is Not Validated Across Baselines

The assumption or constraint. The paper's operational rule-of-thumb for selecting $K$ (Section 6) is: "choose $K$ once per baseline by inspecting the entropy of the baseline policy, rather than the evaluation metric. When entropy is low (risk of mode collapse), choose $K = 0.6$ to inject diversity; when entropy is high (risk of unstable updates), choose $K = 0.4$ to temper exploration." This assumes the practitioner has access to training-phase entropy curves for the baseline method and can make a diagnostic judgment about whether entropy is "low" or "high" — a relative comparison that requires knowing what constitutes normal entropy behavior for the given model, dataset, and training configuration.

The default $K = 0.4$ is empirically validated on Qwen3-8B-Base with DAPO's Clip-Higher ($\epsilon_{\text{high}} = 0.28$) through the sensitivity sweep in Figure 9 (Appendix B.3). No K-sweeps are reported for CLIP-Cov, KL-Cov, or GSPO (Table 2), nor for the 14B or 30B model scales. All non-Clip-Higher configurations use the same default $K = 0.4$ without justification based on those methods' entropy characteristics. The paper states this explicitly in Section 5: "Unless noted, we keep all training and decoding hyper-parameters identical across baselines and our method, changing only the response-level baseline from the mean to a K-quantile (default $K = 0.4$)."

The consequence. The default $K = 0.4$ may be suboptimal for configurations where the baseline entropy dynamics differ from Clip-Higher on Qwen3-8B. The GSPO + QAE configuration on Qwen3-30B-A3B (Table 2) shows the only consistent pass@16 regressions (−2.2% on AMC'23) and a slight pass@1 regression (−0.7% on AMC'23), which could reflect $K = 0.4$ being inappropriate for the GSPO optimization dynamics or the 30B MoE architecture. The KL-Cov + QAE configuration shows a −0.5% pass@1 regression on AIME'24, which could similarly reflect suboptimal K for the KL-regularized entropy regime. Without K-sweeps for these configurations, these results cannot be interpreted: they could mean QAE is genuinely less effective for these baselines, or they could mean the default K is simply wrong.

For a practitioner adopting QAE, the entropy-inspection rule-of-thumb requires running the baseline method first to full convergence to observe its entropy trajectory, then selecting K, then re-running with QAE. This doubles the training cost for K selection (baseline run + QAE run). If the entropy behavior of the baseline is unknown or if training compute is too constrained to run the baseline to convergence, the practitioner must rely on the default $K = 0.4$, which is validated only for one specific configuration and may not generalize.

What evidence exists in the paper. Figure 9 (Appendix B.3) sweeps K for one configuration only (Qwen3-8B-Base + Clip-Higher). The K-sweep shows that performance varies substantially with K — K=0.2 and K=0.8 both produce significantly lower final accuracy than K=0.4 (Figure 9, middle) — confirming that K selection matters. The mask ablation (Figure 6, b–c) demonstrates that the dominant failure mode (explosion vs. collapse) shifts with clipping strength, implying that the optimal K should also shift, but this shift is not quantified through K-sweeps at different clipping strengths. The entropy dynamics for Clip-Cov, KL-Cov, and GSPO are not shown, so there is no diagnostic basis for assessing whether the default K=0.4 is appropriate.

Mitigation status. The paper acknowledges K-tuning as future work in Section 8: "Beyond a fixed K, explore simple schedules or two-phase curricula to better balance exploration and exploitation" and "Adapt K to model state (e.g., success rate, entropy, or gradient variance) to remove manual tuning." The rule-of-thumb (Section 6) provides some practical guidance, but it is validated on only one configuration and requires training-phase access to baseline entropy. The paper does not propose a method for selecting K without a baseline run, nor does it characterize the sensitivity of final performance to K across configurations. This is partially mitigated by the fact that all methods in Table 2 use the same default K=0.4, making the comparisons internally consistent even if not individually optimal — but the absolute performance of QAE in non-Clip-Higher configurations should be understood as a lower bound, not a tuned optimum.

---

6.4 The Two-Sided Entropy Guarantee Is Provable Only Under Simplifying Assumptions That Do Not Hold in Practice

The assumption or constraint. Proposition 4.2 (Section 4.3) proves two-sided entropy safety for the K-quantile baseline under a bandit reduction (each full response is a single action) with first-order softmax logit updates and no clipping, dynamic sampling, KL penalties, or token-level loss normalization. The proof relies on: (a) the entropy–covariance identity $\Delta H(q) \approx -\eta \,\text{Cov}(\log \pi, \pi \hat{A})$, which is a first-order Taylor approximation valid only for small step sizes $\eta$; (b) the monotonicity of $\Delta H(q; b)$ in the baseline $b$, which follows from the strict positivity of $\text{Cov}(\log\pi, \pi)$ for non-uniform softmax policies — a property that assumes the policy is a softmax over the discrete action space of complete responses; and (c) the binary reward structure $r \in \{0, 1\}$ and the threshold reduction of the K-quantile baseline (Eq. 4).

The actual RLVR training violates all of these assumptions. It operates at the token level across sequences of thousands of tokens, uses multiple gradient accumulation steps with mini-batches (16 accumulation steps, batch size 32, global batch 512; Appendix B.1), applies asymmetric clipping ($\epsilon_{\text{low}} = 0.20$, $\epsilon_{\text{high}} = 0.28$) that modifies the effective update magnitude per token, normalizes the loss by the total number of tokens in the group (the $1/Z$ factor in DAPO's objective, Section 2), enforces a dynamic sampling constraint, and uses AdamW optimizer with momentum — none of which are captured by the first-order softmax logit update model. The bandit reduction also collapses the entire autoregressive generation process into a single action, ignoring the sequential nature of language generation where later tokens depend on earlier tokens and entropy manifests across the full sequence.

The consequence. The entropy-safety guarantee in Proposition 4.2 should be understood as a qualitative insight about the structural property of the K-quantile baseline (monotonic relationship between baseline and entropy change) rather than a quantitative bound on actual training entropy. The guarantee cannot predict the magnitude of entropy stabilization, the time horizon over which it holds, or whether it survives the interaction with clipping, gradient accumulation, and AdamW. A practitioner cannot use Proposition 4.2 to, for example, choose $\eta$ and $K$ to guarantee that entropy stays within a specific range — the proof provides an extremal comparison at a single step under idealized conditions, not a trajectory-level guarantee.

This is not a fatal flaw — most theoretical results in deep RL operate under simplifications — but it matters because the paper's narrative strongly emphasizes the theoretical guarantee as a distinguishing feature. The Abstract claims QAE "proves two-sided entropy safety, giving lower/upper bounds on one-step entropy change that curb explosion and prevent collapse." Section 4.3 is titled "Theoretical Analysis: Two-Regime Entropy Safety" and uses the language of formal proof ("Proposition 4.2," "hence," "strictly increasing"). A reader who does not carefully note the bandit reduction and first-order assumptions may overestimate the strength of the guarantee. The empirical results (Figures 5, 10, 11) provide the actual evidence for entropy stabilization; the theoretical analysis provides a conceptual explanation for why stabilization occurs, but not a formal guarantee that it will occur under realistic training conditions.

What evidence exists in the paper. The proof is in Appendix A.2. The assumptions (bandit reduction, first-order softmax updates) are stated in Section 4.3. The paper explicitly contrasts QAE with token-level methods: "Token-level mechanisms only rescale steps and do not change the response-level baseline, so they cannot realize these guarantees" (Section 4.3 Takeaway). This is true within the bandit model, but in the actual token-level training, token-level mechanisms do affect the effective response-level advantage through clipping and importance weighting, blurring the clean separation the proof relies on. The empirical entropy trajectories (Figure 5, middle; Figures 10, 11) are consistent with the qualitative prediction (QAE suppresses negative-advantage-driven entropy growth), but the degree of suppression and the stability of the suppressed regime are empirical findings, not deductive consequences of Proposition 4.2.

Mitigation status. The paper does not discuss the gap between the theoretical model and the actual training setup in Section 8 or elsewhere. The theoretical analysis is presented as a self-contained result without caveats about the bandit reduction or first-order approximation. This is partially standard practice in the RLVR literature (Cui et al., 2025, from which the entropy-covariance identity is adapted, uses similar simplifications), but given the paper's emphasis on the two-sided guarantee as a core contribution, a discussion of the assumptions and their practical limitations would strengthen the paper's transparency.

---

6.5 The Hardest Queries Receive No Meaningful Benefit Across All Methods — QAE Does Not Expand the Frontier of Solvable Problems

The assumption or constraint. The paper demonstrates that QAE stabilizes training and improves pass@1 on math benchmarks, but these improvements come from better allocation of the existing gradient signal across the query difficulty distribution — not from enabling the model to solve problems it fundamentally cannot solve. The paper provides no evidence that QAE helps on the hardest queries (those where the base policy's success rate is near zero).

The consequence. While the paper does not use explicit difficulty bins (unlike some prior RLVR work), the quantile baseline's two-regime structure (Eq. 4) implies that on very hard queries where $p(q) \leq 1-K$, QAE applies the hard-query regime: only rare successes get reinforced, and all failures are masked. If the base policy never produces a correct response for a query ($p(q) = 0$), then QAE cannot assign positive advantage to any response — all samples are masked, no gradient flows, and the policy never improves on that query. This is not a flaw in QAE per se — it is a fundamental property of any advantage-based method with binary rewards — but it means QAE's benefits are concentrated on queries where the model already has at least some non-trivial chance of success. For genuinely out-of-capability problems, QAE provides zero benefit. The paper does not report per-difficulty breakdowns (e.g., accuracy on the hardest vs. easiest subsets of AIME problems), so it is unknown whether QAE's pass@1 gains come primarily from improving moderate-difficulty problems (where the gradient signal is informative) or from a uniform improvement across the difficulty spectrum.

This connects to a broader limitation of RLVR: test-time compute methods (scaling generation budgets, search algorithms, verifier-guided sampling) cannot compensate for a complete absence of correct solutions in the proposal distribution. The paper's Claim in Section 1 — that "baseline design — rather than token-level heuristics — is the primary mechanism for scaling RLVR" — refers to training stability and efficiency of scaling, not to expanding the frontier of solvable problems. A practitioner working on a dataset where most queries are very hard for the base model (pass@1 near zero) should not expect QAE to help; pretraining or supervised fine-tuning on harder data would be more appropriate investments.

What evidence exists in the paper. None directly. The paper does not report difficulty-stratified results. The closest evidence is the AIME'24 and AIME'25 results in Table 2: Qwen3-8B-Base + DAPO achieves 39.69 pass@1 on AIME'24, and QAE improves this to 48.23. If AIME'24 contains a subset of extremely hard problems where the base model scores ~0%, QAE's gains on those problems would be zero, meaning the +21.5% improvement must be driven by problems where DAPO already had non-trivial pass@1. The paper cannot confirm or refute this interpretation without difficulty-bin analysis.

The K-sweep (Figure 9) provides indirect evidence: at all K values, accuracy climbs and then plateaus or climbs slowly — no K value shows an escape from zero on initially-unsolvable problems. This is consistent with the interpretation that QAE redistributes gradient effort across difficulty levels but does not enable solving previously impossible problems.

Mitigation status. The paper does not discuss this limitation. Section 8 (Limitations and Future Work) focuses on K-tuning extensions and PPO integration, not on QAE's difficulty-dependent effectiveness. This is a notable omission given the RLVR literature's increasing attention to difficulty-aware training strategies and the practical importance of knowing which problems benefit from a new method.

---

6.6 Sparsity as a Claimed Benefit Without a Controlled Sparsity Baseline

The assumption or constraint. One of the paper's headline findings is that QAE "sparsifies credit assignment" such that "roughly 80% of responses receive zero advantage" (Abstract, Section 5.2, Figure 5, right). The paper interprets this sparsity as beneficial: "This concentrates computational effort on the most informative samples and revealing a deep redundancy in standard mean-baseline approaches" (Section 1). However, the paper never compares QAE against a controlled baseline that also induces sparsity without the quantile mechanism — for example, randomly masking 80% of advantages regardless of query difficulty, or using a fixed threshold baseline that produces similar sparsity levels.

The consequence. The observed correlation between sparsity (80% zero-advantage responses) and improved training stability/pass@1 does not establish that the quantile-gating mechanism is causal. It could be that any method that reduces the number of active gradient updates per batch — regardless of which updates are masked — would stabilize entropy and improve performance, simply because fewer, larger-magnitude updates reduce gradient noise. If random sparsity also works, then the two-regime gating (hard vs. easy queries, positive vs. negative masking) is not the essential mechanism, and the quantile baseline's theoretical properties (Proposition 4.2) are not load-bearing. If random sparsity does NOT work (i.e., QAE's specific pattern of sparsity matters), then the paper's interpretation is correct but untested.

The mask ablation (Figure 6, b–c) partially addresses the pattern question by comparing POS-MASK and NEG-MASK, but both of these are structured masks derived from the quantile gate — they preserve the task-difficulty-dependent gating but ablate one side. They do not test whether the difficulty-dependence matters. A fixed sparsity baseline (e.g., always mask 80% of responses, chosen randomly, or always use a baseline of $b = 0.5$ and standardize) would test whether the quantile's data-adaptive threshold is necessary.

What evidence exists in the paper. Figure 5 (right) documents the sparsity level. The mask ablation (Figure 6) shows that the pattern matters for the relative importance of positive vs. negative masking under different clipping regimes, but it does not compare structured sparsity against unstructured sparsity. The discriminative reformulation (Proposition 4.1) shows that the quantile baseline produces a specific query-weighting scheme (asymmetric, monotonic, with hard gating), which is structurally different from the symmetric $\sqrt{p(1-p)}$ weighting of the mean baseline — but this doesn't rule out that another weighting scheme (e.g., fixed threshold) might work similarly well.

Mitigation status. The paper does not discuss this limitation, does not propose a random-sparsity or fixed-threshold baseline as an ablation, and does not acknowledge that the sparsity-as-benefit interpretation is correlational rather than causally established. The claim that QAE "sparsifies credit assignment" is descriptive (it documents that sparsity occurs), but the paper's framing in the Abstract and Introduction ("revealing a deep redundancy in standard mean-baseline approaches") implicitly claims that sparsity is the mechanism of improvement, which requires a controlled comparison to verify. Adding a simple random-masking baseline (or a fixed $b = 0.5$ baseline) would substantially strengthen this aspect of the paper's argument.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper changes what variable practitioners think about when debugging RLVR instability. Before QAE, the dominant mental model was that entropy problems in value-free RL are token-level phenomena: if the policy collapses, raise the upper clipping bound (Clip-Higher) or uplift low-probability tokens; if the policy is unstable, lower the learning rate or add KL penalties. The baseline — the $b$ in $\hat{A}_i = (R_i - b) / \sigma$ — was treated as a fixed, uninteresting constant (the group mean), and the entire design space for entropy control was assumed to live in token-level hyperparameters.

QAE reframes the problem entirely. The paper's central theoretical move — proving (Proposition 4.2, Appendix A.2) that under first-order softmax updates, the one-step entropy change $\Delta H(q; b)$ is strictly increasing in the baseline $b$ — establishes that the baseline is not a passive normalization constant but a direct, monotonic entropy knob. This is a genuine conceptual shift: the baseline controls which responses receive non-zero advantage, which determines the sign pattern of policy gradients, which determines whether the policy concentrates or disperses. Token-level clipping modifies magnitudes but leaves the sign pattern — and therefore the entropy direction — unchanged. The paper demonstrates this empirically through Observation 4 (Table 1): sweeping $\epsilon_{\text{high}}$ from 0.20 to 0.28 — the primary token-level knob in DAPO — produces only marginal accuracy changes and does not resolve the late-stage plateau. The bottleneck is not how much to update per token but which responses to update at all.

This reframing makes the paper a diagnostic contribution as much as a method contribution. Prior work documented entropy collapse and proposed fixes, but no prior work identified that the mean baseline simultaneously causes both collapse and explosion through the same structural flaw: symmetric, full-coverage advantage assignment that distributes non-zero gradients to every response in every batch. The discriminative reformulation (Proposition 4.1, Section 4.2) makes this structural flaw explicit: the mean baseline produces a symmetric $\sqrt{p(1-p)}$ query weight that simultaneously pushes the policy toward correct responses and away from incorrect responses on every query, creating a dispersion force (on hard queries, the push away from $G-1$ failures dominates the pull toward one success) and a concentration force (on easy queries, the pull toward $G-1$ successes dominates) that together drive entropy in opposing directions depending on difficulty. QAE's key insight is that nullifying one side per query — masking negatives on hard queries, masking positives on easy queries — breaks this symmetry and provides independent control over each direction.

The paper also provides a unified explanation for conflicting prior findings. The field's inconsistent results with entropy interventions — some methods prevent collapse but induce explosion, some are stable but under-explore, some work on some models but not others — are explained by the observation that all prior methods operate on a common, flawed baseline. Clip-Higher (Yu et al., 2025) prevents early collapse but triggers an entropy spike driven by negative-advantage samples (Observation 3, Figure 4 left); the spike is not a side effect of the clipping mechanism but of the mean baseline that Clip-Higher doesn't modify. Zhu et al. (2025) find that learning primarily from negative samples improves diversity — this makes sense because it partially masks the positive-advantage concentration force, approximating one side of QAE's easy-query gating. Cui et al. (2025) document entropy collapse and propose token-level covariance penalties, but those penalties rescale update magnitudes without changing the baseline — so they can prevent collapse but not explosion. QAE's two-regime gate explains why each prior method helps in some regimes but fails in others: they address one half of the entropy dilemma while leaving the other half unaddressed, and the mean baseline guarantees that the unaddressed half will eventually dominate.

What becomes more attractive as a research direction:

• Baseline design as a first-class design axis. The paper demonstrates that simply swapping the mean for a quantile — a one-line code change — produces 4–22% relative pass@1 improvements without touching any other hyperparameter. This makes baseline design a high-leverage, low-engineering-cost intervention that deserves systematic exploration: what other baseline statistics (trimmed mean, median, Huberized mean, learned functions) produce what entropy dynamics? The entropy–covariance identity (Eq. 18) provides a principled framework for analyzing any candidate baseline, not just the K-quantile.

• Sparse credit assignment in RLVR. The emergent 80/20 rule (Figure 5, right) — approximately 80% of responses receive zero advantage under QAE — suggests that the dominant paradigm of updating on every sample is deeply inefficient and that intelligent selection of which samples to update matters more than the magnitude of those updates. Future RLVR methods should prioritize sample selection mechanisms over adaptive learning rate schemes.

• Entropy as a diagnostic, not just a metric. The paper's operational rule-of-thumb — choose $K$ by inspecting training entropy, not evaluation accuracy — elevates entropy from a passively tracked metric to an actively used diagnostic for hyperparameter selection. This is a practical shift: before QAE, a practitioner seeing high entropy would lower the learning rate; after QAE, they would change the baseline.

What becomes less attractive:

• Blind token-level hyperparameter tuning. Table 1 (sweeping $\epsilon_{\text{high}}$ with marginal effects) and Observation 4 (token-level control yields "homogenized, low-quality exploration," Figure 3) suggest that the entropy-sensitivity of RLVR training is not a token-level problem. Further incremental adjustments to clipping bounds, KL penalties, or per-token importance weights — without addressing the underlying baseline — are unlikely to resolve the fundamental entropy dilemma. The paper is not saying token-level controls are useless (Clip-Higher is necessary to prevent immediate collapse; QAE assumes it), but rather that they are saturated: the remaining instability after applying best-practice token-level methods originates at the baseline level.

• Purely empirical recipe development. The paper demonstrates that a minimal, theoretically-motivated change (K-quantile baseline, justified by the entropy–covariance identity and the two-regime proof) outperforms complex empirical recipes that stack multiple heuristics. This suggests that future RLVR advances will come from re-examining foundational assumptions (what is the right advantage estimator?) rather than from more elaborate combinations of existing components.

---

Follow-Up Research This Work Enables

Cheap, online difficulty estimation to replace the K hyperparameter. The paper's main practical limitation is that $K$ is a fixed hyperparameter that must be selected before training, and the rule-of-thumb (inspect baseline entropy) requires a preliminary training run. A natural next step is to make $K$ adaptive during training: estimate the policy's current entropy state from a small rolling window of recent batches, compare it to a target entropy range, and adjust $K$ online. If entropy is above the target, decrease $K$ to make more queries "hard" and apply the explosion-proof $b_K = 0$ regime; if entropy is below the target, increase $K$ to make more queries "easy" and apply the collapse-proof $b_K = 1$ regime. This is a simple feedback controller (proportional control) that requires no additional forward passes and only minimal bookkeeping (tracking the running mean of per-batch entropy). A strong follow-up would implement this adaptive K schedule on the same Qwen3-8B + DAPO setup used in the paper, compare against fixed $K = 0.4$ across 3 seeds, and report whether adaptive K eliminates the need for manual tuning while matching or exceeding the fixed-K performance. The key question is whether the entropy–K relationship is stable enough across training for simple proportional control, or whether it exhibits hysteresis or non-monotonicity that requires more sophisticated control (PID, model-predictive).

Does QAE generalize to non-binary rewards and non-math domains? The paper's theoretical framework (Proposition 4.2) and the threshold reduction in Equation 4 both assume binary rewards $R_i \in \{0, 1\}$. However, many important RLVR applications use structured or continuous rewards: code generation with pass@k test-case rewards (0 to 1 continuous), dialogue with rubric-based scoring, or multi-objective rewards combining correctness and style. Extending QAE to non-binary rewards requires defining a K-quantile for continuous-valued rewards — which is straightforward (the empirical quantile of a set of real numbers is well-defined) — but the key open question is whether the two-regime entropy safety guarantee survives. For continuous rewards, the baseline is a real number, not necessarily 0 or 1, and the "threshold rule" that partitions queries into hard/easy regimes becomes fuzzy: what does it mean for a query to be "hard" when rewards can be 0.3, 0.7, etc.? A concrete experiment: apply QAE to code generation (e.g., HumanEval, MBPP) where rewards are the fraction of test cases passed, keep the quantile baseline $b_K(q)$ as the empirical K-quantile of the group's continuous reward distribution, and track whether the entropy stabilization pattern from Figure 5 replicates. If it does, the binary-reward assumption is not load-bearing and QAE generalizes. If it doesn't, the binary structure is essential and extensions to continuous rewards require a different mechanism (e.g., quantizing rewards, or using a multi-quantile baseline).

QAE + PRM-guided search: combining baseline-level and verifier-level entropy control. The paper studies entropy control purely within the RLVR training loop. However, prior work (referenced in the broader RLVR literature) has shown that test-time compute strategies — process reward model (PRM) guided beam search, best-of-N verification, sequential revisions — also affect the effective entropy of the model's output distribution. A natural extension is to use QAE during RLVR training to produce a policy with well-regulated entropy, and then apply PRM-guided search at inference to further improve accuracy on hard queries where even the well-regulated policy has low pass@1. The two mechanisms are complementary: QAE ensures the policy doesn't over-concentrate (preserving solution diversity) or over-disperse (preserving solution quality), while PRM search exploits the remaining diversity to find correct solutions. A concrete experiment: train a Qwen3-8B model with DAPO + QAE (default $K=0.4$) on the DAPO-Math-17K dataset, then evaluate on AIME'24 with a PRM-guided best-of-N verifier at inference (N=4, 16, 64), comparing against a DAPO-only (mean baseline) model with the same test-time compute budget. The hypothesis is that QAE-trained models will show better pass@k scaling (larger gap between pass@1 and pass@k) because QAE preserves within-group diversity during training, providing a richer proposal distribution for the test-time verifier to select from.

Interaction between group size G and the quantile baseline. The quantile baseline's behavior depends on the group size $G$ because the empirical success rate $p(q) = (\text{number correct}) / G$ has granularity $1/G$, and the threshold $1-K$ may fall between achievable $p(q)$ values. For small $G$ (e.g., 4 or 8), the gating is coarse-grained — with $G=4$ and $K=0.4$, $1-K=0.6$, and $p(q)$ can be 0, 0.25, 0.5, 0.75, or 1.0, so the threshold lands between 0.5 and 0.75, creating a sharp boundary. For large $G$ (e.g., 32 or 64), the threshold is finer-grained. This matters because DAPO's dynamic sampling constraint ensures $0 < |\{\text{correct}\}| < G$, but doesn't specify $G$ itself, and the optimal $K$ may depend on $G$. A sweep over $G \in \{4, 8, 16, 32\}$ with a fixed $K=0.4$ on the standard Qwen3-8B + DAPO setup would test whether QAE's benefits are robust to group size or whether smaller groups (where the gating is coarser) lose the entropy stabilization effect. If small $G$ degrades QAE, this would motivate either larger group sizes or an adaptive K that compensates for granularity.

QAE in multi-turn and long-horizon RL settings. The paper evaluates QAE on single-turn math reasoning where each response is generated independently and rewarded once. However, many emerging RLVR applications involve multi-turn interactions (dialogue, tool use, iterative refinement) where rewards are delayed and partial. In these settings, the "group" might be a set of complete trajectories rather than independent responses to the same prompt, and the concept of query difficulty ($p(q)$) becomes ambiguous — difficulty is not a static property of the query but emerges over the course of interaction. Adapting QAE to these settings would require redefining the quantile baseline to operate over trajectory-level returns rather than response-level binary rewards, and the two-regime gating (hard = exploitation, easy = exploration) would need to be reinterpreted for sequential credit assignment. A concrete stress-test: apply QAE to a multi-turn math reasoning setup where the model can revise its answer over multiple turns, receiving a binary reward only at the final turn. Use the final reward to compute $p(q)$ for the query, but apply the quantile baseline to advantage estimates at each turn. This tests whether the two-regime structure survives temporal credit assignment — do "hard" queries (low final success rate) benefit from reinforcing intermediate steps that lead to rare successes, even though those intermediate steps are not directly rewarded?

Theoretical analysis: extending the entropy safety guarantee beyond the bandit reduction. Proposition 4.2 is proven under a bandit reduction (complete response = single action) with first-order softmax updates. This abstraction captures the response-level sign pattern but ignores token-level structure, clipping, multi-step optimization, and momentum. A stronger theoretical contribution would derive entropy bounds for the actual token-level autoregressive policy gradient with clipping, perhaps using a Lyapunov-style analysis that shows the entropy of the full sequence distribution contracts toward a bounded interval under the quantile baseline, with the contraction rate depending on $K$ and the clipping parameters. This is a significantly harder problem — the token-level policy is a product of conditional distributions, not a single softmax — but even a partial result (e.g., a bound on the expected entropy change per training step under the quantile baseline vs. the mean baseline, for a simplified 2-token sequence model) would bridge the gap between the paper's clean theory and the messy practice, and would provide guidance on how clipping and K interact.

---

Practical Applications and Downstream Use Cases

Cost-efficient RLVR training for mid-sized open-source models. The most immediate practical application of QAE is as a drop-in stabilizer for existing RLVR pipelines training reasoning models on math, code, or science benchmarks. The implementation cost is essentially zero — replacing mean({R_i}) with quantile({R_i}, K) in the advantage estimation code — and the training compute overhead is negligible (sorting $G$ binary values per batch). For teams training 8B–30B parameter models on mathematical reasoning datasets (the exact scale tested in the paper), QAE's pass@1 improvements of +6.7% to +21.5% (Table 2, Qwen3-8B-Base on AIME'24 and AIME'25) translate directly to higher benchmark scores without additional training steps, larger models, or more training data. The stabilized entropy (Figures 10, 11) also means that training runs are less sensitive to hyperparameter choices — the reduced entropy volatility under QAE implies that early-stopping decisions are less brittle, and the sustained accuracy gains past the DAPO plateau (Figure 5, left) mean that training can continue productively for longer without additional tuning.

Training larger models with bounded response length. A secondary practical benefit is QAE's effect on response length. Figures 10 and 11 show that DAPO's entropy explosion correlates with response length spikes (5,000–9,000 tokens with large oscillations for the 8B model), while QAE stabilizes response length in a tighter range (4,000–5,500 tokens) without sacrificing accuracy. For teams training models at scale (70B+ parameters), response length directly controls inference cost during training — longer responses mean more tokens to generate, more memory to store attention states, and longer backward passes. QAE's length stabilization reduces the variance of per-step training cost, making resource allocation more predictable, and may reduce the average response length if the baseline method's entropy-driven length inflation is substantial. The paper does not quantify the compute savings from shorter responses, but for a 14B model, reducing average response length from ~8,000 to ~6,000 tokens (the approximate gap in Figure 11) represents roughly a 25% reduction in inference FLOPs per training step, which at RLVR training scales (millions of samples) translates to meaningful GPU-hour savings.

Self-improvement pipelines with iterative RLVR fine-tuning. Many production LLM pipelines involve multiple rounds of RLVR training: generate solutions, train on the successful ones, generate better solutions, repeat (e.g., STaR, ReST^EM, or rejection sampling fine-tuning loops). In these settings, the entropy characteristics of the policy can shift dramatically across rounds — early rounds may have high entropy (the base model is uncertain), middle rounds may have moderate entropy (the model is improving), and late rounds may risk collapse (the model overfits to the training distribution). QAE's K parameter provides a single knob to adapt the exploration–exploitation balance across rounds without re-tuning the entire optimization stack: start with $K = 0.6$ in early rounds to encourage exploration (entropy-increasing $b_K = 1$ on easy queries), shift to $K = 0.4$ in middle rounds for balanced updates, and consider $K = 0.2$ in late rounds if the policy shows signs of over-concentration. The paper's K-sweep (Figure 9) demonstrates that this range covers a substantial spread of entropy behaviors (from low-entropy, exploration-poor at $K=0.2$ to high-entropy, volatile at $K=0.8$), providing empirical grounding for a round-dependent K schedule. For a team running 3–5 rounds of RLVR fine-tuning, this replaces per-round hyperparameter grid searches with a single, interpretable schedule.

Open-source RLVR recipes as a standard component. The paper's finding that QAE composes cleanly with Clip-Cov, KL-Cov, and GSPO (Table 2) — without changing their hyperparameters — positions it as a standard add-on for open-source RLVR training scripts, similar to how gradient clipping and learning rate warmup became standard components of supervised fine-tuning pipelines. The implementation simplicity (one-line change) and the consistent pass@1 improvements across 10 out of 12 comparisons in Table 2 make QAE a low-risk, low-effort addition to any value-free RLVR codebase. The paper's default $K=0.4$ is validated for DAPO with Clip-Higher and provides a reasonable starting point for other configurations (even if suboptimal, as discussed in Section 6.3, the gains in Table 2 show it is rarely harmful). For the open-source community maintaining VERL, OpenRLHF, or similar RLVR frameworks, making QAE the default advantage estimator (with a configurable K parameter) would be a high-impact, low-regression-risk change — the paper provides the empirical evidence to justify this, though multi-seed validation would strengthen the case.