Your Group-Relative Advantage Is Biased

🎯 Pitch

Group-relative advantage estimation—the backbone of GRPO and its variants—is provably biased in a way that sabotages learning: it systematically underestimates advantages for hard problems and overestimates them for easy ones. This paper identifies the theoretical root of this failure and introduces a lightweight, plug-and-play correction (HA-DW) that restores balance, yielding consistent gains on multiple math benchmarks—but only when the base model has some foothold on the task.

1. Executive Summary

This paper theoretically analyzes and empirically exposes a fundamental statistical bias in group-relative reinforcement learning algorithms (GRPO, GSPO, DAPO) used for post-training LLMs on mathematical reasoning benchmarks: the group-relative advantage estimator systematically underestimates advantages for hard prompts and overestimates them for easy prompts (with the probability of underestimation exceeding 78% for prompts with expected success below 25%, and deterministic bias at extremes below 1/G or above (G−1)/G), leading to imbalanced exploration and exploitation. To correct this, the authors propose History-Aware Adaptive Difficulty Weighting (HA-DW), which uses a Kalman-style evolving difficulty anchor that aggregates cross-batch reward history, combined with an exponential reweighting factor that amplifies advantage estimates for hard prompts and suppresses them for easy ones — a lightweight plug-and-play module for any group-relative RL algorithm. When integrated into GRPO and its variants on Qwen3-4B/8B-Base and LLaMA-3.2-3B-Instruct across five mathematical reasoning benchmarks (MATH500, AIME25, AMC23, Minerva, OlympiadBench), HA-DW yields consistent accuracy improvements averaging ~2 percentage points, with the gain concentrated on hard MATH500 questions (+3.4%), establishing that correcting biased advantage estimation improves RLVR training but only when the base model possesses non-trivial capability on the targeted difficulty level — the method provides no benefit on problems fundamentally outside the model's reach.

2. Context and Motivation

The Core Problem: We Don't Know If Group-Relative Advantage Estimation Is Actually Correct

This paper addresses a surprisingly fundamental question that has gone unexamined despite widespread adoption: when GRPO-style algorithms compute advantages using only the rewards within a small group of sampled responses, are those advantage estimates statistically well-behaved? The question matters because group-relative RL algorithms — GRPO (Shao et al., 2024) and its numerous variants (GSPO, DAPO, Dr.GRPO, GMPO, etc.) — have become the de facto approach for post-training large language models on reasoning tasks after the success of DeepSeek-R1 (DeepSeek-AI, 2025). Yet, as the paper notes in its introduction, this family of methods "still lacks a detailed theoretical characterization" (Section 1, paragraph 3).

This gap is significant for several practical and theoretical reasons:

Widespread deployment without understanding: GRPO-style algorithms are being used to train reasoning models at scale — the DeepSeek-R1 technical report demonstrated that reinforcement learning with verifier rewards can induce sophisticated reasoning behaviors like self-verification and backtracking. If the core advantage estimation mechanism in these algorithms is systematically biased, then every model trained this way may be learning from distorted signal, with consequences that compound across training steps.
Resource allocation under constrained sampling: A key practical motivation for group-relative methods is that they avoid training a separate critic model (as PPO requires), making them simpler and more memory-efficient. However, this simplicity comes at a cost: each prompt typically gets only a small number of rollouts (commonly $G = 8$ , as the paper notes in Section 2.2, citing Zhang et al., 2025; Liu et al., 2025a; Shen et al., 2025). With such small group sizes, statistical estimators can behave very differently from their asymptotic properties. The paper explicitly motivates its analysis around this practical constraint: "generating multiple rollouts per prompt is computationally expensive in practice" (Section 2.2, before Corollary 1).
The exploration-exploitation tradeoff: If the advantage estimator systematically underestimates advantages on hard prompts, the policy receives a weaker learning signal for the very problems where it most needs to improve. Conversely, if it overestimates advantages on easy prompts, the policy may over-exploit already-mastered behaviors, wasting compute on marginal improvements. This imbalance — under-learning from hard questions while over-exploiting easy ones — is precisely what the paper identifies as the downstream consequence of its discovered bias (Section 1, paragraph after heading "Your advantage estimation is biased").

Why This Problem Was Hidden

The bias the paper uncovers is not obvious from aggregate performance curves. If you train a model with GRPO and plot average accuracy, the curve goes up — the algorithm works. What's hidden is that the learning signal is distorted differently across the difficulty spectrum: hard prompts get a weaker signal than they deserve, easy prompts get a stronger one. This is a distributional effect that average metrics conceal. The paper's contribution is therefore in exposing a structural flaw that manifests not as training failure, but as suboptimal allocation of learning capacity across prompt difficulties.

The bias is also conditional on a subtle but ubiquitous design choice: group-relative algorithms typically discard or ignore groups where all responses are correct or all are incorrect (because then all advantages are zero, producing no gradient). This filtering — which the paper formalizes as conditioning on the "non-degenerate event" $\mathcal{S} = \{1 \leq R \leq G-1\}$ (Equation 6 in Section 2.2) — is precisely what introduces the bias. Intuitively, by excluding uniform-outcome groups, you're conditioning on a subset of the sampling distribution that has a shifted mean. The paper shows that this conditional mean is systematically below the true expected reward $p_t$ when $p_t < 0.5$ , and above it when $p_t > 0.5$ , with equality only at exactly $p_t = 0.5$ (Theorem 1).

This is a subtle point that likely escaped prior notice because (a) the conditioning event $\mathcal{S}$ is not an explicit design choice but an emergent consequence of the algorithm's gradient structure, and (b) the bias direction flips with prompt difficulty, so it doesn't show up as a simple offset in aggregate metrics.

Where Prior Work Falls Short

The paper identifies several categories of prior work that are related but fail to address this specific gap:

1. Algorithm development without theoretical characterization. The GRPO family has evolved rapidly since DeepSeek-R1, with numerous variants proposed to improve stability and performance. The paper lists Dr.GRPO (which removes heuristic normalizations), DAPO (which adds decoupled clipping and dynamic sampling), GSPO (which uses sequence-level ratios), GMPO (which uses geometric mean of token-level rewards), and others (Section 1, paragraph 1; Appendix A). Each of these variants modifies the optimization objective — the clipping strategy, the importance sampling ratio definition, the advantage normalization — but none of them question whether the underlying group-relative advantage estimate is itself statistically correct. They refine how the estimate is used, not whether the estimate is reliable.

This is a classic pattern in ML: when a method works, the research community focuses on engineering improvements (better clipping, better normalization, dynamic hyperparameters) without revisiting the foundational assumptions. The paper's framing — "Your advantage estimation is biased" — is deliberately provocative because it redirects attention upstream of the optimization objective to the estimator itself.

2. Empirical scaling studies without bias analysis. Contemporary work by Xiong et al. (2025) and Tan et al. (2025) studies the scaling behavior of RLVR training, examining how performance varies with model size, training steps, and hyperparameters. These studies provide valuable empirical guidance but do not analyze the statistical properties of the advantage estimator at the core of the algorithms they study. The paper cites these works (Section 1) to establish that "a detailed theoretical characterization" remains absent from the literature.

3. Cross-batch methods in other domains. The paper's proposed solution — an evolving difficulty anchor that aggregates information across training batches — draws inspiration from cross-batch techniques developed in computer vision and recommender systems (Appendix A cites cross-batch memory for embedding learning, cross-batch negative sampling, cross-iteration batch normalization, and cross-batch hard example mining). These methods demonstrate that incorporating historical batch information can improve training dynamics, but they were developed for entirely different problem settings (face recognition, recommendation, representation learning) and do not address the specific statistical bias in group-relative advantage estimation for RLVR. The paper repurposes the cross-batch idea in a novel way: not for better negative samples or normalization statistics, but to construct an evolving difficulty reference that reveals when advantage estimates are systematically off.

4. Bias-variance tradeoff and off-policy correction literature. The paper acknowledges the broader context of biased estimation in machine learning — the bias-variance tradeoff (Hastie et al., 2009; Murphy, 2012), off-policy correction methods like Retrace (Munos et al., 2016), V-trace (Espeholt et al., 2018), and doubly robust estimators (Jiang and Li, 2016). However, these methods address a fundamentally different source of bias: the distribution shift between the behavior policy (which collected the data) and the target policy (which is being evaluated). In RLVR, by contrast, the data is on-policy — responses are sampled from the current policy $\pi_{\theta_t}$ — so there is no off-policy correction needed in the traditional sense. The bias the paper identifies is entirely a finite-sample, conditioning-induced artifact of using within-group averages as baselines. This is a different category of estimation error that prior off-policy methods were not designed to address.

5. Difficulty-aware training methods that treat difficulty as fixed. Several recent works incorporate prompt difficulty into RL training, such as adaptive difficulty curriculum learning (Zhang et al., 2025) and length-aware dynamic sampling (Chen et al., 2025). These methods treat difficulty as a property of the prompt that can be leveraged for curriculum design or sampling strategy. However, the paper's perspective is fundamentally different: difficulty is not a fixed prompt property but is relative to the evolving policy. A prompt that is hard at training step $t=1$ may become easy at step $t=100$ as the model improves. This relativistic view — formalized through the evolving difficulty anchor $C_t$ — is absent from prior work, which implicitly assumes difficulty is static.

How This Paper Positions Itself

The paper positions itself not as proposing yet another GRPO variant with a new clipping strategy or normalization trick, but as identifying and correcting a fundamental statistical flaw that affects all group-relative methods equally. The authors explicitly state their framing (Section 1, final paragraph):

"Our goal is not to model all RLVR settings, but to expose a previously overlooked statistical bias in group-relative algorithms and demonstrate that even lightweight corrections can yield consistent gains."

This distinguishes the paper from the GRPO variant literature in two ways. First, its contribution is architectural rather than algorithmic: HA-DW is a plug-and-play module that wraps around any group-relative objective (Equations 25, 29, 31 in Appendix B show the instantiation for GRPO, GSPO, and DAPO), rather than a new objective function that replaces existing ones. Second, the paper's theoretical contribution (Theorems 1-3) provides a unifying explanation for why certain training phenomena occur — such as insufficient exploration on hard problems or over-optimization on easy ones — that may have been observed empirically but not understood mechanistically.

The paper also explicitly acknowledges its boundaries. It does not claim that HA-DW will help on all prompts — the hardest prompts where the base model's expected reward is near zero will not benefit because there are essentially no correct responses to amplify (the model has nothing to learn from). This is consistent with the broader RLVR literature showing that reinforcement learning can refine existing capabilities but cannot create them from scratch. The paper's contribution is therefore in making the learning signal more faithful to the true advantages on the prompts where learning is actually possible, which represents the majority of training data in practice.

Why This Matters for the Field

The significance of this work extends beyond the specific algorithm it proposes. By providing the first rigorous characterization of bias in group-relative advantage estimation, the paper establishes a new axis for evaluating and improving RLVR algorithms that is orthogonal to the objective function design space explored by prior work. Future GRPO variants should not only consider how to process the advantage estimate (clipping, normalization, token-level vs. sequence-level), but also whether the estimate itself is reliable for the specific prompt difficulty at the current training stage.

Moreover, the paper's finding that the bias is deterministic at extremes — $\hat{A}_{t,i} < A_{t,i}$ surely when $p_t < 1/G$ , and $\hat{A}_{t,i} > A_{t,i}$ surely when $p_t > (G-1)/G$ (Corollary 3) — means that for the most difficult and easiest prompts in a training batch, the learning signal is guaranteed to be wrong. With typical group sizes of $G=8$ , any prompt with expected success below 12.5% or above 87.5% will receive a directionally biased advantage estimate with probability 1. This is not a rare edge case — it covers a substantial fraction of prompts in a typical RLVR training run, especially early in training when the model struggles with most problems.

3. Technical Approach

3.1 Reader Orientation (Approachable Technical Breakdown)

This is primarily a theoretical analysis and algorithm correction paper. The system being built is a lightweight, plug-and-play reweighting module that corrects a previously unnoticed statistical bias in how GRPO-style reinforcement learning algorithms estimate the "advantage" of each generated response during LLM training. The problem it solves is that group-relative advantage estimators systematically give too little credit to good answers on hard problems and too much credit to good answers on easy problems — and the solution has the shape of an adaptive multiplier that amplifies advantages for hard prompts and suppresses them for easy prompts, using a running estimate of the model's overall capability as the reference point for what counts as "hard."

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components that operate inside the existing GRPO training loop:

Base Group-Relative RL Algorithm (GRPO/GSPO/DAPO) — the existing training procedure that samples $G$ responses per prompt, computes group-relative advantages $\hat{A}_{t,i} = r_{t,i} - \hat{p}_t$ , and updates the policy. HA-DW wraps around this without changing its core mechanics.
Evolving Difficulty Anchor ( $C_t$ ) — a Kalman-filter-style running estimate of the model's current success rate that aggregates reward observations across training batches. It serves as a dynamic reference point for what "average difficulty" means as the model improves.
Difficulty-Aware Reweighting ( $\Phi_{t,i}$ ) — an exponential multiplicative factor computed per response that compares the prompt's group success rate $\hat{p}_t$ against the difficulty anchor $C_t^-$ , then amplifies or suppresses the advantage estimate accordingly.
Modified Policy Objective ( $\mathcal{L}_{\text{HA-DW}}$ ) — the original group-relative objective with each advantage term multiplied by $\Phi_{t,i}$ , yielding corrected gradient signals that compensate for the systematic bias.

Information flows as follows: each training batch produces reward observations $y_t$ (the fraction of correct responses in the batch) → the difficulty anchor updates from its prior $C_t^-$ to posterior $C_t^+$ using a Kalman-style update with adaptive forgetting factor $\eta_t$ → the updated anchor becomes the prior for the next batch → within each batch, the prompt-specific reweighting factor $\Phi_{t,i}$ is computed by comparing $\hat{p}_t$ to $C_t^-$ via an exponential function → this factor multiplies each advantage estimate in the policy gradient objective → the result is that hard-prompt advantages are boosted (compensating for systematic underestimation) and easy-prompt advantages are dampened (compensating for overestimation).

3.3 Roadmap for the Deep Dive

First, the theoretical foundation (Theorems 1-2 and Corollaries 1-3): we must understand exactly what the bias is, where it comes from, and what functional form it takes, because every design choice in HA-DW is motivated by this characterization.
Second, the evolving difficulty anchor (Section 3.1): the Kalman-style update mechanism that provides the reference point for reweighting — this is the cross-batch "memory" that makes dynamic difficulty assessment possible.
Third, the HA-DW reweighting factor (Section 3.2): how the anchor is combined with group-level statistics to produce the multiplicative correction $\Phi_{t,i}$ , including the sign logic and exponential form.
Fourth, the integration with existing GRPO variants (Appendix B): the mechanical details of how HA-DW plugs into GRPO, GSPO, and DAPO objectives — since the modification is identical in structure across all three, we can treat this compactly.
Fifth, the theoretical guarantee (Theorem 3 and Lemma 1): the proof that an appropriate choice of scaling parameter $\lambda_{\text{scale}}$ provably reduces bias, connecting the practical design to the theoretical analysis.

3.4 Detailed, Sentence-Based Technical Breakdown

This is an algorithm correction paper whose core idea is that group-relative advantage estimators in GRPO-family algorithms are systematically biased in a difficulty-dependent way — and that a lightweight, adaptive reweighting scheme using cross-batch history can provably reduce this bias and improve empirical performance, without modifying the underlying RL algorithm's clipping, normalization, or importance sampling mechanics.

The Bias: What It Is and Where It Comes From

The paper's entire technical contribution rests on the characterization of a specific estimation bias, so we must understand it precisely before examining the correction.

Setup assumptions. At training step $t$ , a prompt $x_t$ is sampled from the data distribution $\mathcal{D}$ . The current policy $\pi_{\theta_t}$ generates $G$ independent responses $\{y_{t,i}\}_{i=1}^G$ , each receiving a binary reward $r_{t,i} \in \{0, 1\}$ from a verifier. The expected reward of the policy on this prompt is:

$p_t = \mathbb{E}_{y \sim \pi_{\theta_t}(\cdot|x_t)} [r(y)] = P(r(y) = 1 | x_t, \pi_{\theta_t})$

where $p_t$ is the unknown true success probability of the policy on prompt $x_t$ , and $r(y)$ is the binary reward function.

What it computes: the long-run fraction of times the current policy would answer prompt $x_t$ correctly if we sampled infinitely many responses. This is the ground-truth difficulty parameter that the group-relative estimator is trying to approximate.

Why this form: binary rewards (pass/fail from a verifier) are the standard setting in mathematical reasoning RLVR because correctness is objective — the model either produces the right final answer or it doesn't. The Bernoulli model $r_{t,i} \sim \text{Bernoulli}(p_t)$ follows directly from this binary-outcome assumption and is the natural statistical model for repeated independent trials with fixed success probability.

The group-relative advantage estimator. For each response $i$ in the group, the empirical group-relative advantage is:

$\hat{A}_{t,i} = r_{t,i} - \hat{p}_t, \quad \hat{p}_t = \frac{1}{G} \sum_{j=1}^G r_{t,j}$

where $\hat{p}_t$ is the empirical group baseline (the fraction of correct responses in the group of size $G$ ) and $r_{t,i}$ is the binary reward for response $i$ .

What it computes: an estimate of how much better or worse response $i$ is compared to the average response in its group. If $r_{t,i} = 1$ and most other responses are 0, the advantage is positive and large; if $r_{t,i} = 0$ and most others are 1, the advantage is negative. The estimate uses only within-group information — no external critic or value function.

Why this form: group-relative methods avoid training a separate value network (as PPO requires) by using the group mean as a baseline. This is computationally simpler and has become the dominant paradigm after DeepSeek-R1. The implicit assumption is that the group average $\hat{p}_t$ is a reasonable proxy for the true expected reward $p_t$ — which the paper demonstrates is false in a systematic way.

The true (expected) advantage. The paper defines the expected advantage as:

$A_{t,i} = r_{t,i} - p_t$

where $p_t$ is the true expected reward defined above, and $r_{t,i}$ is the observed reward for that specific response.

What it computes: the advantage that would be computed if we knew the policy's true success probability on this prompt — that is, how much better or worse this response is than what the policy typically produces. This is the "correct" advantage that the empirical estimator $\hat{A}_{t,i}$ is trying to approximate.

Why define it this way: the distinction between $\hat{A}_{t,i}$ (using the empirical group mean $\hat{p}_t$ ) and $A_{t,i}$ (using the true expected reward $p_t$ ) is the entire point of the paper. The bias is the systematic difference between these two quantities, and characterizing when and why they diverge is the paper's core contribution.

The non-degenerate event. A crucial detail: GRPO-style algorithms typically discard or implicitly ignore groups where all responses receive the same reward (all correct or all incorrect), because then all advantages $\hat{A}_{t,i}$ are zero and produce no gradient. The paper formalizes this conditioning:

$\mathcal{S} = \{1 \leq R \leq G-1\}$

where $R = \sum_{i=1}^G r_{t,i}$ is the total number of correct responses in the group, and $\mathcal{S}$ is the non-degenerate event where at least one response is correct and at least one is incorrect.

What it defines: the subset of sampled groups that actually contribute to learning. Groups where $R=0$ or $R=G$ are "degenerate" — they produce zero gradients and are effectively filtered out of the training process.

Why this matters: conditioning on $\mathcal{S}$ changes the sampling distribution. Under $\mathcal{S}$ , the empirical group mean $\hat{p}_t = R/G$ is no longer an unbiased estimator of $p_t$ , because the cases where the sample mean would have been 0 or 1 are excluded. This is the root cause of the bias — it's not that the estimator itself is wrong, but that the effective estimator (conditioned on non-degenerate groups) has a shifted expectation. This is a subtle point: the algorithm doesn't explicitly condition on $\mathcal{S}$ ; rather, the gradient structure of the objective means groups with uniform rewards contribute nothing, so the effective update distribution is conditional on $\mathcal{S}$ .

Theorem 1 (expectation-level bias). The central theoretical result characterizes the conditional expectation:

$\mathbb{E}[\hat{A}_{t,i} | \mathcal{S}] \begin{cases} < A_{t,i}, & \text{if } p_t < 0.5 \\ > A_{t,i}, & \text{if } p_t > 0.5 \\ = A_{t,i}, & \text{if and only if } p_t = 0.5 \end{cases}$

What it states: the group-relative advantage estimator is biased downward (underestimates the true advantage) for prompts where the model succeeds less than half the time — i.e., hard prompts — and biased upward (overestimates the true advantage) for prompts where the model succeeds more than half the time — i.e., easy prompts. Unbiasedness holds only at the knife-edge case $p_t = 0.5$ .

Why this result is non-obvious: the bias direction flips with difficulty and vanishes at exactly $p_t = 0.5$ . This means the estimator is not uniformly conservative or uniformly optimistic — it distorts the signal in opposite directions depending on the prompt. Consequently, hard prompts (where the model most needs a strong learning signal) receive a systematically weakened advantage, while easy prompts (where learning is less critical) receive an inflated one. The proof (Appendix D.1) derives the conditional expectation of $\hat{p}_t$ under $\mathcal{S}$ :

$\mathbb{E}[\hat{p}_t | \mathcal{S}] = \frac{p_t - p_t^G}{1 - (1-p_t)^G - p_t^G}$

and shows that this quantity is less than $p_t$ when $p_t < 0.5$ , greater when $p_t > 0.5$ , and equal at 0.5. The intuition: when $p_t$ is small (hard prompt), most groups will have $R = 0$ and be discarded; the retained groups ( $R \geq 1$ ) are disproportionately those where the model got "lucky," so the conditional mean of $\hat{p}_t$ is pulled upward toward the few observed successes, making $\mathbb{E}[\hat{p}_t|\mathcal{S}] > p_t$ , which implies $\mathbb{E}[\hat{A}_{t,i}|\mathcal{S}] = r_{t,i} - \mathbb{E}[\hat{p}_t|\mathcal{S}] < r_{t,i} - p_t = A_{t,i}$ . The opposite logic applies for easy prompts.

Theorem 2 (distribution-level characterization). Theorem 1 only covers expectations; Theorem 2 quantifies the probability of overestimation or underestimation by a specified margin. For hard prompts ( $p_t < 0.5$ ) and any $\epsilon \in (0, \mathbb{E}[\hat{p}_t|\mathcal{S}] - p_t)$ :

$P(\hat{A}_{t,i} < A_{t,i} - \epsilon | \mathcal{S}) = \frac{\sum_{k=\lfloor G(p_t + \epsilon)\rfloor + 1}^{G-1} \binom{G}{k} p_t^k (1-p_t)^{G-k}}{1 - (1-p_t)^G - p_t^G}$

What it computes: the exact probability that the group-relative advantage underestimates the true advantage by at least $\epsilon$ , conditional on the group being non-degenerate. It is a ratio: the numerator is the probability mass of binomial outcomes where the group mean $\hat{p}_t$ exceeds $p_t$ by more than $\epsilon$ (causing underestimation of advantage), and the denominator is the probability of the conditioning event $\mathcal{S}$ .

Why this form: the difference $A_{t,i} - \hat{A}_{t,i} = \hat{p}_t - p_t$ , so the event $\hat{A}_{t,i} < A_{t,i} - \epsilon$ is equivalent to $\hat{p}_t > p_t + \epsilon$ . The sum over $k$ from $\lfloor G(p_t + \epsilon)\rfloor + 1$ to $G-1$ counts all group compositions where the number of correct responses exceeds the threshold $G(p_t + \epsilon)$ , weighted by their binomial probabilities. The denominator excludes the uniform-outcome cases $k=0$ and $k=G$ . The analogous expression for easy prompts involves summing $k$ from 1 to $\lceil G(p_t - \epsilon)\rceil - 1$ .

Corollary 1 (small-G practical bounds). Since real-world RLVR uses small groups (typically $G = 8$ ), the paper specializes Theorem 2 to $G \in [2, 8]$ under the assumption that $p_t$ is uniformly distributed over $[0, 1]$ , yielding numerical bounds:

$P(\hat{A}_{t,i} < A_{t,i} | \mathcal{S}, p_t < 0.5) > 0.63$ $P(\hat{A}_{t,i} > A_{t,i} | \mathcal{S}, p_t > 0.5) > 0.63$

For more extreme difficulties, the probabilities increase: > 0.78 for $p_t < 0.25$ or $p_t > 0.75$ , and deterministically 1.00 for $p_t < 0.125$ or $p_t > 0.875$ .

What it states: with at least 63% probability (and typically much higher), the advantage estimate is directionally biased for any prompt that is not at exactly 50% difficulty. For the hardest quarter of prompts, the probability exceeds 78%.

Why these numbers matter: they quantify how pervasive the bias is under realistic training conditions. With $G=8$ , if 30% of your training prompts are hard ( $p_t < 0.25$ ), then for roughly 78% of those prompts, the advantage of correct responses is being underestimated — meaning the policy is receiving a weaker-than-deserved reinforcement signal on the very problems where it most needs to improve.

Corollary 3 (deterministic bias at extremes). For any $G \geq 2$ :

$\hat{A}_{t,i} < A_{t,i} \text{ surely if } p_t < \frac{1}{G}$ $\hat{A}_{t,i} > A_{t,i} \text{ surely if } p_t > \frac{G-1}{G}$

What it states: when the true success probability is below $1/G$ , the group-relative advantage necessarily underestimates the true advantage (with probability 1, conditional on $\mathcal{S}$ ). Conversely, when it's above $(G-1)/G$ , it necessarily overestimates. With $G=8$ , the deterministic bias thresholds are $p_t < 0.125$ and $p_t > 0.875$ .

Why this is striking: the proof is simple — on $\mathcal{S}$ , $R \geq 1$ , so $\hat{p}_t = R/G \geq 1/G > p_t$ , which means $\hat{p}_t > p_t$ surely, implying $\hat{A}_{t,i} < A_{t,i}$ for all responses. There's no probability involved; the conditioning event guarantees the bias direction. This means that for the most extreme prompts in your training set, the learning signal is not just probably wrong — it is certainly wrong.

Lemma 2 (bias magnitude formula). The exact magnitude of the bias is:

$A_{t,i} - \mathbb{E}[\hat{A}_{t,i} | \mathcal{S}] = \frac{p_t(1-p_t)^G + p_t^{G+1} - p_t^G}{1 - (1-p_t)^G - p_t^G}$

What it computes: the expected difference between the estimated and true advantage, as a function of the true success probability $p_t$ and group size $G$ .

Why this matters for algorithm design: the bias is largest when $p_t$ is far from 0.5 and $G$ is small (as visualized in Figure 2). This directly motivates why HA-DW needs to apply larger corrections for prompts with extreme $\hat{p}_t$ values — the bias magnitude grows with the deviation from $p_t = 0.5$ . The formula also shows that increasing $G$ reduces bias (the denominator grows faster than the numerator), which is why simply using more rollouts would help — but the paper's goal is to correct the bias under the practical constraint of small $G$ .

Difficulty definition. The paper operationalizes "hard" and "easy" through Definition 3:

$x_t$ is hard if $p_t < 0.5 - \Delta$
$x_t$ is moderate if $0.5 - \Delta \leq p_t \leq 0.5 + \Delta$
$x_t$ is easy if $p_t > 0.5 + \Delta$

where $\Delta \in [0, 1)$ is a user-defined threshold. In practice, the true $p_t$ is unknown, so difficulty must be estimated — which motivates the evolving difficulty anchor.

The Evolving Difficulty Anchor ( $C_t$ )

The HA-DW method requires a reference point for what "average difficulty" means at the current stage of training, because difficulty is relative to the model's current capability. The paper proposes a Kalman-filter-style belief state that tracks the model's overall success rate across batches.

The observation signal. For each training batch $t$ with $B_t$ total responses, the batch-level observation is:

$y_t = \frac{K_t}{B_t}, \quad K_t = \sum_{i=1}^{B_t} r_{t,i}$

where $K_t$ is the total number of correct responses in the batch, and $B_t$ is the total number of responses (across all prompts in the batch). So $y_t$ is the overall accuracy fraction for batch $t$ .

What it computes: a single scalar per batch representing what fraction of all responses generated in that batch were correct. This aggregates across multiple prompts, so it captures the model's global performance level rather than prompt-specific difficulty.

Why aggregate across prompts: the difficulty anchor is meant to represent the model's capability, not the difficulty of any specific prompt. By averaging over many prompts in a batch, $y_t$ provides a noisy but unbiased estimate of the model's overall expected accuracy. Individual prompts may deviate above or below this average — and those deviations are exactly what the reweighting scheme uses to identify hard and easy prompts.

Kalman-style belief update. The model's solving capability is treated as a latent belief state $C_t$ that evolves over training. At each step, the prior belief $C_t^-$ (carried forward from the previous batch) is updated to a posterior belief $C_t^+$ using the current observation $y_t$ :

$C_t^+ = (1 - \eta_t) C_t^- + \eta_t y_t, \quad \eta_t \in [0, 1]$

where $\eta_t$ is an adaptive forgetting factor that controls how much weight is given to the new observation versus historical information.

What it computes: an exponentially weighted moving average of batch-level accuracy, where the smoothing parameter $\eta_t$ adapts to training stability. When $\eta_t$ is close to 1, the anchor responds quickly to new observations (high plasticity); when $\eta_t$ is close to 0, the anchor retains historical information (high stability).

Why a Kalman-style update rather than a simple moving average: the adaptive forgetting factor $\eta_t$ is the key innovation. During early training, the model's capability is changing rapidly (the policy is improving), so a high $\eta_t$ ensures the anchor tracks these changes. During later training, the model plateaus, and a low $\eta_t$ reduces noise by averaging over more history. A fixed- $\eta$ moving average would either lag behind rapid early improvements or be overly noisy during stable late training.

State propagation. Between consecutive steps, the posterior becomes the next prior:

$C_t^+ \rightarrow C_{t+1}^-$

This creates a recurrent state that accumulates information across the entire training history, with exponential decay controlled by the sequence of $\eta_t$ values.

Adaptive forgetting factor. The forgetting factor $\eta_t$ is not fixed but is dynamically modulated by the stability of recent accuracy:

$\bar{C}_t = \frac{1}{m} \sum_{j=1}^m C_{t-j}$

$\sigma_t = \sqrt{\frac{1}{m} \sum_{j=1}^m (C_{t-j} - \bar{C}_t)^2}$

$\eta_t = \eta \cdot \sigma_t$

where $m$ is a window size (number of recent batches to consider), $\bar{C}_t$ is the mean belief over that window, $\sigma_t$ is the standard deviation of recent beliefs, and $\eta$ is a task-dependent base hyperparameter.

What it computes: $\eta_t$ is proportional to the recent volatility of the capability estimate. When $\sigma_t$ is large (the model's performance is changing rapidly — typical of early training or distribution shifts), $\eta_t$ is large, and the anchor adapts quickly. When $\sigma_t$ is small (performance is stable — typical of convergence), $\eta_t$ is small, and the anchor is smooth.

Why modulate by standard deviation: this is an implicit change-point detection mechanism. A sudden drop or spike in accuracy increases $\sigma_t$ , which increases $\eta_t$ , which allows the anchor to quickly catch up to the new regime. Once performance stabilizes at the new level, $\sigma_t$ decreases, $\eta_t$ decreases, and the anchor becomes more stable again. This prevents the anchor from oscillating with batch-to-batch noise while still responding to genuine capability shifts.

Hard update variant (Appendix F). For simplicity, the paper also proposes a hard-update variant where the belief is computed as a direct average over the last $h$ batches:

$C_t^+ = \frac{1}{h} \left( \sum_{j=1}^{h-1} y_{t-j} + y_t \right)$

What it computes: the simple arithmetic mean of batch accuracies over a fixed-length window of the most recent $h$ training rounds. No adaptive forgetting factor is used.

Why provide this variant: it eliminates the need to track and update $\sigma_t$ and $\eta_t$ , reducing algorithmic complexity. The tradeoff is that it "ignores short-term oscillations in belief updates" (Appendix F) — it cannot distinguish between genuine capability shifts and batch-to-batch noise. The adaptive version ( $\eta_t = \eta \cdot \sigma_t$ ) is the primary method; the hard variant is offered as a computationally simpler alternative for practitioners who find the adaptive mechanism too complex or who observe that their training dynamics are stable enough that a fixed window suffices.

History-Aware Adaptive Difficulty Weighting (HA-DW)

The second key component — and the namesake of the method — is the reweighting factor that adjusts advantage estimates using the difficulty anchor.

History-based prompt difficulty. The relative difficulty of a prompt is defined as:

$\text{diff}^{\text{his}}_t = \hat{p}_t - C_t^-$

where $\hat{p}_t = R/G$ is the group baseline (the fraction of correct responses in this prompt's group) and $C_t^-$ is the prior difficulty anchor (the model's estimated global accuracy before seeing batch $t$ ).

What it computes: the deviation of this specific prompt's success rate from the model's overall expected performance. A positive $\text{diff}^{\text{his}}_t$ means this prompt is easier than the model's current average capability (the group has more successes than expected); a negative value means it's harder (fewer successes than expected).

Why use $C_t^-$ (the prior) rather than $C_t^+$ (the posterior): the prior represents the model's belief before incorporating the current batch's information. Using the prior ensures that the difficulty assessment for each prompt is relative to the model's pre-existing capability estimate, preventing the current batch's observations from circularly influencing the difficulty assessment. If we used $C_t^+$ , a batch with unusually many hard prompts would temporarily lower the anchor, making those prompts appear less hard — a form of reference frame contamination.

Direction of adjustment. The sign logic determines whether to amplify or suppress the advantage:

$D_{t,i} = -\text{sgn}(\hat{A}_{t,i}) \cdot \text{sgn}(\text{diff}^{\text{his}}_t)$

where $\text{sgn}(\cdot)$ returns $+1$ for positive arguments, $-1$ for negative arguments, and (implicitly) $0$ at zero.

What it computes: the direction of the reweighting adjustment. Let's unpack the four cases:

$\text{sgn}(\hat{A}_{t,i})$	$\text{sgn}(\text{diff}^{\text{his}}_t)$	$D_{t,i}$	Meaning
$+1$ (correct on hard prompt)	$-1$ (hard)	$+1$	Amplify: correct on hard prompt is undervalued
$+1$ (correct on easy prompt)	$+1$ (easy)	$-1$	Suppress: correct on easy prompt is overvalued
$-1$ (incorrect on hard prompt)	$-1$ (hard)	$-1$	Amplify penalty: incorrect on easy to get right?
$-1$ (incorrect on easy prompt)	$+1$ (easy)	$+1$	Suppress penalty: mistakes on easy prompts less informative

Why this sign logic: it implements the bias correction direction identified in Theorem 1. For hard prompts ( $\text{diff}^{\text{his}}_t < 0$ ), correct responses ( $\hat{A}_{t,i} > 0$ ) need their advantage amplified because the group-relative estimator underestimates them. For easy prompts ( $\text{diff}^{\text{his}}_t > 0$ ), correct responses need their advantage suppressed because the estimator overestimates them. The $D_{t,i}$ term captures this as a single scalar that feeds into the exponential reweighting: positive $D_{t,i}$ means amplify, negative means suppress.

The paper's text is somewhat telegraphic on the sign logic — it states $D_{t,i} = -\text{sgn}(\hat{A}_{t,i}) \cdot \text{sgn}(\text{diff}^{\text{his}}_t)$ without exhaustively enumerating cases — but the operational meaning is clear from the stated intentions: "amplifies the estimated advantage for difficult prompts" and "suppresses it for easy prompts."

Magnitude of adjustment. The absolute deviation from the anchor determines how strong the correction should be:

$M_t = |\text{diff}^{\text{his}}_t| = |\hat{p}_t - C_t^-|$

What it computes: the absolute distance between the prompt's group success rate and the model's current capability estimate. A large $M_t$ means the prompt is very far from average difficulty — either very hard or very easy — and thus subject to stronger estimation bias.

Why absolute deviation: Lemma 2 and Figure 2 show that the bias magnitude increases as $p_t$ moves away from 0.5. $M_t$ is the observable proxy for this deviation. Prompts with $M_t$ close to zero (near the model's average capability) should receive minimal adjustment because the bias is small near $p_t \approx C_t^- \approx 0.5$ (assuming the anchor converges near 0.5). Prompts with large $M_t$ (far from average) need stronger correction.

History-aware reweighting factor. The direction and magnitude are combined into a single multiplicative factor:

$\Phi_{t,i} = \lambda_{\text{scale}} \cdot \exp(D_{t,i} \cdot M_t)$

where $\lambda_{\text{scale}}$ is a global scaling constant, $D_{t,i}$ is the signed direction ( $+1$ for amplify, $-1$ for suppress), and $M_t$ is the absolute difficulty deviation.

What it computes: an exponential multiplicative weight applied to the advantage estimate. When $D_{t,i} = +1$ and $M_t$ is large (hard prompt, correct response), $\exp(+M_t) > 1$ , so the advantage is amplified. When $D_{t,i} = -1$ and $M_t$ is large (easy prompt, correct response), $\exp(-M_t) < 1$ , so the advantage is suppressed. The $\lambda_{\text{scale}}$ factor provides a global scaling to keep adjustments in a reasonable range.

Why exponential form: an exponential has several desirable properties. (1) It is always positive, so the sign of the advantage estimate is preserved — the reweighting only changes magnitude, not direction. (2) It is multiplicative, so the adjustment scales proportionally with the original advantage rather than adding a constant offset, which would interact poorly with clipping and normalization in the downstream policy objective. (3) It is smooth and differentiable with respect to $M_t$ , enabling stable training dynamics. (4) It naturally produces large adjustments for large deviations while being near-identity ( $\exp(0) \approx 1$ ) for prompts near the anchor. Alternatives like linear scaling ( $1 + \alpha M_t$ ) could become negative for sufficiently negative $D_{t,i} \cdot M_t$ , which would flip the sign of the advantage — a clearly undesirable property.

The $\lambda_{\text{scale}}$ hyperparameter: this global scaling constant controls the overall aggressiveness of the correction. Theorem 3 provides theoretical guidance on its valid range (discussed below), and the paper's ablation (Appendix E.3, Table 7) empirically tests values $\{0.5, 0.8, 1.0, 1.3, 1.5, 1.7, 2.0\}$ , finding optimal performance at $\lambda_{\text{scale}} = 1.3$ or $1.5$ for Qwen3-4B-Base.

Integration with Group-Relative RL Objectives

HA-DW is designed as a plug-and-play module that wraps around existing group-relative objectives by multiplying each advantage term by $\Phi_{t,i}$ . The general modified objective is:

$\mathcal{L}_{\text{HA-DW}}(\theta) = \frac{1}{G} \sum_{i=1}^G \psi\left(\frac{\pi_\theta(y_{t,i}|x_t)}{\pi_{\theta_{\text{old}}}(y_{t,i}|x_t)}\right) \cdot \phi(\hat{A}_{t,i}) \cdot \Phi_{t,i}$

where $\psi(\cdot)$ is a function applied to the importance sampling ratio (identity, clipping, or logarithmic), and $\phi(\cdot)$ is a function applied to the advantage term (identity or clipped). The specific instantiations are:

GRPO + HA-DW (Equation 25). GRPO applies token-level clipping with group-normalized advantages. The HA-DW-modified objective is:

$\mathcal{J}_{\text{GRPO+HA-DW}}(\theta) = \frac{1}{G} \sum_{i=1}^G \frac{1}{|o_{t,i}|} \sum_{\tau=1}^{|o_{t,i}|} \min\left(r_{t,i,\tau}(\theta) \hat{A}_{t,i,\tau} \cdot \Phi_{t,i}, \text{clip}(r_{t,i,\tau}(\theta), 1-\epsilon, 1+\epsilon) \hat{A}_{t,i,\tau} \cdot \Phi_{t,i}\right)$

where $r_{t,i,\tau}(\theta)$ is the token-level importance sampling ratio, $\hat{A}_{t,i,\tau}$ is the group-normalized advantage assigned uniformly to all tokens in the response, and $\Phi_{t,i}$ is the prompt-level (not token-level) reweighting factor.

What it does: the reweighting factor $\Phi_{t,i}$ multiplies both the unclipped and clipped advantage terms, so the clipping ratio operates on the already-reweighted advantage. This means that for hard-prompt correct responses ( $\Phi_{t,i} > 1$ ), the effective learning signal is amplified before clipping constraints are applied; for easy-prompt correct responses ( $\Phi_{t,i} < 1$ ), it's dampened.

Why per-response, not per-token: $\Phi_{t,i}$ depends on the prompt's group success rate $\hat{p}_t$ , which is a property of the whole group, not individual tokens. Applying $\Phi_{t,i}$ uniformly to all tokens in a response preserves the token-level granularity of the original GRPO objective while correcting the prompt-level bias.

GSPO + HA-DW (Equation 29). GSPO uses sequence-level (not token-level) importance ratios. The modified objective is:

$\mathcal{J}_{\text{GSPO+HA-DW}}(\theta) = \frac{1}{G} \sum_{i=1}^G \min\left(r_{t,i}(\theta) \hat{A}_{t,i} \cdot \Phi_{t,i}, \text{clip}(r_{t,i}(\theta), 1-\epsilon, 1+\epsilon) \hat{A}_{t,i} \cdot \Phi_{t,i}\right)$

where $r_{t,i}(\theta) = \frac{\pi_\theta(y_{t,i}|x_t)}{\pi_{\theta_{\text{old}}}(y_{t,i}|x_t)}$ is the sequence-level probability ratio.

DAPO + HA-DW (Equation 31). DAPO uses token-level processing with decoupled clipping ( $\epsilon$ for lower bound, $\epsilon'$ for upper bound) and length normalization. The modified objective is:

$\mathcal{J}_{\text{DAPO+HA-DW}}(\theta) = \frac{1}{\sum_{i=1}^G |o_{t,i}|} \sum_{i=1}^G \sum_{\tau=1}^{|o_{t,i}|} \min\left(r_{t,i,\tau}(\theta) \hat{A}_{t,i,\tau} \cdot \Phi_{t,i}, \text{clip}(r_{t,i,\tau}(\theta), 1-\epsilon, 1+\epsilon') \hat{A}_{t,i,\tau} \cdot \Phi_{t,i}\right)$

Key observation: in all three instantiations, $\Phi_{t,i}$ multiplies the advantage term before any clipping or min operations. This is crucial — if $\Phi_{t,i}$ were applied after clipping (outside the min), the correction would be partially nullified by the clipping bounds. By placing it inside both branches of the min, the reweighting affects the effective gradient regardless of whether the clipped or unclipped term is selected.

Theoretical Guarantee: Lemma 1 and Theorem 3

The paper provides theoretical justification that HA-DW with an appropriate $\lambda_{\text{scale}}$ provably reduces the bias.

Lemma 1 (baseline rectification). Consider adjusting the empirical baseline with a factor $c$ : $\tilde{p}_t = c \cdot \hat{p}_t$ . Under the assumption $p_t \in [\Delta, 1-\Delta]$ for some $\Delta \in (0, 1/2]$ (i.e., $p_t$ is bounded away from 0 and 1), and given a confidence $\delta \in (0, 1)$ , define:

$\epsilon_\delta = \sqrt{\frac{1}{2G} \log\left(\frac{2}{\delta(1 - (1-\Delta)^G - \Delta^G)}\right)}$

Let $\mathcal{I}_t = [\hat{p}_t - \epsilon_\delta, \hat{p}_t + \epsilon_\delta] \cap [\Delta, 1-\Delta]$ and $A(p) = 1 - (1-p)^G - p^G$ . Define:

$c_{\text{low}} = \sup_{p \in \mathcal{I}_t} \frac{(p - \epsilon) A(p)}{p(1 - p^{G-1})}, \quad c_{\text{high}} = \inf_{p \in \mathcal{I}_t} \frac{(p + \epsilon) A(p)}{p(1 - p^{G-1})}$

Then with probability at least $1-\delta$ conditional on $\mathcal{S}$ , for any $c \in (c_{\text{low}}, c_{\text{high}})$ :

$\mathbb{E}[\tilde{p}_t | \mathcal{S}] \in (p_t - \epsilon, p_t + \epsilon)$

What it states: there exists a range of multiplicative correction factors $c$ such that the adjusted baseline $\tilde{p}_t = c \cdot \hat{p}_t$ has conditional expectation within $\pm\epsilon$ of the true $p_t$ — i.e., the correction removes the bias to within a tolerance $\epsilon$ . The range $(c_{\text{low}}, c_{\text{high}})$ is computable from observable quantities ( $\hat{p}_t$ , $G$ , $\epsilon_\delta$ ) and known parameters ( $\Delta$ , $\delta$ ).

Why this lemma is needed: it bridges the gap between the theoretical characterization of bias (Theorems 1-2) and the practical reweighting scheme (HA-DW). It shows that a multiplicative correction to the baseline can, in principle, achieve near-unbiased estimation — providing a theoretical foundation for the exponential reweighting in HA-DW.

Lemma 3 and Lemma 6 in the proof (Appendix D.4): The proof first derives the exact correction factor that would make the adjusted baseline unbiased:

$c = \frac{1 - (1-p_t)^G - p_t^G}{1 - p_t^{G-1}}$

which depends on the unknown $p_t$ . It then shows that any $c$ in the interval specified above achieves the $\pm\epsilon$ bound. Lemma 4 and Lemma 5 provide concentration inequalities that allow replacing the unknown $p_t$ with the observable $\hat{p}_t$ in the definition of the valid interval, making the result practically applicable.

Theorem 3 (HA-DW provably reduces bias). Suppose there exists a scaling factor $\lambda_{\text{scale}}$ in Equation (16) such that:

$\lambda_{\text{scale}} \in \left(\frac{1 + \frac{(1-c_{\text{high}})\hat{p}_t}{1-\hat{p}_t}}{\exp(D_{t,i} M_t)}, \frac{1 + \frac{(1-c_{\text{low}})\hat{p}_t}{1-\hat{p}_t}}{\exp(D_{t,i} M_t)}\right) \cup \left(\frac{c_{\text{low}}}{\exp(D_{t,i} M_t)}, \frac{c_{\text{high}}}{\exp(D_{t,i} M_t)}\right)$

Then:

$|\mathbb{E}[\hat{A}_{t,i} \cdot \Phi_{t,i} | \mathcal{S}] - A_{t,i}| < |\mathbb{E}[\hat{A}_{t,i} | \mathcal{S}] - A_{t,i}|$

What it states: when $\lambda_{\text{scale}}$ falls within a specific union of intervals (which depend on $c_{\text{low}}$ , $c_{\text{high}}$ , $\hat{p}_t$ , $D_{t,i}$ , and $M_t$ ), the HA-DW-adjusted advantage estimate $\hat{A}_{t,i} \cdot \Phi_{t,i}$ is closer in expectation to the true advantage $A_{t,i}$ than the unadjusted estimate $\hat{A}_{t,i}$ . In other words, HA-DW provably reduces the bias.

Why the union of two intervals: the first interval (with the $(1-c)\hat{p}_t/(1-\hat{p}_t)$ terms) covers the case where $r_{t,i} = 1$ (correct responses), derived from the relation $\Phi_{t,i} \hat{A}_{t,i} = r_{t,i} - c\hat{p}_t$ and solving for $\Phi_{t,i}$ when $r_{t,i} = 1$ : $\Phi_{t,i} = \frac{1 - c\hat{p}_t}{1 - \hat{p}_t}$ . The second interval (with $c_{\text{low}}/\exp(D_{t,i}M_t)$ and $c_{\text{high}}/\exp(D_{t,i}M_t)$ ) covers the case where $r_{t,i} = 0$ (incorrect responses), where $\Phi_{t,i} = c$ . The proof in Appendix D.4.2 explicitly separates these four cases (hard prompt correct, hard prompt incorrect, easy prompt correct, easy prompt incorrect) and shows they all collapse to the union stated in the theorem.

Practical interpretation: Theorem 3 does not give a single optimal $\lambda_{\text{scale}}$ — it provides a feasibility result showing that some $\lambda_{\text{scale}}$ exists that reduces bias. In practice, $\lambda_{\text{scale}}$ is treated as a hyperparameter and tuned via ablation (the paper finds $\lambda_{\text{scale}} = 1.3$ or $1.5$ works best). The theorem's value is in establishing that the exponential reweighting form is not arbitrary — it can, in principle, achieve bias reduction under appropriate parameter settings.

Summary of Key Design Choices and Their Justifications

Kalman-style evolving anchor over fixed threshold: difficulty is relative to the model's current capability, which changes during training. A fixed threshold (e.g., $\hat{p}_t < 0.3$ = hard) would misclassify prompts as the model improves. The anchor tracks capability online.
Adaptive forgetting factor $\eta_t = \eta \cdot \sigma_t$ over constant $\eta$ : early training has rapid capability changes requiring fast adaptation; late training is stable and benefits from averaging over more history to reduce noise. The standard-deviation-based modulation automatically detects these regimes.
Exponential reweighting $\exp(D_{t,i} \cdot M_t)$ over additive or linear correction: preserves the sign of the advantage, is always positive, scales proportionally with the original advantage magnitude, and naturally produces large corrections for large deviations while being near-identity for small deviations. Linear corrections ( $1 + \alpha D_{t,i} M_t$ ) would risk negative weights or sign flips.
Per-prompt (not per-token) reweighting: the bias identified in Theorems 1-2 is a prompt-level phenomenon — it depends on $p_t$ , the prompt's expected success rate. Applying $\Phi_{t,i}$ uniformly to all tokens in a response respects this structure while preserving token-level granularity in the underlying RL objective.
Plug-and-play integration over new objective: by multiplying the advantage term inside the existing objective (before clipping/min operations), HA-DW modifies the learning signal without changing the optimization algorithm's core mechanics. This allows it to be applied to GRPO, GSPO, DAPO, and potentially future variants with minimal adaptation.
Global $\lambda_{\text{scale}}$ over per-prompt $\lambda_t$ : a single scaling constant is simpler, has fewer hyperparameters to tune, and Theorem 3 shows that a single value can work across the range of difficulties encountered in training. The ablation study (Table 7) confirms that performance is not extremely sensitive to $\lambda_{\text{scale}}$ — values from 1.0 to 1.7 all improve over baseline.

Practical Hyperparameters and Configurations (from Appendices C and E)

The training setup uses the VeRL framework on a single node with 8× NVIDIA A100 GPUs. Key hyperparameters (Table 8):

Model: Qwen3-4B-Base, Qwen3-8B-Base, LLaMA-3.2-3B-Instruct
Training data: MATH dataset (7.5k training questions from Hendrycks et al., 2021; Lightman et al., 2024)
Generation: $G = 8$ rollouts per prompt, maximum prompt length 1024 tokens, maximum response length 4096 tokens
Training: batch size 256, mini-batch size 16, micro-batch size 4, 3 epochs (9 for DAPO), learning rate $1 \times 10^{-6}$ , AdamW optimizer, weight decay 0.1, warmup 10 steps, gradient clip 1.0
Clipping: GRPO clip-high/clip-low = 0.2/0.2; GSPO clip-high/clip-low = 0.0004/0.0003; DAPO clip-high/clip-low = 0.28/0.2 (decoupled)
HA-DW specific: $m$ (window for $\sigma_t$ computation) is not explicitly specified in the main text; $\lambda_{\text{scale}}$ tuned over $\{0.5, 0.8, 1.0, 1.3, 1.5, 1.7, 2.0\}$ with optimum at 1.3–1.5; $\eta$ (base forgetting factor) is described as "task-dependent" without a specific value; $h$ (window for hard update variant) also not specified; $\Delta$ (difficulty threshold from Definition 3) is "user-defined" and not given a concrete value in the experiments.

What's unspecified: the paper does not provide concrete values for several of its own hyperparameters ( $m$ , $\eta$ , $h$ , $\Delta$ ), which limits exact reproducibility. The ablation studies focus on $C_t$ vs. fixed thresholds (Table 2) and $\lambda_{\text{scale}}$ (Table 7, Appendix E.3), but the internal parameters of the Kalman update are left as implementation details that must be tuned per setup. This is a practical limitation — the method's theoretical guarantees hold for some parameter settings, but finding those settings requires empirical tuning.

Evaluation: Five benchmarks — MATH500 (Hendrycks et al., 2021), AIME25, AMC23, Minerva, OlympiadBench (He et al., 2024) — evaluated with avg@16 on AIME25 and AMC23 to reduce variance on small benchmark sets. Evaluation frequency: every 5 training steps.

4. Key Insights and Innovations

Innovation 1: Exposing Bias as a Conditional-Expectation Artifact, Not an Optimization Issue

The paper's most fundamental conceptual contribution is identifying that the bias in group-relative advantage estimation does not arise from optimization choices (clipping, normalization, advantage formulation) but from a statistical conditioning effect that has been invisible to the GRPO variant literature. Prior work — Dr.GRPO, DAPO, GSPO, GMPO, MAPO, and the dozen-plus variants catalogued in Appendix A — has focused exclusively on how the advantage estimate is processed after it is computed: how to clip it, how to normalize it, whether to apply it at the token or sequence level, how to handle outlier importance ratios. These are all downstream transformations of an estimator whose statistical properties were assumed unproblematic.

The paper identifies that the estimator itself is systematically wrong, and for a reason that is both subtle and inevitable. The conditioning event $\mathcal{S} = \{1 \leq R \leq G-1\}$ — which discards groups with uniform outcomes — is not an explicit design choice but an emergent consequence of the gradient structure: when $R=0$ or $R=G$ , all advantages are zero, producing zero gradients. No one designed this filter; it fell out of the math. Yet conditioning on $\mathcal{S}$ shifts the sampling distribution in a way that makes $\hat{p}_t$ a biased estimator of $p_t$ , with the bias direction flipping precisely at $p_t = 0.5$ (Theorem 1). This is fundamentally different from the off-policy correction literature (Retrace, V-trace, doubly robust estimators) because there is no off-policy distribution shift — the data is on-policy. The bias is purely a finite-sample, conditioning-induced artifact specific to the group-relative design.

The diagnostic move — distinguishing between the true expected advantage $A_{t,i} = r_{t,i} - p_t$ and the empirical group-relative advantage $\hat{A}_{t,i} = r_{t,i} - \hat{p}_t$ , then analyzing $\mathbb{E}[\hat{A}_{t,i}|\mathcal{S}]$ — is conceptually clean but had not been done before. This reframes the entire GRPO improvement agenda: before asking "how should we process the advantage?", one must ask "is the advantage estimate reliable in the first place?" The paper's title — "Your Group-Relative Advantage Is Biased" — is provocative precisely because it redirects attention upstream of the optimization objective, to the estimator itself.

Significance beyond performance: This is primarily a theoretical diagnostic advance rather than an algorithmic one. It explains why certain training phenomena occur — under-exploration on hard problems, over-exploitation on easy ones — that practitioners may have observed empirically but attributed to other causes (inadequate exploration, poor reward shaping, suboptimal learning rates). The deterministic bias at extremes (Corollary 3: $\hat{A}_{t,i} < A_{t,i}$ surely when $p_t < 1/G$ ) means that for the hardest prompts, the learning signal is guaranteed to point in the wrong direction — not probably, but certainly. This is a stronger claim than mere variance or noise; it is a structural flaw.

Comparison to prior work: Prior GRPO variants (DAPO's dynamic sampling, Dr.GRPO's removal of heuristics, GSPO's sequence-level ratios) all operate on the assumption that $\hat{A}_{t,i}$ is an unbiased estimate of $A_{t,i}$ . The paper demonstrates this assumption is false, and the falsity has a specific functional form (Lemma 2) that depends on $p_t$ and $G$ in a way that is invisible to aggregate training metrics. This is not an incremental refinement of an existing approach — it is a fundamental finding that recontextualizes the entire family of group-relative methods. The evidence is anchored in Figure 2, which visualizes the exact bias magnitude $|A_{t,i} - \mathbb{E}[\hat{A}_{t,i}|\mathcal{S}]|$ as a function of $p_t$ for $G \in \{2,4,6,8\}$ , showing the bias peak at extremes and the vanishing point at $p_t = 0.5$ .

Innovation 2: Difficulty as a Relative, Dynamic Construct — Not a Static Prompt Property

The paper's second intellectual contribution is redefining difficulty as relative to the model's evolving capability rather than as a fixed property of the prompt. This is a conceptual reframing with practical consequences that distinguishes HA-DW from all prior difficulty-aware training methods.

Prior work that incorporates difficulty into RL training — adaptive difficulty curriculum learning (Zhang et al., 2025), length-aware dynamic sampling (Chen et al., 2025), dynamic clipping policy optimization (Yang et al., 2025) — treats difficulty as a static prompt attribute. A problem from MATH Level 5 is always hard; a problem from Level 1 is always easy. These methods design curricula or sampling strategies based on these fixed labels. The paper's perspective is fundamentally different: difficulty is defined by $p_t$ , the probability that the current policy answers correctly. A prompt that is hard at training step $t=0$ (the model almost never gets it right) may become easy at step $t=100$ (the model has learned to solve it reliably). The difficulty anchor $C_t$ — a Kalman-filter-style running estimate of the model's global accuracy — operationalizes this relativistic view by providing a dynamic reference point against which individual prompt difficulties are measured.

This relativistic definition is not merely philosophical; it is necessary for the bias correction to work. If difficulty were defined statically, a prompt that the model initially found hard but has since mastered would continue to receive amplified advantages, overcorrecting and potentially destabilizing training. The evolving anchor ensures that as the model improves, the set of prompts classified as "hard" (relative to current capability) naturally shrinks, focusing the correction where it is genuinely needed.

Significance beyond performance: This reframing has implications for the broader RLVR and curriculum learning literature. If difficulty is relative, then any method that uses fixed difficulty labels — whether from dataset metadata or pre-computed statistics — is using a stale signal that may be actively harmful in later training stages. The paper's adaptive forgetting factor $\eta_t = \eta \cdot \sigma_t$ , which increases when the model's performance is volatile and decreases when it stabilizes, is an elegant solution to the tension between tracking genuine capability shifts and filtering out batch-to-batch noise. It implicitly performs change-point detection without requiring explicit change-point modeling.

The ablation in Table 2 provides evidence for the relativistic claim: training with a fixed threshold (0.4, 0.5, or 0.6) consistently underperforms the dynamic $C_t$ anchor, with the gap being roughly 0.5–1.0 percentage points in average accuracy for Qwen3-4B-Base with GRPO. The fixed-threshold methods can partially mitigate bias (they all outperform the baseline of 46.5% average accuracy), but they cannot adapt to the model's improving capability, causing the difficulty classification to become increasingly misaligned as training progresses.

Comparison to prior work: Curriculum learning methods (Zhang et al., 2025) organize training data by difficulty, but the ordering is pre-computed and static. The paper's approach is closer to online curriculum learning, where the curriculum is determined by the model's current state, but implemented at the level of advantage reweighting rather than data ordering. This is an incremental but important conceptual shift: it moves difficulty from the data pipeline to the optimization objective itself.

Innovation 3: A Unified Bias Correction Framework That Is Orthogonal to Objective Design

The paper's third contribution is demonstrating that bias correction in group-relative RL is an orthogonal axis of improvement to the objective function design space that the GRPO variant literature has been exploring. This is significant because it establishes a new category of algorithmic improvement that is composable with existing and future variants — not a replacement for them.

The GRPO variant literature has been exploring one dimension: how to process the advantage estimate once it is computed. Clipping strategies (symmetric vs. decoupled), importance ratio definitions (token-level vs. sequence-level), normalization schemes (with or without standard deviation, with or without heuristics), and advantage formulations (geometric mean, percentile-based, trajectory-certainty-weighted) — all of these operate on $\hat{A}_{t,i}$ after it is computed. HA-DW operates on a different dimension entirely: whether $\hat{A}_{t,i}$ is a reliable estimate of $A_{t,i}$ in the first place. This makes it architecturally orthogonal: any GRPO variant can incorporate HA-DW by multiplying its advantage term by $\Phi_{t,i}$ inside the existing objective, without modifying the variant's clipping, normalization, or importance sampling mechanics.

The evidence for orthogonality is in Table 1: HA-DW improves performance when integrated into GRPO, GSPO, and DAPO — three algorithms with substantially different objective formulations. The average improvement across all three algorithms and all five benchmarks is roughly 1–3 percentage points, and the pattern is consistent: HA-DW helps regardless of the base algorithm. This is not because HA-DW is somehow "better" than these algorithms, but because it corrects a flaw that all of them share — the biased advantage estimator at their core.

Significance beyond performance: This is primarily an architectural contribution with implications for how the field should develop future group-relative methods. The paper's framework suggests that future work should consider two independent design axes: (1) how to process the advantage (the GRPO variant space), and (2) whether the advantage estimate is reliable (the bias correction space). Improvements along these axes should be multiplicative — a better-processed estimate is still limited by the quality of the underlying estimate, and a better estimate benefits from better processing. The paper does not fully explore this multiplicative potential (it only tests HA-DW with three base algorithms at one set of hyperparameters), but the conceptual framework it establishes invites such exploration.

Comparison to prior work: Prior GRPO variants modify the objective function; HA-DW modifies the estimator that feeds into the objective. This is analogous to the distinction in statistics between improving an estimator and improving how that estimator is used in a decision rule — both can improve outcomes, but through independent mechanisms. The paper's plug-and-play design (Equations 25, 29, 31 in Appendix B) makes this composability concrete: HA-DW is a single-line modification to any group-relative objective.

Innovation 4: Empirical Confirmation That the Bias Matters Precisely Where Learning Matters Most

The paper's fourth contribution is not the theoretical characterization of bias (Innovation 1) but the empirical demonstration that correcting it yields gains concentrated on exactly the prompts where improvement is most valuable. This transforms the bias from a theoretical curiosity into a practically consequential problem.

Figure 1(c) shows that GRPO+HA-DW outperforms GRPO by 3.4% on hard MATH500 prompts (Levels 4–5), while performance on Easy (Level 1) and Mid (Levels 2–3) prompts is comparable. This is a clean confirmation of the theory: the bias causes under-learning on hard prompts, and correcting it unlocks gains specifically on those prompts. The fact that easy prompt performance does not degrade (which might happen if HA-DW were over-suppressing easy-prompt advantages) suggests the correction is well-calibrated — it neither overcorrects nor undercorrects.

The training dynamics in Figure 4 provide additional corroboration. RL algorithms with HA-DW converge to higher accuracy plateaus and higher training rewards than their base counterparts, and they produce longer responses. The longer-response finding is particularly suggestive: if the bias causes the model to under-learn from hard prompts, and hard prompts require longer, more sophisticated reasoning chains (Jin et al., 2024; DeepSeek-AI, 2025), then correcting the bias should naturally incentivize the development of more elaborate reasoning. The paper observes exactly this — response length increases under HA-DW — providing a mechanistic link between the bias correction and the observed accuracy gains.

The ablation in Table 3 further reinforces the practical significance: GRPO+HA-DW with $G=8$ rollouts outperforms GRPO with $G=16$ rollouts (which is impossible due to memory constraints with $G=32$ ). This means HA-DW provides a compute-efficient alternative to increasing group size for bias reduction. Since scaling $G$ is the naive solution to estimation bias (more samples → better estimator), the fact that HA-DW with $G=8$ beats GRPO with $G=16$ demonstrates that intelligent correction can substitute for brute-force sampling — a practically important finding given the computational cost of generating rollouts from large language models.

Significance beyond performance: This is both an empirical validation of the theoretical analysis and a practical demonstration that lightweight bias correction can be more cost-effective than scaling up the obvious algorithmic lever (group size). The finding that the gains are concentrated on hard prompts — precisely where the bias is largest per Corollary 1 — provides a satisfying coherence between theory and experiment that strengthens both. It also suggests a diagnostic: if a GRPO-trained model underperforms on hard prompts relative to its overall capability, biased advantage estimation may be a contributing factor.

5. Experimental Analysis

Evaluation Methodology

Dataset. All experiments use the MATH dataset (Hendrycks et al., 2021; Lightman et al., 2024), specifically the 7.5k training questions from the split released by Lightman et al. (2024) for RL training. Evaluation is conducted on five benchmarks: MATH500 (Hendrycks et al., 2021), AIME25, AMC23, Minerva, and OlympiadBench (He et al., 2024). For small benchmark sets (AIME25 and AMC23), the paper reports avg@16 to mitigate high variance and obtain reliable results.
Base model(s). Three model configurations are tested: Qwen3-4B-Base and Qwen3-8B-Base (Team, 2025), representing moderate-scale base models across two sizes, and LLaMA-3.2-3B-Instruct, representing a different model family and the instruct-tuned variant. The choice spans models of varying scale (3B to 8B parameters) and family (Qwen vs. LLaMA), testing whether the bias phenomenon and correction generalize across architectures and pretraining regimes. The models are deliberately chosen as "base" (rather than already instruction-tuned on reasoning) to study RLVR post-training from a neutral starting point.
Metrics. The primary metric throughout is benchmark accuracy (%) — the fraction of test questions answered correctly. For MATH500, the paper also reports accuracy stratified by difficulty levels (Easy: Level 1, Mid: Levels 2–3, Hard: Levels 4–5) in Figure 1(c). For AIME25 and AMC23, accuracy is averaged over 16 independent evaluation runs (avg@16). Training dynamics are monitored via training reward (mean reward across batches), response length (mean token count of generated responses), and average benchmark accuracy across all five evaluation sets, shown as time-series curves in Figure 4.
Baselines. The paper compares against the base group-relative RL algorithms without HA-DW: GRPO (Shao et al., 2024), GSPO (Zheng et al., 2025), and DAPO (Yu et al., 2025). Each baseline uses its standard objective formulation and hyperparameters as specified in Appendix C, Table 8. For the ablation on difficulty thresholds (Table 2), additional baselines include GRPO with fixed thresholds at 0.4, 0.5, and 0.6 (replacing the dynamic $C_t$ anchor with static values). For the group size ablation (Table 3), GRPO with $G=16$ rollouts serves as a higher-compute baseline against GRPO+HA-DW with $G=8$ .
Generation budget / compute accounting. The primary unit of test-time compute is the rollout count per prompt — the number of responses $G$ generated for each training prompt. All main experiments use $G = 8$ , which the paper identifies as the practical standard in RLVR (Section 2.2, citing Zhang et al., 2025; Liu et al., 2025a; Shen et al., 2025). The ablation in Table 3 compares $G=8$ (with and without HA-DW) against $G=16$ (without HA-DW) to assess whether HA-DW's correction can substitute for increased sampling. Training itself is constrained to a single node with 8× NVIDIA A100 GPUs using the VeRL framework (Sheng et al., 2024), with $G=32$ reported as out-of-memory for the GRPO baseline on Qwen3-4B-Base (Table 3 note). Training hyperparameters — batch size, learning rate, epochs, clipping bounds — are held constant across all comparisons within each algorithm family (Table 8). HA-DW introduces a single additional hyperparameter $\lambda_{\text{scale}}$ (tuned over {0.5, 0.8, 1.0, 1.3, 1.5, 1.7, 2.0} in Table 7), plus internal Kalman filter parameters ( $m$ , $\eta$ , $\Delta$ ) that are described as task-dependent but not given concrete values in the main text.
Cross-validation / statistical protocol. There is no explicit cross-validation or statistical significance testing reported. Evaluation is conducted periodically (every 5 training steps, per Table 8) and the paper reports final benchmark accuracies after training completes, with avg@16 for small benchmarks to reduce evaluation variance. The difficulty-binned analysis in Figure 1(c) uses the MATH dataset's predefined difficulty levels (1–5) — not the paper's own Definition 3 difficulty bins — to stratify performance. The advantage distribution analysis in Appendix E.1 (Figure 6) samples 50 prompts per category, evaluates them at $G=128$ rollouts to estimate "true" difficulty, and compares against the $G=8$ estimates used during training — this is an observational verification, not a controlled experiment.

Main Quantitative Results

Overall Performance Across Algorithms, Models, and Benchmarks

Table 1 presents the paper's central empirical result: HA-DW consistently improves performance when integrated into GRPO, GSPO, and DAPO across all three model configurations and five benchmarks. The headline numbers, averaged across all five benchmarks:

Qwen3-4B-Base: GRPO achieves 46.5% average accuracy; GRPO+HA-DW reaches 48.7% (+2.2 percentage points). GSPO: 47.1% → 49.2% (+2.1 pp). DAPO: 46.8% → 49.5% (+2.7 pp).
Qwen3-8B-Base: GRPO: 49.6% → 52.5% (+2.9 pp). GSPO: 50.2% → 51.7% (+1.5 pp). DAPO: 50.7% → 53.4% (+2.7 pp).
LLaMA-3.2-3B-Instruct: GRPO: 25.7% → 27.1% (+1.4 pp). GSPO: 24.9% → 25.8% (+0.9 pp). DAPO: 26.5% → 28.1% (+1.6 pp).

The improvements are remarkably consistent: HA-DW helps in 15 out of 15 algorithm-model comparisons (3 algorithms × 3 models = 9 comparisons, but each comparison spans 5 benchmarks — the AVG column shows HA-DW higher in all 9 cases, and the per-benchmark results show HA-DW higher in the vast majority of individual cells). The magnitude of improvement is modest (typically 1–3 percentage points in average accuracy) but systematic. The largest single-benchmark gains include: GRPO+HA-DW on Qwen3-8B-Base at AMC23 (64.2% → 72.8%, +8.6 pp), GSPO+HA-DW on Qwen3-4B-Base at AMC23 (62.2% → 68.6%, +6.4 pp), and DAPO+HA-DW on Qwen3-8B-Base at MATH500 (79.2% → 82.8%, +3.6 pp).

Key observation: the gains are not uniform across benchmarks. AMC23 shows the largest improvements across almost all configurations, while AIME25 and OlympiadBench show more modest gains. The Qwen3-8B-Base results are particularly notable — the larger model benefits more from HA-DW in absolute terms (2.7 pp gain for DAPO vs. 2.7 pp for the same algorithm on 4B, but with higher baseline accuracy), suggesting the correction compounds with model capability rather than being subsumed by it.

Difficulty-Stratified Performance (Figure 1(c))

The paper's central mechanistic claim — that the bias causes under-learning on hard prompts — is tested by stratifying MATH500 performance into Easy (Level 1), Mid (Levels 2–3), and Hard (Levels 4–5) using Qwen3-4B-Base trained with GRPO vs. GRPO+HA-DW. The results in Figure 1(c) show: on Easy and Mid prompts, performance is comparable between the two methods (exact percentages are not reported numerically in the text; the figure shows bars of similar height). On Hard prompts, GRPO+HA-DW outperforms GRPO by 3.4%. The paper states this is "due to our history-based dynamic reweighting strategy, which enhances exploration on hard prompts while reducing unnecessary exploitation on easy ones" (Section 5.1).

Interpretation: this is a clean confirmation of the paper's theoretical predictions. Theorem 1 and Corollary 1 predict that the advantage estimator underestimates $A_{t,i}$ for hard prompts ( $p_t < 0.5$ ). Correcting this underestimation should produce the largest gains on exactly the subset of prompts where the bias is largest — the hard ones. The fact that Easy and Mid performance does not degrade (which would indicate overcorrection) is an important sanity check: HA-DW suppresses advantages on easy prompts (where the bias causes overestimation), and the fact that Easy performance remains comparable suggests the suppression is well-calibrated — it removes unnecessary exploitation without harming correct behavior.

Limitation: the difficulty stratification uses the MATH dataset's predefined difficulty levels, not the relativistic difficulty definition the paper advances in Definition 3 (which depends on the model's current $p_t$ ). A Level 4 problem from the MATH dataset may not be "hard" for a well-trained model — it might have $p_t > 0.5$ by the end of training. Using the relativistic difficulty bins would provide a more direct test of the theory (are gains concentrated on prompts where $p_t$ is extreme relative to the current $C_t$ ?), but this is not reported. The paper's empirical verification in Appendix E.1 (Figure 6) provides complementary evidence using the relativistic definition by tracking how many prompts with extreme $G=8$ outcomes remain extreme at $G=128$ — but this is observational, not linked to performance gains.

Training Dynamics (Figure 4)

Figure 4 shows three time-series panels for Qwen3-4B-Base and Qwen3-8B-Base: (1) average accuracy across five benchmarks, (2) training reward, and (3) response length. The curves compare GRPO, GSPO, and DAPO, each with and without HA-DW.

Accuracy: RL algorithms with HA-DW "converge to a higher performance plateau" (Section 5.1). The curves show separation emerging early in training and persisting throughout. For Qwen3-4B-Base, DAPO+HA-DW (the top-performing configuration per Table 1) shows the clearest separation from DAPO alone. GRPO+HA-DW also consistently tracks above GRPO.

Training reward: HA-DW methods "acquired higher reward compared to the original RL algorithms" (Section 5.1). The reward curves show a clear upward trend over training steps, with HA-DW variants maintaining a consistent advantage over their base counterparts. The paper attributes this to the enhanced exploration signal on hard prompts making more efficient use of the training data.

Response length: This is the most novel dynamic finding. The paper states that HA-DW "encourages longer reasoning, greatly improving its reasoning abilities" and "is capable of incentivizing the model to produce more sophisticated reasoning chain of thoughts to tackle more challenging tasks" (Section 5.1). The response length curves in Figure 4 show HA-DW variants producing longer responses than their base counterparts, with the gap widening over training. This is mechanistically consistent: if the bias correction amplifies the advantage of correct responses on hard prompts, and hard prompts benefit from longer reasoning chains (Jin et al., 2024; DeepSeek-AI, 2025), then the policy should naturally learn to allocate more tokens to reasoning. The paper does not provide quantitative measures of this effect (e.g., average response length at start vs. end of training), but the qualitative trend is visible.

Relevance to claims: the training dynamics support the paper's narrative that HA-DW improves both efficiency (higher reward for the same training steps) and capability (higher plateau accuracy, longer reasoning chains). However, the response length increase is a double-edged finding — longer responses are not necessarily better responses, and the paper does not demonstrate that the additional tokens contribute to correctness rather than reflecting verbosity bias in the verifier. The accuracy gains provide indirect evidence that the longer responses are genuinely more capable, but a direct analysis of reasoning quality (e.g., step-by-step correctness, not just final answer) is absent.

Group Size Ablation: HA-DW vs. More Rollouts (Table 3)

Table 3 compares three configurations for Qwen3-4B-Base with GRPO: (1) $G=8$ rollouts (baseline), (2) $G=16$ rollouts (increased sampling, the naive bias-reduction strategy), and (3) $G=8$ + HA-DW (intelligent correction at the original budget). The results, reproduced:

Dataset	G=8 (GRPO)	G=16 (GRPO)	G=8 + HA-DW
MATH500	75.4	76.2	78.0
AIME25	19.6	19.2	20.4
AMC23	60.3	61.6	63.4
Minerva	33.8	34.2	36.8
OlympiadBench	43.5	43.9	44.7

GRPO+HA-DW with $G=8$ outperforms GRPO with $G=16$ on all five benchmarks. The advantage is largest on AMC23 (+1.8 pp over G=16) and Minerva (+2.6 pp over G=16). Meanwhile, $G=32$ with GRPO is reported as out-of-memory for this hardware configuration (8× A100 GPUs).

Significance: this is the paper's most practically impactful empirical result. It demonstrates that HA-DW is not merely a cheaper approximation to "use more rollouts" — it is more effective than doubling the group size, while using half the generation budget. Since generating rollouts from LLMs is the dominant computational cost in RLVR training, this represents a genuine efficiency gain: better results at lower cost. The finding that $G=32$ is impossible due to memory constraints underscores the practical relevance — for many practitioners, increasing group size is not even an option, making algorithmic bias correction the only viable path to improved advantage estimation.

Caveat: the comparison is only for GRPO, not for GSPO or DAPO, so the finding's generality across algorithms is not established. Additionally, $G=16$ GRPO was not given the same hyperparameter tuning that $G=8$ GRPO+HA-DW received (the $\lambda_{\text{scale}}$ ablation in Table 7 represents substantial empirical optimization), so the comparison may slightly favor HA-DW. A fairer comparison would tune learning rate and other hyperparameters for $G=16$ as extensively as $\lambda_{\text{scale}}$ was tuned for HA-DW.

$\lambda_{\text{scale}}$ Sensitivity Analysis (Table 7, Appendix E.3)

Table 7 ablates the scaling parameter $\lambda_{\text{scale}}$ over seven values {0.5, 0.8, 1.0, 1.3, 1.5, 1.7, 2.0} for Qwen3-4B-Base with GRPO+HA-DW. The average accuracy across five benchmarks peaks at $\lambda_{\text{scale}} = 1.3$ (48.7%) and $\lambda_{\text{scale}} = 1.5$ (48.6%), with performance degrading on both sides. At $\lambda_{\text{scale}} = 0.5$ , average accuracy drops to 46.5% — identical to the GRPO baseline without HA-DW — indicating that overly conservative correction provides no benefit. At $\lambda_{\text{scale}} = 2.0$ , accuracy drops to 47.3% — still above baseline but below the optimum, suggesting that overly aggressive correction introduces new distortions.

The performance curve is relatively flat between 1.0 and 1.7 (range: 47.2%–48.7%), indicating that HA-DW is not extremely sensitive to $\lambda_{\text{scale}}$ — values within a reasonable range all improve over baseline. This is practically important because it means practitioners do not need to tune $\lambda_{\text{scale}}$ with high precision; a grid search over a handful of values suffices. The optimal range (1.3–1.5) is > 1.0, confirming that the correction needs to amplify (on average) rather than merely attenuate — consistent with the theory that hard-prompt underestimation is the more consequential direction of bias during training, since hard prompts are where learning matters most.

Per-benchmark variation: the optimal $\lambda_{\text{scale}}$ varies slightly by benchmark. MATH500 peaks at 1.3 (78.0%), AIME25 at 1.5 (20.8%), AMC23 ties at 1.3 and 1.7 (63.4%), Minerva at 1.5 (37.1%), OlympiadBench at 1.3 (44.7%). The paper does not explore per-benchmark tuning; a single global $\lambda_{\text{scale}}$ is used for all benchmarks in a training run. This is appropriate since RL training optimizes for a general reasoning capability, not benchmark-specific performance.

Ablation Studies and Robustness Checks

Dynamic $C_t$ vs. fixed threshold (Table 2): The evolving difficulty anchor $C_t$ is compared against fixed thresholds at 0.4, 0.5, and 0.6 on Qwen3-4B-Base with GRPO. Dynamic $C_t$ achieves the highest average accuracy (48.7%), outperforming fixed 0.4 (48.1%), fixed 0.5 (47.8%), and fixed 0.6 (48.0%). All four configurations outperform the GRPO baseline (46.5%), confirming that any difficulty-aware reweighting — even with a static threshold — partially mitigates the bias. However, the dynamic anchor provides an additional 0.6–0.9 percentage point gain over fixed thresholds by adapting to the model's evolving capability. The fixed thresholds all perform similarly (range: 47.8–48.1%), suggesting that the specific threshold value matters less than the fact of having some difficulty-based correction — but the dynamic adaptation provides a consistent, if modest, additional benefit.
Advantage distribution verification (Figure 6, Appendix E.1): To empirically verify that $G=8$ rollouts produce systematically biased difficulty estimates, the paper evaluates selected prompts from MATH and DAPO-Math-17k at both $G=8$ and $G=128$ rollouts on Qwen3-4B-Base. For prompts with exactly 1 correct response at $G=8$ (candidates for "hard"), 24 out of 50 on MATH and 15 out of 50 on DAPO-Math-17k have fewer than 16 correct responses at $G=128$ — confirming these prompts are genuinely hard, and the $G=8$ estimate was directionally informative but imprecise. Conversely, for prompts with exactly 1 incorrect response at $G=8$ (candidates for "easy"), 12 out of 50 on MATH and 21 out of 50 on DAPO-Math-17k have fewer than 16 incorrect responses at $G=128$ . The paper frames this as evidence that "the advantage of correct responses for these prompts are underestimated at rollout=8" and that "distinct responses in these most challenging prompts are crucial for pushing the model's capability frontier" (Appendix E.1). This is an observational sanity check, not a causal demonstration, but it provides empirical grounding for the theoretical claim that small- $G$ advantage estimates are unreliable at the extremes.
$\lambda_{\text{scale}}$ hyperparameter sweep (Table 7): Discussed in detail above under Main Quantitative Results. The key finding is a relatively flat optimum around 1.3–1.5, with performance degrading gracefully on both sides and remaining above baseline for all tested values ≥ 0.8. At $\lambda_{\text{scale}} = 0.5$ , the correction is too weak to provide benefit (performance equals baseline), confirming that $\Phi_{t,i}$ must be meaningfully different from 1.0 to have an effect.
Hard update variant (Appendix F): The paper proposes but does not experimentally evaluate the hard update variant of $C_t$ . This is a notable gap — the claim that the hard variant "significantly simplifies the overall algorithm" (Appendix F) is made without evidence that the simplification does not degrade performance. The adaptive $\eta_t = \eta \cdot \sigma_t$ mechanism adds complexity (tracking $\sigma_t$ , tuning $\eta$ and $m$ ); a direct comparison showing whether the hard variant achieves comparable results would strengthen the paper's practical recommendations.
Non-binary reward extension (Appendix D.5): The paper extends its theoretical analysis to continuous bounded reward distributions (Beta and truncated Gaussian) in Theorem 4 and Corollaries 4–5, showing mathematically that the bias phenomenon persists under these more general reward models. However, no experiments are conducted with non-binary rewards — all training uses binary verifier outcomes (pass/fail on mathematical reasoning). The theoretical extension is valuable for establishing generality, but the empirical validation is limited to the binary case. Whether HA-DW would provide similar gains with soft reward signals (e.g., partial credit from a PRM) is untested.

Critical Assessment

Claim 1: Group-relative advantage estimators are systematically biased, underestimating advantages for hard prompts and overestimating for easy prompts.

The theoretical characterization (Theorems 1–2, Corollaries 1–3) provides a rigorous mathematical demonstration under the Bernoulli reward model with conditioning on the non-degenerate event $\mathcal{S}$ . The empirical advantage distribution analysis in Appendix E.1 (Figure 6) provides observational evidence consistent with the theory: prompts that appear hard at $G=8$ remain hard at $G=128$ in a substantial fraction of cases, and prompts that appear easy at $G=8$ remain easy at $G=128$ . However, this verification is limited in several ways:

Small sample size: only 50 prompts per category are analyzed, from only two datasets (MATH and DAPO-Math-17k), on a single model (Qwen3-4B-Base) at a single training stage. The binary classification ("1 correct at G=8" vs. "1 incorrect at G=8") captures only extreme cases. A more comprehensive verification would analyze the full distribution of $\hat{p}_t$ errors across all prompts and training stages.
No direct measurement of $p_t$ : the "true" difficulty at $G=128$ is still an empirical estimate, not the actual expected reward $p_t$ . With $G=128$ , the standard error on $\hat{p}_t$ is roughly $\sqrt{p_t(1-p_t)/128} \leq 0.044$ , so prompts near the 0.5 boundary could be misclassified. A truly definitive verification would require orders of magnitude more rollouts or access to the analytic $p_t$ .
Observational, not interventional: the analysis shows that $G=8$ estimates are noisy at the extremes, but does not demonstrate that correcting this noise via HA-DW causes the observed performance improvements. The link between bias characterization (theory) and bias correction (HA-DW performance gains) is established by the overall results in Table 1, but the Appendix E.1 analysis is only a consistency check, not a causal mediation analysis.

Claim 2: HA-DW corrects this bias and improves reasoning performance consistently across algorithms, model scales, and benchmarks.

Supported with qualifications. Table 1 provides strong evidence for consistency: HA-DW improves average accuracy in all 9 algorithm-model combinations tested, across 5 benchmarks. The improvements are modest (typically 1–3 pp) but systematic. However:

Single training dataset: all experiments use the MATH training set (7.5k questions). Whether HA-DW generalizes to other RLVR training data (code generation, scientific reasoning, multi-modal reasoning) is untested. The MATH dataset has specific properties (competition math, clean verifier signal, well-defined difficulty levels) that may interact with the bias correction in ways that do not transfer.
Single RL framework: all experiments use the VeRL framework with fixed hyperparameters. The interaction between HA-DW and training hyperparameters (learning rate, batch size, KL penalty, number of training steps) is not explored. It is possible that the apparent HA-DW benefit could be replicated by better hyperparameter tuning of the base algorithms — a standard concern in ML when a new method is compared against baselines that may not be optimally tuned. The paper does tune HA-DW's $\lambda_{\text{scale}}$ extensively (Table 7), but does not report equivalent tuning effort for the base algorithms.
No statistical significance testing: the paper reports point estimates without confidence intervals, standard errors, or significance tests. For a 500-question test set (MATH500), a 1–2 percentage point difference represents 5–10 questions — small enough that run-to-run variance could matter. The avg@16 protocol for AIME25 and AMC23 partially addresses evaluation variance, but training variance (different random seeds producing different final policies) is not accounted for. Re-running with multiple random seeds and reporting variance would substantially strengthen the reliability of the claimed improvements.
The LLaMA-3.2-3B-Instruct results are weaker: the gains on LLaMA (0.9–1.6 pp) are smaller than on Qwen models (1.5–2.9 pp). This could indicate that the bias correction is more beneficial for certain model architectures or pretraining regimes, or that the instruct-tuned starting point already has more calibrated uncertainty estimates that partially mitigate the bias. The paper does not discuss this discrepancy or hypothesize about its cause.

Claim 3: HA-DW with G=8 rollouts outperforms GRPO with G=16 rollouts, providing a compute-efficient alternative to increasing group size.

Supported with an important caveat. Table 3 demonstrates this for GRPO on Qwen3-4B-Base across all five benchmarks. The superiority of G=8+HA-DW over G=16 is unambiguous in the reported numbers. However, the comparison is asymmetric in terms of hyperparameter optimization effort:

HA-DW's $\lambda_{\text{scale}}$ was tuned over seven values (Table 7), representing substantial empirical optimization.
It is unclear whether the G=16 GRPO baseline received equivalent tuning. Doubling the group size changes the effective batch size (more tokens per prompt) and the statistical properties of the advantage estimates (lower variance, lower bias per the theory), which might interact with the optimal learning rate, clipping bounds, and other hyperparameters. The paper reuses the same hyperparameters for G=8 and G=16 (Table 8 notes only the rollout count changes), which may disadvantage the G=16 baseline.

A fairer comparison would tune GRPO's hyperparameters at G=16 as extensively as HA-DW's $\lambda_{\text{scale}}$ was tuned at G=8, or would compare both methods at their respective optimal hyperparameter settings. The practical claim that HA-DW is "more efficient" than increasing group size is plausible but not rigorously demonstrated by the current evidence.

Claim 4: The gains from HA-DW are concentrated on hard prompts where the bias is largest, as predicted by the theory.

Supported by one specific datapoint. Figure 1(c) shows a 3.4% gain on Hard MATH500 prompts for GRPO+HA-DW vs. GRPO on Qwen3-4B-Base. This is the only difficulty-stratified performance analysis in the paper. Missing analyses that would strengthen this claim:

Stratification by the paper's own relativistic difficulty definition: the MATH difficulty levels are static dataset properties. The theoretical prediction is that bias is a function of $p_t$ , the model's current success probability, which evolves during training. Do the gains concentrate on prompts where $\hat{p}_t$ (estimated at some point during training) is far from 0.5? This would be a more direct test of the mechanism.
Difficulty-stratified results for other algorithms and models: the claim that HA-DW helps "by enhancing exploration on hard prompts" should hold for GSPO and DAPO as well, and for other model scales. Without these results, the difficulty-stratified finding is limited to one algorithm-model combination.
Temporal analysis: do the hard-prompt gains emerge specifically when the bias is expected to be largest (early training, when many prompts have $p_t$ far from 0.5), or are they uniform throughout training? The training dynamics in Figure 4 are aggregated across all difficulties, so they cannot answer this question.

Missing experiments that would substantially strengthen the paper:

Multiple random seeds: repeating the full training pipeline with 3–5 seeds and reporting mean ± std would address concerns about run-to-run variance and establish whether the 1–3 pp gains are statistically reliable.
HA-DW with non-binary rewards: Appendix D.5 theoretically extends the bias analysis to continuous rewards, but no experiments test whether HA-DW helps with soft verifier signals (e.g., reward models that give partial credit, PRM scores). This is important because many RLVR pipelines use learned reward models, not just binary verifiers.
Ablation on the Kalman filter components: the paper ablates $C_t$ vs. fixed threshold (Table 2) and $\lambda_{\text{scale}}$ (Table 7), but does not ablate the adaptive forgetting factor $\eta_t = \eta \cdot \sigma_t$ vs. a constant $\eta$ , or the window size $m$ , or compare the Kalman-style update against the hard update variant in Appendix F. These internal design choices are asserted to be important but not empirically validated. A reader cannot determine whether the adaptive forgetting factor is a necessary ingredient or an unnecessary complexity.
Comparison against alternative bias correction methods: the paper's theoretical analysis (Lemma 1) shows that a multiplicative correction $c \cdot \hat{p}_t$ to the baseline can reduce bias. HA-DW implements one specific form of this correction (exponential reweighting with a learned anchor). Are there simpler corrections that achieve similar gains? For example, directly estimating the bias using Lemma 2's formula with $\hat{p}_t$ as a proxy for $p_t$ and applying the exact analytical correction factor? Or using a simple additive correction based on the deviation from 0.5? Without such comparisons, we cannot assess whether HA-DW's complexity (Kalman filter, adaptive forgetting factor, exponential reweighting) is justified over simpler alternatives.
Training on diverse datasets: all training uses MATH questions. Testing on alternative RLVR training sets (code generation, scientific reasoning, GSM8K) would establish whether the bias phenomenon and HA-DW's correction generalize beyond mathematical reasoning.
Interaction with verifier quality: the bias analysis assumes binary rewards from a perfect verifier. In practice, verifiers can be noisy (false positives and false negatives). How does verifier noise interact with the group-relative bias? Does HA-DW amplify the effects of verifier errors on hard prompts? The paper does not address this.
Direct measurement of $p_t$ evolution: tracking the actual expected reward $p_t$ for a fixed set of prompts across training steps (by evaluating with many rollouts at checkpoints) would directly validate the claim that $C_t$ tracks capability, and that the set of "hard" prompts (by the relativistic definition) shrinks as training progresses. Without this, the evolving anchor's benefit over fixed thresholds is only demonstrated by the modest accuracy gain in Table 2, not by mechanistic evidence.

Overall assessment: The experiments provide consistent evidence that HA-DW yields modest but reliable accuracy improvements across multiple algorithms, model scales, and benchmarks — a pattern that is unlikely to be purely coincidental. The difficulty-stratified result (Figure 1(c)) provides suggestive evidence that the mechanism operates as theorized (gains concentrated on hard prompts). The group-size ablation (Table 3) demonstrates practical value as a compute-efficient correction. However, the experimental methodology has meaningful limitations: no statistical significance testing, asymmetric hyperparameter tuning between HA-DW and baselines, limited difficulty-stratified analysis (one algorithm, one model, static difficulty bins), no ablation of the Kalman filter's internal components, no test with non-binary rewards, and no comparison against simpler correction strategies. These gaps mean the paper demonstrates that HA-DW works, but not as conclusively why it works or whether its specific design choices (adaptive forgetting factor, exponential reweighting, Kalman-style update) are necessary rather than incidental. The theoretical analysis is rigorous; the empirical validation is suggestive but incomplete.

6. Limitations and Trade-offs

Difficulty Estimation Relies on a Costly and Unvalidated Kalman Filter Design

The assumption or constraint. The entire HA-DW framework depends on the evolving difficulty anchor $C_t$ to provide a reference point for what counts as "hard" or "easy" at the current training stage. The paper proposes a Kalman-style update with an adaptive forgetting factor $\eta_t = \eta \cdot \sigma_t$ , where $\sigma_t$ is the standard deviation of recent beliefs over a window of size $m$ . The paper states that $\eta$ is a "task-dependent hyperparameter" (Section 3.1, after Equation 11) and provides no concrete values for $m$ or $\eta$ in its experimental configuration (Appendix C, Table 8). The hard update variant in Appendix F is described as a simplification but is never experimentally tested or compared against the adaptive version.

The consequence. A practitioner attempting to deploy HA-DW faces an unspecified tuning burden: three interrelated hyperparameters ( $m$ , $\eta$ , and the base forgetting factor) control the behavior of the difficulty anchor, and none have recommended values or documented sensitivity. If $\eta_t$ is set too high, the anchor tracks batch-to-batch noise rather than genuine capability shifts, causing the difficulty classification to oscillate and producing unstable reweighting. If $\eta_t$ is set too low, the anchor lags behind the model's improving capability, causing prompts that were initially hard but have become easy to continue receiving amplified advantages — an overcorrection that could destabilize late-stage training. The paper's own ablation (Table 2) shows that fixed thresholds underperform $C_t$ by 0.6–0.9 percentage points, but this does not reveal whether the Kalman filter design (with adaptive $\eta_t$ ) outperforms simpler alternatives like an exponential moving average with fixed decay, a simple windowed average, or direct use of the batch accuracy $y_t$ without any smoothing.

What evidence exists in the paper. Table 2 compares dynamic $C_t$ against fixed thresholds (0.4, 0.5, 0.6) and shows the dynamic anchor is best by a modest margin. However, there is no ablation of the Kalman filter's internal components: no comparison of adaptive $\eta_t$ vs. constant $\eta$ , no sweep over window sizes $m$ , no empirical comparison of the soft Kalman update vs. the hard update variant from Appendix F. The paper asserts that "a larger $\eta_t$ is used during early training stages to capture rapid capability shifts, while a smaller $\eta_t$ is adopted in later, more stable stages" (Section 3.1) as if this behavior is guaranteed by the $\eta_t = \eta \cdot \sigma_t$ design, but no evidence is presented that $\sigma_t$ actually decreases over training or that the adaptation behaves as described. The advantage distribution analysis in Appendix E.1 was conducted on a single model (Qwen3-4B-Base) at a single (implicit) training stage and does not track how $C_t$ or $\sigma_t$ evolve.

Mitigation status. The paper does not address this limitation. The hard update variant in Appendix F is offered as a complexity-reducing alternative but is described as ignoring "short-term oscillations in belief updates" — a claim made without evidence. Future work could systematically compare anchor designs (fixed EMA, windowed average, Kalman with fixed vs. adaptive forgetting, direct batch accuracy) to determine which components of the Kalman filter design are necessary and which are incidental. A sensitivity analysis over $m$ and $\eta$ analogous to the $\lambda_{\text{scale}}$ ablation in Table 7 would help practitioners set these hyperparameters. Until such analysis is available, the difficulty anchor's design remains an untested bundle of mechanisms whose individual contributions are unknown.

The Method Is Validated on a Single Training Dataset and Task Family (Mathematical Reasoning with Binary Verifiers)

The assumption or constraint. All RL training experiments use the MATH dataset (7.5k training questions from Hendrycks et al., 2021; Lightman et al., 2024) with binary verifier rewards (pass/fail on final answer correctness). The evaluation benchmarks (MATH500, AIME25, AMC23, Minerva, OlympiadBench) are all mathematical reasoning tasks. The paper acknowledges this implicitly by not claiming generality beyond mathematical reasoning, but also does not discuss how the bias phenomenon or HA-DW's correction might behave on other task families. The theoretical analysis in Appendix D.5 extends the bias characterization to continuous bounded reward distributions (Beta, truncated Gaussian), but no experiments are conducted with non-binary reward signals.

The consequence. Mathematical reasoning with binary verifiers has specific properties that may affect both the bias magnitude and the correction's effectiveness: (1) rewards are exactly 0 or 1, with no partial credit — this maximizes the variance of individual responses and creates sharp distinctions between correct and incorrect that the bias analysis exploits; (2) the verifier is assumed perfect — there are no false positives or false negatives, meaning the only source of noise is sampling variance, not verifier error; (3) MATH problems have well-defined difficulty ordering that correlates with the model's actual success probability. On tasks with continuous or learned reward signals (e.g., RLHF with a reward model, code generation with test-case-based scoring, dialogue with human preference scores), the Bernoulli assumption that underlies Theorems 1–2 breaks down. The paper's Appendix D.5 shows that the bias can persist under continuous rewards, but the functional form (and thus the optimal correction) would differ. Additionally, on tasks where the verifier itself is noisy (e.g., learned reward models that overfit to superficial features), the bias from group-relative estimation could compound with verifier bias in unpredictable ways — HA-DW amplifies advantages on hard prompts, which might also amplify the effects of verifier errors if hard prompts are precisely where the verifier is least reliable.

What evidence exists in the paper. All experimental results in Tables 1–3 and Figures 1, 4 are specific to mathematical reasoning benchmarks with binary verifier rewards. The non-binary reward extension in Appendix D.5 is purely theoretical — it derives formulas for the bias under Beta and truncated Gaussian reward distributions (Corollaries 4–5) and shows a numerical illustration in Figure 5 confirming the bias persists, but no training experiments are conducted with such rewards. The paper does not discuss verifier quality, verifier noise, or the interaction between verifier errors and HA-DW's reweighting.

Mitigation status. Not addressed. The paper's stated goal is "not to model all RLVR settings" (Section 1, final paragraph), which is a reasonable scope limitation for a paper introducing a new problem. However, the absence of even a single experiment on a non-math task or with non-binary rewards means the generality of both the bias phenomenon and HA-DW's correction is entirely untested. A practitioner working on code generation RLVR or multi-modal reasoning cannot determine from this paper whether HA-DW would help, harm, or have no effect. Future work should replicate the analysis on diverse RLVR training pipelines — at minimum, on a code generation task (where rewards can be binary pass/fail on test cases, providing a natural extension) and on a task with learned reward models (where reward noise and calibration become relevant).

Performance Gains Are Modest and Lack Statistical Reliability Quantification

The constraint. The paper reports all results as point estimates — single-number benchmark accuracies without confidence intervals, standard deviations, or statistical significance tests. Training and evaluation are conducted once per configuration; there is no mention of multiple random seeds, repeated runs, or variance estimation. The test sets are relatively small: MATH500 has 500 questions, and when stratified into Easy/Mid/Hard in Figure 1(c), the Hard subset (Levels 4–5) likely contains only 100–200 questions, meaning the reported 3.4% gain corresponds to roughly 3–7 additional correct answers. AIME25 and AMC23 are even smaller (the paper reports avg@16 specifically to mitigate variance on these).

The consequence. The paper cannot distinguish between a genuine algorithmic improvement and run-to-run training variance. The typical gains of 1–3 percentage points in average accuracy (Table 1) correspond to, e.g., 5–15 additional correct answers on MATH500, 1–2 on AIME25, or 3–9 on AMC23. With a single training run per configuration, these differences could plausibly arise from: (1) different random seeds for model initialization or data ordering, (2) different random sampling during rollout generation (which introduces substantial variance with $G=8$ ), (3) different checkpoint selection, or (4) interaction between HA-DW's reweighting and the specific prompts sampled in each batch. The paper's own advantage distribution analysis (Appendix E.1) demonstrates that $G=8$ rollouts produce noisy difficulty estimates — this noise propagates into training dynamics, making single-run comparisons inherently unreliable. The consistency of HA-DW's improvement across 9 algorithm-model combinations is suggestive but does not substitute for formal statistical quantification: if each configuration has, say, a 70% chance of outperforming its baseline by chance alone (given the noise), observing 9/9 wins is statistically meaningful, but we do not know the null distribution and cannot assess this.

What evidence exists in the paper. The paper does not report standard deviations, confidence intervals, p-values, or the number of training runs. The evaluation protocol mentions avg@16 for AIME25 and AMC23 (Appendix C) to reduce evaluation variance (by averaging over 16 decoding runs), but this does not address training variance (different training runs producing different policies). The difficulty-stratified result in Figure 1(c) reports a single percentage (3.4%) without error bars. The training dynamics curves in Figure 4 show single trajectories per method, leaving the reader unable to assess whether the separation between HA-DW and baseline curves is larger than the typical run-to-run variation.

Mitigation status. Not addressed. This is a methodological weakness that is common in the RLVR literature (many papers report single-run results due to the computational cost of LLM training) but is particularly consequential here because the effect sizes are small relative to plausible training variance. The paper could partially address this by: (1) reporting evaluation variance (standard deviation across multiple evaluation checkpoints or decoding runs) to give a lower bound on noise, (2) running a subset of configurations with multiple seeds to establish variance estimates, or (3) reporting per-benchmark statistical tests against the null hypothesis of no improvement. Absent this, a practitioner evaluating whether to adopt HA-DW faces uncertainty about whether the 1.4 percentage point gain on LLaMA-3.2-3B-Instruct (Table 1) would replicate in their setting or is within the noise floor of their training pipeline.

The Method Cannot Create Capability Where None Exists — Hardest Prompts Remain Unimproved

The assumption or constraint. HA-DW corrects the bias in how the model learns from the responses it already generates, but it does not change what responses the model can generate. If the policy's expected success probability $p_t$ is near zero on a prompt — the model essentially never produces a correct answer — then there is no correct response whose advantage can be amplified. The group-relative advantage for all responses will be near zero (since $\hat{p}_t \approx 0$ in groups where no response is correct, and such groups are discarded by the conditioning event $\mathcal{S}$ ), and HA-DW's reweighting factor $\Phi_{t,i}$ multiplies zeros, producing zeros. The paper's theoretical analysis (Corollary 3) shows that the bias is deterministic at extremes ( $p_t < 1/G$ ), but does not claim that correcting the bias will make the model succeed on these prompts — the correction can only improve the learning signal when there is a signal to improve.

The consequence. On prompts where the base model's pass@1 is effectively zero, HA-DW provides no benefit regardless of how well it corrects the bias. This is the same fundamental limitation that affects all RLVR methods: reinforcement learning can refine and amplify existing capabilities but cannot create them from scratch. The paper's experiments confirm this implicitly: the absolute accuracy on the hardest benchmarks (AIME25, OlympiadBench) remains low even with HA-DW — for Qwen3-4B-Base, GRPO achieves 19.6% on AIME25 and GRPO+HA-DW reaches 20.4%, a gain of only 0.8 percentage points (Table 1). On OlympiadBench, the gain is 1.2 pp (43.5% → 44.7%). These small absolute improvements on the most challenging benchmarks suggest that many AIME25 and OlympiadBench problems are in the $p_t \approx 0$ regime for a 4B-parameter model, where bias correction cannot help because there are virtually no correct responses to learn from. The method's value is concentrated on the subset of prompts where the model has non-trivial but imperfect capability — the "medium-hard" regime where $p_t$ is small but non-zero, and the bias causes systematic under-learning. This is an important but bounded domain of applicability.

What evidence exists in the paper. The difficulty-stratified result in Figure 1(c) shows gains concentrated on Hard prompts (+3.4%) with comparable performance on Easy and Mid prompts. However, "Hard" in this classification refers to MATH Levels 4–5 — problems that are challenging but still within the model's partial capability (the baseline GRPO accuracy on Hard prompts is non-zero, since it gains 3.4 pp). The paper does not separately analyze the hardest-of-the-hard subset (Level 5 only, or prompts where $p_t$ is estimated to be near zero at some training stage) to test whether HA-DW helps or not. The AIME25 and OlympiadBench results in Table 1 provide indirect evidence: these benchmarks are more difficult than MATH500, the absolute accuracy is lower, and the HA-DW gains are smaller (0.2–2.1 pp on AIME25, 0.3–2.8 pp on OlympiadBench) compared to MATH500 (0.8–3.6 pp). This pattern is consistent with the hypothesis that harder benchmarks contain more prompts in the $p_t \approx 0$ regime where bias correction is irrelevant. The FLOPs-matched context: no comparison to scaling model size or training data is provided, so we cannot assess how the gains from HA-DW compare to simply using a larger model that would have non-zero $p_t$ on more prompts.

Mitigation status. The paper does not explicitly discuss this capability bound, though it is implied by the mechanism. The statement that HA-DW "enhances exploration on hard prompts" (Section 5.1) should be qualified: it enhances exploration on hard prompts where the model sometimes succeeds. For prompts where the model never succeeds, no amount of advantage reweighting will help. This limitation is fundamental to RLVR and is not specific to HA-DW, but the paper's framing could mislead practitioners into expecting uniform improvements across all difficulty levels when the actual gains are contingent on the base model's existing capability distribution. A more precise characterization — e.g., measuring HA-DW's benefit as a function of estimated $p_t$ at the start of training, and showing where the gains go to zero — would establish the applicability boundary more rigorously.

The $\lambda_{\text{scale}}$ Hyperparameter Requires Per-Setup Tuning Without Transferable Guidance

The assumption or constraint. HA-DW introduces a global scaling hyperparameter $\lambda_{\text{scale}}$ that controls the overall aggressiveness of the advantage reweighting (Equation 16). Theorem 3 proves that some value of $\lambda_{\text{scale}}$ in a specified union of intervals can provably reduce bias, but does not provide a single optimal value — the intervals depend on $c_{\text{low}}$ and $c_{\text{high}}$ , which in turn depend on the unknown true $p_t$ , the observed $\hat{p}_t$ , the confidence level $\delta$ , and the difficulty bound $\Delta$ (Lemma 1). In practice, $\lambda_{\text{scale}}$ is treated as a tunable hyperparameter, and the paper's ablation (Table 7) sweeps seven values {0.5, 0.8, 1.0, 1.3, 1.5, 1.7, 2.0}, finding the optimum at 1.3 or 1.5 for Qwen3-4B-Base with GRPO.

The consequence. A practitioner deploying HA-DW on a new model, dataset, or RL algorithm must conduct a hyperparameter sweep over $\lambda_{\text{scale}}$ to find the effective range. The paper provides no transferable guidance: the optimal value may depend on model size (Qwen3-8B-Base might prefer a different $\lambda_{\text{scale}}$ than Qwen3-4B-Base, but this was not tested), the base RL algorithm (GRPO vs. GSPO vs. DAPO might have different optimal scales due to different advantage normalization), the training dataset difficulty distribution (a harder dataset might need more aggressive amplification, i.e., higher $\lambda_{\text{scale}}$ ), or the group size $G$ (larger $G$ reduces the bias per Lemma 2, so the optimal $\lambda_{\text{scale}}$ should be closer to 1.0). The paper's ablation in Table 7 shows that $\lambda_{\text{scale}} = 0.5$ (very weak correction) achieves baseline-level performance, while $\lambda_{\text{scale}} = 2.0$ (aggressive correction) degrades performance compared to the optimum — confirming that the value matters and cannot be arbitrarily set. The computational cost of sweeping $\lambda_{\text{scale}}$ over 5–7 values is non-trivial: each setting requires a full RL training run (potentially hours to days on 8× A100 GPUs), making the tuning overhead comparable to the cost of simply running the baseline with more rollouts (the very cost HA-DW claims to avoid).

What evidence exists in the paper. Table 7 provides a $\lambda_{\text{scale}}$ sweep for exactly one configuration (Qwen3-4B-Base, GRPO). The performance is relatively flat between 1.0 and 1.7 (range: 47.2%–48.7% average accuracy), suggesting some robustness. However, this sweep was done only after the method was designed and the theoretical analysis was complete — it represents within-paper tuning, not transferable guidance. The paper does not report $\lambda_{\text{scale}}$ values used for other configurations (Qwen3-8B-Base, LLaMA-3.2-3B-Instruct, GSPO, DAPO), nor does it state whether a common $\lambda_{\text{scale}}$ was used across all experiments or whether per-configuration tuning was performed. If different $\lambda_{\text{scale}}$ values were used for different configurations, the reported gains in Table 1 represent post-hoc optimal performance rather than default performance, and a practitioner using a single $\lambda_{\text{scale}}$ (e.g., 1.3, the best from the 4B sweep) might see smaller gains on other configurations.

Mitigation status. Not addressed. The paper does not discuss $\lambda_{\text{scale}}$ transferability, provide a heuristic for setting it (e.g., " $\lambda_{\text{scale}}$ should increase with model size" or "start at 1.3 and adjust based on whether hard-prompt accuracy improves"), or propose an adaptive scheme that would eliminate the hyperparameter (e.g., setting $\lambda_{\text{scale}}$ based on the running estimate of bias magnitude from Lemma 2). The theoretical intervals in Theorem 3 are expressed in terms of quantities ( $c_{\text{low}}$ , $c_{\text{high}}$ , $\hat{p}_t$ , $D_{t,i}$ , $M_t$ ) that are in principle computable during training, suggesting that a data-driven $\lambda_{\text{scale}}$ selection might be possible, but this is not explored. In its current form, HA-DW replaces one form of computational overhead (generating more rollouts) with another (hyperparameter tuning), and it is not clear which is more costly or more reliable in practice.

The Paper Does Not Test HA-DW in Combination with PRM Search or Iterative Revision — Only Base GRPO-Style Training

The constraint. HA-DW is evaluated exclusively as a modification to the base RLVR training loop (GRPO, GSPO, DAPO) where advantages are computed from binary verifier rewards on single-shot responses. The paper does not test HA-DW in combination with other test-time compute strategies that are commonly used alongside or as alternatives to RLVR — specifically, process reward model (PRM) guided search (beam search, best-of-N weighted) or iterative revision models (where the model conditions on previous incorrect attempts to produce improved answers). These methods operate on different principles (verifier-guided selection vs. policy gradient updates) and might interact with HA-DW in non-obvious ways.

The consequence. The paper positions HA-DW as a general bias correction for "group-relative RL algorithms" (Section 1), but the scope of evaluation is limited to a specific training paradigm: on-policy RL with binary verifier rewards, where the group-relative advantage is computed from a small number of independently sampled responses. Several important RLVR variants that also use group-relative advantage estimation are not tested: methods that incorporate PRM scores as dense rewards (where the advantage at each step depends on step-level verifier scores, not just final outcomes), methods that combine RL with rejection sampling (where responses are filtered by a verifier before contributing to the gradient), or methods that use iterative revision (where responses within a group are not independent but form a chain of revisions). In these settings, the Bernoulli reward assumption underlying Theorems 1–2 may not hold, the group size $G$ might have different semantics (e.g., $G$ beam-search beams vs. $G$ independent samples), and the conditioning event $\mathcal{S}$ might have a different structure. The paper's claim that HA-DW is a "plug-and-play module" for "GRPO and its variants" (Section 3.2) is tested only on three closely related variants (GRPO, GSPO, DAPO) that differ primarily in clipping and importance ratio design, not in the fundamental structure of advantage computation.

What evidence exists in the paper. None. The related work section (Appendix A) lists numerous GRPO variants (DaGRPO, MAPO, LSPO, GMPO, DCPO, FAPO, SPO, KTAE) but none are tested with HA-DW. The paper does not discuss how HA-DW would interact with step-level PRM rewards, with rejection sampling, with iterative revision, or with off-policy data. The experimental setup (Appendix C) is a standard VeRL training loop with binary verifier rewards on MATH — one of the simplest RLVR configurations.

Mitigation status. Not addressed. This is primarily a scope limitation rather than a flaw — the paper's contribution is in identifying and correcting the bias, not in exhaustively testing combinations with every RLVR variant. However, for practitioners using more complex RLVR pipelines, the lack of evidence about interactions is a meaningful gap. A future study could test whether the bias is larger or smaller when rewards are dense (per-step PRM scores might reduce the conditioning-induced bias by providing more granular signal), whether HA-DW's correction is still effective when responses in a group are not independent (revision chains have correlated outcomes), and whether combining HA-DW with other bias-reduction strategies (larger $G$ , off-policy corrections, verifier ensembles) yields additive or sub-additive gains.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a new axis of evaluation and improvement for group-relative RL algorithms that is orthogonal to — and therefore composable with — the objective-function design space that has dominated the GRPO variant literature. Prior work on improving GRPO (Dr.GRPO, DAPO, GSPO, GMPO, MAPO, DCPO, FAPO, SPO, KTAE — catalogued in Appendix A) has focused exclusively on how to process the advantage estimate once it is computed: clipping strategies, normalization schemes, token-level vs. sequence-level ratios, and advantage reweighting based on trajectory certainty. All of these methods implicitly assume that the underlying advantage estimator $\hat{A}_{t,i}$ is a reasonable approximation of the true advantage $A_{t,i}$ . This paper demonstrates that this assumption is false in a specific, quantifiable, and practically consequential way: the group-relative estimator is systematically biased, with the bias direction flipping at exactly $p_t = 0.5$ and the magnitude growing as $p_t$ deviates from this midpoint (Theorem 1, Lemma 2, Figure 2).

The conceptual shift is not that GRPO-family algorithms have a bug — they work, and their aggregate performance metrics improve with training. Rather, the shift is in recognizing that the learning signal itself is distorted in a difficulty-dependent way that aggregate metrics conceal. The practical manifestation — under-learning from hard prompts where improvement is most valuable, over-exploitation of easy prompts where marginal gains are minimal — was likely observed empirically by practitioners (manifesting as slow progress on challenging benchmarks or diminishing returns from extended training) but attributed to other causes: inadequate exploration, suboptimal reward design, or learning rate decay. The paper provides a mechanistic explanation for these phenomena that is rooted in the statistical properties of the estimator, not in optimization hyperparameters.

This is not a paradigm shift in the sense of introducing a new learning algorithm or objective function. It is better characterized as a diagnostic reframing: before asking "how should we process the advantage?", one must ask "is the advantage estimate reliable in the first place?" This reframing has several downstream consequences for how the field develops:

Reconciling conflicting empirical observations. Practitioners who observed that GRPO training plateaus on hard benchmarks despite continued improvements on training metrics now have a candidate explanation: the advantage signal on hard prompts is systematically weakened, so the model receives less gradient information from exactly the problems where it most needs to improve. Conversely, the observation that GRPO can overfit to easy patterns might be explained by overestimated advantages on easy prompts, causing the policy to over-exploit behaviors that already work rather than exploring new ones. These are not separate phenomena requiring separate fixes — they are two sides of the same statistical coin.

Changing the improvement agenda. The GRPO variant literature has been exploring improvements along one dimension (objective function design). This paper establishes a second, independent dimension (estimator reliability). Improvements along these dimensions should be multiplicative: a better-processed advantage estimate is still limited by the quality of the underlying estimate, and a better estimate benefits from better processing. The paper's empirical demonstration that HA-DW improves GRPO, GSPO, and DAPO — three algorithms with substantially different objective formulations — provides direct evidence for this orthogonality. Future research on group-relative RL should therefore consider both axes simultaneously, rather than treating objective design as the sole lever.

Redirecting attention from scaling compute to improving estimation. A natural response to the bias problem is to increase the group size $G$ — more rollouts produce better estimates, per standard statistical theory. The paper anticipates and tests this response (Table 3), showing that HA-DW with $G=8$ rollouts outperforms GRPO with $G=16$ rollouts while using half the generation budget. This has practical significance because generating rollouts from LLMs is the dominant computational cost in RLVR training, and memory constraints often prevent scaling $G$ beyond modest values ( $G=32$ is reported as out-of-memory for Qwen3-4B-Base on 8× A100 GPUs). The paper's finding redirects investment from brute-force sampling (which hits hardware limits quickly) toward algorithmic bias correction (which has essentially zero additional inference cost — HA-DW requires only a few scalar operations per batch). This is an efficiency argument that strengthens the case for investing in estimator quality over raw compute.

Establishing difficulty as a dynamic, relative construct. The paper's evolving difficulty anchor $C_t$ operationalizes a definition of difficulty that is relative to the model's current capability — a prompt is "hard" not because of its intrinsic properties, but because the current policy has a low probability of answering it correctly. This relativistic view has implications beyond HA-DW. Any method that uses difficulty labels — curriculum learning, adaptive sampling, dynamic clipping — should consider whether static labels (dataset difficulty levels, pre-computed success rates) become stale as the model improves. A prompt that is Level 5 (hard) in the MATH dataset may have $p_t > 0.5$ for a well-trained model, at which point the bias direction flips and the prompt would benefit from advantage suppression rather than amplification. Methods that use static difficulty labels may inadvertently apply corrections in the wrong direction during later training stages. The paper's finding that dynamic $C_t$ outperforms fixed thresholds by 0.6–0.9 percentage points (Table 2) is modest in absolute terms but conceptually important: it demonstrates that difficulty relativity matters, even if the performance gain from getting it right is incremental.

Limitations of the landscape change. The paper's impact is bounded by its scope: all experiments are on mathematical reasoning with binary verifier rewards, using a single training dataset (MATH). The bias phenomenon is characterized theoretically under the Bernoulli reward model, and the extension to continuous rewards (Appendix D.5) is purely theoretical. Whether the bias is equally consequential — and whether HA-DW provides similar gains — on tasks with learned reward models, dense step-level rewards (PRM scores), non-independent responses (revision chains), or entirely different modalities (code generation, multi-modal reasoning) is unknown. The paper's contribution is therefore best understood as opening a new research direction (estimator reliability in group-relative RL) rather than closing one (solving the bias problem definitively across all settings). It provides the conceptual framework, theoretical characterization, and proof-of-concept correction for one important setting; generalizing to the broader RLVR landscape is the task of follow-up work.

Follow-Up Research This Work Enables

1. Direct measurement of $p_t$ evolution during training to validate the relativistic difficulty claim. The paper's central conceptual innovation — that difficulty is relative to the model's evolving capability — is asserted but not directly measured. A follow-up study would take periodic checkpoints of a GRPO-trained model (e.g., every 50 training steps), evaluate each checkpoint on a fixed set of 200–500 MATH prompts with $G=512$ rollouts to obtain high-precision estimates of $p_t$ , and track how the set of prompts classified as "hard" ( $p_t < 0.5 - \Delta$ ) and "easy" ( $p_t > 0.5 + \Delta$ ) changes over training. Key predictions to test: (1) the fraction of prompts in the hard regime should decrease as training progresses (the model improves), (2) the advantage bias magnitude (computed via Lemma 2 using the measured $p_t$ ) should correlate with the performance gap between a HA-DW-corrected and uncorrected training run at each checkpoint, and (3) the Kalman anchor $C_t$ should track the mean of the $p_t$ distribution across prompts. This experiment would provide mechanistic validation of the paper's theoretical framework and quantify how much of HA-DW's benefit comes from correcting bias at different training stages (early vs. late). If the bias is largest early in training — when many prompts have $p_t$ far from 0.5 — and HA-DW's gains are concentrated there, this would support the claim that early-stage correction is the primary mechanism.

2. Ablation of the Kalman filter design to identify necessary vs. incidental components. The paper's difficulty anchor uses a Kalman-style update with adaptive forgetting factor $\eta_t = \eta \cdot \sigma_t$ (Equations 8–11), but no experiment isolates the contribution of the adaptive component. A minimal ablation study would compare four anchor designs on the same Qwen3-4B-Base + GRPO setup: (a) the full Kalman anchor with adaptive $\eta_t$ , (b) a Kalman anchor with constant $\eta_t = 0.1$ (no adaptation), (c) a simple exponential moving average $C_t = \alpha y_t + (1-\alpha) C_{t-1}$ with fixed $\alpha$ , and (d) the direct batch accuracy $y_t$ with no smoothing. Each variant would use the same HA-DW reweighting (Equation 16) with $\lambda_{\text{scale}}$ tuned independently. The key metric is whether the adaptive forgetting factor provides gains beyond a well-tuned constant EMA. If the constant EMA matches the adaptive version, the Kalman filter's complexity is unnecessary, and practitioners can use a simpler design. If the adaptive version significantly outperforms, the paper's claim that $\sigma_t$ "captures rapid capability shifts" (Section 3.1) is supported. This ablation would also include the hard update variant from Appendix F (fixed window average) as a fifth condition. A negative result — the hard update matching the adaptive Kalman — would simplify practical deployment substantially.

3. Testing HA-DW on code generation RLVR with test-case-based binary rewards. The paper's theoretical analysis and empirical validation are confined to mathematical reasoning. Code generation provides a natural extension domain with several attractive properties: (a) rewards are typically binary (all test cases pass or not), matching the Bernoulli assumption of Theorems 1–2; (b) the verifier is objective and deterministic (unit tests), eliminating concerns about verifier noise interacting with bias correction; (c) code generation problems have a natural difficulty spectrum (from simple function implementations to complex algorithmic problems); (d) the RLVR training pipeline for code (e.g., using the APPS or CodeContests datasets) is structurally identical to math RLVR. A strong follow-up would replicate the Table 1 experimental design — GRPO, GSPO, DAPO, each with and without HA-DW — on a code generation benchmark (HumanEval, MBPP, LiveCodeBench) using a standard base model (e.g., DeepSeek-Coder-Base or Qwen2.5-Coder-Base) trained on a code-specific dataset. The critical test is whether the difficulty-stratified gains replicate: does HA-DW improve performance specifically on hard coding problems (where the model's pass@1 is low but non-zero) while leaving easy problems unchanged? A negative result — HA-DW providing no benefit on code despite the theoretical prediction — would suggest that the bias phenomenon depends on domain-specific properties of mathematical reasoning (e.g., the distribution of $p_t$ values across training prompts, the relationship between prompt difficulty and response length) that do not transfer to code.

4. Interaction between HA-DW and verifier quality — a stress test with noisy or learned reward models. The paper assumes a perfect binary verifier: pass/fail on final answer correctness with no false positives or false negatives. In many RLVR deployments, the verifier is a learned reward model (e.g., trained on human preference data or outcome-based labels) that has non-trivial error rates, especially on hard prompts where the model's outputs are unusual or the verifier's training data is sparse. A critical stress test would introduce controlled verifier noise — e.g., flipping a fraction $\epsilon$ of reward labels, where $\epsilon$ is higher for hard prompts (simulating a verifier that is less reliable on out-of-distribution responses) — and measure whether HA-DW's gains persist, diminish, or reverse. The theoretical prediction is ambiguous: HA-DW amplifies advantages on hard prompts, which could amplify the impact of verifier errors (if the verifier incorrectly labels a good response as bad, HA-DW would amplify the incorrect penalty). Conversely, HA-DW might provide a regularizing effect by dampening advantages on easy prompts where verifier errors are less consequential. A systematic experiment varying $\epsilon \in \{0, 0.05, 0.1, 0.2\}$ and measuring HA-DW's benefit (or harm) at each noise level would establish the method's robustness to a realistic deployment condition that the paper does not address. This experiment could also test whether the bias magnitude changes under verifier noise — if the verifier has systematic errors (e.g., false negatives on certain reasoning patterns), the effective $p_t$ distribution is distorted, potentially changing which prompts are classified as "hard" and how the bias manifests.

5. Combining HA-DW with larger group sizes to test for additive vs. sub-additive gains. The paper shows that HA-DW with $G=8$ outperforms GRPO with $G=16$ (Table 3), but does not test HA-DW with $G=16$ . This leaves open the question of whether the benefits are additive — does HA-DW at $G=16$ provide gains beyond HA-DW at $G=8$ , or do the two strategies (larger groups + bias correction) address overlapping variance? The theoretical prediction is that larger $G$ reduces the bias (Lemma 2 and Figure 2 show the bias magnitude decreases with $G$ ), so the marginal benefit of HA-DW should be smaller at larger $G$ . Testing this requires running GRPO with $G=16$ + HA-DW and comparing against GRPO with $G=16$ alone, and ideally also against $G=32$ if memory permits (using gradient accumulation or model parallelism to circumvent the OOM constraint reported in the paper). If HA-DW at $G=16$ provides negligible additional gain over GRPO at $G=16$ , this would confirm the theoretical prediction and establish a practical guideline: HA-DW is most valuable when $G$ is small (the computationally constrained regime), and its benefit diminishes as $G$ increases. If HA-DW at $G=16$ provides substantial gains over both GRPO at $G=16$ and HA-DW at $G=8$ , this would suggest the bias correction addresses variance sources beyond what larger $G$ alone can fix, making HA-DW valuable even in high-compute settings. This experiment would also help practitioners decide whether to invest in HA-DW or in scaling up $G$ (hardware permitting).

6. A simpler baseline: direct analytical bias correction using Lemma 2. HA-DW uses an exponential reweighting scheme with a learned difficulty anchor to correct the bias. A simpler alternative — suggested but not tested by the paper's own Lemma 2 — is to estimate the bias magnitude directly and apply the analytical correction. Lemma 2 provides the exact formula for the expected bias: $\mathbb{E}[\hat{A}_{t,i}|\mathcal{S}] - A_{t,i} = \frac{p_t(1-p_t)^G + p_t^{G+1} - p_t^G}{1 - (1-p_t)^G - p_t^G}$ . Substituting the observed $\hat{p}_t$ as a proxy for the unknown $p_t$ (or using a Bayesian estimate with a Beta prior) yields a computable bias correction: multiply each advantage by the inverse of the expected bias factor. This approach would (a) eliminate the Kalman filter entirely (no $C_t$ , no $\sigma_t$ , no $\eta_t$ ), (b) require no evolving state across batches, (c) have only the group size $G$ as a known parameter, and (d) be directly justified by the paper's own theoretical analysis. A strong follow-up would implement this analytical correction and compare it against HA-DW on the same Qwen3-4B-Base + GRPO setup. The key comparison is: does a simple plug-in estimator (using $\hat{p}_t$ in Lemma 2's formula) match or exceed HA-DW's performance? If it does, HA-DW's complexity (Kalman filter, adaptive forgetting, exponential reweighting) is unnecessary for the settings tested. If it underperforms — perhaps because $\hat{p}_t$ is too noisy at $G=8$ to serve as a reliable plug-in estimate — this would justify HA-DW's cross-batch smoothing. Either result would sharpen our understanding of why HA-DW works: whether the key ingredient is the cross-batch difficulty anchor (stabilizing the estimate of $p_t$ ) or the exponential reweighting form (providing a specific functional correction). This experiment would also directly test the paper's theoretical framework by determining whether the Lemma 2 formula, which is a mathematical identity given $p_t$ , is practically useful when $p_t$ must be estimated from small- $G$ data.

Practical Applications and Downstream Use Cases

1. Cost-efficient RLVR training with small rollout budgets on consumer or academic hardware. The paper's most directly actionable finding for practitioners is that HA-DW with $G=8$ rollouts achieves better performance than doubling the rollout budget to $G=16$ (Table 3), while $G=32$ is out-of-memory on 8× A100 GPUs. For research labs, startups, or individual practitioners training reasoning models on limited GPU budgets (e.g., 4–8 consumer GPUs, single-node academic clusters), this translates to: (a) reduced generation cost — half the inference compute for the same or better downstream accuracy, (b) the ability to train on hardware where $G=32$ (or even $G=16$ in some configurations) would be impossible due to memory constraints, and (c) faster experimental iteration — each training step processes fewer tokens, reducing wall-clock time. The specific numbers: on Qwen3-4B-Base, GRPO+HA-DW at $G=8$ achieves 78.0% on MATH500 vs. 76.2% for GRPO at $G=16$ , while using half the generation budget. For a typical RLVR training run processing 7.5k prompts over hundreds of steps, the generation savings are substantial — roughly a 2× reduction in the dominant cost. HA-DW's implementation is a few lines of code added to the existing loss computation (Equations 25, 29, 31 in Appendix B), requiring no additional model components, no extra forward/backward passes, and only scalar operations on already-computed batch statistics. The main practical barrier is tuning $\lambda_{\text{scale}}$ (the paper suggests 1.3–1.5 works well, with the method being relatively insensitive within 1.0–1.7 per Table 7), which can be done once per model-dataset combination.

2. Targeted improvement on hard evaluation subsets for benchmark-driven development. The paper demonstrates that HA-DW's gains are concentrated on hard prompts: +3.4% on MATH500 Hard (Levels 4–5) with comparable performance on Easy and Mid prompts (Figure 1(c)). For practitioners whose evaluation benchmarks are skewed toward challenging problems — e.g., AIME, Olympiad-level math, or hard subsets of standard benchmarks — HA-DW offers a way to improve exactly where it matters most, without trading off performance on easier problems. This is particularly relevant for models targeting competition-level benchmarks (AIME, IMO, Codeforces) where the majority of problems are in the "hard" regime for current models. The mechanism — amplifying the learning signal on prompts where the model occasionally succeeds but usually fails — directly targets the capability frontier. The implication for training strategy: rather than training on the full dataset uniformly, practitioners could use HA-DW in combination with data filtering (focusing training on prompts in the $p_t \in [0.1, 0.4]$ range where the bias is largest and correction is most impactful) to maximize hard-prompt improvements within a fixed compute budget.

3. Plug-and-play enhancement for any GRPO-based training pipeline without architectural changes. HA-DW is designed as a wrapper that multiplies the advantage term inside the existing policy objective, without modifying the underlying RL algorithm's clipping, normalization, or importance sampling mechanics. This makes it directly applicable to existing GRPO training codebases (VeRL, OpenRLHF, TRL, custom implementations) with minimal integration effort — essentially inserting $\Phi_{t,i}$ into the loss computation and adding a few lines to track $C_t$ and $y_t$ across batches. For teams that have already invested in GRPO training infrastructure and hyperparameter tuning, HA-DW can be added as an incremental improvement without discarding prior work. The paper's demonstration that HA-DW improves GRPO, GSPO, and DAPO (Table 1) — three algorithms with different clipping strategies, importance ratio definitions, and normalization schemes — suggests the correction is robust to the specific objective formulation, reducing the risk that HA-DW will interact poorly with algorithm-specific design choices. The consistent improvement across model scales (3B, 4B, 8B) and families (Qwen, LLaMA) further supports the claim of broad applicability within the mathematical reasoning domain. The main integration requirement — tracking batch-level accuracy $y_t$ and maintaining the Kalman anchor state — is lightweight (a few scalar variables per training run) and does not require distributed communication or synchronization beyond what standard data-parallel training already provides.

When to Prefer This Method

The paper explicitly positions HA-DW against two alternatives: (1) not using any bias correction (the standard GRPO-family baseline), and (2) increasing the group size $G$ (the naive bias-reduction strategy of generating more rollouts per prompt). The decision framework that emerges from the paper's theoretical and empirical results is:

Prefer HA-DW over standard GRPO-family algorithms when: the training dataset contains a non-trivial fraction of prompts where the model's expected success probability $p_t$ is far from 0.5 — specifically, when many prompts fall in the $p_t < 0.25$ (hard) or $p_t > 0.75$ (easy) regimes, where Corollary 1 shows the probability of directional bias exceeds 78%. This condition is likely met in most RLVR training runs on mathematical reasoning, especially early in training or when using base models without prior reasoning-specific fine-tuning. The paper's experiments (Table 1) show HA-DW provides consistent 1–3 percentage point gains across 9 algorithm-model configurations, making it a low-regret addition when computational budgets permit tuning $\lambda_{\text{scale}}$ .
Prefer HA-DW with $G=8$ over scaling to $G=16$ (or higher) when: (a) GPU memory constraints prevent increasing the group size — the paper reports $G=32$ as out-of-memory on 8× A100 GPUs for Qwen3-4B-Base, making algorithmic correction the only viable path to improved advantage estimation; (b) generation throughput is the bottleneck — HA-DW provides compute-equivalent improvements to doubling $G$ (Table 3) while using half the generation budget, making it strictly more efficient when inference dominates training cost; (c) training latency matters — fewer rollouts per step means faster iteration. The paper's finding that HA-DW at $G=8$ outperforms GRPO at $G=16$ across all five evaluation benchmarks establishes that the correction is more effective than simply adding more samples in this regime.
Prefer standard GRPO without correction when: (a) the model is already near the performance ceiling on the training distribution ( $p_t \approx 0.5$ for most prompts), at which point the bias is small (Theorem 1: zero at exactly $p_t = 0.5$ ) and the correction provides minimal benefit; (b) the deployment scenario involves minimal tuning budget — HA-DW introduces at least one new hyperparameter ( $\lambda_{\text{scale}}$ ) that requires empirical optimization (Table 7 sweeps 7 values), and the Kalman filter parameters ( $m$ , $\eta$ ) add additional unspecified tuning burden; (c) the RLVR pipeline uses continuous or learned rewards where the Bernoulli bias model may not apply — while Appendix D.5 suggests the bias persists under more general reward distributions, no experiments validate this claim, and the correction's effectiveness in these settings is unknown.

The paper does not position HA-DW against other bias-reduction strategies (e.g., off-policy correction methods like V-trace or Retrace, or alternative advantage estimators that don't condition on $\mathcal{S}$ ), so no decision framework for those comparisons is supported by the evidence presented.