Self-Hinting Language Models Enhance Reinforcement Learning

ArXiv: 2602.03143

🎯 Pitch

On hard prompts, GRPO silently collapses: when no rollout in a group succeeds, advantage estimates vanish and the model stops learning. SAGE injects self-generated hints during training—lossy compressions of reference solutions—to rescue within-group outcome diversity under the exact same reward function, yielding consistent gains (e.g., +2.0 on Llama-3.2-3B) while requiring zero privileged information at test time.

1. Executive Summary

This paper introduces SAGE (Self-hint Aligned GRPO with PrivilEged Supervision), an on-policy reinforcement learning framework that prevents GRPO training from stalling on hard prompts by injecting privileged hints during training—lossy compressions of reference solutions—to reshape the rollout distribution while keeping the terminal verifier reward unchanged. Experiments across 6 mathematical benchmarks with 3 LLMs (Llama-3.2-3B-Instruct, Qwen2.5-7B-Instruct, Qwen3-4B-Instruct) demonstrate that SAGE consistently outperforms GRPO, yielding average accuracy improvements of +2.0 on Llama-3.2, +1.2 on Qwen2.5, and +1.3 on Qwen3. The approach is complemented by a policy-dependent hint-strength scheduler that activates hints only when within-group reward variance collapses—yielding an automatic curriculum that tracks the learner's current bottlenecks—and an online self-hinting mechanism that periodically refreshes the hint distribution to maintain calibration to the evolving policy. The analysis formalizes GRPO collapse as a gate-opening probability under Bernoulli rewards, establishing that privileged hinting increases the frequency of non-degenerate updates precisely when the no-hint policy's success probability is too low for finite groups to contain mixed outcomes, and that test-time deployment requires no hints or privileged information whatsoever.

2. Context and Motivation

The Core Problem: GRPO Stalls Silently on Hard Problems

The fundamental problem this paper addresses is a finite-sample degeneracy in Group Relative Policy Optimization (GRPO) that causes training to silently stall on difficult prompts. This isn't a theoretical limitation of the objective function—the expected gradient would be perfectly informative—but rather a practical pathology of how GRPO estimators behave with limited samples. Understanding why this happens requires a brief look at how GRPO works.

GRPO refresher. GRPO is a reinforcement learning algorithm for aligning language models with verifiable reward signals. For a given prompt $x$, the policy model $\pi_\theta$ generates a group of $G$ complete responses (trajectories), each receiving a reward $R(x, \tau) \in \{0, 1\}$ from a binary verifier (e.g., a math answer checker). Rather than training a separate critic network to estimate advantages (as PPO does), GRPO computes advantages directly from within-group reward statistics. The key operation is standardization: each rollout's advantage is computed as

$A_i = \frac{R_i - \bar{R}}{s + \epsilon}$

where $\bar{R}$ is the group mean reward and $s$ is the group standard deviation. This standardization serves as a baseline—it centers and scales rewards so that the policy receives positive updates for above-average rollouts and negative updates for below-average rollouts, all relative to its own current performance on that prompt.

Where the degeneracy creeps in. The standardization step has a critical failure mode. When a prompt is hard—meaning the policy's success probability $p_\theta(x)$ is very small—a randomly sampled group of $G$ rollouts will frequently contain only incorrect answers. If every rollout in the group scores $R_i = 0$, then $\bar{R} = 0$, $s = 0$, and every advantage $A_i = 0$. The policy-gradient estimate for that prompt becomes identically zero. No update occurs. The model receives no signal about what went wrong, which rollouts were slightly better than others, or even that the prompt exists.

The paper formalizes this starkly. Let $p_\theta(x)$ be the probability that a single rollout from $\pi_\theta(\cdot \mid x)$ correctly solves prompt $x$. For a group of size $G$, the probability that the group contains mixed outcomes (at least one success and at least one failure)—which is necessary for $s > 0$ and therefore for any update signal—is:

$\Pr[s > 0 \mid x] = 1 - (1 - p_\theta(x))^G - p_\theta(x)^G$

In the sparse regime where $p_\theta(x) \ll 1$, the all-ones term $p_\theta(x)^G$ is negligible, and we can approximate:

$\Pr[s > 0 \mid x] \approx 1 - (1 - p_\theta(x))^G \approx G \cdot p_\theta(x)$

Training therefore receives useful signal only when $G \cdot p_\theta(x)$ is not tiny. When the group size is $G = 8$ (a common setting, used throughout this paper's main experiments) and $p_\theta(x) = 0.01$, the probability of getting any learning signal from that prompt is approximately $8 \times 0.01 = 0.08$—meaning 92% of the time, that prompt contributes absolutely nothing to training. The gradient doesn't get noisier; it vanishes entirely.

This is what the paper means by a gate: GRPO updates are gated by whether the group contains mixed outcomes. When the gate is closed (all rewards identical), the prompt is wasted. The prompt is still seen, compute is still spent generating rollouts, but the resulting gradient is zero.

Why this is not just a theoretical curiosity. Figure 2 in the paper provides concrete empirical evidence of the scale of this problem. Across three model families trained on 64k prompts with $G = 8$:

• For Llama-3.2-3B-Instruct (the weakest model), approximately 57% of prompts have their correct trajectories never sampled over the entire course of training. These prompts are essentially dead weight—they consume compute but provide zero learning signal.
• For Qwen2.5-7B-Instruct (moderate capability), roughly 30% of prompts are wasted.
• Even for Qwen3-4B-Instruct (the strongest, with extensive math-focused RL pretraining), about 22% of prompts never produce a correct trajectory.

This is not a minor inefficiency. For the weakest model, more than half the training data is effectively invisible to the learning algorithm. The model cannot improve on these prompts because it never receives any gradient information about them, creating a perverse dynamic where the prompts that would benefit most from learning (the hard ones) are precisely the ones that contribute least to training.

Why This Problem Matters

The practical stakes are significant for several reasons, some explicitly discussed in the paper and others implicit in the broader RL-for-LLMs landscape.

Difficult prompts are disproportionately valuable for learning. Prior work has established that prompts with lower pass rates are more informative for RL training (Xiong et al., 2025b; Yu et al., 2025). Easy prompts—those the model already solves consistently—provide little room for improvement. Hard prompts, where the model struggles, are where genuine capability growth occurs. If GRPO's finite-sample degeneracy selectively silences the hardest prompts, it systematically biases training away from the most valuable learning opportunities. The model overfits to a small set of solvable prompts while stagnating on the rest.

The problem compounds during training. As training progresses and the model improves on easy prompts, the remaining unsolved prompts become, by definition, the hardest ones—exactly those most susceptible to GRPO stall. This creates a self-reinforcing dynamic: early training solves the easy prompts, leaving behind a residue of hard prompts that provide no signal, causing later training to plateau prematurely. The paper's Figure 2 shows this isn't just a startup transient—the percentage of "dead" prompts remains stubbornly high across multiple epochs, indicating the model genuinely cannot break through to these problems without some intervention.

Wasted compute at scale. RL training for LLMs is extraordinarily expensive, often requiring thousands of GPU-hours. When more than half the prompts in a batch contribute zero gradient, a corresponding fraction of the forward and backward passes are wasted. This isn't a problem that resolves with more compute—running more GRPO steps doesn't help if the gate never opens on those prompts. Indeed, the paper shows (Table 2) that even after full training, GRPO leaves 40% of Llama-3.2 prompts and 10% of Qwen2.5 prompts completely untouched by any correct trajectory. These prompts are "seen" but never "learned from."

It explains contradictory findings in the literature. Some papers report strong GRPO results on reasoning benchmarks, while others observe frustrating plateaus. This paper's analysis suggests a reconciliation: the effectiveness of GRPO depends critically on the difficulty distribution of the training data relative to the base model's capabilities. A training set that is mostly solvable (high $p_\theta$) will train smoothly; a training set with many hard prompts will stall. Since different papers use different training data and base models, they implicitly operate at different points on this spectrum, leading to conflicting conclusions about GRPO's reliability.

Where Prior Approaches Fall Short

The paper identifies several existing strategies for dealing with GRPO stall, each with significant limitations.

Skipping degenerate groups (resampling). The simplest fix is to detect groups where all rewards are identical, discard them, and resample new prompts (Yu et al., 2025; Xiong et al., 2025a). This prevents wasted computation on degenerate groups but has a fundamental flaw: it biases the training distribution toward easier prompts. Prompts that consistently produce degenerate groups get systematically excluded, which means the model never sees—and therefore never learns from—the hardest problems. The training distribution shifts toward prompts the model already handles reasonably well, leaving capability gaps unaddressed. This is a form of survivorship bias in the training data: only prompts that survive the degeneracy filter contribute to learning.

Adaptive sampling and curriculum scheduling. More sophisticated approaches allocate additional rollouts or training emphasis to difficult prompts (Yao et al., 2025; Xiong et al., 2025b; Li et al., 2025; Zhang et al., 2025c). While principled, these methods face an inherent tension: they try to solve the finite-sample degeneracy by increasing the sample size ($G$), following the logic that $\Pr[s > 0] \approx G \cdot p_\theta(x)$. However, the required $G$ to reliably open the gate on very hard prompts ($p_\theta(x) < 0.01$) can be impractically large—needing $G > 100$ just to have a reasonable chance of seeing one success. Moreover, these approaches don't fundamentally change what the model samples; they just sample more of it. If the model's proposal distribution on hard prompts is genuinely poor (generating almost entirely nonsensical or systematically wrong answers), more samples from that same distribution may yield only marginal improvements.

Leveraging offline data or external teachers. Another family of approaches injects correct trajectories from stronger models (Zhang et al., 2025a; Yan et al., 2025; Zhang et al., 2025b). For example, LUFFY (Yan et al., 2025) replaces one on-policy rollout in each group with a correct trajectory from a stronger teacher (DeepSeek-R1 in the paper's experiments). While this guarantees at least one positive reward per group—breaking the degeneracy—it introduces off-policy distribution mismatch. The policy model $\pi_\theta$ is trained on trajectories it didn't generate, under contexts it may not have seen. The paper's results (Figure 4, Table 1) show this empirically: LUFFY-trained models exhibit unstable training dynamics, with Llama-3.2 showing excessively high entropy and oscillatory response lengths, and Qwen3 suffering from very low rewards early in training. The stronger model's reasoning patterns may not align with what the learner model can actually produce, causing the policy to chase a distribution it cannot effectively represent.

Scaf-GRPO and external hint generators. Scaf-GRPO (Zhang et al., 2025b) attempts a more nuanced intervention: it uses hints generated by a stronger external model (GPT-5.2 in the paper's experiments) to scaffold learning on difficult prompts. When a group collapses, Scaf-GRPO augments the batch with hinted trajectories. The paper identifies three problems with this approach. First, it mixes contexts within a single group—some rollouts come from $\pi_\theta(\cdot \mid x)$ while others come from $\pi_\theta(\cdot \mid x, h)$—which blurs the interpretation of groupwise advantage normalization. GRPO's standardization assumes all rollouts in a group are drawn from the same distribution; mixing hinted and non-hinted rollouts violates this assumption. Second, external hint generators may be miscalibrated to the learner's capabilities. A hint that is perfectly informative for GPT-5.2 may be too revealing or too cryptic for a 3B-parameter learner. Third, as Figure 4 shows, Scaf-GRPO exhibits the lowest entropy among all methods, suggesting it overly constrains exploration—the external hints may be so strong that the model converges prematurely to a narrow solution strategy rather than developing robust reasoning skills.

A deeper issue: none of these approaches maintain strict on-policy learning. The paper emphasizes that GRPO is designed as an on-policy algorithm—the policy that generates rollouts should be the same policy being optimized. LUFFY and Scaf-GRPO both inject off-policy data (from stronger teachers or hinted contexts) into what is supposed to be an on-policy update. This isn't just a theoretical concern; the paper's Figure 4 shows concrete training instabilities: LUFFY causes entropy explosions and reward collapses, Scaf-GRPO suppresses exploration, and both exhibit more erratic training dynamics than pure GRPO (despite GRPO's own stall problem). The challenge, then, is to break the degeneracy without breaking the on-policy contract.

How This Paper Positions Itself

The paper frames SAGE as a complementary approach that addresses GRPO stall through a fundamentally different mechanism: rather than discarding hard prompts, resampling, or injecting external data, SAGE reshapes the rollout distribution on hard prompts by conditioning the policy on privileged hints during training. The key insight is that the finite-sample degeneracy is a property of the sampling distribution, not the objective function. If $p_\theta(x)$ is too small for a group of size $G$ to contain mixed outcomes, the fix is to temporarily increase $p_\theta(x)$ by providing additional context—not to change the reward, not to import off-policy data, not to skip the prompt.

The privileged hinting philosophy. Hints are "lossy compressions" of reference solutions ($\tau^\star$)—for example, a high-level plan or a key algebraic insight—that guide the model toward correct reasoning without revealing the final answer. Critically, the task reward $R(x, \tau)$ is unchanged. A rollout that produces the correct final answer still scores 1; an incorrect one still scores 0. Hints don't cheat the verifier; they simply increase the probability that, under finite sampling, at least one rollout in the group follows a path that leads to the correct answer. This makes $p_\theta(x, h) > p_\theta(x, \emptyset)$, raising $G \cdot p_\theta(x, h)$ above the threshold where mixed-outcome groups become common.

Three design principles distinguish SAGE from prior work. The paper explicitly positions these as its unique contributions:

1. On-policy conditioning: Hints are appended to the prompt as part of the conditioning context, so rollouts are drawn from $\pi_\theta(\cdot \mid x, h)$. The policy-gradient loss uses $\log \pi_\theta(\tau \mid x, h)$—the log-probability under the same context that generated the rollout. This keeps training strictly on-policy for the augmented context. The paper includes an ablation (Section 5.2, Figure 5) showing that a variant that samples with hints but evaluates $\log \pi_\theta(\tau \mid x)$ (dropping $h$ from the log-probability) performs substantially worse, confirming that matching the sampling and optimization contexts matters.

2. Policy-dependent scheduling: Hints aren't applied uniformly. A scheduler activates hints only when the group collapses (all rewards identical), and increases hint strength ($\ell$) only when the current hint level still fails to produce a positive rollout. This creates an automatic curriculum: prompts that the model can already solve receive no hints ($\ell = 0$, the deployable no-hint setting); prompts that stall at $\ell = 0$ get a weak hint ($\ell = 1$); if that still fails, a stronger hint ($\ell = 2$); and so on up to a maximum level $L$. The scheduler is policy-dependent—it uses statistics from recent rollouts under the current $\pi_\theta$ to decide whether to escalate hint strength.

3. Online self-hinting: The hint generator $q_\phi(h \mid x, \tau^\star, \ell)$ is periodically refreshed using a copy of the current policy $\pi_\theta$, rather than remaining frozen at initialization. This ensures the hint distribution remains calibrated to the learner's evolving capabilities. An initial policy might need detailed step-by-step guidance, while a partially trained policy might only need a nudge toward the right approach. Fixed hints from a stronger external model (as in Scaf-GRPO) cannot adapt to the learner's changing skill level.

The analysis formalizes why this works. The paper provides a clean theoretical framing in Section 3: standardized GRPO behaves as a gated update procedure where the gate opens only when the group contains mixed outcomes. The gate-opening probability under Bernoulli rewards is $u(p) = 1 - (1-p)^G - p^G$, which is strictly concave and maximized at $p = 1/2$. This has two implications. First, hinting is useful precisely when it moves $p_\theta$ out of the $Gp \ll 1$ regime where the gate is almost always closed. Second, and more subtly, hints should not be too strong—if $p_\theta(x, h) \approx 1$, the group will be all-ones and the gate closes again. The optimal hint distribution places mass on hints that make $p_\theta(x, h) \approx 1/2$, where the gate-opening probability is maximized. This explains why the adaptive strength scheduler (which escalates only until the gate opens) is better than always using a strong hint, and why online hint refreshing (which tracks the learner's current $p_\theta$) outperforms fixed hints.

Deployment requires no privileged information. This is the crucial practical property. During training, SAGE uses reference solutions $\tau^\star$ to generate hints—these are available because the training data includes (prompt, solution) pairs. At test time, the model is deployed with $\ell = 0$, meaning $h = \emptyset$, and runs as a standard language model on the prompt alone. There is no hint generation, no reference solution access, no additional inference-time computation beyond what the base model would require. The gains from hinting during training transfer to the no-hint policy at test time because the model has learned to internalize the reasoning patterns that hints previously guided it toward.

Relationship to prior work in a broader context. The paper connects SAGE to the long history of privileged information and reward shaping in reinforcement learning (Ng et al., 1999; Szepesvári, 2022), but notes that LLM-specific applications have been largely heuristic. SAGE provides a principled instantiation: the privilege is a hint derived from a reference solution, the shaping is indirect (it modifies the sampling distribution, not the reward), and the deployment gap is closed by a policy-dependent scheduler that gradually removes the privilege as the learner improves. This distinguishes it from prior LLM hinting approaches that either used static external hints, mixed hinted and non-hinted contexts, or failed to maintain on-policy updates.

3. Technical Approach

3.1 Reader Orientation

This paper develops SAGE, a training framework that wraps around standard GRPO to prevent it from silently failing on hard math problems by having the language model generate its own reference-solution-derived hints during training, conditioning its rollouts on those hints to increase the chance of seeing both correct and incorrect attempts within each group, and then deploying the model without any hints at test time. The system solves the problem of GRPO's "vanishing gradient on hard prompts" pathology—where entire groups of rollouts receive identical zero rewards, causing the policy update to be numerically zero—by temporarily making hard prompts easier through hint conditioning, then gradually removing hints as the model improves, all while keeping the verifier reward function completely unchanged.

3.2 Big-Picture Architecture (Diagram in Words)

The SAGE system has five tightly coupled components that operate within an epoch-based training loop:

1. Policy Model ($\pi_\theta$): The LLM being trained—generates complete solutions (trajectories) conditioned on a prompt and optionally a hint. At test time, deployed without hints ($h = \emptyset$).

2. Hint Generator ($q_\phi$): A copy of the policy model (or a lagged version) prompted to produce a compact plan/hint from a reference solution $\tau^\star$. The hint is a lossy compression—it captures the key reasoning approach without revealing the final answer. Refreshed periodically to stay calibrated to the evolving policy.

3. Strength Scheduler: A policy-dependent gate that decides, for each prompt at each epoch, what hint strength level $\ell \in \{0, 1, \ldots, L\}$ to use. $\ell = 0$ means no hint (deployable setting); higher $\ell$ means progressively more detailed hints. Strength increases only when the current level fails to produce a positive rollout—implemented either through epoch-level accuracy tracking (SAGE-LIGHT) or per-group degeneracy detection (SAGE).

4. GRPO Update Engine: The standard GRPO machinery—samples $G$ rollouts per prompt, computes standardized advantages from within-group rewards, and updates the policy via a clipped policy-gradient loss plus optional KL penalty. The critical modification is that rollouts are sampled from $\pi_\theta(\cdot \mid x, h)$ rather than $\pi_\theta(\cdot \mid x)$, making this an on-policy objective for the hint-conditioned context.

5. Reference Solution Bank ($\mathcal{D} = \{(x, \tau^\star)\}$): The training dataset, where each prompt $x$ is paired with a verified correct solution $\tau^\star$ (generated by DeepSeek-R1 and verified by Math-Verify). These solutions exist only during training and are used exclusively to generate hints—they never appear in the model's context directly.

Information flows as follows at each epoch: for each prompt $x$, the scheduler decides the hint level $\ell$ (based on previous epoch's accuracy or current probe group degeneracy) → the hint generator $q_\phi$ produces $h \sim q_\phi(\cdot \mid x, \tau^\star, \ell)$ → the policy samples $G$ rollouts $\tau_i \sim \pi_\theta(\cdot \mid x, h)$ → the verifier assigns binary rewards $R_i \in \{0, 1\}$ based on whether each $\tau_i$'s final answer matches the ground truth → GRPO computes standardized advantages $A_i = (R_i - \bar{R})/(s + \epsilon)$ and updates $\theta$ via the policy gradient. At deployment, the scheduler is bypassed, $\ell$ is fixed at 0, $h = \emptyset$, and the model generates solutions from $\pi_\theta(\cdot \mid x)$ alone.

3.3 Roadmap for the Deep Dive

• First, the formalization of GRPO's finite-sample degeneracy as a gate-opening probability (Section 3.1 of the paper)—this establishes the mathematical language for understanding why hinting helps and what an optimal hint distribution looks like.
• Second, the on-policy conditioning requirement—why hints must appear in both the sampling context and the log-probability computation, and what goes wrong when this contract is violated.
• Third, the hint generation mechanism—how $q_\phi$ produces progressive levels of hints from reference solutions, and why online refreshing is necessary for calibration.
• Fourth, the two policy-dependent scheduling schemes (SAGE-LIGHT and SAGE)—how each decides when to escalate hint strength using policy-derived statistics, and the accuracy-vs-efficiency tradeoff between them.
• Fifth, the complete SAGE training algorithm (Algorithm 1)—a walkthrough of the nested sampling, scheduling, and optimization loop that ties all components together.
• Sixth, the Jensen inequality insight (Remark 3.4)—why sampling a single hint per prompt per epoch is actually better than sampling multiple hints, counter to intuition about diversity.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a method paper that proposes a training framework (SAGE) to prevent GRPO from stalling on hard prompts under sparse binary rewards. The core idea is to condition rollouts on privileged hints during training to reshape the sampling distribution, increasing the probability that a finite group of rollouts contains mixed outcomes (both successes and failures), while keeping the verifier reward unchanged and deploying without hints.

Formalizing GRPO Collapse as a Gate-Opening Probability

The paper's analysis (Section 3.1) provides the theoretical foundation for SAGE by characterizing exactly when and why GRPO updates vanish. This analysis is essential for understanding what the scheduler and hint generator are trying to achieve.

The setup: Bernoulli rewards and standardized advantages. Consider a single prompt $x$ and a hint $h$ (which may be empty, $h = \emptyset$, for the no-hint case). The policy $\pi_\theta$ generates $G$ independent rollouts $\tau_1, \ldots, \tau_G$, each receiving a binary reward from the verifier:

$R_i = R(x, \tau_i) \in \{0, 1\}$

The success probability—the chance that a single rollout from the hint-conditioned policy solves the problem correctly—is defined as:

$p_\theta(x, h) = \Pr_{\tau \sim \pi_\theta(\cdot \mid x, h)}[R(x, \tau) = 1]$

where $p_\theta(x, h)$ is the per-rollout success rate of the current policy $\pi_\theta$ when conditioned on prompt $x$ and hint $h$. This is not a fixed dataset property; it evolves as the policy improves during training.

GRPO standardizes advantages within each group using the empirical mean $\bar{R} = \frac{1}{G}\sum_i R_i$ and standard deviation $s = \sqrt{\frac{1}{G}\sum_i (R_i - \bar{R})^2}$:

$A_i = \frac{R_i - \bar{R}}{s + \epsilon}$

where $\epsilon > 0$ is a numerical stabilizer (preventing division by zero when $s = 0$).

The gate-opening probability. The key insight is that GRPO's update signal for a given prompt is entirely controlled by whether the group contains mixed outcomes—at least one success and at least one failure. When $s = 0$ (all rewards identical), $A_i = 0$ for every rollout and the policy gradient contribution from that prompt is exactly zero.

The paper formalizes this as a gate: the "gate opens" when the rollout group has mixed outcomes ($s > 0$), and remains closed when all rewards are identical ($s = 0$). Proposition 3.2 gives the gate-opening probability under Bernoulli rewards:

$\Pr[s > 0 \mid x, h] = 1 - (1 - p_\theta(x, h))^G - p_\theta(x, h)^G$

where $p_\theta(x, h)$ is the success probability defined above, and $G$ is the group size (number of rollouts per prompt per training step).

What this equation computes: the probability that a group of $G$ independent Bernoulli trials contains both at least one success (probability $1 - (1-p)^G$) and at least one failure (probability $1 - p^G$). The formula subtracts the two degenerate cases—all-zero ($(1-p)^G$) and all-one ($p^G$)—from 1 to get the probability of non-degenerate (mixed) outcomes. For example, with $G=8$ and $p=0.01$, the gate opens with probability approximately $1 - (0.99)^8 - (0.01)^8 \approx 1 - 0.923 - 0 \approx 0.077$, meaning only about 8% of groups from that prompt produce any learning signal.

Why this form matters. In the sparse regime where $p_\theta(x, h) \ll 1$, the $p^G$ term is negligible (e.g., $(0.01)^8 = 10^{-16}$), and we can approximate:

$\Pr[s > 0 \mid x, h] \approx 1 - (1 - p_\theta(x, h))^G \approx G \cdot p_\theta(x, h)$

This linear approximation reveals the critical threshold: training stalls whenever $G \cdot p_\theta(x, h) \ll 1$. For the paper's default $G = 8$, any prompt with $p_\theta < 0.01$ has less than an 8% chance of providing signal per step. Over multiple steps, the expected number of informative updates from such a prompt is tiny.

The advantage energy as a continuous gate signal. Corollary 3.1 shows that the magnitude of the update (not just whether it is zero) is directly controlled by the same gate. Define the "advantage energy":

$E = \frac{1}{G}\sum_{i=1}^G A_i^2$

where $A_i$ are the standardized advantages from Equation (2).

Then for $\epsilon > 0$:

$E = \frac{s^2}{(s + \epsilon)^2} \in [0, 1)$

What this computes: a single scalar measuring how much "signal energy" the group provides for the policy update. When $s = 0$ (degenerate group), $E = 0$—no energy, no update. When $s > 0$, $E$ increases monotonically with $s$, meaning groups with more reward diversity produce stronger updates. The denominator $(s+\epsilon)^2$ provides a smooth transition rather than a hard threshold, with $\epsilon$ controlling how quickly the energy rises from zero as $s$ increases.

Why this form: standardization turns the raw reward differences $(R_i - \bar{R})$ into a normalized signal whose energy depends only on the within-group variance $s^2$ relative to the stabilizer $\epsilon$. This means the prompt-level update magnitude is completely determined by whether the group is non-degenerate ($s > 0$)—there is no partial credit for groups where all rollouts fail but some were "closer" to success. This all-or-nothing property is what makes the gate-opening probability so critical: if the gate is closed, the prompt contributes literally nothing, regardless of how close the incorrect answers might have been.

The shape of the gate-opening function. The function $u(p) = 1 - (1-p)^G - p^G$ has important properties that inform the design of SAGE:

• Symmetry: $u(p) = u(1-p)$—the gate opens equally often at $p$ and $1-p$. This is intuitive: a group of all successes ($p=1$) is just as uninformative as a group of all failures ($p=0$).
• Strict concavity: $u''(p) < 0$ for $G \geq 2$—the function curves downward, meaning the gate-opening probability increases fastest when moving away from extreme $p$ values.
• Maximum at $p = 1/2$: the gate opens most frequently when the success rate is 50%. This is where groups are most likely to contain mixed outcomes.

These properties motivate a key design principle: hints should aim to push $p_\theta(x, h)$ toward $1/2$, not toward 1. An overly strong hint that makes the prompt trivially solvable ($p_\theta \approx 1$) closes the gate just as effectively as having no hint on an impossibly hard prompt ($p_\theta \approx 0$). SAGE's adaptive scheduler is designed to find this sweet spot—escalating hint strength only until the gate opens, not beyond.

The expected gate-opening rate as a training objective. Proposition 3.3 frames hint selection as an optimization problem. For a given policy $\theta$ and prompt $x$, let $u(p_\theta(x, h)) = \Pr[s > 0 \mid x, h]$ be the gate-opening probability with hint $h$. For any distribution over hints $q(\cdot \mid x)$, the expected gate-opening rate is:

$J_x(\theta, q) = \mathbb{E}_{h \sim q(\cdot \mid x)}[u(p_\theta(x, h))]$

where $J_x(\theta, q)$ is the expected fraction of groups that will contain mixed outcomes when hints are sampled from $q$ and rollouts from $\pi_\theta(\cdot \mid x, h)$. Maximizing $J_x(\theta, q)$ over $q$ means finding the hint distribution that most reliably opens the gate.

What this computes: an objective function for evaluating hint distributions—higher $J_x$ means fewer degenerate groups, which means more prompts contribute to training, which means better sample efficiency. The paper shows that the optimal $q^\star_\theta$ concentrates on hints that make $p_\theta(x, h) \approx 1/2$—calibrating hints that make the task neither trivially easy nor impossibly hard.

Why this form: it separates the hint selection problem from the policy optimization problem. $J_x(\theta, q)$ measures only whether the gate opens, not whether the resulting gradients are useful—it's a necessary condition for learning, not a sufficient one. The policy update itself handles turning the mixed outcomes into directional learning. This modularity is what allows SAGE to be a wrapper around GRPO rather than a replacement for it.

The policy-dependence of optimal hints. A critical insight from Proposition 3.3: the set of calibrating hints depends on $\theta$. As the policy improves during training, $p_\theta(x, h)$ changes for any fixed hint $h$—what was a perfectly calibrating hint ($p_\theta \approx 1/2$) early in training may become too weak ($p_\theta \ll 1/2$) or too strong ($p_\theta \gg 1/2$) later. This means a fixed hint distribution (e.g., offline hints generated once before training) cannot remain optimal throughout training. This motivates SAGE's online hint refreshing: periodically update $q_\phi$ using the current policy so that hints track the learner's evolving capabilities.

On-Policy Conditioning: Why Hints Must Be in the Log-Probability Context

A subtle but crucial design choice in SAGE is that hints appear in both the sampling context and the optimization objective. The policy generates rollouts conditioned on the hint: $\tau_i \sim \pi_\theta(\cdot \mid x, h)$. The policy-gradient loss then evaluates log-probabilities under the same context:

$\mathcal{L}_{\text{pg}} = -\frac{1}{G}\sum_{i=1}^G A_i \sum_{t=1}^{T_i} \log \pi_\theta(y_{i,t} \mid x, h, y_{i,<t})$

where $A_i$ are the standardized advantages, $y_{i,t}$ is the $t$-th token of the $i$-th rollout, $T_i$ is the length of that rollout, and $y_{i,<t}$ denotes all tokens before position $t$ in that rollout. The outer sum averages over the $G$ rollouts in the group; the inner sum accumulates token-level log-probabilities weighted by the rollout-level advantage.

What this computes: the standard REINFORCE-style policy gradient, but with the conditioning context augmented by the hint $h$. For each rollout, the loss increases the probability of token sequences that led to above-average rewards ($A_i > 0$) and decreases the probability of sequences that led to below-average rewards ($A_i < 0$). The magnitude of the update per rollout is proportional to $|A_i|$, which itself depends on the reward variance within the group.

Why this form matters (on-policy vs. off-policy). The paper explicitly contrasts this with an off-policy variant where rollouts are sampled with the hint ($\tau \sim \pi_\theta(\cdot \mid x, h)$) but the loss evaluates log-probabilities without the hint ($\log \pi_\theta(\tau \mid x)$). This mismatch breaks the on-policy contract: the gradient no longer corresponds to the derivative of any well-defined objective under the actual sampling distribution. In practice, the paper's ablation (Section 5.2, Figure 5) shows this off-policy variant performs substantially worse than on-policy SAGE (56.5 vs. 59.2 average accuracy on Qwen3-4B-Instruct), and even underperforms the single hint-level baseline (58.3). The on-policy variant ensures that the policy learns to generate correct solutions given the hint, which transfers to the no-hint setting because the reasoning patterns that produce correct solutions with a hint are genuinely useful reasoning patterns that the model can internalize.

The KL penalty (optional but included for completeness). The full SAGE loss includes a KL divergence term to prevent the policy from diverging too far from a reference policy:

$\mathcal{L}(\theta) = \mathcal{L}_{\text{pg}} + \beta \cdot \mathbb{E}\left[\text{KL}\left(\pi_\theta(\cdot \mid x, h) \parallel \pi_{\text{ref}}(\cdot \mid x, h)\right)\right]$

where $\beta$ is the KL penalty weight and $\pi_{\text{ref}}$ is the reference policy (typically the initial policy or a frozen copy). Following DAPO (Yu et al., 2025), the paper's main experiments set $\beta = 0$ (disabling the KL term) and use asymmetric clipping with $\epsilon_{\text{low}} = 0.2$ and $\epsilon_{\text{high}} = 0.28$ on the policy ratio $r_t(\theta) = \pi_\theta(y_t \mid x, h, y_{<t}) / \pi_{\theta_{\text{old}}}(y_t \mid x, h, y_{<t})$ to stabilize training.

Hint Generation: Progressive Compression of Reference Solutions

SAGE generates hints from reference solutions $\tau^\star$ using a prompted language model. The hint generator $q_\phi(h \mid x, \tau^\star, \ell)$ takes the prompt $x$, the reference solution $\tau^\star$, and a discrete strength level $\ell \in \{0, 1, \ldots, L\}$, and produces a textual hint $h$ that captures aspects of the solution approach without revealing the final answer. $\ell = 0$ is the no-hint setting, corresponding to $h = \emptyset$ deterministically. Higher $\ell$ values produce progressively more detailed hints.

Implementation of $q_\phi$. The paper implements the hint generator by prompting the policy model $\pi_\theta$ (or a copy thereof) with a specific system prompt that instructs it to act as a "tutoring assistant that generates progressive hints." The system prompt (provided in Appendix B) specifies:

"Given a question and its solution, generate 3 levels of hints that progressively guide the student toward solving the problem independently."

The model is instructed to produce a JSON object with three fields: level_1 (minimal hint pointing to the key concept), level_2 (medium hint providing more direction on the method), and level_3 (detailed hint giving substantial guidance while still requiring the student to complete the solution). Each level builds on the previous one, and the final answer is never revealed.

The user prompt for hint generation simply concatenates:

Question: {problem}
Solution: {solution}

The generated JSON is parsed, and the hint corresponding to the requested level $\ell$ is extracted.

Why progressive levels. The three-level structure (minimal, medium, detailed) maps directly to the scheduler's escalation logic. Level 1 provides a high-level pointer to the key concept or approach ("Rewrite the base-b numerals as ordinary integers in terms of b, then turn the divisibility condition into a statement about a simple linear expression"). Level 2 gives more direction on the intermediate method ("Convert the expressions to the form Ab + C. If (b+7) divides (9b+7), it also divides any linear combination that cancels the b-term"). Level 3 provides substantial guidance while still requiring completion ("Compute 17b = b+7 and 97b = 9b+7. Subtract a multiple of (b+7) from (9b+7) to eliminate b. The condition becomes '(b+7) divides a constant'. Enumerate divisors and keep only b > 9").

The paper sets $L = 3$ in all experiments, meaning there are four possible hint levels: $\ell = 0$ (no hint), $\ell = 1$ (minimal), $\ell = 2$ (medium), $\ell = 3$ (detailed). The graded escalation ensures the model first tries to solve the problem with no help, then with increasingly specific guidance, and only uses the strongest hint when absolutely necessary.

Online vs. offline hint generation. The paper studies three variants of $q_\phi$:

1. Offline self-hinting: $q_\phi$ is derived from the initial policy $\pi_{\theta_0}$ and frozen before training begins. All hints are generated once in a preprocessing step. The hint generator never updates.

2. Online self-hinting (SAGE): $q_\phi$ is derived from the current policy $\pi_\theta$ and refreshed periodically during training. When the scheduler requires a hint for a prompt at level $\ell$, the current policy generates it on-the-fly. This ensures the hint distribution tracks the learner's evolving capabilities.

3. External teacher hints (Scaf-GRPO baseline): $q_\phi$ is produced by a stronger frozen model (GPT-5.2 in the paper's experiments). This provides high-quality hints but without calibration to the learner.

Figure 3 (left) evaluates all three variants on Qwen3-4B-Instruct using a set of 4.5k extremely hard prompts (those whose correct trajectories were never sampled during standard training, per Figure 2). At all hint levels ($\ell = 1, 2, 3$), online self-hinting consistently achieves the highest accuracy. At $\ell = 2$, online self-hinting reaches 58.3% average accuracy, compared to 55.4% for offline self-hinting and 56.5% for GPT-5 hints. The gap between online and offline is particularly informative: increased hint diversity alone (ablation in Figure 5, "offline with more hints") only partially closes the gap (+0.9 over standard offline), suggesting that calibration to the learner's current capabilities matters more than hint diversity. Online hints are better calibrated because they reflect what the current policy finds helpful, not what the initial policy found helpful.

The hint injection format. During RL training, the hint is appended to the prompt using a simple template:

{problem}
Here is a hint to help you:
{hint}

The model then generates its solution starting from this augmented context. The system prompt for RL is simply: "Please reason step by step, and put your final answer within \boxed{}."

Why the hint is about process, not answer. Critically, SAGE's hints are "procedural"—they describe reasoning approaches ("convert to base-10", "subtract a multiple to cancel the b-term") rather than revealing the final answer. This is essential for two reasons. First, it means the verifier reward is genuinely earned—the model still must execute the reasoning to get the correct answer. Second, it means the policy learns transferable reasoning patterns, not just answer-copying. The case study in Appendix A illustrates this: with a Level 2 hint, the model is steered toward the key cancellation step $9b+7 - 9(b+7) = -56$, after which it must still enumerate divisors and sum bases—genuine problem-solving, not answer regurgitation.

Policy-Dependent Scheduling: When and How Strong to Hint

The scheduler is the component that decides, for each prompt at each training epoch, what hint strength level $\ell$ to use. This is the mechanism that creates SAGE's automatic curriculum—escalating hint strength only for prompts where the current policy fails, and never hinting on prompts the policy already handles.

The paper presents two scheduling schemes with different accuracy-vs-efficiency tradeoffs.

Scheme 1: SAGE-LIGHT (epoch-level accuracy threshold). This scheme uses aggregate accuracy statistics from the previous epoch to decide hint levels for the current epoch. It is "light" because it requires no additional rollouts beyond the standard training procedure.

Formally, let $\hat{p}_{t-1}(x)$ be the empirical success rate of prompt $x$ measured in epoch $t-1$, computed from the $G$ rollouts sampled during standard training (where $G$ is typically 8). Given a threshold $\alpha \in (0, 1)$ (set to $\alpha = 0.35$ in the paper's experiments), the hint level for prompt $x$ at epoch $t$ is:

$\ell_t(x) = \min\{\ell_{t-1}(x) + 1, L\} \quad \text{if } \hat{p}_{t-1}(x) < \alpha$

and $\ell_t(x) = \ell_{t-1}(x)$ otherwise. All prompts start at $\ell_0(x) = 0$ (no hint). The hint level never decreases—once escalated, it stays escalated.

What this computes: a simple threshold-based escalation rule. If the policy succeeded on a prompt less than 35% of the time in the previous epoch, the hint strength increases by one level for the current epoch. If the prompt already has a success rate above 35%, the hint level stays where it is. The min with $L = 3$ caps the maximum hint strength.

Why this form: the 35% threshold is chosen to detect prompts that are providing sparse signal—if fewer than roughly 1 in 3 rollouts succeeds, the gate-opening probability is low and the update signal is weak. Escalating by one level at a time (rather than jumping to $L$) creates a smooth curriculum where the model first tries to learn from minimal hints, then medium, then detailed. The never-decrease policy is a simplification for efficiency—in principle, hint levels could decrease as the model improves, but tracking this would require probing at lower hint levels, which SAGE-LIGHT avoids.

Limitations of SAGE-LIGHT. The epoch-level update means SAGE-LIGHT reacts slowly. If a prompt suddenly becomes harder mid-epoch (e.g., because the policy's distribution shifted), the scheduler won't respond until the next epoch. The $\alpha$ threshold is also a coarse hyperparameter—the optimal threshold likely depends on the model, the task, and the training stage, but the paper uses a fixed $\alpha = 0.35$ throughout.

Scheme 2: SAGE (group-degeneracy trigger). This scheme uses a more local and reactive trigger: it probes the current hint level at the start of each epoch with a small group of rollouts, and escalates only if that probe group is completely degenerate (all rewards zero). This directly targets GRPO's failure mode at the level of individual groups rather than aggregate statistics.

Formally, for each prompt $x$ at epoch $t$, SAGE samples a small probe group of $G$ rollouts from the policy with the current hint level $\ell_{t-1}(x)$:

$\tilde{\tau}_i \sim \pi_\theta(\cdot \mid x, \tilde{h}), \quad \tilde{h} \sim q_\phi(\cdot \mid x, \tau^\star, \ell_{t-1}(x))$

The group degeneracy indicator is:

$z(x) = \mathbb{I}\left[\sum_{i=1}^G R(x, \tilde{\tau}_i) = 0\right]$

where $\mathbb{I}[\cdot]$ is the indicator function (1 if true, 0 if false). If $z(x) = 1$ (the probe group contains no positive rollouts), the hint level escalates:

$\ell_t(x) = \min\{\ell_{t-1}(x) + 1, L\} \quad \text{if } z(x) = 1$

and $\ell_t(x) = \ell_{t-1}(x)$ otherwise. The hint $h$ used for the actual training rollouts is then sampled at the (possibly escalated) level $\ell_t(x)$. Algorithm 1 in the paper integrates this into the training loop: for each prompt, the algorithm tries $\ell = 0, 1, \ldots, L$ sequentially, at each level sampling a hint and probe rollouts, and breaks at the first level where the probe group contains at least one success ($\sum_i \tilde{R}_i > 0$) or when it reaches the maximum level $L$.

What this computes: a direct test for whether the current hint level opens the GRPO gate. If a probe group of size $G$ contains any positive rollout, the gate has a non-zero probability of opening during training, so the current hint level is sufficient. If the probe group is all-zero, the gate would be closed during training, so the hint strength must increase.

Why this form: the no-positives trigger directly targets the finite-sample pathology identified in Section 3.1. Rather than relying on aggregate accuracy (which may be a noisy estimator of gate-opening probability, especially when $p_\theta$ is very small), SAGE tests the actual condition that matters: can the policy at this hint level produce at least one correct trajectory in a finite group? The sequential probing at $\ell = 0, 1, \ldots, L$ ensures the minimum necessary hint strength is used—if the policy can already solve the prompt at $\ell = 0$, no hint is ever sampled. The computational cost is the probe rollouts, which are additional to the training rollouts.

Comparison of the two schemes. SAGE-LIGHT is compute-efficient because it reuses training rollouts for accuracy estimation and only updates hint levels once per epoch. The paper reports that SAGE-LIGHT requires 53% of SAGE's training time (Table 3: 1.2× GRPO's time for SAGE-LIGHT vs. 2.3× for SAGE on Qwen2.5-7B-Instruct). However, SAGE-LIGHT's epoch-level granularity means it can miss transient collapses or escalate too slowly. SAGE is more reactive and directly tests the gate-opening condition, but incurs additional probe rollout computation, especially when the policy is weak (probing at $\ell = 0, 1, 2$ before finding a working level). The paper reports results for both schemes, with SAGE consistently achieving higher accuracy (e.g., +0.9 over SAGE-LIGHT on Llama-3.2, +0.4 on Qwen2.5, +1.1 on Qwen3 in Table 1) at higher computational cost.

The adaptive curriculum in action. Figure 6 visualizes the scheduler's behavior on Llama-3.2-3B-Instruct over 500 training steps. The number of prompts using hints decreases over time: at step 0, many prompts require hints (especially at levels $\ell = 1$ and $\ell = 2$); by step 500, the "use hint" count drops significantly. This is the curriculum at work—as the policy improves, more prompts become solvable without hints, and the scheduler automatically stops escalating (or stays at $\ell = 0$). The three hint levels ($\ell = 1, 2, 3$) show a distribution: level 1 hints are most commonly used (sufficient for mildly hard prompts), level 2 hints are used for moderately hard prompts, and level 3 hints are used sparingly for the hardest prompts. This distribution shifts toward lower levels as training progresses.

Why never-decrease works (despite being suboptimal in theory). Both schemes use a never-decrease policy for hint levels—once escalated to $\ell$, the hint level never returns to a lower value. In principle, as the policy improves, a lower hint level might suffice (and would be preferable, since weaker hints provide a harder learning signal closer to the deployable $\ell = 0$ setting). The paper doesn't explicitly justify the never-decrease choice, but it can be understood as a practical simplification: probing whether a lower hint level would work requires additional rollouts at that level, increasing computational cost. Since the policy improves monotonically (or close to it) on most prompts, the hint level needed typically doesn't decrease—the model either learns to solve the prompt at the current hint level or continues to need it. The never-decrease policy also prevents oscillation where the scheduler repeatedly escalates and de-escalates as the policy's success rate fluctuates near the threshold.

The Complete SAGE Training Algorithm

Algorithm 1 in the paper provides the full pseudocode for SAGE training. Here we walk through the algorithm step by step, explaining what each component does and why it's arranged this way.

Initialization. The algorithm starts with:

• A training dataset $\mathcal{D} = \{(x, \tau^\star)\}$ of (prompt, reference solution) pairs (15k prompts subsampled from OpenR1-Math-220k, with DeepSeek-R1 traces verified by Math-Verify and filtered to <8192 tokens).
• A policy model $\pi_\theta$ with initial parameters $\theta$ (Llama-3.2-3B-Instruct, Qwen2.5-7B-Instruct, or Qwen3-4B-Instruct-2507, depending on the experiment).
• A per-prompt hint level map $\ell(x) \leftarrow 0$ for all $x \in \mathcal{D}$ (initially, no prompts use hints).
• A hint generator $q_\phi$ based on $\pi_\theta$ (initially the base model).
• Hyperparameters: group size $G = 8$ (or $G = 4$ for Qwen3-4B due to slower training from long response lengths), KL weight $\beta = 0$ (following DAPO), stabilizer $\epsilon > 0$, maximum hint level $L = 3$, threshold $\alpha = 0.35$ (SAGE-LIGHT only).

Epoch loop. Training proceeds in epochs. Within each epoch:

Step 1: Hint level selection (SAGE-LIGHT variant). For SAGE-LIGHT, the hint level is updated once per epoch, before any minibatches are processed. For each prompt $x$, if $\text{epoch} > 1$ (not the first epoch) and the previous epoch's empirical success rate $\bar{R}$ (averaged over the $G$ training rollouts) falls below $\alpha = 0.35$, the hint level increments: $\ell(x) \leftarrow \min\{\ell(x) + 1, L\}$. This step requires no additional computation—it reuses the training rewards from the previous epoch.

Step 2: Hint level selection (SAGE variant). For SAGE, hint levels are determined per-minibatch, not per-epoch. For each prompt $x_b$ in the current minibatch, the algorithm iterates over possible hint levels $\ell = 0, 1, \ldots, L$:

• Sample a hint $\tilde{h}_b \sim q_\phi(\cdot \mid x_b, \tau^\star_b, \ell)$.
• Sample $G$ probe rollouts $\tilde{\tau}_{b,i} \sim \pi_\theta(\cdot \mid x_b, \tilde{h}_b)$.
• Compute rewards $\tilde{R}_{b,i} = R(x_b, \tilde{\tau}_{b,i})$ (via the verifier).
• If $\sum_i \tilde{R}_{b,i} > 0$ (at least one success) or $\ell = L$ (maximum level reached), accept this hint and break: $h_b \leftarrow \tilde{h}_b$, $\tau_{b,i} \leftarrow \tilde{\tau}_{b,i}$, $R_{b,i} \leftarrow \tilde{R}_{b,i}$, $\ell(x_b) \leftarrow \ell$.
• Otherwise, continue to the next hint level.

This probing starts at $\ell = 0$ (no hint), so if the policy can already solve the prompt without hints, no escalation occurs and no hint is ever sampled. The probe rollouts at the accepted level become the training rollouts for that prompt in this step—no additional sampling is needed.

Step 3: Hint sampling (SAGE-LIGHT variant). For SAGE-LIGHT, after the hint level is determined at epoch start, hints are sampled once per prompt: $h_b \sim q_\phi(\cdot \mid x_b, \tau^\star_b, \ell(x_b))$.

Step 4: Training rollouts (SAGE-LIGHT variant). For SAGE-LIGHT, training rollouts are sampled separately from the hint level selection (unlike SAGE, which reuses probe rollouts): $\tau_{b,i} \sim \pi_\theta(\cdot \mid x_b, h_b)$ for $i = 1, \ldots, G$.

Step 5: Advantage computation. For each prompt in the minibatch, compute:

• Mean reward: $\bar{R}_b = \frac{1}{G}\sum_{i=1}^G R_{b,i}$.
• Standard deviation: $s_b = \sqrt{\frac{1}{G}\sum_{i=1}^G (R_{b,i} - \bar{R}_b)^2}$.
• Standardized advantages: $A_{b,i} = \frac{R_{b,i} - \bar{R}_b}{s_b + \epsilon}$.

The stabilizer $\epsilon$ ensures the advantages are well-defined even when $s_b = 0$ (in which case $A_{b,i} = 0$ for all $i$, producing no update—this is the degenerate case SAGE aims to prevent).

Step 6: Policy gradient loss. The policy gradient loss is:

$\mathcal{L}_{\text{pg}} = -\frac{1}{B G}\sum_{b=1}^B \sum_{i=1}^G A_{b,i} \sum_{t=1}^{T_{b,i}} \log \pi_\theta(y_{b,i,t} \mid x_b, h_b, y_{b,i,<t})$

where $B$ is the minibatch size (128 in the paper's experiments, reduced to 64 for the PPO minibatch), $G$ is the group size, $T_{b,i}$ is the length of rollout $i$ for prompt $b$, and $y_{b,i,t}$ is the $t$-th token. The triple sum averages over minibatches, group members, and token positions. Each token's log-probability is weighted by the rollout-level advantage $A_{b,i}$—tokens in above-average rollouts ($A_{b,i} > 0$) get their probabilities increased; tokens in below-average rollouts ($A_{b,i} < 0$) get their probabilities decreased.

Step 7: KL penalty (optional). If $\beta > 0$, a KL penalty term is added:

$\mathcal{L}_{\text{kl}} = \frac{1}{B}\sum_{b=1}^B \text{KL}(\pi_\theta(\cdot \mid x_b, h_b) \parallel \pi_{\text{ref}}(\cdot \mid x_b, h_b))$

where $\pi_{\text{ref}}$ is the reference policy. Following DAPO, the paper sets $\beta = 0$ in all main experiments, relying on asymmetric clipping ($\epsilon_{\text{low}} = 0.2$, $\epsilon_{\text{high}} = 0.28$) for stabilization instead of KL regularization.

Step 8: Parameter update. The model parameters are updated via gradient descent:

$\theta \leftarrow \theta - \eta \nabla_\theta(\mathcal{L}_{\text{pg}} + \beta \mathcal{L}_{\text{kl}})$

where $\eta$ is the learning rate.

Step 9: Online hint generator refresh. Periodically (the paper doesn't specify the exact frequency, but Figure 3 implies refreshing occurs during training), the hint generator $q_\phi$ is updated using a copy of the current policy $\pi_\theta$. This ensures that hints generated in future epochs reflect the current policy's capabilities, not the initial policy's.

Deployment. At test time, $\ell$ is forced to $0$ for all prompts, meaning $h = \emptyset$. The model runs as $\pi_\theta(\cdot \mid x)$—a standard language model with no privileged information. All benchmark evaluations in Table 1 use this no-hint deployment.

Key hyperparameters and configurations (from the paper's Section 5 and Appendix C):

• Training set: 15k prompts subsampled from OpenR1-Math-220k, filtered to DeepSeek-R1 traces <8192 tokens and verified by Math-Verify.
• Group size: $G = 8$ (Llama-3.2, Qwen2.5), $G = 4$ (Qwen3, due to longer response lengths causing slower training).
• Total training steps: 500.
• Batch size: 128 prompts per step; PPO mini-batch size: 64.
• Maximum response length: 8096 tokens (main results; required by LUFFY baseline), 2048 tokens (remaining experiments).
• Asymmetric clipping: $\epsilon_{\text{low}} = 0.2$, $\epsilon_{\text{high}} = 0.28$.
• KL weight: $\beta = 0$ (disabled).
• Maximum hint level: $L = 3$.
• SAGE-LIGHT threshold: $\alpha = 0.35$.
• Evaluation: every 50 steps, temperature 0.6, top-p 0.95. Best average accuracy over all checkpoints reported.
• Hardware: 8 A100 GPUs; training framework: verl (Sheng et al., 2025) for training, vLLM (Kwon et al., 2023) for sampling.

Why the SAGE probing reuses rollouts but SAGE-LIGHT doesn't. In SAGE, the probe rollouts at the accepted hint level become the training rollouts—they are not discarded. This means SAGE's additional cost comes only from the rejected probe levels ($\ell$ that produced all-zero groups). If the policy can solve the prompt at $\ell = 0$, the cost is identical to GRPO. If it takes $\ell = 2$ before finding a positive rollout, the algorithm pays for probing at $\ell = 0$ and $\ell = 1$ (discarded) plus training at $\ell = 2$. In contrast, SAGE-LIGHT always pays for one set of training rollouts (at the scheduled $\ell$) with no probing overhead—but may use a suboptimal hint level because it can only update once per epoch.

Why Sample a Single Hint Per Prompt Per Epoch? The Jensen Inequality Insight

A non-obvious design choice in SAGE is that each prompt receives exactly one hint per epoch, and all $G$ training rollouts for that prompt share the same hint. Intuition might suggest that sampling multiple diverse hints per prompt would increase the chance of finding a useful hint—more variety, more likely to hit a hint that opens the gate. Remark 3.4 shows this intuition is mathematically wrong under the gate-opening objective.

The Jensen inequality argument. Recall the gate-opening probability for a given hint $h$: $u(p_\theta(x, h)) = 1 - (1-p_\theta(x, h))^G - p_\theta(x, h)^G$. For $G \geq 2$, the function $u(p)$ is strictly concave on $[0, 1]$:

$u''(p) = -G(G-1)\left(p^{G-2} + (1-p)^{G-2}\right) \leq 0$

where $u''(p)$ is the second derivative of $u$ with respect to $p$. A strictly concave function has the property that the expected value of the function is less than or equal to the function evaluated at the expected value of its argument—this is Jensen's inequality.

Applying this to the hint sampling problem: let $Z = p_\theta(x, h)$ be the random success probability induced by sampling $h \sim q(\cdot \mid x)$ from some distribution. Then:

$\mathbb{E}_{h \sim q}\left[u(p_\theta(x, h))\right] \leq u\left(\mathbb{E}_{h \sim q}[p_\theta(x, h)]\right)$

What this inequality means: the left side is the expected gate-opening probability when we sample one hint per prompt, draw $G$ rollouts conditioned on that hint, and check if the group has mixed outcomes—averaged over the hint distribution $q$. The right side is the gate-opening probability we would get if we could somehow use the average success probability over all hints as a single "effective" success probability.

Concavity of $u$ means that variability in $p_\theta(x, h)$ across different hints reduces the expected gate-opening rate, compared to using a single hint that achieves the average success probability. In operational terms: if you have a set of hints that collectively give an average success rate of 0.3, you'll open the gate more reliably by always using the hint that gives exactly 0.3 than by randomly picking hints that sometimes give 0.1 and sometimes 0.5. The all-0.1 hints keep the gate closed most of the time (because $G \cdot 0.1$ is small), dragging down the average even if some hints perform better.

Why this doesn't contradict the value of online hint refreshing. The Jensen argument says: at a given training step, given a fixed distribution over hints, sampling one hint and using it for all $G$ rollouts maximizes the expected gate-opening rate (compared to sampling different hints for different rollouts within the same group). It does not say that the hint distribution should be fixed across training. Online hint refreshing changes the hint distribution $q$ over time to track the policy's evolving $p_\theta$, which is a different mechanism—it's about shifting the mean of the distribution toward the calibrating regime, not about adding within-step variability.

Practical consequence. SAGE samples one hint realization per prompt per epoch (or per step, depending on the scheduler), and all $G$ rollouts for that prompt share that hint context. This design choice is enshrined in Algorithm 1, where for each prompt, a single $h$ is sampled and used for all $G$ training rollouts. This reduces unnecessary variance from hints while maximizing the probability that the resulting group opens the GRPO gate.

Connection to the gate-maximization objective. Proposition 3.3 shows that the optimal hint distribution $q^\star_\theta$ concentrates its mass on hints that achieve $p_\theta(x, h) \approx 1/2$. Remark 3.4 reinforces this by showing that, even if the optimal distribution must spread mass across multiple hints (because no single hint achieves exactly 1/2), concentrating the per-prompt sampling on a single hint is better than diluting across multiple hints within the same group. The per-prompt sampling is a design choice about how to use a given hint distribution; online refreshing is about improving the distribution itself.

4. Key Insights and Innovations

Innovation 1: GRPO's Finite-Sample Degeneracy Is a Gate-Opening Problem, Not a Variance Problem

The paper's most distinctive conceptual contribution is reframing GRPO's training difficulties on hard prompts as a gated update procedure rather than a high-variance gradient estimation problem. This diagnostic move is subtle but consequential.

Prior work on policy-gradient methods in RL has long understood that sparse rewards cause high variance. The standard remedies—larger batch sizes, better baselines, importance sampling corrections—all treat the problem as one of reducing estimator variance around a non-zero expected gradient. What SAGE identifies is qualitatively different: under GRPO's standardization with finite groups, the gradient isn't noisy—it is identically zero with probability approaching 1. This isn't high variance; it's a binary gate that remains closed most of the time.

The gate-opening probability $u(p) = 1 - (1-p)^G - p^G$ gives this idea mathematical precision (Proposition 3.2). When $G \cdot p_\theta(x) \ll 1$, the gate is almost always closed, and no amount of variance reduction helps—the estimator itself provides zero signal, not just noisy signal. This distinguishes the pathology from standard RL variance problems. A larger batch size in PPO would reduce noise; a larger group size in GRPO might eventually open the gate, but the required $G$ is impractically large (needing $G > 100$ for $p_\theta = 0.01$).

The symmetry of $u(p)$—peaking at $p = 1/2$, collapsing at both $p \approx 0$ and $p \approx 1$—is the key insight that drives SAGE's design. It reveals that the fix is not to make prompts easier (pushing $p \to 1$), which would close the gate again, but to make them appropriately challenging. This is a fundamental shift from prior work: LUFFY (Yan et al., 2025) injects correct trajectories to guarantee at least one positive reward (implicitly aiming for $p \approx 1$ in the mixed group), while Scaf-GRPO (Zhang et al., 2025b) uses powerful external hints that can make prompts trivially solvable. Both approaches risk overshooting the optimal gate-opening regime. SAGE's adaptive scheduler, by contrast, escalates hint strength only until the gate opens—explicitly targeting the concave maximum rather than monotonic improvement in success rate.

The theoretical framing in Section 3.1 (Corollary 3.1, Propositions 3.2 and 3.3, Remark 3.4) constitutes a self-contained diagnostic toolkit for understanding when and why GRPO stalls. The advantage energy $E = s^2/(s+\epsilon)^2$ provides a continuous, computable signal of gate status during training. Proposition 3.3's characterization of the optimal hint distribution as calibrating toward $p \approx 1/2$ gives a precise objective for hint generation that prior heuristic approaches lacked. Remark 3.4's Jensen inequality argument—that per-prompt hint variability reduces expected gate-opening rate—is a genuinely non-obvious result that contradicts the intuitive appeal of hint diversity, and is validated empirically (Figure 5: offline with multiple hints underperforms online self-hinting despite having more diverse hints).

This is a fundamental conceptual contribution, not an incremental refinement. It changes how one thinks about the problem: from "GRPO has high variance on hard prompts" (which suggests variance-reduction remedies) to "GRPO is a gated procedure whose gate closes on hard prompts" (which suggests interventions that open the gate). The diagnostic framework is reusable beyond SAGE—any future method addressing GRPO's finite-sample behavior can be analyzed through the lens of gate-opening probability.

Innovation 2: Privileged Hinting as On-Policy Distribution Shaping Without Reward Modification

SAGE introduces a clean separation between shaping the rollout distribution and modifying the reward function—a distinction that prior approaches to sparse-reward RL for LLMs blurred or violated entirely.

The dominant paradigm for dealing with sparse rewards in RL is reward shaping (Ng et al., 1999): add intermediate rewards to guide the agent toward the terminal goal. In LLM training, this manifests as process reward models (PRMs) that score intermediate reasoning steps, or as dense reward signals from learned verifiers. These approaches change the optimization landscape—the policy now optimizes a shaped reward that may not perfectly align with the true objective.

SAGE takes a fundamentally different path. The terminal verifier reward $R(x, \tau) \in \{0, 1\}$ is completely unchanged. A correct final answer still scores 1; an incorrect one still scores 0. The hint $h$ does not appear in the reward computation, does not provide partial credit, and does not alter the verifier's judgment. Instead, hints operate exclusively on the sampling side: they condition the policy to generate rollouts from $\pi_\theta(\cdot \mid x, h)$ rather than $\pi_\theta(\cdot \mid x)$, reshaping which trajectories are likely to appear in a finite group. The optimization objective (Equation 5) remains a standard GRPO policy gradient, just evaluated under an augmented context.

This matters because it preserves the integrity of the verifier signal. If hints leaked into the reward—for instance, by giving partial credit for following the hinted approach—the policy could learn to exploit the hint structure rather than genuinely solving problems. The case study in Appendix A illustrates the principle: even with a strong Level 3 hint providing the algebraic cancellation $9b+7 - 9(b+7) = -56$, the model still must correctly enumerate divisors and sum bases. The hint guides the approach but doesn't guarantee success; the verifier still demands a correct final answer.

The paper makes a second, equally important commitment: the on-policy contract must be maintained. This distinguishes SAGE from LUFFY (which injects off-policy trajectories from a different model into GRPO groups) and from Scaf-GRPO (which mixes hinted and non-hinted rollouts within a single group, breaking the assumption that all group members are drawn from the same distribution). SAGE keeps training strictly on-policy by conditioning rollouts on $(x, h)$ and evaluating log-probabilities under the same $(x, h)$ context. The ablation in Figure 5 of the main experiments—the off-policy variant that samples with hints but evaluates $\log \pi_\theta(\tau \mid x)$—shows a clear performance drop (56.5 vs. 59.2 for on-policy SAGE on the hard-prompt subset). This is not merely a theoretical nicety; the empirical gap demonstrates that maintaining the on-policy objective matters for learning stability and final performance.

The conceptual advance here is in recognizing that distribution shaping and reward shaping are independent axes of intervention, and that the former can address finite-sample degeneracies without the risks of the latter (reward hacking, misalignment between shaped and true objectives). Prior work in LLM training—curriculum learning, adaptive sampling, rejection sampling—shapes distributions but rarely with the explicit goal of opening a gate for policy-gradient estimation while preserving the terminal reward and on-policy learning. SAGE's synthesis of these constraints into a coherent framework is a genuine methodological contribution.

Innovation 3: Policy-Dependent Hint Scheduling Creates an Automatic, Difficulty-Tracking Curriculum

SAGE's hint scheduler is not merely a heuristic for deciding when to use hints—it embodies a principled approach to difficulty tracking that maintains the policy at the edge of its current capabilities. This contrasts with both fixed curricula (where difficulty progression is predetermined) and purely data-driven curricula (which select prompts based on loss or success rate without modifying the prompts themselves).

The insight is that the scheduler's escalation logic—increasing hint strength only when the current level fails to produce a positive rollout—creates a per-prompt difficulty controller that automatically adjusts to the learner's evolving capabilities. Unlike prior curriculum methods that select which prompts to train on (Xiong et al., 2025b; Li et al., 2025; Zhang et al., 2025c), SAGE modifies how each prompt is presented. Hard prompts aren't discarded or deferred; they're made temporarily easier through hinting, then progressively stripped of hints as the policy improves.

Figure 6 provides the empirical signature of this curriculum: the number of prompts requiring hints decreases monotonically over training. At step 0, many prompts need hints at levels $\ell = 1$ and $\ell = 2$; by step 500, the hint usage has dropped substantially. This isn't because hints are being abandoned—it's because the policy is learning to solve prompts without them. The scheduler's never-decrease policy (hint levels never go down) means the decreasing hint usage reflects genuine capability improvement: prompts that previously required hints at step 200 may still be at $\ell = 1$ at step 500, but the scheduler never escalates because the gate now opens at that level.

The comparison between SAGE and SAGE-LIGHT (Table 1, Table 3) reveals an accuracy-vs-efficiency tradeoff that is itself an interesting finding. SAGE-LIGHT (epoch-level accuracy threshold, $\alpha = 0.35$) is significantly faster (1.2× GRPO's training time vs. 2.3× for SAGE on Qwen2.5-7B-Instruct) but achieves lower accuracy (+1.0 over GRPO for Qwen2.5 vs. +4.5 for SAGE). The gap suggests that the per-group degeneracy trigger captures something that epoch-level accuracy misses—likely the stochastic nature of gate-opening on very hard prompts, where even a modest average success rate can mask frequent all-zero groups.

The prior work comparison is instructive. Adaptive sampling methods (Yao et al., 2025) allocate more rollouts to hard prompts but don't change the prompts themselves—if $p_\theta$ is too small, more samples from the same distribution may still fail to open the gate. LUFFY and Scaf-GRPO modify prompts (by injecting correct trajectories or external hints) but use fixed intervention strategies that don't adapt to the learner. SAGE combines prompt modification (via hints) with policy-dependent adaptation (via the scheduler), creating a feedback loop where the intervention strength tracks the policy's current needs. This is an incremental advance over fixed-curriculum or fixed-intervention approaches, but a significant one—it removes the need to tune per-prompt difficulty schedules, which would be impractical at scale.

Innovation 4: Hints Generated by the Learner Outperform Hints from Stronger Teachers

A counterintuitive empirical finding with significant practical implications: self-generated hints from the current policy outperform hints from a much stronger external model (GPT-5.2 in the Scaf-GRPO baseline).

Figure 3 (left) establishes this clearly on Qwen3-4B-Instruct trained on the hardest 4.5k prompts. Across all hint levels ($\ell = 1, 2, 3$), online self-hinting achieves the highest accuracy: 56.7 vs. 55.9 vs. 55.6 for online, GPT-5, and offline self-hinting at $\ell = 1$; 58.3 vs. 56.5 vs. 55.4 at $\ell = 2$; 57.1 vs. 56.3 vs. 57.0 at $\ell = 3$. The gap between online and GPT-5 hints is small but consistent, and the online variant's advantage grows with extended training (Figure 3, right: online self-hinting at $\ell = 2$ shows steady improvement over 400 steps while GPT-5 hints plateau).

This is surprising because GPT-5.2 is vastly more capable than a 4B-parameter model. One would expect its hints to be more instructive, better structured, and more likely to guide the learner toward correct solutions. The paper's explanation—that online hints are better calibrated to the learner's current capabilities—has deeper implications than it might first appear.

The calibration argument is that hints must match the learner's "zone of proximal development": too cryptic and they don't help (the gate stays closed); too revealing and they make the task trivial (the gate closes again because all rollouts succeed). A stronger teacher model, even when prompted to generate progressive hints, produces hints calibrated to its own understanding of what constitutes a minimal, medium, or detailed hint. These may not align with what a 4B-parameter learner finds helpful. An online self-hint generator shares the learner's "conceptual vocabulary"—its hints reference reasoning patterns and intermediate representations that the learner can actually produce, while GPT-5's hints may reference concepts or strategies that are natural to GPT-5 but alien to the learner.

The ablation with "offline self-hinting with more hints" (Figure 5) teases apart calibration from diversity. Generating 10 diverse hints offline (to match the variety that online generation provides) improves over single-hint offline (+0.9), but still underperforms online by 2.0 points. This suggests that diversity helps—consistent with the idea that different prompts benefit from different hint styles—but that calibration to the current policy matters more. As training progresses, the policy's capabilities shift, and the offline hint generator (frozen at initialization) becomes increasingly miscalibrated.

This finding has practical consequences for the growing ecosystem of LLM training pipelines that rely on stronger teacher models for data generation. Scaf-GRPO's approach—using GPT-5.2 as an external hint generator—is natural and appealing, especially when a capable teacher is available. SAGE's results suggest it may be strictly better to use the learner itself as the hint generator, refreshed online. This eliminates dependency on external models, simplifies the training pipeline (no separate teacher inference), and produces better results. The implication extends beyond SAGE: any method that uses teacher-generated guidance during RL training should consider whether self-generated, online-refreshed guidance might be more effective.

This is an empirical finding with theoretical grounding (Proposition 3.3 formalizes why fixed hint distributions drift from optimality), but its significance is primarily practical—it changes what practitioners should do. The performance gap is not enormous (1–2 points on the hard-prompt subset), but the direction is consistent and the elimination of external model dependency is a substantial engineering simplification.

5. Experimental Analysis

Evaluation Methodology

The paper evaluates SAGE primarily on mathematical reasoning, using three LLM families across six in-distribution benchmarks and two out-of-distribution benchmarks to assess generalization. All methods share the same training data, base models, and evaluation protocol, with the only variation being whether and how privileged hints are used during RL training.

• Dataset. Training data is drawn from OpenR1-Math-220k (Hugging Face, 2025), which uses prompts from NuminaMath 1.5 (Li et al., 2024) with reasoning traces generated by DeepSeek-R1 (DeepSeek-AI et al., 2025). The initial dataset contains 94k prompts. After filtering with Math-Verify (Kydlíček) to remove prompts whose DeepSeek-R1 traces are incorrectly verified, 64k prompts remain. Due to resource constraints and a requirement from the LUFFY baseline, the authors further subsample 15k prompts from this set, restricting corresponding DeepSeek-R1 traces to fewer than 8,192 tokens. No filtering by pass rate is applied, so the resulting 15k prompts span a wide range of difficulty levels. The 64k prompt set is used only for Figure 2 (the analysis of degenerate prompts); all training and evaluation uses the 15k subset.

• Base models. Three LLMs with varying degrees of math specialization are used: Llama-3.2-3B-Instruct (Meta, 2024), representing a general-purpose model with limited math capability; Qwen2.5-7B-Instruct (Yang et al., 2024), representing moderate capability with some math-focused optimization; and Qwen3-4B-Instruct-2507 (Yang et al., 2025), representing high math capability trained extensively via RL. This span allows the paper to test whether SAGE's benefits depend on base model strength. The Qwen3-4B model is notably stronger—its base accuracy on MATH-500 is 93.6% compared to 44.7% for Llama-3.2 (Table 1, "Base" row)—meaning the 15k training prompts that are hard for Llama-3.2 (~57% with no correct trajectories ever sampled, per Figure 2) are substantially easier for Qwen3 (~22%).

• Metrics. The primary metric is accuracy on each benchmark, computed as the fraction of test problems for which the model's final answer matches the ground truth. Answers are extracted from the \boxed{} format and compared via exact match or mathematical equivalence (the paper uses Math-Verify for verification during training, implying the same tool for evaluation). For in-distribution benchmarks (AIME24, AIME25, AMC23, MATH-500, Minerva Math, OlympiadBench), accuracy is reported individually and averaged. For out-of-distribution benchmarks (GPQA-diamond, MMLU-Pro), accuracy is reported individually and averaged as a separate group. The paper also reports training dynamics: training rewards (mean reward per step, though with the caveat that adding hints changes prompt difficulty, making absolute reward values non-comparable across methods), response length (average number of tokens per rollout, plotted over training steps as a proxy for reasoning depth), and entropy (average token-level entropy of the policy distribution, as a measure of exploration).

• Baselines. Four baselines are compared:

  • Base LLM: the pretrained model without any fine-tuning or RL, evaluated directly.
  • SFT (Supervised Fine-Tuning): the base model fine-tuned on reasoning traces from DeepSeek-R1 using OpenRLHF (Hu et al., 2024) with learning rate 5e-5, batch size 64, 10% warmup ratio, and 3 epochs.
  • GRPO (Shao et al., 2024): standard Group Relative Policy Optimization without any hints, sharing the same hyperparameters as SAGE (batch size 128, 8 trajectories per prompt, 500 training steps, asymmetric clipping with ε_low=0.2 and ε_high=0.28, β=0 following DAPO).
  • LUFFY (Yan et al., 2025): replaces one on-policy trajectory in each group with the corresponding correct trajectory from DeepSeek-R1. Reproduced using the open-source implementation with batch size 128 and PPO mini-batch size 64.
  • Scaf-GRPO (Zhang et al., 2025b): incorporates hints generated by GPT-5.2 under a low-reasoning-effort setting. Reproduced using the open-source implementation. SFT, LUFFY, and Scaf-GRPO all rely on a stronger external LLM (DeepSeek-R1 or GPT-5.2), whereas SAGE learns only from self-generated hints. SAGE-LIGHT is included as a compute-efficient SAGE variant using epoch-level accuracy thresholds (α = 0.35).

• Generation budget / compute accounting. Compute is measured primarily through training time (wall-clock hours on 8 A100 GPUs, reported in Table 3 for Qwen2.5-7B-Instruct) rather than FLOPs. GRPO serves as the baseline at 1.0× (25.3 hours). LUFFY requires 1.2×, Scaf-GRPO 1.5×, SAGE-LIGHT 1.2×, and SAGE 2.3×. The paper acknowledges this latency as a limitation (Section 5.2). Within each method, the generation budget is controlled by keeping batch size (128), number of trajectories per prompt (G=8 for Llama-3.2 and Qwen2.5, G=4 for Qwen3), and total training steps (500) identical. For SAGE, the additional cost comes from probe rollouts during hint level selection—if a prompt requires probing at multiple hint levels before finding one that opens the gate, the discarded probe rollouts represent overhead. SAGE-LIGHT avoids this overhead by updating hint levels once per epoch using existing training statistics, requiring no additional rollouts.

• Cross-validation / statistical protocol. The paper does not use cross-validation or report confidence intervals. Instead, it evaluates every 50 training steps over 500 total steps and reports the best average accuracy over all checkpoints for each method-benchmark combination. This is a standard protocol in RL-for-LLMs papers (e.g., DeepSeek-R1, DAPO) but means that the reported numbers represent peak performance rather than final convergence, and that checkpoint selection is based on test-set performance (an implicit form of test-set optimization). No statistical significance testing is reported. For the hard-prompt subset analysis (Figure 3, Table C.1), the paper trains for 200 steps by default and 400 steps for methods marked with an asterisk; the best checkpoint is selected from this extended run.

Main Quantitative Results

Aggregate performance across benchmarks and models (Table 1)

Table 1 reports accuracy for all methods on all eight benchmarks across three base models. The headline result is that SAGE achieves the highest average accuracy in every comparison group:

• Llama-3.2-3B-Instruct: SAGE average 23.9% on in-distribution benchmarks (+6.1 over the base model's 17.8%), 34.0% on out-of-distribution (+11.5 over 22.5%). SAGE-LIGHT achieves 23.0% (+5.2) and 33.2% (+10.7). GRPO achieves 21.9% (+4.1) and 33.1% (+10.6). SFT, LUFFY, and Scaf-GRPO all underperform the base model on average (SFT: 8.2% in-distribution, -9.6; LUFFY: 14.7%, -3.1; Scaf-GRPO: 21.5%, +3.7).

• Qwen2.5-7B-Instruct: SAGE average 42.3% on in-distribution (+4.5 over 37.8%), 48.6% on out-of-distribution (+1.9 over 46.7%). SAGE-LIGHT achieves 41.9% (+4.1) and 47.7% (+1.0). GRPO achieves 41.1% (+3.3) and 47.4% (+0.7). SFT performs substantially worse (23.2%, -14.6 in-distribution). LUFFY (41.7%, +3.9) and Scaf-GRPO (41.0%, +2.2) are competitive with but below SAGE.

• Qwen3-4B-Instruct: SAGE average 70.0% on in-distribution (+4.2 over 65.8%), 65.2% on out-of-distribution (+0.9 over 64.3%). SAGE-LIGHT achieves 68.9% (+3.1) and 64.5% (+0.2). GRPO achieves 68.7% (+2.9) and 64.5% (+0.2). SFT again performs worst (41.5%, -24.3). LUFFY (60.6%, -5.2) underperforms the base model on in-distribution tasks, while Scaf-GRPO (68.5%, +2.7) is comparable to GRPO.

Several patterns emerge from Table 1:

The relative benefit of SAGE over GRPO decreases as base model capability increases. The in-distribution improvement is +2.0 for Llama-3.2 (17.8% → 23.9%), +4.5 for Qwen2.5 (37.8% → 42.3%), and +4.2 for Qwen3 (65.8% → 70.0%). The absolute gain for Qwen3 is comparable to Qwen2.5, but the relative gain over GRPO narrows: +6.1 vs. +4.1 for Llama (a 2-point margin), +4.5 vs. +3.3 for Qwen2.5 (a 1.2-point margin), +4.2 vs. +2.9 for Qwen3 (a 1.3-point margin). This aligns with the mechanism: SAGE helps most on hard prompts where GRPO stalls, and stronger models have fewer such prompts. Table 2 confirms this: the percentage of prompts that never produce a correct trajectory drops from 40.2% (GRPO on Llama-3.2) to 10.3% (Qwen2.5) to 1.3% (Qwen3). SAGE reduces these to 30.0%, 8.2%, and 1.0% respectively—a larger absolute reduction on weaker models.

SAGE-LIGHT consistently underperforms SAGE but by a small margin, while requiring 53% of SAGE's training time (Table 3). The gap is +0.9 for Llama-3.2 (23.9% vs. 23.0% in-distribution), +0.4 for Qwen2.5 (42.3% vs. 41.9%), and +1.1 for Qwen3 (70.0% vs. 68.9%). SAGE-LIGHT remains above all baselines for Llama-3.2 and Qwen2.5, and ties or exceeds GRPO for Qwen3.

Out-of-distribution results mirror in-distribution trends, with SAGE achieving the best average on GPQA and MMLU-Pro across all models. The gains are particularly large for Llama-3.2 (+11.5 over base, +0.8 over GRPO) and more modest for Qwen3 (+0.9 over base, +0.7 over GRPO). This suggests that the reasoning patterns learned through privileged hinting transfer to non-mathematical tasks, though the paper does not analyze why—the relationship between mathematical reasoning and GPQA/MMLU-Pro performance is not explored.

SFT performs worst across all models and benchmarks. The paper attributes this to overfitting: SFT memorizes the DeepSeek-R1 reasoning traces without the exploration and self-correction that RL provides. The drop is catastrophic for Qwen3 (-24.3 in-distribution, from 65.8% to 41.5%), suggesting that for a model already strong at math, forcing it to imitate a different model's reasoning patterns is actively harmful—it overwrites the model's own effective strategies with off-policy patterns that it cannot generalize.

LUFFY shows unstable, model-dependent behavior. It outperforms GRPO on Qwen2.5 (+3.9 vs. +3.3) but underperforms the base model on both Llama-3.2 (14.7% vs. 17.8%) and Qwen3 (60.6% vs. 65.8%). The paper attributes this to off-policy distribution mismatch—the DeepSeek-R1 trajectories that LUFFY injects may align with Qwen2.5's reasoning style but clash with Llama-3.2's and Qwen3's. For Qwen3 specifically, the model already has strong math RL training; injecting a different model's reasoning patterns appears to interfere with its existing capabilities rather than augmenting them.

Scaf-GRPO performs competitively but below SAGE. On Llama-3.2, Scaf-GRPO achieves 21.5% in-distribution (+3.7 over base) compared to SAGE's 23.9% (+6.1). On Qwen2.5, 41.0% (+2.2) vs. SAGE's 42.3% (+4.5). On Qwen3, 68.5% (+2.7) vs. SAGE's 70.0% (+4.2). The consistent gap of 1.5-2.5 points, combined with Figure 4 showing Scaf-GRPO's lower entropy, supports the paper's claim that external hints from a stronger model overly constrain exploration compared to self-generated, online-refreshed hints.

Training dynamics analysis (Figure 4, Figure 2, Table 2)

Figure 4 plots training rewards, response length, and entropy over 500 steps for GRPO, LUFFY, Scaf-GRPO, and SAGE across all three models. These curves reveal qualitative differences in how each method trains, beyond the final accuracy numbers.

Training rewards: The paper cautions that training reward values are not directly comparable across methods because adding hints (SAGE, Scaf-GRPO) makes prompts easier, inflating rewards, and LUFFY injects correct off-policy trajectories that guarantee positive rewards. The trend is what matters. SAGE shows stable, monotonic reward growth for Llama-3.2 and Qwen2.5. LUFFY shows severe instability: for Llama-3.2, rewards oscillate; for Qwen3, rewards start very low and recover slowly (the paper attributes this to distribution mismatch between the policy model and DeepSeek-R1). Scaf-GRPO shows moderate growth but with lower final rewards than SAGE for Llama-3.2 and Qwen2.5.

Response length: SAGE exhibits faster response-length growth than GRPO for both Llama-3.2 and Qwen2.5, which the paper interprets as learning from hard prompts that fail to provide signal under GRPO—the model is developing longer, more detailed reasoning chains. LUFFY shows dramatically faster length growth early in training, reflecting imitation of the (long) DeepSeek-R1 traces, but this is paired with instability (oscillatory lengths for Llama-3.2). For Qwen3, response lengths are already high at initialization and grow modestly for all methods.

Entropy: Scaf-GRPO exhibits the lowest entropy among all methods, consistent with the paper's claim that external hints overly constrain exploration. LUFFY shows the highest entropy for Llama-3.2, indicating the policy is uncertain and struggling to reconcile its own distribution with the injected off-policy trajectories. SAGE maintains entropy comparable to GRPO for Qwen2.5 and Qwen3, and somewhat lower (but not as low as Scaf-GRPO) for Llama-3.2, suggesting it preserves exploration while providing useful guidance.

Dead prompt analysis (Table 2, Figure 2). Table 2 quantifies the percentage of the 15k training prompts that never yield a single correct trajectory during the entire training procedure:

The reduction from GRPO to SAGE represents prompts that were "rescued" by privileged hinting—prompts that would have contributed zero gradient throughout training but instead provided learning signal because hinting increased the probability of sampling a correct trajectory. The effect is largest for Llama-3.2 (-10.2 percentage points), moderate for Qwen2.5 (-2.1), and small for Qwen3 (-0.3). This directly validates SAGE's mechanism: it makes hard prompts learnable, and the number of such prompts depends on how many hard prompts exist in the training data for a given model.

Hard-prompt subset analysis (Figure 3, Table C.1). The paper further isolates SAGE's effect by evaluating on a subset of 4.5k extremely hard prompts—those whose correct trajectories were never sampled during standard training (per Figure 2). On this subset, with group size increased to G=32 (to encourage exploration) and training extended to 400 steps for the online self-hinting variant at ℓ=2:

• No hint (GRPO): 54.1% average accuracy across six benchmarks (Table C.1). Training without hints only slightly improves over the base model's 53.3%, and Figure 3 (right) shows performance degrading over training steps, consistent with the model overfitting to the few solvable prompts while receiving no signal from the rest.

• Online self-hinting (SAGE) at ℓ=2: 58.3% average accuracy, with steady improvement over 400 steps (Figure 3, right).

• GPT-5 hints at ℓ=2: 56.5%.

• Offline self-hinting at ℓ=2: 55.4%.

• Online self-hinting at ℓ=1: 56.7%.

• Online self-hinting at ℓ=3: 57.1%.

The 4.2-point gap between online self-hinting (58.3%) and no hint (54.1%) on this hard-prompt subset is larger than SAGE's overall improvement on the full 15k-prompt set, confirming that SAGE's benefits concentrate on the hardest prompts—exactly where GRPO's finite-sample degeneracy is most severe. The ordering of hint types (online > GPT-5 > offline) is consistent across ℓ=1, 2; at ℓ=3, offline (57.0%) narrowly edges online (57.1%), but the difference is within noise.

SAGE vs. constant hint level (Figure 5). SAGE's adaptive scheduler (escalating hint strength only when the current level fails) is compared against using a fixed hint level throughout training. On Qwen3-4B-Instruct with the hard-prompt subset: online ℓ=2 (fixed) achieves 58.3%, while SAGE (adaptive) achieves 59.2% (+0.9). This demonstrates that adaptive escalation—starting at ℓ=0 and only increasing when needed—is better than always using a medium-strength hint. The improvement is modest (~1 point) but directionally consistent with the theory: some prompts don't need hints at all, some need only ℓ=1, and always using ℓ=2 may overshoot for prompts that would learn fine with weaker hints.

Hint usage decreases during training (Figure 6). Figure 6 plots the number of prompts using hints at each training step on Llama-3.2-3B-Instruct. The "use hint" count (dark blue bars) decreases from a high initial value to near zero by step 500. Within the hinted prompts, level ℓ=1 (light blue) is most common, ℓ=2 (orange) is less common, and ℓ=3 (green) is rare. This decreasing trend is the empirical signature of SAGE's curriculum: as the policy improves, more prompts become solvable without hints, and the scheduler stops escalating (or stays at the minimum effective level for prompts that still need hints). The paper does not provide analogous plots for Qwen2.5 or Qwen3, which would likely show faster decay (since those models start with fewer dead prompts).

Ablation Studies and Robustness Checks

Online vs. offline vs. external hint generation (Figure 3, Figure 5): Online self-hinting consistently outperforms both offline self-hinting and GPT-5 hints across all hint levels on the hard-prompt subset (Figure 3 left). The ablation with "offline with more hints" (Figure 5) uses 10 diverse hints per prompt generated offline at temperature=1.0, with a different hint used at each training step. This improves over standard offline self-hinting (+0.9, from 55.4% to 56.3%) but still underperforms online self-hinting by 2.0 points, indicating that calibration to the current policy matters more than hint diversity. The paper argues this is because online hints track the learner's evolving capabilities—what constitutes a helpful hint at step 0 may be too cryptic or too revealing at step 400.

On-policy vs. off-policy conditioning (Figure 5): An off-policy variant samples rollouts with the hint (τ ~ π_θ(· | x, h)) but evaluates log-probabilities without the hint (log π_θ(τ | x)). This achieves 56.5% on the hard-prompt subset with Qwen3-4B, compared to 58.3% for online ℓ=2 (on-policy) and 59.2% for SAGE (adaptive, on-policy). The 2.7-point gap confirms that maintaining the on-policy contract—matching the sampling context to the optimization context—is important for learning stability. The off-policy variant even underperforms the fixed-level online baseline (58.3%), suggesting the mismatch creates a systematic disadvantage rather than just adding noise.

SAGE (per-group degeneracy trigger) vs. SAGE-LIGHT (epoch-level accuracy threshold) (Table 1, Table 3): SAGE achieves higher accuracy across all models (+0.9 on Llama-3.2, +0.4 on Qwen2.5, +1.1 on Qwen3 in-distribution averages) but at 2.3× the training time of GRPO, compared to 1.2× for SAGE-LIGHT. The accuracy gap is modest, especially for Qwen2.5 where SAGE-LIGHT (41.9%) is only 0.4 points behind SAGE (42.3%). This suggests that for models with moderate base capability, the simpler epoch-level scheduling is nearly as effective as the more expensive per-group probing, and the additional cost of SAGE's probe rollouts may not be justified. For Llama-3.2 and Qwen3, the gap is larger, suggesting the per-group trigger provides more benefit at the extremes (very weak or very strong base models).

Effect of asymmetric clipping and KL penalty: The paper follows DAPO (Yu et al., 2025) in setting β=0 (no KL penalty) and using asymmetric clipping with ε_low=0.2, ε_high=0.28. No ablation on these choices is reported, nor on the DAPO-specific modifications (e.g., filtering prompts with all-zero or all-one rewards, which DAPO does but SAGE does not need to, since hinting prevents all-zero groups). The paper does not compare against DAPO as a baseline, which would be a natural comparison point given the shared hyperparameters. This is a missing ablation: it is unclear whether SAGE's gains over GRPO are partly attributable to the DAPO-style clipping and KL=0 recipe, or whether standard GRPO with KL penalty would show different behavior.

Group size sensitivity (G=8 vs. G=4): Qwen3-4B uses G=4 instead of G=8 "due to slower training caused by its long response length." This means the gate-opening probability ~G·p_θ is halved for Qwen3 compared to the other models, which should make GRPO stall more severe on Qwen3 (counter to the observed pattern where Qwen3 has fewer dead prompts). The resolution is that Qwen3's base p_θ is much higher on the training prompts, so G·p_θ is still large enough for most prompts. The paper does not ablate G to test sensitivity—would SAGE provide larger gains over GRPO at very small G (e.g., G=2) where the gate-opening threshold is harder to reach, or at very large G (e.g., G=16) where the gate opens more often even without hints? This is a missing experiment that would strengthen the connection between the theory (Section 3) and the empirical results.

Number of hint levels (L=3): The paper uses exactly three hint levels (plus ℓ=0 for no hint). No ablation on L=2 or L=4 is reported. Intuitively, more levels allow finer-grained escalation, but also increase the potential probing cost in SAGE (more levels to probe through before finding one that works). Fewer levels are cheaper but coarser. The choice of L=3 appears pragmatic rather than principled.

Hint strength threshold for SAGE-LIGHT (α=0.35): SAGE-LIGHT uses α=0.35, meaning hints escalate when the previous epoch's success rate falls below 35%. No sensitivity analysis on α is reported. The choice is not obviously motivated—why 0.35 rather than 0.5 (the gate-opening maximum) or 0.1 (a more conservative threshold)? A sweep over α would reveal whether SAGE-LIGHT's performance is robust to this hyperparameter or requires careful tuning.

Maximum response length (8096 vs. 2048): The main results use 8096 tokens as the maximum response length (required by the LUFFY baseline, which uses full DeepSeek-R1 traces that can be long). The remaining experiments use 2048. The paper does not report whether SAGE's benefits depend on response length—on the one hand, shorter responses might make the task harder (less room for reasoning), increasing the need for hints; on the other hand, shorter responses reduce the sampling cost, potentially allowing larger G within the same compute budget.

Prompt set size (15k vs. 64k): Training uses 15k prompts subsampled from the original 64k, with the constraint that corresponding DeepSeek-R1 traces are <8192 tokens. This filtering could introduce a bias—prompts with longer reference solutions might be systematically harder, and excluding them might make the training set easier, reducing the need for hinting. The paper does not analyze whether the 15k subset is representative of the full 64k in terms of difficulty distribution.

Online hint generator refresh frequency: The paper states that the hint generator q_φ is "periodically refreshed using a copy of the current policy π_θ" but does not specify the refresh frequency. This matters because refreshing too often could introduce noise (the policy is still changing rapidly) while refreshing too rarely could cause calibration drift. The experiments in Figure 3 use Qwen3-4B with 400 training steps, but the refresh schedule is not reported, making reproduction difficult.

Temperature and sampling for evaluation: Evaluation uses temperature=0.6 and top_p=0.95 (Appendix C). The paper does not report pass@1 under greedy decoding (temperature=0), which would test whether the learned reasoning patterns are robust or rely on stochastic exploration. This is a standard practice in reasoning benchmarks and its absence is a minor weakness.

Evaluation checkpoint selection: The paper reports the "best average accuracy over all checkpoints" evaluated every 50 steps. This means a method that peaks briefly at step 350 and then degrades would be reported at its peak, while a method that converges slowly but steadily might look worse if its peak is lower. SAGE's training curves (Figure 3 right, Figure 4) show monotonic or near-monotonic improvement, suggesting this is not a major concern, but the protocol favors methods with higher variance or earlier peaks. Reporting final-checkpoint accuracy alongside best-checkpoint accuracy would provide a clearer picture of convergence behavior.

Critical Assessment

The experimental section provides substantial evidence for SAGE's effectiveness, but the strength of the evidence varies across the paper's claims, and several important questions remain unaddressed.

Claim: SAGE consistently outperforms GRPO across three models and six benchmarks (Table 1). This is the paper's central empirical claim, and it is robustly supported within the scope of the experiments. SAGE achieves higher average accuracy than GRPO for every model on both in-distribution and out-of-distribution benchmarks. The margins are +2.0, +4.5, and +4.2 in-distribution for Llama-3.2, Qwen2.5, and Qwen3 respectively. These are not enormous gains—they represent a ~9% relative improvement for Llama-3.2 (from 21.9% to 23.9%), ~3% for Qwen2.5, and ~2% for Qwen3—but they are consistent and are achieved without any external teacher model, stronger verifier, or test-time privilege.

However, the claim's generality is constrained by the experimental scope. All results are on mathematical reasoning benchmarks with binary verifiable rewards. The paper does not test SAGE on code generation (another domain with verifiable rewards via unit tests), on multi-modal tasks, or on tasks with non-binary reward structures (e.g., partial-credit grading). The out-of-distribution generalization to GPQA and MMLU-Pro is encouraging but the gains are small (+0.9 for Qwen3 on out-of-distribution) and the connection between math RL training and general QA performance is not analyzed—the improvement could be due to improved reasoning patterns that transfer, or it could be an artifact of the specific training data and model families used.

Claim: SAGE addresses GRPO stall by increasing the proportion of prompts that provide learning signal (Table 2). The dead-prompt analysis directly supports this mechanism. SAGE reduces the percentage of prompts that never produce a correct trajectory from 40.2% to 30.0% for Llama-3.2, from 10.3% to 8.2% for Qwen2.5, and from 1.3% to 1.0% for Qwen3. The effect is largest where the problem is most severe (Llama-3.2) and smallest where GRPO already works well (Qwen3). This validates the paper's core diagnosis: the finite-sample degeneracy is real, it affects weaker models more, and hinting mitigates it.

However, the dead-prompt analysis uses any correct trajectory during training as the criterion, not the gate-opening criterion (mixed outcomes in a group). A prompt could have one lucky correct trajectory at some point in training but still have most of its groups be all-zero (gate closed). The gate-opening analysis from Section 3 would predict that SAGE helps when $G·p_\theta$ is small but non-zero—the dead-prompt analysis counts prompts that never have a success, which is a stricter criterion. A more precise ablative analysis would measure the actual gate-opening frequency (fraction of groups with mixed outcomes) for GRPO vs. SAGE and correlate this with per-prompt learning progress. This analysis is missing and would more directly validate the theoretical framework.

Claim: Online self-hinting outperforms external teacher hints and offline self-hinting (Figure 3). Supported for the hard-prompt subset on Qwen3-4B, with online achieving 58.3% vs. GPT-5's 56.5% and offline's 55.4% at ℓ=2 with extended training. The result is robust to hint level (online wins at ℓ=1 and ℓ=2, ties at ℓ=3) and to extended training (Figure 3 right shows online steadily improving while GPT-5 plateaus). The ablation with "offline with more hints" teases apart calibration from diversity and suggests calibration is the dominant factor.

However, this comparison is only done on Qwen3-4B with the hard-prompt subset (G=32, 400 steps). The paper doesn't report analogous comparisons for Llama-3.2 or Qwen2.5, where the gap between online and external hints might differ (a weaker model might benefit more from stronger external hints, or might be more sensitive to calibration drift). The "GPT-5" baseline represents a specific external teacher at a specific capability level—results might differ with a weaker or stronger teacher, or with hints generated by the same model family at a larger scale (e.g., Qwen2.5-72B hints for Qwen2.5-7B).

Claim: The policy-dependent scheduler creates an effective automatic curriculum (Figure 6). The decreasing hint usage over training steps is consistent with a curriculum effect—the model learns to solve prompts with less help. However, Figure 6 only shows Llama-3.2; the paper does not provide analogous plots for Qwen2.5 or Qwen3. The never-decrease policy means that prompts at ℓ=1 stay at ℓ=1 even if the model could now solve them at ℓ=0. Since SAGE probes starting from ℓ=0 at each epoch (Algorithm 1), it would discover that ℓ=0 works and not escalate, but ℓ(x) in the algorithm stores the accepted level, not the minimum effective level. The paper could report the distribution of accepted hint levels over training steps (not just whether any hint is used) to show that the curriculum progressively shifts toward lower levels, not just toward "hint not needed." Without this, the decreasing "use hint" count could partially reflect the scheduler getting stuck at ℓ=0 for prompts that still need hints (because the probe group at ℓ=0 happened to have a success, though the true $p_\theta$ is still very low).

Missing experiments and baselines. Several experiments would strengthen the paper's claims:

• DAPO as a baseline. SAGE uses DAPO's hyperparameters (β=0, asymmetric clipping, filtering modifications). Comparing against DAPO directly would clarify whether SAGE's gains are additive to or redundant with DAPO's own improvements for sparse-reward GRPO. DAPO already filters prompts with all-zero or all-one rewards; SAGE prevents all-zero groups from occurring in the first place. Are the two approaches complementary, or does DAPO's filtering achieve much of what SAGE achieves with less complexity?

• Comparison with simply increasing G. The gate-opening probability scales as ~G·p_θ. An alternative to hinting is to increase the group size G, which also increases the chance of mixed outcomes. The paper doesn't compare SAGE (G=8 with hints) against GRPO with larger G (e.g., G=16, G=32) at equivalent total generation budget. If the cost of SAGE's probe rollouts is comparable to the cost of doubling G, this comparison would reveal whether hinting is more sample-efficient than simply sampling more rollouts.

• Sensitivity to G. The paper uses G=8 for Llama-3.2 and Qwen2.5, and G=4 for Qwen3. This means the gate-opening threshold differs across models, confounding cross-model comparisons. Running Llama-3.2 with G=16 would test whether the gate-opening problem can be solved by larger groups alone, and running Qwen3 with G=2 would test whether SAGE helps even when the gate is very hard to open (G·p_θ very small).

• Per-prompt accuracy improvement vs. hint usage. The paper shows aggregate dead-prompt reduction (Table 2) and aggregate hint usage decrease (Figure 6), but does not show a per-prompt scatter plot of accuracy improvement vs. hint usage. This would directly test whether the prompts that received the most hinting are the ones that improved the most—a causal test of SAGE's mechanism.

• Comparison with process reward models (PRMs). The paper contrasts SAGE with reward shaping approaches but doesn't compare against PRM-based training (e.g., using a learned process reward model to provide dense rewards instead of terminal-only 0/1). This is a natural alternative for addressing sparse rewards and would contextualize SAGE's distribution-shaping approach against the more traditional reward-shaping approach.

Weaknesses in experimental design:

• Single training data source. All training uses DeepSeek-R1 reasoning traces from OpenR1-Math-220k. The quality and style of these traces (long, verbose, self-reflective) may interact with SAGE's hinting mechanism in ways that wouldn't generalize to other data sources. For example, if the reference solutions are particularly detailed, the hints extracted from them may be unusually informative.

• No standard errors or significance testing. The paper reports point estimates for accuracy without confidence intervals, standard deviations, or statistical tests. With 500 evaluation steps and checkpoint selection every 50 steps, the variance in "best average accuracy" is unknown. The margins between SAGE and baselines are sometimes small (e.g., +0.4 for SAGE vs. SAGE-LIGHT on Qwen2.5 in-distribution); without error bars, it's unclear whether these differences are statistically reliable.

• Best-checkpoint selection is a form of test-set optimization. By evaluating on the test benchmarks every 50 steps and selecting the checkpoint with the best average accuracy, the reported numbers are optimistic relative to a protocol that selects the final checkpoint or uses a held-out validation set for checkpoint selection. This is standard practice in the field but worth flagging—the gains over GRPO might be smaller if checkpoint selection were done on a validation set rather than the test set.

• Training set construction filters by trace length (<8192 tokens). This excludes prompts with very long reference solutions, which are likely among the hardest problems. If SAGE's benefits are largest on hard prompts, this filtering works against SAGE—the training set is artificially easier, reducing the need for hinting. SAGE's gains on the full 64k set (without length filtering) might be larger than reported.

• The paper does not isolate SAGE's components. SAGE combines three innovations: privileged hinting, policy-dependent scheduling, and online self-hinting. The ablations test these individually (online vs. offline hinting, SAGE vs. SAGE-LIGHT for scheduling, on-policy vs. off-policy conditioning), but not all combinations. For example, there is no result for "online self-hinting with SAGE-LIGHT scheduling" or "offline self-hinting with SAGE scheduling," which would reveal which component contributes most to the overall gain.

Summary. The experiments convincingly demonstrate that SAGE improves over GRPO on mathematical reasoning benchmarks, with the largest gains on the weakest model and the hardest prompts, consistent with the proposed mechanism of mitigating finite-sample degeneracy. The evidence is strongest for the claim that hinting helps (it does) and weakest for the claim that online self-hinting is substantially better than external teacher hinting (the gap is small and tested on only one model). The paper would benefit from comparisons with DAPO, ablations on group size, per-prompt causal analysis, and standard errors. The practical significance of the +2.0 to +4.5 point gains should be weighed against SAGE's 2.3× training time overhead—for practitioners, SAGE-LIGHT's 1.2× overhead with slightly lower gains may be the more relevant operating point.

6. Limitations and Trade-offs

Training Cost Overhead: SAGE Is 2.3× More Expensive Than GRPO

The constraint. SAGE's per-group degeneracy trigger (Scheme 2) requires probing at multiple hint levels before finding one that opens the gate. For prompts where the model fails at ℓ = 0, the algorithm must sample and discard rollouts at ℓ = 0, then potentially at ℓ = 1, before reaching a hint strength that produces a positive rollout. Each discarded probe level costs one full group of G rollouts (sampled and verified, but contributing no gradient if the gate remains closed). The paper reports this explicitly in Table 3: on Qwen2.5-7B-Instruct, SAGE requires 2.3× the training time of GRPO (25.3 hours vs. an estimated ~58 hours for SAGE on 8 A100 GPUs). The authors are transparent about this:

"A potential limitation of SAGE is its latency, as it must generate and use hints on the fly when a correct trajectory of the prompt can't be sampled." (Section 5.2)

The consequence. The 2.3× overhead substantially complicates the value proposition. If a practitioner has a fixed compute budget (e.g., 48 hours on 8 GPUs), running SAGE for 500 steps costs the same as running GRPO for ~1,150 steps. The paper does not compare SAGE at 500 steps against GRPO at equivalent wall-clock time (more steps or larger batch), which is the relevant baseline for resource-constrained practitioners. The headline improvements (+2.0 to +4.2 points in-distribution) must be discounted against the opportunity cost of the additional compute. For the stronger models (Qwen3-4B, where SAGE gains +1.3 over GRPO in-distribution), the cost-benefit ratio is particularly unfavorable—a 2.3× compute multiplier for a ~2% relative improvement may not justify deployment in cost-sensitive settings.

Evidence in the paper. Table 3 reports training times normalized to GRPO = 1.0× (25.3h). SAGE is 2.3×, SAGE-LIGHT is 1.2×, Scaf-GRPO is 1.5×, LUFFY is 1.2×. The paper does not report a FLOPs-matched or wall-clock-matched comparison (e.g., SAGE at step k vs. GRPO at step 2.3k). Table 1 reports accuracy with all methods run for 500 steps regardless of cost. The overhead is concentrated on hard prompts—Section 5.2 notes that "for highly complex prompts, SAGE may sample hints across multiple levels (from ℓ = 0 to ℓ = 3), which increases computational overhead." Figure 6 shows that hint usage decreases during training on Llama-3.2, suggesting the overhead is front-loaded (early training when the model is weakest).

Mitigation status. The paper offers SAGE-LIGHT as a partial mitigation: it achieves 1.2× overhead (comparable to GRPO) by using epoch-level accuracy thresholds instead of per-group probing, and still outperforms GRPO by +1.0 to +3.1 points in-distribution (Table 1). The accuracy gap between SAGE and SAGE-LIGHT is modest (+0.4 to +1.1 points in-distribution), suggesting the 2.3× variant may not be worth its cost in practice. However, SAGE-LIGHT introduces its own limitation: the epoch-level update (α = 0.35 threshold, checked once per epoch) reacts slowly to transient reward collapse. The paper does not explore what fraction of SAGE's overhead comes from probing cost vs. hint generation cost, or whether cheaper probing strategies (e.g., G=2 probe groups instead of full G=8) could reduce overhead while retaining the per-group trigger's reactivity.

Hard Problems Remain Fundamentally Out of Reach

The constraint. SAGE increases the probability of sampling correct trajectories by conditioning on hints, but this mechanism has an inherent ceiling: if the base model cannot produce a correct solution even with the strongest available hint, no amount of hinting helps. The reference solution τ⋆ is only useful if there exists some hint h derived from τ⋆ such that π_θ(· | x, h) can generate the correct answer with non-negligible probability. For problems where the base model's capability gap is too large—it lacks the necessary knowledge, reasoning patterns, or representational capacity—even Level 3 detailed hints cannot bridge the gap.

The paper models this explicitly through the gate-opening probability u(p) = 1 − (1−p)^G − p^G. If p_θ(x, h) remains near zero even at the maximum hint level ℓ = L, then G · p_θ(x, h) ≪ 1 persists, the gate stays closed, and the prompt contributes no signal. The paper's theory (Proposition 3.3) shows hints are only useful when they move p_θ into the regime where u(p) is non-negligible—but there is no guarantee such a hint exists for every prompt.

The consequence. SAGE cannot solve the hardest tier of problems that fall outside the base model's fundamental capability range. This is structurally analogous to the limitation that Section 7 of Snell et al. (2024) identifies for test-time compute scaling: test-time strategies amplify existing capability but cannot create it from nothing. For SAGE, the parallel is that hinting amplifies the model's ability to find correct reasoning paths that are within its representational reach, but cannot teach it reasoning patterns it fundamentally cannot execute.

The paper does not characterize which prompts fall into this unsolvable category, what properties distinguish them from solvable prompts, or what fraction of a typical training set they represent. This matters for practitioners because it means SAGE cannot substitute for pretraining on harder data distributions—if a training set contains a substantial fraction of problems beyond the base model's capability, those prompts will remain dead weight regardless of hinting.

Evidence in the paper. Table 2 shows that even with SAGE, 30.0% of training prompts for Llama-3.2-3B-Instruct still never produce a single correct trajectory during the entire training procedure (vs. 40.2% for GRPO). For Qwen2.5-7B, 8.2% remain dead (vs. 10.3% for GRPO). For Qwen3-4B, 1.0% remain dead (vs. 1.3% for GRPO). These residual dead prompts represent the fraction of the training set that SAGE cannot rescue—problems where even the strongest available hints fail to enable the model to produce a correct solution. The paper does not analyze these residual dead prompts: Are they systematically harder (more steps, more complex reasoning) than the rescued prompts? Would stronger hints (L > 3) help, or is the limitation fundamental to the model capacity? Are they concentrated in particular mathematical domains?

Figure 3 (left) shows diminishing returns from increasing hint strength: on the hard-prompt subset with Qwen3-4B, online self-hinting achieves 56.7% at ℓ = 1, 58.3% at ℓ = 2, and 57.1% at ℓ = 3—the gain from ℓ = 2 to ℓ = 3 is actually negative, suggesting that beyond a certain point, stronger hints don't help and may even hurt (by making some prompts trivially easy, closing the gate from the p ≈ 1 side). This is consistent with a capability ceiling: for the hardest prompts, no hint level within the {1, 2, 3} range achieves a useful p_θ.

Mitigation status. The paper does not address this limitation directly. The theory (Proposition 3.3, Remark 3.4) characterizes the optimal hint calibration target (p ≈ 1/2) and the gate-opening concavity, but does not provide conditions under which such a hint exists or how to detect prompts where no hint will suffice. The adaptive scheduler (Algorithm 1) will probe up to ℓ = L and if the gate still doesn't open, will accept the degenerate group (line 12: "if sum > 0 or ℓ = L, break") and proceed with an all-zero update—effectively reverting to GRPO stall behavior for that prompt. A more principled response would be to detect such prompts early and either (a) remove them from training, (b) defer them to a later training stage when the model is stronger, or (c) use an entirely different intervention (e.g., SFT on the reference solution directly, as a one-time injection of the correct reasoning pattern). The paper does not explore these options, leaving the residual dead-prompt problem unresolved.

Reference Solutions Are Required During Training, Limiting Applicability

The constraint. SAGE generates hints from reference solutions τ⋆—verified correct answers with reasoning traces. The training data must therefore consist of (prompt, reference solution) pairs where the solution is known to be correct and contains detailed reasoning steps from which a hint generator can extract procedural guidance. The paper uses DeepSeek-R1 traces verified by Math-Verify as these reference solutions, and explicitly notes their availability:

"The training data are drawn from OpenR1-Math-220k, using prompts from NuminaMath 1.5 and reasoning traces generated by DeepSeek-R1." (Section 5)

This assumption is satisfied for mathematical reasoning benchmarks where ground-truth answers exist and reasoning traces can be generated by a capable model, but it does not hold for many important RL-for-LLMs applications.

The consequence. SAGE cannot be directly applied to settings where:

• Only binary reward signals are available, without reference solutions. Many RL applications for LLMs use learned reward models (e.g., helpfulness, harmlessness) or programmatic verifiers (e.g., unit tests for code that don't provide a correct implementation). SAGE requires not just a reward signal but a full solution trace that the hint generator can compress.
• The reference solutions are generated by a model no stronger than the learner. SAGE's online self-hinting uses the learner itself as the hint generator, but this only works if the learner can generate a correct solution when given the prompt and a strong enough hint—the reference solution must exist externally (from a stronger model or ground truth) to bootstrap the process. The paper uses DeepSeek-R1, a much stronger model, for initial reference solutions. If a practitioner only has access to the base model being trained and a binary verifier, SAGE has no source of reference solutions.
• The task is open-ended or has no single correct answer. Creative writing, dialogue, summarization—tasks where "correctness" is subjective or multi-dimensional—don't admit clean reference solutions from which to extract procedural hints.

The reliance on reference solutions also creates a subtle dependency: the quality and style of the reference solutions influence the quality and style of the generated hints, which in turn influences what the policy learns. If the reference solutions use reasoning patterns that don't align with what the learner model can effectively represent (e.g., DeepSeek-R1's verbose, self-reflective style for a 3B-parameter model), the hints may guide the model toward patterns it cannot internalize, similar to the off-policy mismatch that the paper identifies in LUFFY.

Evidence in the paper. The paper's entire experimental setup (Section 5, Appendix C) assumes access to DeepSeek-R1 reasoning traces. The hint generation prompt (Appendix B) explicitly requires a solution: "Given a question and its solution, generate 3 levels of hints..." The paper does not include any experiment where reference solutions are unavailable, or where they are generated by the learner itself from scratch (rather than extracted from pre-existing traces). The LUFFY baseline also uses DeepSeek-R1 traces (it injects them directly into training groups), meaning the comparison against LUFFY partially controls for the "access to a strong teacher model" variable—but the paper doesn't isolate whether SAGE's gains come from having any reference solution access vs. having specifically self-generated hints.

Mitigation status. The paper acknowledges this limitation implicitly through its framing of hints as "privileged" information available only during training, but does not discuss the data requirements as a limitation of applicability. There is no experiment testing whether reference solutions could be generated by the base model itself (e.g., via rejection sampling: sample many solutions, keep the correct ones, use those as reference solutions). This would close the loop—making SAGE applicable whenever a binary verifier exists, without requiring a stronger teacher model—but would introduce a cold-start problem: at initialization, the base model may have near-zero pass rate on hard prompts, producing no correct solutions to serve as reference. The paper does not explore whether SAGE could bootstrap from easier prompts (where the base model does generate correct solutions) to harder prompts, or whether synthetic reference solutions from the base model are sufficient for effective hinting.

Evaluation Is Confined to Mathematical Reasoning with Binary Verifiable Rewards

The constraint. All experiments in the paper use mathematical reasoning benchmarks with binary 0/1 verifiable rewards (exact match of final answer against ground truth). The six in-distribution benchmarks (AIME24, AIME25, AMC23, MATH-500, Minerva Math, OlympiadBench) and two out-of-distribution benchmarks (GPQA-diamond, MMLU-Pro) are all question-answering tasks with objectively correct answers. The training data (OpenR1-Math-220k) is similarly mathematical. The paper does not evaluate on code generation (where unit tests provide binary rewards), multi-step agentic tasks, or any domain outside formal reasoning.

This matters because the GRPO stall pathology that SAGE addresses depends on the reward structure. With binary 0/1 rewards, the gate is either fully open (mixed outcomes) or fully closed (all identical). With continuous or multi-level rewards (e.g., partial credit for partially correct solutions, rubric-based grading), the within-group variance s² may be non-zero even when all rewards are below 1, meaning the gate-opening analysis from Section 3.1 would need modification. SAGE's design—particularly the no-positives trigger in Scheme 2—assumes the gate is closed when no rollout receives a reward of 1. On tasks with continuous rewards, groups may contain useful gradient signal even without any perfect solutions, potentially reducing the need for hinting.

The consequence. A practitioner cannot confidently apply SAGE to non-binary-reward tasks, or to tasks where the reward structure differs meaningfully from mathematical exact-match, without additional validation. The paper's theoretical framework (Proposition 3.2, gate-opening probability) is derived specifically for Bernoulli rewards R_i ∈ {0, 1}. Extending the analysis to categorical rewards (multi-class correctness), continuous rewards (BLEU, ROUGE, learned reward model scores), or structured rewards (per-subtask success) would require a different gate-opening criterion and potentially a different scheduler design. The empirical results provide no evidence for generalizability beyond the math domain, and the out-of-distribution benchmarks (GPQA, MMLU-Pro) are tested only at inference time after math-focused RL training—they demonstrate that math RL training transfers to some general QA performance, not that SAGE works when applied directly to non-math RL training.

Evidence in the paper. The paper's entire experimental section (Section 5, Tables 1-3, Figures 2-6) uses only mathematical reasoning data for training and primarily mathematical benchmarks for evaluation. The out-of-distribution GPQA and MMLU-Pro results (Table 1, right columns) show SAGE's math-trained models generalize to science and general knowledge QA, but this is a test of transfer learning, not of SAGE's effectiveness when the training objective itself is non-mathematical. The paper does not include experiments on code generation benchmarks (HumanEval, MBPP), which would be the most natural extension given that code tasks also have binary verifiable rewards (pass/fail on unit tests) and are a major application domain for RL-trained LLMs. The theory section (Section 3) explicitly assumes Bernoulli rewards and groupwise standardization; the extension to other reward structures is not discussed.

Mitigation status. The paper does not claim applicability beyond mathematical reasoning with binary rewards, but it also does not flag this as a limitation. The title ("Self-Hinting Language Models Enhance Reinforcement Learning") and abstract are phrased generally, suggesting broader applicability than what is empirically demonstrated. The connection to prior work (GRPO, DAPO, LUFFY) is similarly framed in general RL-for-LLMs terms, not math-specific terms. A more precise scoping—"for verifiable mathematical reasoning tasks with binary rewards"—would better reflect the evidence base. The paper does not propose a framework for extending SAGE to continuous rewards or non-math domains, leaving this as entirely future work.

The Scheduler's Never-Decrease Policy May Leave Prompts Permanently Over-Hinted

The constraint. Both SAGE and SAGE-LIGHT use a never-decrease policy for hint levels: once ℓ(x) is incremented for a prompt, it never returns to a lower value. Algorithm 1 formalizes this with ℓ(x_b) ← min{ℓ(x_b) + 1, L} on escalation, and no mechanism for de-escalation. The paper states this explicitly in the SAGE-LIGHT description:

"The hint level never decreases—once escalated, it stays escalated." (Section 4.3, implicit in the min{ℓ_{t-1}(x) + 1, L} update)

The intuition is that as the policy improves, prompts become easier at their current hint level, so de-escalation isn't needed. But this is an assumption about monotonic improvement that may not hold in practice—policy optimization is non-monotonic, and a prompt that was hard at epoch t may become easier at epoch t+1 (the model generalizes from other prompts), making the escalated hint level unnecessarily strong.

The consequence. Prompts can become permanently over-hinted. If a prompt is escalated to ℓ = 2 at epoch 50 because the model was struggling, but by epoch 200 the model has improved enough that ℓ = 1 (or even ℓ = 0) would suffice, the prompt remains at ℓ = 2. The model continues training on an unnecessarily easy version of the prompt, which has two negative effects. First, the gate-opening probability u(p) is symmetric and concave—if the escalated hint pushes p_θ too close to 1, the gate-opening probability decreases, potentially re-closing the gate from the high-success side (Proposition 3.2: u(p) → 0 as p → 1). Second, training with overly strong hints reduces the learning signal that transfers to the deployable ℓ = 0 setting—the model practices reasoning with detailed guidance that won't be available at test time.

This is a direct consequence of prioritizing simplicity (never-decrease is easy to implement) over optimality (de-escalation would require probing at lower hint levels to verify they now work, adding cost). The tradeoff is that SAGE sacrifices some efficiency on prompts that the model outgrows, in exchange for avoiding the probing overhead of de-escalation checks.

Evidence in the paper. Figure 6 shows hint usage decreasing over training on Llama-3.2—but this reflects prompts transitioning from "needs hint" to "produces positive rollouts at ℓ = 0" (so the scheduler doesn't escalate), not prompts de-escalating from higher ℓ to lower ℓ. The paper does not report the distribution of hint levels among prompts that still use hints at late training stages—are they all at ℓ = 1 (would have been fine without hints by now?), or are they genuinely hard prompts that still need ℓ = 2 or ℓ = 3? Without this breakdown, it's impossible to assess how much over-hinting occurs. The "same level hint vs. SAGE" ablation in Figure 5 compares adaptive SAGE (59.2%) against fixed ℓ = 2 (58.3%), showing that adaptive scheduling helps—but this comparison only tests whether starting at ℓ = 0 and escalating is better than always using ℓ = 2. It doesn't test whether a policy that de-escalates from ℓ = 2 to ℓ = 1 or ℓ = 0 when the model improves would outperform the never-decrease policy.

Mitigation status. The paper does not address this limitation or propose a de-escalation mechanism. A natural extension would be to periodically probe at ℓ − 1 (if ℓ > 0) and de-escalate if the gate opens at the lower level, but this would increase the probing overhead and complicate the scheduler. The SAGE probing loop (Algorithm 1, lines 9-13) always starts probing from ℓ = 0, regardless of the stored ℓ(x), which means it could discover that ℓ = 0 works and not escalate—but this applies only to the current epoch, and the stored ℓ(x) is updated to the accepted level, so if ℓ = 0 works at epoch t+1, the probe succeeds at ℓ = 0 and ℓ(x) stays at 0. The issue is for prompts where the stored ℓ(x) is already > 0: the probing always starts from ℓ = 0, so if the model has improved, it will succeed at ℓ = 0 and the stored level effectively de-escalates to 0. This means the never-decrease policy is not as strict as it appears—SAGE's probing loop implicitly tests whether a lower hint level would work and will use it if so. The paper does not clarify this interaction, making it unclear whether over-hinting is a real problem in practice or only a theoretical concern under the stated never-decrease design. Analysis of the distribution of accepted hint levels (not just "use hint yes/no") over training would resolve this ambiguity.

Adaptive Scheduling Effectiveness Relies on Unvalidated Threshold Heuristics

The constraint. The two scheduling schemes rely on threshold parameters whose values are set without systematic tuning or sensitivity analysis. SAGE-LIGHT uses α = 0.35 as the epoch-level accuracy threshold below which hint strength escalates. SAGE uses a binary trigger: escalate if the probe group contains no positive rollouts (∑ R̃_i = 0), which is equivalent to a threshold of exactly 0 successes in G probe rollouts. Both thresholds are plausible but their optimality is unexplored, and the effectiveness of the entire adaptive curriculum depends on them.

For SAGE-LIGHT, the choice of α = 0.35 is not motivated in the paper. Why 0.35 rather than 0.5 (the gate-opening maximum from Proposition 3.2)? Or 0.125 (roughly the threshold where G·p ≈ 1 for G=8, below which the gate is almost always closed)? Or 0.01 (only escalate on extremely hard prompts)? The threshold determines how aggressively the scheduler escalates hint strength—a lower α means fewer prompts get hints (only the very hardest), while a higher α means more prompts get hints (including moderately challenging ones that might learn fine without them). Over-escalation wastes hint budget and risks over-hinting; under-escalation leaves some prompts stalled longer than necessary.

For SAGE, the binary trigger (∑ R̃_i = 0) is equivalent to escalating whenever the probe group contains zero successes in G trials. This is a noisy estimator of whether G·p_θ is small—a prompt with p_θ = 0.1 and G = 8 has probability (1−0.1)^8 ≈ 0.43 of producing an all-zero probe group, meaning the trigger would escalate on ~43% of epochs for a prompt that actually has a reasonable chance of opening the gate. Conversely, a prompt with p_θ = 0.001 has probability ≈ 0.992 of an all-zero probe, so the trigger escalates with ~99% reliability. The binary trigger is conservative (it rarely misses a truly hard prompt) but has a high false-positive rate for marginally hard prompts, causing unnecessary escalation.

The consequence. Without sensitivity analysis, a practitioner cannot assess whether SAGE's gains depend on careful threshold tuning or are robust across a wide range. If SAGE-LIGHT's performance is highly sensitive to α, reproducing the paper's results requires the exact same threshold, which may not transfer to different models, tasks, or training data distributions. If SAGE's binary trigger escalates too aggressively on marginally hard prompts, it may be over-hinting on a substantial fraction of the training set, inflating training cost without commensurate benefit.

The binary trigger's false-positive rate also interacts with the never-decrease policy discussed above. A prompt that gets escalated unnecessarily at an early epoch (because a random all-zero probe group occurred despite p_θ being non-negligible) will remain at an elevated hint level for the rest of training if the sequencing logic in Algorithm 1 doesn't implicitly de-escalate. This could permanently handicap learning on that prompt by making it easier than necessary.

Evidence in the paper. The paper provides no ablation, sweep, or sensitivity analysis for α or for alternative trigger thresholds in SAGE. The SAGE-LIGHT threshold α = 0.35 appears in Section 4.3 without justification beyond the intuitive description that "we increase ℓ only when the current policy provides insufficient learning signal on a prompt." The paper does not report what fraction of prompts fall below this threshold at different training stages, or how this fraction correlates with actual gate-opening frequency. For SAGE, the binary trigger is motivated by the theory (Section 3.1: the gate closes when the group has no positive rollouts) but no comparison is made against softer triggers (e.g., escalate if fewer than k successes, or if the probe group's empirical p̂ is below some threshold). The comparison between SAGE and SAGE-LIGHT in Table 1 shows SAGE-LIGHT achieving comparable accuracy at lower cost, suggesting the binary trigger's additional reactivity provides only modest benefit—but this could be because the binary trigger is too sensitive, causing unnecessary escalation that the per-group probing cost doesn't justify.

Mitigation status. The paper does not address this limitation. The threshold choices are presented as fixed hyperparameters without discussion of their sensitivity or transferability. A thorough analysis would sweep α (for SAGE-LIGHT) and alternative trigger formulations (for SAGE: escalate if fewer than 1, 2, or ⌈G/2⌉ successes) and report the accuracy-cost Pareto frontier. This would reveal whether the published thresholds are optimal, near-optimal, or arbitrary. The paper also does not discuss whether the optimal threshold depends on the base model's capability, the training set difficulty, or the training stage (adapting α over time as the model improves). Given that the scheduler is the mechanism that creates SAGE's adaptive curriculum, the lack of threshold analysis is a significant gap in validating the method's robustness.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper causes a diagnostic reframing rather than a paradigm shift. It does not introduce a fundamentally new learning algorithm, nor does it claim to replace GRPO or on-policy RL for LLMs. What it changes is how the field understands a specific failure mode—GRPO's silent stalling on hard prompts—and it provides a principled, reusable framework for analyzing that failure mode that extends beyond SAGE itself.

The core conceptual move is recasting GRPO's finite-sample degeneracy from a variance problem ("the gradient is noisy on hard prompts") to a gated update problem ("the gradient is identically zero with probability approaching 1"). This is not merely relabeling. Variance problems admit solutions like larger batch sizes, better baselines, and importance sampling corrections—all of which operate by reducing estimator noise around a non-zero expected gradient. A gated update problem, by contrast, has an expected gradient that is itself zero because the gate is closed most of the time; no amount of variance reduction helps because there is no signal to recover. The paper formalizes this through the gate-opening probability $u(p) = 1 - (1-p)^G - p^G$ (Proposition 3.2), which gives the field a precise, computable diagnostic for when and why GRPO stalls. Any future method that modifies GRPO's sampling, reward structure, or advantage computation can be analyzed through this lens: does it increase the gate-opening probability for prompts where $G \cdot p_\theta \ll 1$?

The practical implication of this reframing is that it redirects research attention away from variance-reduction techniques and toward gate-opening interventions. Prior work on sparse-reward GRPO focused largely on data filtering (skipping degenerate groups, resampling easier prompts), adaptive sampling (allocating more rollouts to hard prompts), or external guidance (injecting correct trajectories from stronger teachers). These approaches either bias the training distribution toward easier prompts or introduce off-policy mismatch. SAGE demonstrates a third path: modify the conditioning context to temporarily increase $p_\theta$ on hard prompts while maintaining the on-policy contract and the original reward function. This opens a design space—privileged conditioning during training, removed at deployment—that prior work did not systematically explore.

The symmetry and concavity of $u(p)$ (peaking at $p = 1/2$, collapsing at both $p \to 0$ and $p \to 1$) provides a counterintuitive design principle that contradicts common intuition. Practitioners typically assume that making hard prompts easier is always beneficial—stronger hints, more scaffolding, better teachers. SAGE's analysis shows that overly strong hints can re-close the gate from the $p \approx 1$ side, because a group of all-correct answers is just as uninformative as a group of all-incorrect ones. The optimal hint distribution (Proposition 3.3) calibrates toward $p \approx 1/2$, not $p \to 1$. This explains why Scaf-GRPO's external GPT-5.2 hints—which are highly informative—can suppress exploration (Figure 4 shows Scaf-GRPO has the lowest entropy among all methods) and underperform self-generated hints that are naturally calibrated to the learner's current capabilities. The principle that hints should make prompts appropriately challenging, not trivially easy is a genuinely non-obvious insight with immediate practical consequences for anyone designing scaffolded RL training pipelines.

The paper also reconciles contradictory intuitions about whether stronger teacher models help during RL training. LUFFY (Yan et al., 2025) and Scaf-GRPO (Zhang et al., 2025b) both rely on stronger external models (DeepSeek-R1, GPT-5.2) to provide guidance. SAGE's online self-hinting—where the learner generates its own hints—consistently outperforms both external-teacher approaches (Figure 3: online self-hinting at ℓ = 2 achieves 58.3% on Qwen3's hard-prompt subset vs. GPT-5's 56.5%, Figure 5: offline with more hints underperforms online by 2.0 points). This suggests that for RL training, calibration to the learner's current capabilities matters more than the absolute quality of the guidance. A weaker model's hints, when refreshed online to track its evolving skill level, are more effective than a stronger model's hints that are static or miscalibrated. This finding complicates the prevailing narrative that better teacher models always yield better student models, and it opens a new axis of investigation: not "how to generate the best possible guidance," but "how to generate guidance that maximizes learning progress for a specific learner at a specific point in training."

Perhaps most significantly for practitioners, SAGE demonstrates that privileged information during training can transfer to deployment without any test-time cost. The gains in Table 1 (+2.0 to +4.5 points in-distribution over base models, +0.9 to +11.5 points out-of-distribution) are achieved with a model that, at test time, runs as a standard language model on the prompt alone—no hints, no reference solutions, no additional inference computation. This is in contrast to test-time compute scaling approaches (e.g., best-of-N, beam search, revision models) that improve performance at the cost of additional inference FLOPs. SAGE's approach—investing compute during training to make the model more capable at test time—is complementary to test-time scaling and suggests a more nuanced total-budget optimization: some compute should be spent on smarter training (privileged hinting), some on smarter inference (search, revision), and the optimal allocation likely depends on the deployment scenario (inference volume, latency requirements, prompt difficulty distribution).

The limitations are equally important in shaping the landscape. SAGE requires reference solutions during training (DeepSeek-R1 traces in the paper's setup), which restricts applicability to domains where such solutions exist or can be generated. The 2.3× training time overhead (Table 3) means SAGE is not a drop-in replacement for GRPO in cost-sensitive settings. The residual dead-prompt problem (30.0% of Llama-3.2 prompts still never produce a correct trajectory, Table 2) shows that hinting cannot rescue prompts completely outside the base model's capability range—there is a fundamental floor that privileged conditioning cannot break through. These boundaries define where SAGE is useful (moderate-to-hard prompts within the model's reach, training budgets that can tolerate 2× overhead, domains with verifiable binary rewards and available reference solutions) and where alternative approaches (larger pretraining, different RL algorithms, test-time scaling) are necessary.

Follow-Up Research This Work Enables

Extending SAGE to code generation with unit-test verifiers. The paper's entire evaluation is on mathematical reasoning with exact-match binary rewards. Code generation is the most natural next domain: unit tests provide binary pass/fail signals, reference solutions (correct implementations) can be sourced from strong code models or human-written corpora, and the gap between a weak code model and a strong one mirrors the math capability gap that SAGE addresses. A direct replication would train a small code model (e.g., CodeLlama-7B) on competitive programming prompts with DeepSeek-Coder or GPT-4 generated reference solutions, use SAGE with the same adaptive scheduling and online self-hinting, and evaluate on HumanEval, MBPP, and LiveCodeBench. The key question is whether procedural hints for code (e.g., "use a sliding window to maintain the maximum", "precompute prefix sums to answer queries in O(1)") transfer to the no-hint policy as effectively as mathematical hints. A negative result—SAGE helps on math but not on code—would reveal that the mechanism depends on properties of mathematical reasoning (step-by-step derivability from hints) that don't generalize to algorithmic problem-solving.

Principled de-escalation: bidirectional hint strength scheduling. SAGE's never-decrease policy for hint levels is a pragmatic choice that the paper acknowledges may be suboptimal. A direct follow-up would implement and evaluate a bidirectional scheduler that probes at both ℓ + 1 (to detect closed gates) and ℓ − 1 (to detect when de-escalation is safe). The experiment would compare three variants on the same 15k-prompt math training set: (a) SAGE's current never-decrease scheduler, (b) a "probe-down" variant that periodically (e.g., every 5 epochs) tests whether ℓ − 1 now opens the gate and de-escalates if so, and (c) an "oracle" variant that always uses the minimum ℓ that achieves $G \cdot p_\theta \geq 1$ (estimated via large-scale sampling). The key metric is the final no-hint test accuracy vs. total training FLOPs (probing cost included). If bidirectional scheduling reduces over-hinting and improves accuracy at similar cost, it would refine SAGE into a more efficient method. If it doesn't help (suggesting over-hinting is rare in practice, or that the probing cost of de-escalation outweighs its benefit), that finding would validate the paper's simpler design and provide guidance for practitioners.

Bootstrapping SAGE without an external reference model. The paper's experiments rely on DeepSeek-R1 reasoning traces as reference solutions, which assumes access to a model substantially stronger than the learner. An important extension would test whether SAGE can bootstrap from the base model itself: for each training prompt, sample $N$ solutions from the base model, use the verifier to identify correct ones, and use those as reference solutions for hint generation. This would close the loop—making SAGE applicable whenever a binary verifier exists, without requiring a stronger teacher. The challenge is the cold-start problem: for hard prompts where the base model's pass@N is near zero, no correct solutions can be found to serve as references. A practical experiment would use a curriculum: start with prompts where the base model achieves non-trivial pass@N (e.g., >5%), use bootstrapped reference solutions to train with SAGE, then periodically re-sample from the improved policy to generate reference solutions for harder prompts. The comparison would be against (a) SAGE with external reference solutions (DeepSeek-R1) and (b) GRPO without any hinting, all at equivalent total generation budget (including the cost of rejection sampling for reference solutions). A positive result would dramatically expand SAGE's applicability; a negative result would clarify the minimum reference solution quality needed for effective hinting.

Gate-opening analysis as a diagnostic for RL training health. The paper's theoretical framework (Section 3) provides a precise, computable metric—the gate-opening probability $u(p_\theta(x))$ or its empirical estimate from training rollouts—that could serve as a real-time diagnostic for GRPO training health. A follow-up engineering contribution would instrument a GRPO training run with per-prompt gate-opening statistics (fraction of groups with $s > 0$, binned by estimated $p_\theta$) and correlate these with downstream metrics (per-prompt accuracy improvement, final benchmark scores). The key finding would be whether gate-opening frequency predicts learning progress: do prompts with persistently low gate-opening rates fail to improve, and does an increase in gate-opening rate precede accuracy gains? If so, the diagnostic could be used for online training decisions—pausing training when the aggregate gate-opening rate drops below a threshold, reallocating compute budget to prompts where the gate is closed, or triggering curriculum interventions. This would transform SAGE's theoretical insight into an operational tool that any GRPO practitioner could use, independent of whether they adopt hinting as the intervention.

Stress-testing SAGE on deliberately miscalibrated hints. The paper's online-vs-offline hint comparison (Figure 3) shows that online self-hinting outperforms offline, but the mechanism (calibration to the learner's capabilities) is inferred rather than directly tested. A controlled experiment would deliberately miscalibrate hints and measure the impact: (a) hints that are too weak for the current policy (generated by a much earlier checkpoint), (b) hints that are too strong (generated by a fully trained model or an external teacher prompted to be overly detailed), and (c) hints that are appropriately calibrated (online self-hinting, the SAGE default). The prediction from Proposition 3.3 is that both under- and over-strong hints should reduce the gate-opening rate and hurt final performance, with the optimal hint distribution calibrated to $p_\theta \approx 1/2$. Measuring $p_\theta(x, h)$ directly for each hint variant (via large-scale sampling at multiple checkpoints) would test whether the gate-opening probability $u(p)$ is indeed the mediating variable—does performance degrade precisely when hints push $p_\theta$ away from 1/2, and does the U-shaped $u(p)$ curve explain the degradation quantitatively? A strong confirmation of this mechanism would elevate SAGE from an empirical recipe to a principled method with predictable behavior. A failure to find the predicted U-shape—e.g., if performance improves monotonically with hint strength, contradicting the concavity argument—would suggest the gate-opening framework is an incomplete model and that other factors (hint quality, exploration diversity, off-policy transfer) dominate.

Combining SAGE with test-time compute scaling. SAGE invests training compute (hint generation, probing) to produce a stronger no-hint policy. Test-time compute scaling methods (best-of-N, beam search, revision models) invest inference compute to improve outputs from a fixed policy. These are complementary axes, and the optimal allocation between them is unexplored. A follow-up would train a model with SAGE on math reasoning, then apply test-time compute strategies (e.g., majority voting with $N = 4, 8, 16, 32$, or weighted best-of-N with a process reward model) at evaluation. The key comparison is: does a SAGE-trained model + test-time compute outperform (a) a GRPO-trained model + test-time compute with the same total (training + inference) FLOPs, and (b) a larger model trained without SAGE but with test-time compute? The paper's FLOPs-matched comparison is missing (SAGE + GRPO steps vs. GRPO at equivalent wall-clock time), and combining with test-time scaling would provide a more complete picture of the total compute-optimal strategy. A finding that SAGE's training investment pays off even when test-time compute is plentiful would strengthen its practical value; a finding that test-time compute obviates the need for hinting (by achieving similar accuracy with simpler training) would narrow SAGE's applicability to scenarios where inference-time compute is constrained.

Practical Applications and Downstream Use Cases

Cost-efficient RL training for small-to-medium math models. For organizations training math-specialized LLMs in the 3B-7B parameter range—a common deployment tier for on-device or low-latency applications—SAGE offers a concrete recipe for improving accuracy without increasing model size, inference cost, or dependency on external teacher models. On Llama-3.2-3B-Instruct, SAGE improves in-distribution math accuracy from 17.8% to 23.9% (+6.1 points, a 34% relative improvement, per Table 1) and out-of-distribution from 22.5% to 34.0% (+11.5 points). These gains come from the same model size and same inference cost as the base model; the additional investment is 2.3× training time. For a deployment where inference dominates total cost (many queries served by a trained model), the one-time training overhead of SAGE amortizes favorably against the per-query savings of using a 3B model instead of a 7B or larger alternative. SAGE-LIGHT provides a cheaper path with 1.2× training overhead and gains of +5.2 in-distribution and +10.7 out-of-distribution for Llama-3.2, making it the pragmatic choice when training budget is the primary constraint.

Improving RL data efficiency on skewed difficulty distributions. In many practical RL-for-LLMs deployments, the training data contains a long tail of hard prompts that the base model solves rarely or never. Standard GRPO wastes compute on these prompts (they produce all-zero groups, contributing no gradient), and filtering them biases training toward easier prompts. SAGE directly addresses this: Table 2 shows it reduces the fraction of "dead" prompts from 40.2% to 30.0% for Llama-3.2 and from 10.3% to 8.2% for Qwen2.5. For an organization training on a large, naturally distributed dataset where manual difficulty filtering is impractical, SAGE's ability to automatically recover learning signal from hard prompts means less data curation is needed and more of the available data contributes to training. The practical workflow is: collect a broad dataset of (prompt, verifiable answer) pairs, generate reference solutions using a capable model (or bootstrap from the base model if a teacher is unavailable), and apply SAGE with online self-hinting. The scheduler automatically identifies which prompts need hints and at what strength, eliminating the need for manual difficulty binning or curriculum design.

Self-improvement pipelines with closed-loop hint generation. The online self-hinting mechanism enables a self-contained training loop where the model generates its own curriculum. As the policy improves, its self-generated hints track its evolving capabilities, providing appropriately calibrated guidance without external intervention. This is particularly valuable for iterative self-improvement setups (e.g., STaR, ReST$^{EM}$, or online RL from human feedback) where the model is periodically retrained on its own outputs. A SAGE-based self-improvement pipeline would: (1) train with online self-hinting using reference solutions from a bootstrap phase (or a frozen initial model), (2) after training converges, use the improved policy to generate new reference solutions for prompts it now solves correctly, (3) retrain with SAGE using these self-generated references, creating a virtuous cycle where the model teaches itself to solve progressively harder prompts. The paper's finding that online self-hinting outperforms fixed external hints (Figure 3) suggests this loop could be more effective than relying on a static external teacher, and the decreasing hint usage during training (Figure 6) suggests the model genuinely internalizes the guided reasoning patterns rather than depending on hints.

When to Prefer This Method

The paper explicitly positions SAGE against GRPO (standard, no intervention), SFT (supervised fine-tuning on reference solutions), LUFFY (off-policy trajectory injection from a stronger model), and Scaf-GRPO (external-teacher hinting). The decision rules below are grounded in the paper's empirical comparisons (Table 1, Table 2, Table 3, Figures 3-6) and the theoretical framework (Section 3).

• Prefer SAGE over standard GRPO when the training set contains a substantial fraction of hard prompts where the base model's pass rate is very low (evidence: Table 2 shows SAGE recovers 10.2% of dead prompts for Llama-3.2, Figure 3 shows largest gains on the hard-prompt subset), AND a 2.3× training time overhead is acceptable (Table 3), AND reference solutions are available for hint generation (Section 4.2 requires (x, τ⋆) pairs). The benefit is largest for weaker base models (Llama-3.2: +6.1 in-distribution, Qwen2.5: +4.5) and diminishes for already-strong models (Qwen3: +4.2, but the absolute gain over GRPO narrows to 1.3 points averaging across all benchmarks).

• Prefer SAGE-LIGHT over SAGE when training compute budget is the primary constraint and a 1.2× overhead is the maximum acceptable (Table 3), AND the slightly lower accuracy (+3.1 to +5.2 in-distribution vs. SAGE's +4.2 to +6.1) is tolerable. SAGE-LIGHT's epoch-level scheduling is less reactive but substantially cheaper, making it the pragmatic default unless the additional accuracy from per-group probing justifies the 2× cost multiplier.

• Prefer SAGE over LUFFY when maintaining on-policy learning stability is important. LUFFY exhibits training instability for Llama-3.2 (excessively high entropy, oscillatory response lengths, Figure 4) and Qwen3 (very low initial rewards), and underperforms the base model on both (Table 1: 14.7% vs. 17.8% for Llama-3.2, 60.6% vs. 65.8% for Qwen3). SAGE preserves on-policy updates and shows smooth training dynamics. The exception is Qwen2.5, where LUFFY (41.7%) is competitive with SAGE-LIGHT (41.9%)—but this is a single data point and the paper does not explain why LUFFY works well for this specific model.

• Prefer SAGE over Scaf-GRPO when no external teacher model (e.g., GPT-5.2) is available or when the teacher's hint distribution is poorly calibrated to the learner. Scaf-GRPO requires access to a stronger external model for hint generation; SAGE uses the learner itself. Even when a teacher is available, SAGE's online self-hinting outperforms GPT-5.2 hints on the hard-prompt subset (Figure 3: 58.3% vs. 56.5% at ℓ=2), and Scaf-GRPO's lower entropy (Figure 4) suggests external hints overly constrain exploration.

• Prefer SFT or pretraining scale-up over SAGE when the base model fundamentally cannot solve a large fraction of the training prompts even with the strongest available hints. SAGE leaves 30.0% of Llama-3.2 prompts still dead (Table 2)—these are prompts where even ℓ=3 hints cannot enable a correct solution. For such capability gaps, larger-scale pretraining or SFT on diverse demonstrations (to expand the model's reasoning repertoire) is necessary before RL fine-tuning. SAGE amplifies existing capability; it does not create new capability from nothing.

• Do not prefer SAGE when the domain lacks binary verifiable rewards (the gate-opening analysis in Section 3.1 assumes Bernoulli R_i ∈ {0,1}), lacks reference solutions for hint generation (Section 4.2 requires τ⋆), or when inference-time compute is so abundant that test-time strategies (best-of-N, revision) can achieve the desired accuracy without modifying the training procedure. The paper does not compare SAGE against inference-time scaling at equivalent total compute, and no FLOPs-matched training-vs-inference analysis is provided.