ArXiv: 2602.09000

🎯 Pitch

iGRPO equips language models with the ability to learn from their own best draft, a training-time refinement loop not found in standard RL. This self-feedback mechanism pushes AIME24 accuracy to a new state-of-the-art 85.6% by teaching the policy to systematically improve upon its strongest prior attempt.

1. Executive Summary

This paper introduces Iterative Group Relative Policy Optimization (iGRPO), a two-stage extension of GRPO that incorporates dynamic self-conditioning by appending the policy's own highest-reward draft as in-context feedback before performing a GRPO-style update on draft-conditioned refinements (e.g., Stage 1 selects the best exploratory completion, Stage 2 trains on refinements conditioned on that draft). Across base models including Nemotron-H-8B-Base-8K, DeepSeek-R1-Distilled, and OpenMath-Nemotron at 7B–14B scales evaluated on AIME24/25, MATH500, AMC23, GSM8K, and Minerva Math, iGRPO consistently outperforms vanilla GRPO and self-improvement baselines under matched rollout budgets, with gains concentrated on multi-step competition benchmarks (e.g., +4.06 points on DeepSeek-R1-Distill-Qwen-14B for AIME24). Applied to OpenReasoning-Nemotron-7B trained on AceReason-Math, iGRPO achieves new state-of-the-art results of 85.62% and 79.64% on AIME24 and AIME25, establishing that self-feedback-driven refinement improves verifiable reasoning across model families and scales while delaying premature entropy collapse, but only when the training architecture couples exploration with conditioned exploitation within a single optimization step.

2. Context and Motivation

The Core Problem: Single-Shot Optimization Misses Iterative Self-Improvement

The fundamental gap this paper addresses is that current reinforcement learning algorithms for LLM reasoning treat each generation as an isolated attempt, ignoring the value of the model's own prior outputs as scaffolding for improvement. In the dominant paradigm exemplified by GRPO, each training step samples multiple completions independently, computes rewards, normalizes advantages within the group, and updates the policy — but never conditions future generations on the model's own best attempts. This is a missed opportunity because, as the authors note in Section 1, humans "rarely solve nontrivial problems in a single pass: they often iterate on initial drafts, identify mistakes, and refine their solutions based on internal feedback." The paper aims to inject this iterative refinement loop directly into the RL optimization process itself, not just as an inference-time strategy.

This gap matters for several practical and theoretical reasons the paper develops:

  • Multi-step reasoning tasks are brittle under single-shot optimization. On benchmarks like AIME, a solution that is 90% correct but contains a single algebraic error receives zero reward under standard verifiable outcome scoring (Equation 6). The model never receives a signal that it was "close" — it only learns that its output was wrong. An iterative mechanism that conditions on a mostly-correct draft and trains the model to refine it could systematically recover from such near-misses, converting partially-correct trajectories into fully-correct solutions during training.

  • Standard in-context learning uses static demonstrations that don't co-evolve with the policy. The paper explicitly contrasts iGRPO with ICL in Section 3.2.1, noting that ICL conditions on fixed examples ee that remain unchanged throughout training. As the policy improves, these static demonstrations become stale — they don't reflect the model's growing capabilities. The paper frames this as a "fundamental limitation" because the conditioning signal doesn't adapt to the evolving policy.

  • Existing self-improvement methods typically separate generation, critique, and refinement into distinct phases or agent roles. Methods like Self-Verification and Critique-GRPO ask the model to allocate capacity to auxiliary behaviors (generating verification rationales or natural-language critiques) that are only indirectly optimized by the outcome reward. iGRPO instead keeps the training loop tightly aligned with the verifiable objective: Stage 1 supplies a high-quality draft as an explicit conditioning signal, and Stage 2 directly optimizes a refinement policy using the same outcome reward.

Why This Matters: The Limits of Scaling RL Without Iteration

The paper's motivation is strengthened by the observation that RL fine-tuning on strong base models can plateau quickly. In Section 4.2, the authors show that for OpenMath-Nemotron-7B — a math-specialized model starting at 74.83% average accuracy — standard GRPO yields only a 0.19-point improvement (to 75.02%). This suggests that when a base model already produces correct solutions for most problems, single-shot RL optimization offers diminishing returns: the model already knows the material, so what it needs is not to learn new problem-solving strategies but to learn when its own outputs contain errors and how to fix them. iGRPO addresses this by creating a training signal that explicitly rewards the model for producing answers better than its own best prior attempt, which pushes performance beyond what one-shot optimization can achieve.

The paper also provides a theoretical lens on why this should work. Proposition 3.1 establishes that under binary rewards, the expected quality of the selected Stage 1 draft increases monotonically with the policy's success probability: E[Rφ(d^θ(q))]=1−(1−Vθ(q))N\mathbb{E}[R_\varphi(\hat{d}_\theta(q))] = 1 - (1 - V_\theta(q))^N, where Vθ(q)V_\theta(q) is the expected reward under the current policy. This creates a bootstrapping dynamic: as the policy improves, the drafts it selects as conditioning become better, which provides more informative in-context guidance for Stage 2, which enables further policy improvement. The paper refers to this as "dynamic self-conditioning" — the training context automatically improves as the policy improves.

Conflicting Demands: Exploration vs. Exploitation in RL for Reasoning

A subtler motivation concerns the exploration-exploitation tradeoff during RL training. The paper's entropy analysis (Figure 3) reveals that standard GRPO on DeepSeek-R1-Distill-Qwen-7B trained on MATH causes rapid policy entropy collapse: starting at 2.45 nats, entropy drops to 0.60 nats at 10% of training and 0.42 nats by 30%. This premature collapse means the model quickly concentrates its probability mass on a narrow set of generation patterns, limiting its ability to explore alternative solution strategies or recover from errors.

The authors present iGRPO as partially addressing this without requiring explicit entropy regularization (the paper uses no entropy bonus; Section 4.1 specifies "no entropy regularization"). By conditioning Stage 2 on the best draft, the model is encouraged to explore around a strong scaffold rather than converging immediately to the highest-reward output distribution. The entropy curves in Figure 3 confirm this: iGRPO decays more gradually (0.80 at 15%, 0.48 at 30%) and remains slightly higher through mid-training (0.46 at 60%) before converging near GRPO at the end. The paper interprets this as evidence that "iGRPO delays premature mode collapse," allowing the policy to "preserve alternative completions long enough to recover from near-miss reasoning traces."

Where Existing Approaches Fall Short

The paper identifies specific limitations in prior work along several axes:

GRPO and its variants focus on stabilizing the objective, not on what the model sees. The related work section (Section 2) positions iGRPO as orthogonal to methods like Dr. GRPO, DAPO, and GSPO, which "primarily focus on stabilizing or correcting the underlying optimization objective." These methods modify how advantages are computed, how clipping is applied, or how sequence-level rewards are normalized — but they all operate on independently sampled completions. iGRPO instead "shap[es] the data distribution seen by the optimizer" by introducing a two-stage mechanism that changes what prompts the model trains on.

Self-Verification and Critique-GRPO add auxiliary behaviors that dilute the training signal. Self-Verification (Zhang et al., 2025a) trains the model to both solve problems and verify its own answers, using verification scores to reweight or aggregate solutions at inference time. Critique-GRPO (Zhang et al., 2025b) augments GRPO with natural-language critiques, generating critique-conditioned refinements and optimizing over both initial answers and refinements. The paper acknowledges these as strong baselines (Table 1 shows they outperform vanilla GRPO), but argues they have a structural disadvantage: they "ask the model to allocate capacity to additional behaviors (verifying, critiquing, or producing auxiliary text) that are only indirectly optimized by the outcome reward." In contrast, iGRPO's Stage 1 draft selection uses the same scalar reward signal as optimization — there is no separate critic, verifier, or critique generator to train or coordinate. The model's only task is to produce a better answer than the draft it receives.

Self-play and self-improvement methods blur the roles of reward provider and recipient. Methods like SPIN, Self-Rewarding Language Models, and verifier-based alignment (Chen et al., 2024; Yuan et al., 2024; Kirchner et al., 2024) use the model itself or a closely coupled variant as an internal evaluator. The paper notes (Section 2) that "unreliable rewards can hinder complex reasoning" in such setups. iGRPO "maintains a clearer separation between the model's generation process and the reward signal" by using externally evaluated best-prior drafts — the reward function remains the same rule-based checker used for optimization, not a model-generated proxy.

Existing iterative prompting methods (Self-Refine, Reflexion) operate at inference time, not during RL training. The paper cites Madaan et al. (2023) and Shinn et al. (2023) as evidence that self-feedback mechanisms can improve multi-step reasoning, but notes that "existing RL frameworks do not capitalize on this iterative refinement process." The key distinction is that those methods apply iterative refinement as an inference-time strategy (generate, critique, revise) while the policy itself is trained with single-shot objectives. iGRPO instead bakes the refinement loop into the training objective, so the model learns to be a better refiner as part of its core RL optimization.

How This Paper Positions Itself

The paper's positioning can be understood through three design principles that distinguish iGRPO from prior work:

  1. Iteration within optimization, not just at inference. Unlike Self-Refine or Reflexion, which use iterative prompting as a post-hoc strategy, iGRPO makes iterative refinement part of the training signal. The model learns to improve upon its own outputs because the GRPO objective rewards it for doing so in Stage 2.

  2. Self-feedback as in-context scaffolding, not as a separate task. Unlike Self-Verification or Critique-GRPO, which require the model to produce auxiliary outputs (verification scores, critiques) that must be learned, iGRPO simply appends the best draft as additional context. The model doesn't need to learn to critique — it only needs to learn to refine. The ablation in Table 3 showing that iGRPO works with a GPT-5 judge reinforces this: the mechanism doesn't depend on the model generating the feedback, only on receiving it.

  3. The same reward signal drives both draft selection and policy optimization. This is a subtle but important design choice. Because both stages use the identical rule-based reward RφR_\varphi (Equation 6), there is no mismatch between what determines draft quality and what determines optimization success. The paper explicitly contrasts this with methods that use separate evaluators or learned critics, which can introduce reward gaming or distribution shift.

The paper also positions its contribution as a general refinement wrapper rather than a GRPO-specific modification. Table 2 demonstrates that applying the same two-stage self-feedback mechanism to DAPO and GSPO yields consistent improvements (+1.19 and +1.11 macro-average, respectively). This suggests that the core insight — conditioning training-time generations on the model's own best prior outputs — is separable from the specific choice of group-based PPO variant and represents a general principle for self-improvement in verifiable reasoning domains.

The Mathematical Reasoning Focus

The paper deliberately focuses on mathematical reasoning benchmarks (MATH, AIME, AMC, etc.) for a specific reason: these tasks have clean, automated verifiability through exact answer matching (Equation 6). This is not incidental — it is what makes the self-feedback mechanism tractable. Because the reward function is deterministic and objective, Stage 1 draft selection via arg⁡max⁡iRφ(di)\arg\max_i R_\varphi(d_i) is reliable: there is no ambiguity about which draft is "best." In domains where reward signals are noisy, subjective, or model-generated, the bootstrapping dynamic described in Proposition 3.1 could break down because the selected draft might not actually be better than alternatives.

The paper acknowledges this implicitly by testing generalization beyond pure math in Figure 2, where iGRPO applied to OpenReasoning-Nemotron-7B trained on AceReason-Math transfers gains to GPQA (+1.84) and MMLU-Pro (+0.91). This suggests the refinement behaviors learned during RL training on math generalize to adjacent reasoning domains, but the paper does not claim the method applies to open-ended generation or tasks without clear verifiability.

Summary of the Gap

The paper's core motivation can be distilled to a simple observation: current RL for reasoning treats generation as a one-shot process, but the problems being solved require iterative refinement. The gap is not that iterative refinement is unknown — it is well-studied at inference time and in human problem-solving. The gap is that no existing RL algorithm for LLM reasoning integrates this iterative refinement into the training loop in a way that (a) uses the same reward signal as optimization, (b) avoids requiring auxiliary learned behaviors, and (c) creates a bootstrapping dynamic where the conditioning signal co-evolves with the policy. iGRPO aims to fill exactly this gap.

3. Technical Approach

3.1 Reader Orientation

iGRPO is a reinforcement learning algorithm that trains a language model to solve math problems by first generating a "best attempt" draft from the model itself and then training the model to produce an even better answer when shown that draft as context. It solves the problem that standard RL treats each generation as an isolated attempt — never leveraging the model's own near-misses as scaffolding for improvement — by splitting each training step into an exploration phase (generate multiple drafts, keep the best one) and a refinement phase (generate new answers conditioned on that best draft, and optimize the policy to surpass it).

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components, though two of them (the base model and the reward function) are shared with standard GRPO:

  1. Base Policy ($\pi_\theta$) — the language model being trained, initialized from a pretrained checkpoint (e.g., DeepSeek-R1-Distill-Qwen-7B). It serves as both the draft generator in Stage 1 and the refinement generator in Stage 2, though only Stage 2 outputs receive gradient updates.

  2. Policy Snapshot ($\pi_{\theta_{\text{old}}}$) — a frozen copy of the policy taken at the start of each iteration, used for stable off-policy sampling in both stages. All completions (Stage 1 drafts and Stage 2 refinements) are sampled from this snapshot, while importance ratios in the objective are computed relative to it.

  3. Reward Function ($R_\varphi$) — a rule-based binary checker that extracts the final answer from a completion and compares it to the ground-truth reference answer, returning 1 for a match and 0 otherwise. This same scalar reward is used for three purposes: selecting the best Stage 1 draft ($\arg\max$), computing Stage 2 advantages (group normalization), and defining the optimization objective. No separate learned verifier or critic is needed.

  4. Stage 1: Exploratory Draft Generator — samples $N$ candidate completions from $\pi_{\theta_{\text{old}}}$ given the original prompt $q$, scores each with $R_\varphi$, and selects the maximum-reward draft $\hat{d}$. This draft becomes the self-feedback signal.

  5. Stage 2: Conditioned Refinement Sampler — constructs an augmented prompt $q' = \text{Concat}(q, \hat{d})$ by appending the best draft to the original prompt, then samples $G$ new completions from $\pi_{\theta_{\text{old}}}$ conditioned on $q'$. These Stage 2 completions are scored and used to compute group-normalized advantages and the PPO-style clipped surrogate objective, and their tokens are the only ones that receive gradient updates.

Information flows through the system in a fixed two-stage pipeline per training iteration: (1) the original prompt enters Stage 1 → $N$ drafts are generated and scored → the best draft is selected; (2) the best draft is appended to the original prompt → $G$ refinements are generated from the augmented prompt and scored → advantages are computed via group normalization → the policy is updated using the clipped surrogate objective with a KL penalty. The policy snapshot is refreshed at the start of each iteration, and across iterations the quality of the selected drafts increases as the policy improves, creating the bootstrapping dynamic.

3.3 Roadmap for the Deep Dive

  • First, the standard GRPO objective — how it works, what it optimizes, and its key mathematical components — because iGRPO is a direct extension and shares the same advantage estimation, clipping, and KL penalty machinery. Understanding GRPO's mechanics is prerequisite to understanding what iGRPO changes and why.

  • Second, the core conceptual innovation: dynamic self-conditioning — what it means, how it differs from static in-context learning, and the bootstrapping dynamic it creates. This is the "idea" behind iGRPO before we get to the equations.

  • Third, Proposition 3.1 and the theoretical characterization of why bootstrapping works — the mathematical guarantee that the expected quality of selected drafts increases monotonically with policy performance under binary rewards, and why this property is sufficient to create a virtuous cycle.

  • Fourth, the complete iGRPO objective and algorithm — how the two stages are formalized mathematically, what the augmented prompt construction looks like, how the importance ratios and advantage estimates are computed with respect to the augmented prompt, and the pseudocode that ties everything together.

  • Fifth, the computational cost analysis — how iGRPO achieves comparable generation cost to GRPO by redistributing the same total sampling budget across two stages rather than increasing it, and the specific budget allocation used in experiments.

  • Sixth, the reward function and its role — how the binary rule-based reward drives draft selection, advantage computation, and optimization, and why using the identical reward signal for both stages avoids the reward mismatch problems that affect critic-based or learned-verifier approaches.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a method paper whose core idea is that conditioning RL training updates on the policy's own highest-reward prior output — within the same optimization step — creates a bootstrapped refinement capability that single-shot objectives cannot produce. The technical contribution is the two-stage sampling and optimization procedure, along with the theoretical justification for why the selected drafts improve in expectation as the policy improves.

Standard GRPO: The Baseline Objective

GRPO (Group Relative Policy Optimization) is a value-function-free variant of PPO that replaces the learned critic (value function) with a group-based relative advantage estimate. To understand what iGRPO changes and why those changes matter, we need to understand GRPO's mechanics in detail, because iGRPO inherits all of GRPO's optimization machinery and adds only the two-stage sampling structure.

Group sampling. Given a prompt $q$, GRPO samples a group of $G$ candidate completions from the old policy snapshot $\pi_{\theta_{\text{old}}}$:

oi∼πθold(⋅∣q),i=1,…,G.o_i \sim \pi_{\theta_{\text{old}}}(\cdot \mid q), \quad i = 1, \ldots, G.

where $o_i$ is the $i$-th sampled completion (a sequence of tokens) and $\pi_{\theta_{\text{old}}}$ is the frozen policy snapshot at the start of the current iteration, used to ensure off-policy stability.

What it computes: For each prompt in the training batch, $G$ independent completions are drawn from the same policy. Each completion is a full autoregressive generation, with tokens sampled sequentially from $\pi_{\theta_{\text{old}}}(\cdot \mid q, \text{prefix})$. The result is a set of $G$ complete answer strings.

Why this form: Sampling from a frozen snapshot rather than the live policy prevents the distribution drift that would occur if each gradient update changed the sampling distribution mid-batch. This is standard PPO practice — the policy being optimized ($\pi_\theta$) may diverge from the sampling policy within an iteration, and importance sampling ratios correct for this divergence.

Reward evaluation and group normalization. Each completion $o_i$ is evaluated by a reward function $R_\varphi$ (in this paper's case, a binary rule-based checker that returns 1 if the extracted answer matches the ground truth and 0 otherwise). The resulting scores $\{R_1, \ldots, R_G\}$ are normalized within the group to compute a scalar advantage per completion:

A^i,t=Ri−mean({R1,…,RG})std({R1,…,RG})\hat{A}_{i,t} = \frac{R_i - \text{mean}(\{R_1, \ldots, R_G\})}{\text{std}(\{R_1, \ldots, R_G\})}

where $R_i = R_\varphi(o_i)$ is the scalar reward for the $i$-th completion, $\text{mean}(\cdot)$ and $\text{std}(\cdot)$ are the sample mean and sample standard deviation of the group's $G$ reward scores, and $t$ indexes token positions within completion $o_i$. If $\text{std}(\{R_1, \ldots, R_G\}) = 0$, the normalized advantage is set to 0 (or a small constant $\delta$ is added to the denominator). Note that all tokens $t$ in a given completion $o_i$ share the identical advantage $\hat{A}_{i,t} = \hat{A}_i$ — there is no per-token reward decomposition.

What it computes: For each completion in the group, its reward is centered by subtracting the group mean and scaled by dividing by the group standard deviation. A completion with above-average reward gets a positive advantage; a below-average completion gets a negative advantage. The magnitude reflects how many standard deviations above or below the mean the completion falls. Since all tokens in a completion are assigned this same single scalar advantage, every token in a high-reward completion is up-weighted equally, and every token in a low-reward completion is down-weighted equally.

Why this form: This replaces the learned value function that PPO normally uses as a baseline. The group mean serves as an empirical baseline — it estimates the expected reward under the current policy for that prompt — without requiring a separate critic model. The division by standard deviation normalizes the advantage scale across prompts with different reward variances, preventing prompts where all completions are similarly scored from dominating the gradient. For binary rewards specifically, the mean is the empirical success rate and the standard deviation is $\sqrt{p(1-p)}$ where $p$ is that rate, so the advantage takes values proportional to the z-score relative to the group's performance. This makes the optimization signal adaptive: on prompts where the model mostly succeeds or mostly fails, the advantage magnitudes are small (little to learn); on prompts with mixed success, the advantages are large (stronger learning signal).

Why value-function-free matters for iGRPO: The absence of a separate critic model is important context for iGRPO because it means the method cannot rely on learned intermediate value estimates to guide draft selection or refinement. iGRPO inherits this constraint — its Stage 1 draft selection uses the same scalar reward function, not a learned value function. This constraint is actually a design choice that keeps the system simple and avoids the reward hacking that can arise from optimizing against a learned proxy.

Importance sampling ratio. For each token at position $t$ in completion $o_i$, GRPO computes the ratio of probabilities under the current (optimized) policy versus the old (sampling) policy:

ri,t(θ)=πθ(oi,t∣q,oi,<t)πθold(oi,t∣q,oi,<t)r_{i,t}(\theta) = \frac{\pi_\theta(o_{i,t} \mid q, o_{i,<t})}{\pi_{\theta_{\text{old}}}(o_{i,t} \mid q, o_{i,<t})}

where $\pi_\theta(o_{i,t} \mid q, o_{i,<t})$ is the probability assigned to token $o_{i,t}$ by the current policy given the prompt $q$ and the preceding tokens $o_{i,<t}$, and the denominator is the same probability under the frozen snapshot.

What it computes: A per-token correction factor that accounts for the fact that completions were sampled from $\pi_{\theta_{\text{old}}}$ but we want to optimize $\pi_\theta$. A ratio of 1 means the current policy assigns the same probability as the snapshot; a ratio greater than 1 means the current policy assigns higher probability (more likely under the optimized policy); a ratio less than 1 means lower probability.

Why this form: This is the standard importance sampling weight in PPO. It allows unbiased (up to clipping) estimation of the policy gradient under $\pi_\theta$ using samples drawn from $\pi_{\theta_{\text{old}}}$. Without this ratio, optimizing on data from the old policy would push $\pi_\theta$ in directions that maximize reward under the old distribution, not the current one. The ratio corrects for the distribution shift, ensuring that tokens that become more probable under $\pi_\theta$ contribute positively to the gradient and tokens that become less probable contribute negatively.

Clipped surrogate objective. GRPO maximizes a clipped surrogate that prevents excessively large policy updates:

JGRPO(θ)=Eq∼P(Q),{oi}i=1G∼πθold[1G∑i=1G1∣oi∣∑t=1∣oi∣[min⁡(ri,t(θ)A^i,t,clip(ri,t(θ),1−ϵ,1+ϵ)A^i,t)−βD^KL(i,t)]]\mathcal{J}_{\text{GRPO}}(\theta) = \mathbb{E}_{q \sim P(Q), \{o_i\}_{i=1}^G \sim \pi_{\theta_{\text{old}}}} \left[ \frac{1}{G} \sum_{i=1}^G \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \left[ \min\left(r_{i,t}(\theta) \hat{A}_{i,t}, \text{clip}(r_{i,t}(\theta), 1 - \epsilon, 1 + \epsilon) \hat{A}_{i,t}\right) - \beta \hat{D}^{(i,t)}_{\text{KL}} \right] \right]

where $\epsilon$ is the PPO clipping parameter (controls how far the policy can move per update), $\beta$ is the KL divergence regularization coefficient, $|o_i|$ is the number of tokens in completion $o_i$, and $\hat{D}^{(i,t)}_{\text{KL}}$ is a per-token KL penalty estimator defined as:

D^KL(i,t)=πref(oi,t∣q,oi,<t)πθ(oi,t∣q,oi,<t)−log⁡πref(oi,t∣q,oi,<t)πθ(oi,t∣q,oi,<t)−1\hat{D}^{(i,t)}_{\text{KL}} = \frac{\pi_{\text{ref}}(o_{i,t} \mid q, o_{i,<t})}{\pi_\theta(o_{i,t} \mid q, o_{i,<t})} - \log\frac{\pi_{\text{ref}}(o_{i,t} \mid q, o_{i,<t})}{\pi_\theta(o_{i,t} \mid q, o_{i,<t})} - 1

where $\pi_{\text{ref}}$ is a reference policy (typically the initial pretrained model, frozen during RL training).

What the objective computes (step by step):

  1. For each prompt, sample $G$ completions from the old policy and compute their rewards.
  2. For each completion, compute per-token importance ratios $r_{i,t}(\theta)$ and a shared advantage $\hat{A}_i$ (from group normalization of rewards).
  3. For each token in each completion, compute the clipped surrogate: $\min(r_{i,t} \hat{A}_i, \text{clip}(r_{i,t}, 1-\epsilon, 1+\epsilon) \hat{A}_i)$. The clip function constrains $r_{i,t}$ to the interval $[1-\epsilon, 1+\epsilon]$. The $\min$ operation ensures that (a) if the advantage is positive, we don't reward the policy for increasing probability beyond $1+\epsilon$ times the snapshot probability (prevents over-optimism), and (b) if the advantage is negative, we don't penalize the policy for decreasing probability beyond $1-\epsilon$ times the snapshot probability (prevents over-pessimism).
  4. Subtract the KL penalty $\beta \hat{D}^{(i,t)}_{\text{KL}}$ to prevent the policy from diverging too far from the reference (pretrained) distribution — this preserves general language capabilities.
  5. Average over all tokens in all completions, with each completion weighted inversely by its length ($1/|o_i|$) so that long completions don't dominate the gradient.

Why this form: The clipped surrogate is the defining PPO mechanism for stable policy updates. Without clipping, a large importance ratio (e.g., the policy suddenly assigns probability 0.9 to a token that previously had probability 0.01, giving $r_{i,t} = 90$) could cause a destructively large gradient step. The clip bounds the effective ratio, ensuring that no single token can change the policy by more than $\epsilon$ in probability space per iteration. The KL penalty adds an additional constraint: even within the clip bounds, the policy shouldn't drift far from the reference distribution. The $1/|o_i|$ normalization ensures that prompts producing longer completions don't receive proportionally larger gradients, which would cause the loss to be dominated by verbose outputs rather than correct ones.

The KL estimator form: The paper uses Schulman's non-negative KL estimator, which is guaranteed to be $\geq 0$ for any $\pi_\theta$ and $\pi_{\text{ref}}$. This is an f-divergence approximation: $\hat{D}_{\text{KL}} = r - \log r - 1$ where $r = \pi_{\text{ref}} / \pi_\theta$. When $\pi_\theta = \pi_{\text{ref}}$, $r = 1$, $\log r = 0$, and the estimator is $0$. When the policies diverge, the estimator increases, penalizing the divergence. The expectation of this estimator under samples from $\pi_\theta$ equals the true KL divergence $D_{\text{KL}}(\pi_\theta \parallel \pi_{\text{ref}})$, making it an unbiased sample-based penalty. The paper's experiments set $\beta = 0$ (Section 4.1, Table S.4), effectively disabling the KL penalty, since the ablation in Appendix E shows minimal sensitivity to $\beta$ values (range 0 to 0.01 changes the score by less than 1 percentage point).

Dynamic Self-Conditioning: The Core Conceptual Innovation

The fundamental difference between GRPO and iGRPO is not in the optimization objective itself — both use the same clipped surrogate, the same group-based advantage estimation, and the same KL penalty structure. The difference is in what the model conditions on when generating completions during training.

Standard ICL is static. In standard in-context learning (ICL), the model conditions on a fixed demonstration $e$ during training:

JICL(θ)=Eq∼P(Q)[Eo∼πθ(⋅∣q,e)[Rφ(o)]]\mathcal{J}_{\text{ICL}}(\theta) = \mathbb{E}_{q \sim P(Q)} \left[ \mathbb{E}_{o \sim \pi_\theta(\cdot \mid q, e)} [R_\varphi(o)] \right]

where $e$ is a static example that never changes throughout training. This is what happens, for instance, when training data includes few-shot examples: the prompt $q$ is prepended with demonstrations, but those demonstrations are the same for every training step. The paper identifies this as a "fundamental limitation" because as the policy improves, the static demonstrations become stale — they don't reflect the model's growing capabilities. A model that now achieves 80% accuracy on AIME problems is still training on demonstrations that may contain mistakes or suboptimal reasoning patterns from an earlier, weaker version of itself.

Dynamic self-conditioning is policy-dependent. iGRPO replaces the static $e$ with a policy-generated signal $\hat{d}_\theta(q)$:

JiGRPO(θ)=Eq∼P(Q)[Eo∼πθ(⋅∣qθ′(q))[Rφ(o)]]\mathcal{J}_{\text{iGRPO}}(\theta) = \mathbb{E}_{q \sim P(Q)} \left[ \mathbb{E}_{o \sim \pi_\theta(\cdot \mid q'_\theta(q))} [R_\varphi(o)] \right]

where $q'_\theta(q) = \text{Concat}(q, \hat{d}_\theta(q))$ and $\hat{d}_\theta(q)$ is the best draft generated by the current policy. Note the subscript $\theta$ on both the draft selection and the conditioning prompt — this is what makes the system "dynamic." As $\theta$ changes across training iterations, so does the distribution of selected drafts $\hat{d}_\theta(q)$, and so does the augmented prompt $q'_\theta(q)$ that Stage 2 completions condition on.

What this creates: a coupled dynamical system. The policy $\pi_\theta$ determines the quality of drafts, the selected draft determines the conditioning context for refinement, the refinement objective shapes how $\pi_\theta$ updates, and the updated $\pi_\theta$ generates better drafts in the next iteration. This is a feedback loop, not a feedforward process. The paper calls this a "bootstrapping effect" because the model is literally pulling itself up by its own bootstraps — using its own outputs as scaffolding to reach higher performance.

Why the policy snapshot matters for stability. The paper is careful to note (Section 3.2.1) that although the conceptual description uses $\pi_\theta$ for clarity, the actual implementation follows the PPO convention: "at each iteration, we take a snapshot $\pi_{\theta_{\text{old}}}$, sample both Stage 1 drafts and Stage 2 completions from $\pi_{\theta_{\text{old}}}$, and update $\theta$ using importance ratios computed relative to $\pi_{\theta_{\text{old}}}$." This means the draft selection and refinement generation are off-policy within a single iteration — they use the frozen snapshot — but across iterations the snapshot is updated, so the distribution of conditioning signals shifts. Stage 1 is not differentiated through (the $\arg\max$ is non-differentiable anyway), so the gradient flow is: Stage 2 completions → importance ratios → policy update → new snapshot → new drafts in the next iteration. The coupling is across iterations, not within a single backward pass.

Training-only vs. inference-time usage. A critical design choice: dynamic self-conditioning is used only during training. At inference time, the trained policy is used in standard single-shot mode — it generates directly from the original prompt $q$ without any draft generation, conditioning, or specialized selection scheme (Section 3.2.1). The model learns to be a better reasoner because it was trained to refine drafts, but it doesn't require drafts at test time. This is intentional: the refinement skill is learned as a capability that generalizes to first-shot generation, not as a separate inference-time procedure that would require extra computation.

Proposition 3.1: Why Bootstrapping Works (Binary Rewards)

The bootstrapping dynamic described above depends on a simple premise: as the policy gets better, the drafts it selects as conditioning should also get better. Proposition 3.1 provides a theoretical guarantee for this premise under the binary reward function used throughout the paper.

Formal statement. Assume the reward is binary, $R_\varphi(o) \in \{0, 1\}$, and Stage 1 drafts $\{d_i\}_{i=1}^N$ are sampled i.i.d. from $\pi_\theta(\cdot \mid q)$. Define $V_\theta(q) = \mathbb{E}_{o \sim \pi_\theta(\cdot \mid q)} [R_\varphi(o)]$ as the expected reward under policy $\pi_\theta$ for prompt $q$. Then:

E[Rφ(d^θ(q))]=1−(1−Vθ(q))N\mathbb{E}\left[ R_\varphi(\hat{d}_\theta(q)) \right] = 1 - (1 - V_\theta(q))^N

where $\hat{d}_\theta(q) = \arg\max_{i \in \{1,\ldots,N\}} R_\varphi(d_i)$ is the maximum-reward draft among $N$ samples.

What the equation means in words: The probability that the best-of-$N$ draft is correct is 1 minus the probability that all $N$ drafts are incorrect. If each draft has success probability $V_\theta(q)$ (independently), then all $N$ fail with probability $(1 - V_\theta(q))^N$, so at least one succeeds — and gets selected by $\arg\max$ — with probability $1 - (1 - V_\theta(q))^N$. The expected reward of the selected draft equals this probability because the reward is binary (it's either 0 or 1, so its expectation is the probability of being 1).

Why this is sufficient for a virtuous cycle: The function $f(x) = 1 - (1 - x)^N$ is monotonically increasing in $x$ for $x \in [0, 1]$. This means:

  • If the policy improves — i.e., $V_\theta(q)$ increases — then $\mathbb{E}[R_\varphi(\hat{d}_\theta(q))]$ also increases.
  • The expected quality of the draft used as Stage 2 conditioning is always at least as high as the policy's own expected performance: $\mathbb{E}[R_\varphi(\hat{d}_\theta(q))] \geq V_\theta(q)$ because $1 - (1 - x)^N \geq x$ for $x \in [0, 1]$ (the best-of-N amplifies the base success rate).
  • As $N$ increases, the amplification becomes stronger: with $N = 8$ and $V_\theta(q) = 0.3$, the expected draft quality is $1 - 0.7^8 \approx 0.94$ — far higher than the base rate. This means even a modestly capable policy can provide high-quality conditioning for Stage 2.

What this proposition does NOT say: It guarantees that the selected draft is good in expectation, but not that Stage 2 will successfully refine it. If the model cannot learn to produce answers better than the draft it conditions on, the bootstrapping stalls — the draft quality improves (by the proposition) but the policy doesn't learn to surpass it. The empirical results (Table 1) demonstrate that this refinement capability does emerge in practice, but Proposition 3.1 alone does not guarantee it.

Why binary rewards enable this analysis: With binary rewards, $R_\varphi(\hat{d}_\theta(q))$ is 1 if and only if at least one of the $N$ drafts achieves reward 1. The $\arg\max$ operator selects any draft with reward 1 if one exists. With continuous or multi-level rewards, the analysis would be more complex — you'd need to reason about the distribution of the maximum of $N$ samples from an arbitrary distribution — but the core monotonicity property (better policies → better best-of-N drafts in expectation) would still hold under mild conditions. The paper uses binary rewards throughout (Equation 6), so the simplified analysis applies directly.

The Complete iGRPO Objective and Algorithm

Building on the GRPO formulation from Section 3.1 and the dynamic self-conditioning concept from Section 3.2.1, the full iGRPO objective has the same structure as Equation 1 but with one critical difference: the prompt that Stage 2 completions condition on is the augmented prompt $q'$, not the original prompt $q$.

Stage 1: Draft generation and selection. For each training prompt $q$, sample $N$ candidate drafts from the policy snapshot:

di∼πθold(⋅∣q),i=1,…,Nd_i \sim \pi_{\theta_{\text{old}}}(\cdot \mid q), \quad i = 1, \ldots, N

Then select the draft with the highest reward:

d^=arg⁡max⁡i∈{1,…,N}Rφ(di)\hat{d} = \arg\max_{i \in \{1,\ldots,N\}} R_\varphi(d_i)

where $N$ is the number of Stage 1 drafts per prompt. In the paper's experiments, $N = 4$ when the total sampling budget is 8 completions per prompt (Section 4.1 states "generating eight completions per prompt (halved in each stage for iGRPO)"). The $\arg\max$ selection is non-differentiable and no gradient flows through it — Stage 1 exists only to construct the conditioning context for Stage 2.

Stage 2: Augmented prompt construction and conditioned refinement. Form the augmented prompt by appending the selected draft to the original prompt:

q′=Concat(q,d^)q' = \text{Concat}(q, \hat{d})

Then sample $G$ conditioned completions from the same policy snapshot:

oj∼πθold(⋅∣q′),j=1,…,Go_j \sim \pi_{\theta_{\text{old}}}(\cdot \mid q'), \quad j = 1, \ldots, G

where $G$ is the number of Stage 2 completions. In the paper's matched-budget experiments, $G = 4$ when the total is 8 (so $N + G = 8$). The concatenation uses a fixed prompt template (provided in Appendix C), which includes instructions like "If the prompt contains feedback or a prior draft, treat it as guidance, not as ground truth. Use it to produce a strictly improved final answer."

Advantage computation (identical to GRPO but on augmented prompts). Each Stage 2 completion $o_j$ is scored with $R_\varphi$, and advantages are computed via group normalization over the $G$ Stage 2 completions:

A^j=Rφ(oj)−mean({Rφ(o1),…,Rφ(oG)})std({Rφ(o1),…,Rφ(oG)})\hat{A}_j = \frac{R_\varphi(o_j) - \text{mean}(\{R_\varphi(o_1), \ldots, R_\varphi(o_G)\})}{\text{std}(\{R_\varphi(o_1), \ldots, R_\varphi(o_G)\})}

Note: the advantages are computed within the group of Stage 2 completions only — the Stage 1 drafts are not included in the group normalization. This means the baseline for Stage 2 is the average performance of the refinement attempts, not the average of all generations across both stages.

Why normalize within Stage 2 only: Including Stage 1 drafts in the group normalization would mix two different tasks — unconditional generation and draft-conditioned refinement — which have different expected reward distributions. The draft-conditioned completions should, in principle, have higher expected reward because they benefit from the in-context scaffolding. Pooling them with unconditional drafts would artificially inflate advantages for unconditional drafts (which look good relative to the mixture) and deflate advantages for conditioned drafts. By normalizing within Stage 2, the advantage for a refinement is computed relative to other refinements of the same draft — so the model learns which refinements improve upon the draft and which don't, not just which completions are better than the overall average.

Importance sampling ratio (augmented prompt). The ratio is computed with respect to the augmented prompt, not the original prompt:

rj,t(θ)=πθ(oj,t∣q′,oj,<t)πθold(oj,t∣q′,oj,<t)r_{j,t}(\theta) = \frac{\pi_\theta(o_{j,t} \mid q', o_{j,<t})}{\pi_{\theta_{\text{old}}}(o_{j,t} \mid q', o_{j,<t})}

where $o_{j,t}$ is the $t$-th token of the $j$-th Stage 2 completion, $o_{j,<t}$ are the preceding tokens in that completion, and the conditioning context is $q'$ (the augmented prompt containing the original problem and the best draft).

What this means operationally: The policy $\pi_\theta$ is evaluated on the augmented prompt — it sees both the original problem AND the best draft as context, and must assign a probability to each token given this full context. The importance ratio measures how much more (or less) likely the current policy is to generate each refinement token compared to the snapshot policy, given the same augmented context. This is crucial: the optimization signal encourages the policy to become better at generating tokens conditional on having seen a high-quality draft, which is precisely the refinement skill.

Full objective. The complete iGRPO surrogate objective is:

JiGRPO(θ)=Eq∼P(Q)E{di}i=1N∼πθold(⋅∣q),  d^=arg⁡max⁡iRφ(di),  q′=Concat(q,d^),  {oj}j=1G∼πθold(⋅∣q′)[1G∑j=1G1∣oj∣∑t=1∣oj∣[min⁡(rj,t(θ)A^j,clip(rj,t(θ),1−ϵ,1+ϵ)A^j)−βD^KL(j,t)]]\mathcal{J}_{\text{iGRPO}}(\theta) = \mathbb{E}_{q \sim P(Q)} \mathbb{E}_{\{d_i\}_{i=1}^N \sim \pi_{\theta_{\text{old}}}(\cdot \mid q), \; \hat{d} = \arg\max_i R_\varphi(d_i), \; q' = \text{Concat}(q, \hat{d}), \; \{o_j\}_{j=1}^G \sim \pi_{\theta_{\text{old}}}(\cdot \mid q')} \left[ \frac{1}{G} \sum_{j=1}^G \frac{1}{|o_j|} \sum_{t=1}^{|o_j|} \left[ \min\left(r_{j,t}(\theta) \hat{A}_j, \text{clip}(r_{j,t}(\theta), 1 - \epsilon, 1 + \epsilon) \hat{A}_j\right) - \beta \hat{D}^{(j,t)}_{\text{KL}} \right] \right]

where the KL penalty is also computed under the augmented prompt:

D^KL(j,t)=πref(oj,t∣q′,oj,<t)πθ(oj,t∣q′,oj,<t)−log⁡πref(oj,t∣q′,oj,<t)πθ(oj,t∣q′,oj,<t)−1\hat{D}^{(j,t)}_{\text{KL}} = \frac{\pi_{\text{ref}}(o_{j,t} \mid q', o_{j,<t})}{\pi_\theta(o_{j,t} \mid q', o_{j,<t})} - \log\frac{\pi_{\text{ref}}(o_{j,t} \mid q', o_{j,<t})}{\pi_\theta(o_{j,t} \mid q', o_{j,<t})} - 1

What is the structural difference from GRPO? Everything in the $\min$ and KL penalty terms is identical to GRPO — the same clipping mechanism, the same advantage normalization, the same KL estimator. The only difference is that all Stage 2 probabilities are conditioned on $q'$ (the augmented prompt) rather than $q$ (the original prompt). The expectation is taken over a two-stage sampling process that GRPO does not have: first sample drafts and select the best, then sample refinements conditioned on that best draft. The gradient only flows through the Stage 2 terms — Stage 1 contributes to the expectation of $q'$ but not to the gradient with respect to $\theta$ within a single iteration.

Why the two-stage structure creates a different learning signal: In GRPO, every completion in the group faces the same prompt and the same expected difficulty. The advantage for a completion is simply whether it did better or worse than the average attempt. In iGRPO, the Stage 2 completions face an easier prompt — they have the best draft as scaffolding. The advantage for a Stage 2 completion measures whether the model successfully refined the draft (producing a better answer than other refinement attempts) or failed to refine it (producing a worse answer or repeating the draft verbatim). The model learns to do something GRPO never asks it to do: produce an answer that is strictly better than a provided high-quality reference. This reframing of the learning problem — from "generate a correct answer" to "improve upon this draft" — is the mechanism by which iGRPO teaches refinement.

Algorithm 1 (Pseudocode) in plain English:

  1. Initialize the policy $\pi_\theta$ from the pretrained model and set the reference policy $\pi_{\text{ref}}$ to the same initial weights.
  2. For each training iteration: a. Freeze a snapshot: $\pi_{\theta_{\text{old}}} \leftarrow \pi_\theta$. b. Sample a batch of $S$ prompts with their reference answers from the training set. c. For each prompt in the batch:
    • Stage 1: Sample $N$ drafts from $\pi_{\theta_{\text{old}}}(\cdot \mid q)$, compute their rewards against the reference answer, and select the highest-reward draft $\hat{d}$.
    • Stage 2: Construct the augmented prompt $q' = \text{Concat}(q, \hat{d})$, sample $G$ refinements from $\pi_{\theta_{\text{old}}}(\cdot \mid q')$, compute their rewards, and compute advantages via group normalization (Equation 4, equivalent to the GRPO advantage formula). d. Compute the iGRPO objective (Equation 5) over all Stage 2 completions and update $\theta$ using gradient ascent: $\theta \leftarrow \theta + \eta \nabla_\theta \mathcal{J}_{\text{iGRPO}}(\theta)$.
  3. Return the trained policy $\pi_\theta$.

Important detail — what is NOT learned: Stage 1 is pure exploitation with no learning signal. The model is not trained to produce better drafts because no gradient flows through Stage 1 tokens. The improvement in draft quality happens only indirectly: Stage 2 trains the model to be better at generating refinements, which leads to a better policy overall, which produces better drafts in the next iteration's Stage 1 (because the frozen snapshot is updated at the start of each iteration). The draft selection criterion ($\arg\max$ over reward) is a fixed rule, not a learned behavior.

The policy gradient derivation (Appendix A): The paper provides a formal policy gradient derivation showing that the iGRPO gradient has the same form as the GRPO gradient but with the augmented prompt:

∇θJiGRPO(θ)=E[⋯ ]1G∑j=1G1∣oj∣∑t=1∣oj∣[Ij,t(θ)A^jrj,t(θ)+β(ρj,t(θ)−1)]∇θlog⁡πθ(oj,t∣q′,oj,<t)\nabla_\theta \mathcal{J}_{\text{iGRPO}}(\theta) = \mathbb{E}[\cdots] \frac{1}{G} \sum_{j=1}^G \frac{1}{|o_j|} \sum_{t=1}^{|o_j|} \left[ \mathbb{I}_{j,t}(\theta) \hat{A}_j r_{j,t}(\theta) + \beta (\rho_{j,t}(\theta) - 1) \right] \nabla_\theta \log \pi_\theta(o_{j,t} \mid q', o_{j,<t})

where $\mathbb{I}_{j,t}(\theta)$ is the indicator for whether token $(j,t)$ falls in the unclipped branch of the PPO objective (1 if the ratio hasn't hit the clip boundary, 0 otherwise), and $\rho_{j,t}(\theta) = \pi_{\text{ref}}(o_{j,t} \mid q', o_{j,<t}) / \pi_\theta(o_{j,t} \mid q', o_{j,<t})$ is the reference-to-policy ratio used in the KL penalty gradient. This derivation confirms that the gradient mechanically decomposes into: (1) an advantage-weighted, ratio-scaled, clipped term that pushes the policy toward tokens from high-reward completions, and (2) a KL penalty term that pulls the policy toward the reference distribution. All of this is computed under the augmented prompt $q'$, which is the only structural difference from GRPO's gradient.

Computational Cost Analysis

The paper explicitly addresses the concern that a two-stage method would be more expensive than single-stage GRPO, arguing that iGRPO can be run at "essentially the same dominant generation cost" by redistributing the same total sampling budget.

Cost model. Let $C_{\text{gen}}$ be the cost of producing one sampled completion, which includes prompt encoding, autoregressive decoding of the full response, and reward evaluation. The dominant cost in both GRPO and iGRPO is the generation of sampled completions — everything else (forward passes for log-probability computation, gradient computation and application) is comparatively small.

GRPO cost. With $G_{\text{GRPO}}$ completions sampled per prompt, the per-prompt rollout cost is:

CGRPO≈GGRPO⋅CgenC_{\text{GRPO}} \approx G_{\text{GRPO}} \cdot C_{\text{gen}}

iGRPO cost. With $N$ Stage 1 drafts and $G$ Stage 2 refinements, the per-prompt rollout cost is:

CiGRPO≈(N+G)⋅CgenC_{\text{iGRPO}} \approx (N + G) \cdot C_{\text{gen}}

Matched-budget constraint. The paper's experiments keep the total sampling budget fixed by setting $N + G = G_{\text{GRPO}}$. This means iGRPO redistributes the same number of rollouts across two stages rather than increasing the total. Specifically, with $G_{\text{GRPO}} = 16$ in GRPO, the paper uses $N = 8$ and $G = 8$ in iGRPO, yielding:

CiGRPO≈(N+G)⋅Cgen=GGRPO⋅Cgen≈CGRPOC_{\text{iGRPO}} \approx (N + G) \cdot C_{\text{gen}} = G_{\text{GRPO}} \cdot C_{\text{gen}} \approx C_{\text{GRPO}}

The paper states this explicitly: "iGRPO redistributes the same number of rollouts across Stage 1 and Stage 2 rather than increasing them" (Section 3.2.5). In the main experiments (Section 4.2), the total is 8 completions per prompt, split as $N = 4$ and $G = 4$.

Empirical measurements (Appendix D). The paper provides actual measurements on an 80 GB A100 setup with DeepSeek-R1-Distill-Qwen-7B trained on MATH:

  • Peak memory: 54.9286 GB (GRPO) vs. 54.9349 GB (iGRPO) — a difference of 0.0063 GB, which is negligible. The two-stage design does not require storing additional model copies or intermediate activations beyond what GRPO already stores.
  • Throughput: 0.41 samples/sec (GRPO) vs. 0.34 samples/sec (iGRPO) — a roughly 20% reduction. This is attributed to the additional round of generation (Stage 1 drafts) and the fact that Stage 2 refinement prompts are longer (they include the draft), which increases the cost of both encoding and generation.
  • Total GPU hours for a full training run: 83.3 GPU hours (GRPO) vs. 94.1 GPU hours (iGRPO) — approximately a 13% increase in wall-clock training time.

Why the throughput reduction is acceptable: The paper argues that this 13% time overhead is a "modest increase" that is "favorable" given the performance gains — several percentage points on AIME benchmarks (e.g., +4.06 points on AIME24 for DeepSeek-R1-Distill-Qwen-14B). The tradeoff is between slightly longer training time and substantially higher reasoning accuracy, and the paper presents this as a positive value proposition.

Why not just increase the GRPO budget? A natural question: if iGRPO uses the same total samples as GRPO but takes 13% longer, what if we instead gave GRPO a 13% larger sampling budget? The paper's ablation in Appendix E (Table S.4, bottom) addresses this indirectly by varying the number of total completions in iGRPO: performance improves from 67.79% (4 completions) to 69.87% (8 completions) to 70.17% (16 completions) to 70.33% (32 completions). The gain from 8 to 16 completions is 0.30 points, which is substantially smaller than the gap between iGRPO and GRPO at 8 completions (roughly 1.5 points based on the DeepSeek-R1-Distill-Qwen-7B averages in Table 1). This suggests that simply increasing the GRPO budget would not close the gap — the two-stage structure provides a qualitatively different training signal that more samples alone cannot replicate.

The Reward Function and Its Dual Role

The reward function used throughout is a rule-based binary checker:

Rφ(o)=1[extract(o)=a]R_\varphi(o) = \mathbf{1}[\text{extract}(o) = a]

where $\text{extract}(o)$ parses the final answer from the model's completion string (the portion after a delimiter like <answer>...</answer>), $a$ is the ground-truth reference answer from the training instance, and $\mathbf{1}[\cdot]$ is the indicator function (1 if the extracted answer matches the reference, 0 otherwise).

What it computes: A binary correctness signal. There is no partial credit — a solution that is 99% correct but has a single arithmetic error at the end receives exactly the same reward (0) as a completely nonsensical response. This is the standard verifiable reasoning setup: the reward is deterministic, objective, and can be computed automatically from the ground-truth answer. The $\text{extract}$ function handles formatting variation (the model might output the answer as "x = 42" or "42" or "The answer is 42") by isolating the mathematical expression and comparing it to the reference.

Why binary: For competition math problems like AIME, the answer is a single integer, making exact-match verification straightforward and reliable. The binary nature simplifies both the theoretical analysis (Proposition 3.1 relies on it) and the practical optimization (no need to design a continuous reward scale). The downside — zero signal for near-misses — is exactly the problem iGRPO aims to mitigate by using the best draft as a scaffold that can capture partially-correct reasoning.

The dual role of the reward function:

  1. Draft selection (Stage 1): The reward function is used as a ranking criterion to select the best draft via $\arg\max$. Because the reward is binary, the selection logic is: if any draft achieves reward 1, select one of those (they're all equally "best"); if no draft achieves reward 1, select any draft (they're all equally "worst" with reward 0). This means that on hard problems where no draft is correct, the selected draft is essentially random among the $N$ candidates — the conditioning signal provides no useful guidance. The bootstrapping dynamic only works when the policy is good enough to produce at least some correct drafts, which is consistent with the empirical finding (Section 4.2) that gains are largest on problems within the model's capability range.

  2. Advantage computation and optimization (Stage 2): The same reward function scores Stage 2 completions, and these scores are used for group normalization (Equation 4) and for defining the optimization objective (Equation 5). Because the reward used for optimization is identical to the reward used for draft selection, there is no mismatch between what defines a "good draft" and what defines optimization success. If a different reward signal were used for draft selection (e.g., a learned verifier or a model-generated score), there would be a risk of selecting drafts that score highly under the selection criterion but don't actually represent correct solutions — a form of reward hacking at the draft selection stage.

Generative judge as an extension (Table 3): The paper demonstrates that iGRPO is compatible with richer reward signals by replacing the binary rule-based checker with a GPT-5 judge that scores each solution on a $[0, 1]$ scale. The improvement (+0.94 macro-average) is attributed to partial credit for near-miss solutions, which allows them to survive Stage 1 selection (they receive non-zero scores and can be the $\arg\max$ even if not fully correct) and be refined into correct answers in Stage 2. This is presented as a modular extension — the iGRPO framework only requires a scalar reward for draft selection and advantage computation, and the reward can be rule-based, model-based, or any other source as long as it provides a meaningful ranking signal.

Hyperparameter Configuration and Design Choices

The paper specifies a concrete set of hyperparameters used across all experiments (Section 4.1 and Appendix C):

Training configuration:

  • Learning rate: $1 \times 10^{-6}$ with a cosine schedule (minimum learning rate ratio 0.1)
  • KL divergence coefficient $\beta$: 0 (no KL penalty, based on the ablation in Table S.4 showing minimal sensitivity to $\beta$)
  • No entropy regularization (explicitly stated)
  • One training epoch for all experiments
  • Global batch size: 128 (per-device batch size 16 with gradient accumulation steps 8 on 7B models; per-device batch size 4 with accumulation steps maintaining total 128 on 14B models)
  • Maximum prompt length: 1,024 tokens
  • Maximum completion length: 4,096 tokens
  • Total completions per prompt: 8 (split as $N = 4$ Stage 1 drafts and $G = 4$ Stage 2 refinements)

Generation configuration:

  • Temperature: 0.7 for training rollouts (encourages exploration and diversity in drafts)
  • No top-k or top-p specified for training (though evaluation uses temperature 0.6 and top-p 0.95)

Training datasets:

  • MATH: 7,500 step-by-step competition problems (used for the main controlled experiments in Table 1)
  • AceReason-Math: 9,400 problems (used for the state-of-the-art results with OpenReasoning-Nemotron-7B in Figure 2)

Infrastructure:

  • 7B models: 2 nodes, each with 8 × NVIDIA A100 GPUs, one node reserved for vLLM generation
  • 14B models: 5 nodes of 8 × NVIDIA A100 GPUs each, one node reserved for vLLM
  • Precision: bfloat16
  • Attention implementation: FlashAttention-2
  • Reward weights: accuracy reward weight 1.0, format reward weight 1.0 (two reward components summed)

Why two reward components: The paper uses both an accuracy reward (the binary answer match) and a format reward (checking that the model follows the required output format, e.g., including reasoning in a thinking... section and the final answer in <answer>...</answer> tags). The format reward encourages the model to produce parseable outputs, which is necessary for the extract function to work reliably. Both components have equal weight (1.0), so a well-formatted correct answer gets total reward 2, a well-formatted incorrect answer gets 1, and a malformed answer gets 0.

Why no KL penalty: The ablation in Table S.4 shows that varying $\beta$ from 0 to 0.01 changes performance by less than 1 percentage point (range: 69.31% to 70.23%). Setting $\beta = 0$ simplifies the training pipeline without significant performance loss. This is somewhat unusual — PPO typically relies on KL regularization to prevent catastrophic forgetting — but the paper's explanation is likely that (a) the training is only one epoch on math-specific data, which limits the degree of distribution shift, and (b) the MATH dataset is similar enough to the models' pretraining distribution (math-heavy) that catastrophic forgetting is less of a concern than it would be for, say, fine-tuning on a completely different domain.

Why temperature 0.7 for training: The relatively high temperature (0.7) encourages diverse draft generation in Stage 1, which is important because the bootstrapping dynamic depends on having enough exploration to occasionally produce correct drafts even on hard problems. Lower temperatures would produce more deterministic outputs, reducing the probability of finding a correct draft in Stage 1 and thus limiting the quality of the conditioning signal for Stage 2.

Why 4+4 split instead of other allocations: The paper does not ablate the Stage 1 vs. Stage 2 allocation directly (e.g., comparing $N=2, G=6$ vs. $N=6, G=2$ at the same total budget of 8). This is a notable omission — the choice of equal split seems arbitrary. The ablation on total completions (Table S.4, bottom) shows that increasing from 4 to 8 total completions (which would correspond to, e.g., $N=2, G=2$ vs. $N=4, G=4$) yields a clear improvement (67.79% → 69.87%), but the allocation ratio within the total budget is not studied. In principle, one might expect that harder problems benefit from more Stage 1 drafts (to increase the chance of finding a correct draft) while easier problems benefit from more Stage 2 refinements (to learn to polish already-good drafts), but the uniform allocation treats all prompts identically.

The prompt template (Appendix C): The prompt used for training includes explicit instructions for how to handle the draft in Stage 2:

"If the prompt contains feedback or a prior draft, treat it as guidance, not as ground truth. Use it to produce a strictly improved final answer: fix mistakes, fill gaps, strengthen justification, and improve clarity. Do not repeat the draft verbatim. If the feedback is wrong or incomplete, correct it and proceed."

This template is important because it explicitly instructs the model to (a) not copy the draft verbatim (preventing reward hacking by simply repeating a correct draft), (b) produce a strictly improved answer (encouraging refinement beyond the draft), and (c) correct errors in the draft if present (teaching the model to be critical of its own outputs). The inclusion of this template is a design choice — it shapes the model's behavior in Stage 2 beyond what the reward signal alone would teach, by providing explicit natural-language instructions about the desired refinement behavior.

Summary of Design Choices and Their Justifications

  • Two stages within one optimization step, not sequential training phases: Having both stages occur within the same iteration — using the frozen snapshot for both — means the policy learns to refine its own current best outputs, not outputs from a stale earlier version. This enables the bootstrapping dynamic described in Proposition 3.1 within a single training run.

  • Same reward function for draft selection and optimization: Eliminates the mismatch between selection criteria and optimization criteria that would arise if different reward signals were used at different stages. Also avoids the need for a separate learned verifier or critic.

  • Group normalization within Stage 2 only (excluding Stage 1): Ensures that Stage 2 advantages measure refinement quality (improvement over the draft) rather than absolute quality relative to unconditional generation. The baseline for a refinement is other refinements of the same draft, not all completions.

  • $\arg\max$ selection with no gradient through Stage 1: Keeps the optimization simple — only Stage 2 tokens contribute to gradients — while still enabling the draft quality to improve across iterations as the policy snapshot is updated. The non-differentiable selection avoids the complexity of learning to select good drafts.

  • Matched rollout budget ($N + G = G_{\text{GRPO}}$): Ensures that comparisons between iGRPO and GRPO reflect the effect of the two-stage structure rather than the effect of more sampling. The 13% training time overhead is acknowledged as a tradeoff for the accuracy gains.

  • Binary reward with no partial credit: Simplifies both the theoretical analysis and the practical implementation, at the cost of providing no signal for near-misses. The Stage 1→Stage 2 structure is designed to partially compensate for this by capturing promising-but-incorrect drafts as scaffolding for refinement.

  • No KL penalty in practice ($\beta = 0$): Based on ablation showing insensitivity to the KL coefficient, the simpler configuration is used. This may be dataset-specific — MATH problems are within the pretraining distribution for math-capable models, so KL regularization against the pretrained policy may be less necessary.

  • Temperature 0.7 for training generation: Balances exploration (needed for Stage 1 to find correct drafts) and quality (needed for Stage 2 to learn from those drafts). Higher temperature would increase diversity but might reduce the overall correctness rate; lower temperature would increase correctness but might limit the exploration needed to find good drafts on hard problems.

  • Explicit prompt template instructions for refinement behavior: Supplements the reward-driven optimization with natural-language guidance about the desired refinement behavior (improve, don't copy, correct errors). This is an inductive bias that speeds up learning of the refinement skill by clarifying the task to the model.

4. Key Insights and Innovations

Innovation 1: Training-Time Iteration as a First-Class Optimization Signal

The paper's most fundamental conceptual move is reframing the problem from "optimize the policy to produce correct answers" to "optimize the policy to improve upon its own best attempts." This is not merely an inference-time strategy layered on top of an existing optimizer — it restructures the RL training loop itself to make iterative refinement the objective, not a post-hoc procedure.

What the field did before. Prior self-improvement methods for LLM reasoning operated almost entirely at inference time. Self-Refine (Madaan et al., 2023) and Reflexion (Shinn et al., 2023) generate an initial output, critique it, and produce a revision — but the underlying policy is trained with single-shot objectives and the refinement loop is an external scaffold. Even methods that incorporate self-generated feedback into RL training, such as Self-Verification (Zhang et al., 2025a) and Critique-GRPO (Zhang et al., 2025b), treat the feedback generation (verification rationales, natural-language critiques) as an auxiliary behavior that the model must learn alongside the primary reasoning task. The optimization signal remains: "generate a correct answer." The refinement is a separate skill, trained with a potentially misaligned objective.

What's distinctive. iGRPO redefines the optimization target itself. Stage 2 completions are not evaluated on "did you produce the correct answer?" but on "did you produce an answer that is better than what the model itself could do without conditioning?" The advantage $\hat{A}_j$ for a Stage 2 completion is computed relative to other refinement attempts of the same draft, not relative to absolute correctness. This means the model learns that the task is not merely to be right — it is to be better than the draft it was given. The objective is inherently comparative and self-referential: the baseline for success shifts upward as the policy improves, because the drafts being conditioned on are themselves getting better (Proposition 3.1).

This is a fundamental shift in how the learning problem is posed. Standard RL for reasoning asks: "given this problem, produce the correct answer." iGRPO asks: "given this problem and a high-quality attempt that may still contain errors, produce a strictly improved answer." The latter is a meta-skill — refinement — that standard RL never explicitly teaches. The model doesn't just learn to solve problems; it learns to be a problem-solver that can improve upon its own work.

Evidence. The gains on AIME benchmarks are the clearest validation of this claim. On DeepSeek-R1-Distill-Qwen-14B, vanilla GRPO achieves 60.26% on AIME24 while iGRPO reaches 64.06% (Table 1) — a +3.80 point gain under matched rollout budgets. AIME problems are long-horizon multi-step reasoning tasks where small algebraic errors near the end of an otherwise correct solution produce a final answer that fails the binary reward check. In a single-shot training regime, such near-misses receive zero reward and provide no learning signal for how to fix them. iGRPO's Stage 1 captures these near-miss solutions as drafts (they are incorrect but structurally close to correct, making them valuable scaffolds), and Stage 2 learns to convert them into fully correct answers. The +4.06 gain on DeepSeek-R1-Distill-Qwen-14B for AIME24 (60.26% → 64.06%) is not just "better one-shot accuracy" — it represents the model learning to recover from the specific class of errors that plague long reasoning chains, a capability that standard GRPO never targets.

The theoretical characterization in Proposition 3.1 is not merely a proof of correctness — it is a diagnostic tool that explains when the method works and when it doesn't. The formula $\mathbb{E}[R_\varphi(\hat{d}_\theta(q))] = 1 - (1 - V_\theta(q))^N$ tells us that the expected quality of the selected draft increases monotonically with the base policy's success rate, but it also reveals that if $V_\theta(q)$ is near zero — the model almost never produces a correct answer for a given prompt — then even best-of-$N$ selection cannot provide useful conditioning. The draft will be wrong, the refinement will be conditioned on a wrong scaffold, and the bootstrapping cycle stalls. This explains the empirical pattern visible across all experiments: gains are largest on AIME, AMC, and MATH500 (problems within the model's approximate capability range) and minimal on GSM8K (where baseline performance is already high) and hard subsets where the model rarely succeeds. The theory predicts exactly this difficulty-dependent behavior, converting what could be post-hoc rationalization into a principled boundary condition.

Innovation 2: Self-Feedback as In-Context Scaffolding, Not as a Learned Auxiliary Skill

A second conceptual innovation is the architectural separation of feedback generation from feedback utilization. In prior self-improvement methods — Self-Verification, Critique-GRPO, SPIN, Self-Rewarding Language Models — the model must learn to produce its own feedback. It generates verification scores, critique text, or self-rewards, and then must learn to interpret and act on that feedback. This couples two distinct capabilities: (1) recognizing what makes an output good or bad, and (2) improving that output. The model must become both critic and reviser within a single policy, often with different (and potentially conflicting) optimization signals.

What the field did before. Self-Verification (Zhang et al., 2025a) trains the model to assign verification scores to its own answers, then uses those scores at inference time to reweight or aggregate solutions. The verification behavior is trained within the same RL framework as the reasoning behavior, so the model's verification capability and its reasoning capability co-evolve. Critique-GRPO (Zhang et al., 2025b) goes further: the model generates natural-language critiques of its initial answers, then generates refinements conditioned on those critiques, and the RL objective optimizes over both the initial answers and the refinements. In both cases, the model's capacity is split between the primary task (solving the problem) and the auxiliary task (evaluating or critiquing its own work), and the optimization signal for the auxiliary task is indirect — the outcome reward tells the model whether the final answer is correct, not whether the critique was accurate or the verification score was well-calibrated.

What's distinctive. iGRPO eliminates the auxiliary task entirely. The model does not evaluate itself. It does not produce critiques, verification scores, or self-rewards. The feedback — the best draft $\hat{d}$ — is selected by the same external reward function $R_\varphi$ that drives optimization. The model's only job in Stage 2 is to produce a better answer than the draft it receives. The feedback evaluation is completely decoupled from the feedback utilization: the reward function handles evaluation (Stage 1), the policy handles refinement (Stage 2), and these two processes share only the scalar reward signal, not any learned internal representations.

This is an architectural innovation rather than a purely algorithmic one. It reframes self-improvement from a monolithic "model does everything" paradigm to a modular design where feedback quality is guaranteed by an external verifier and the model focuses exclusively on acting on that feedback. The practical implication is significant: iGRPO's refinement wrapper does not require the model to have any critique or verification capability whatsoever. It works with any base model because the feedback is externally provided, not internally generated. This is why the wrapper generalizes to DAPO and GSPO (Table 2) without modification — those optimizers have different internal mechanics, but they all operate on generation outputs, and the self-feedback mechanism only requires that the model can condition on an appended draft.

The compatibility with a GPT-5 judge (Table 3) further reinforces this modularity. Swapping the rule-based reward for a model-based scalar evaluator changes the quality and granularity of the feedback (partial credit for near-misses → gains of +0.94 macro-average), but it requires zero changes to the iGRPO algorithm itself. The model doesn't know or care where the draft came from or how it was scored — it only sees the draft as conditioning text. This suggests that iGRPO can seamlessly benefit from improvements in reward modeling without requiring any architectural retooling, a property that self-critique methods lack because they build the evaluator into the model itself.

Evidence. The consistent outperformance of iGRPO over Self-Verification and Critique-GRPO across all model scales in Table 1 (e.g., 45.04% vs. 43.39% for Nemotron-H-8B, 69.87% vs. 69.14% for DeepSeek-R1-Distill-Qwen-7B) is evidence for the efficiency of decoupling. The auxiliary behaviors required by the baselines consume model capacity and training signal that could be directed entirely toward the primary reasoning task. iGRPO avoids this split entirely — every bit of capacity is devoted to producing better answers, because better answers are all that the reward function evaluates. The gains are not enormous in absolute terms (0.73–1.65 points over the baselines), but they are consistent across model families and scales, suggesting a structural advantage rather than a chance improvement on a particular configuration.

This is a fundamental reframing of the self-improvement problem. Prior work asked: "How can we train the model to be its own critic?" iGRPO asks: "Given that we already have a reliable critic (the reward function), how can we design the training loop to make the model a better refiner?" The second question leads to a simpler system — no auxiliary objectives, no learned verifiers, no dual-role training — that performs better empirically. It is a case where restricting the model's responsibilities (removing the critic role) actually increases its capabilities (improving refinement performance).

Innovation 3: Difficulty-Aware Learning Through Self-Selected Scaffolding

A subtler but important conceptual contribution is that iGRPO creates an implicit difficulty-adaptive curriculum without any explicit difficulty estimation. The Stage 1 draft selection mechanism naturally provides different scaffolding quality for different problems, and this variation in scaffolding quality shapes what the model learns in Stage 2.

What the field did before. In standard GRPO, every prompt is treated identically during training. The group-based advantage normalization adjusts the magnitude of the learning signal based on within-group reward variance, but the type of signal is the same for all prompts: "generate a correct answer." On easy problems where the model almost always succeeds, the reward variance is low, the advantages are near zero, and there is little to learn. On hard problems where the model almost always fails, the situation is symmetric — low variance, near-zero advantages, no learning signal. The prompts that produce the strongest gradient are those where the model's performance is mixed, creating high reward variance and large-magnitude advantages. But even on these prompts, the model is only learning to distinguish correct from incorrect completions — it never learns why some attempts fail or how to recover from near-misses.

What's distinctive. iGRPO creates a structured variation in training difficulty that depends on the model's own capability level. For a given prompt:

  • If the model is strong (high $V_\theta(q)$), Stage 1 will frequently select a correct draft ($\mathbb{E}[R_\varphi(\hat{d})] \approx 1$). Stage 2 then asks: "given a correct solution, produce another correct solution." This is a relatively easy task — the model can learn to replicate correct reasoning patterns — and it reinforces reliable performance on problems the model already handles.
  • If the model is moderate (moderate $V_\theta(q)$), Stage 1 will sometimes select a correct draft and sometimes an incorrect one. When the draft is incorrect, Stage 2 asks: "given an attempt that contains errors, produce a corrected solution." This is the most valuable training signal — it teaches the model to identify and fix the specific types of errors it makes, converting near-misses into successes.
  • If the model is weak (very low $V_\theta(q)$), Stage 1 will almost always select an incorrect draft — and the draft quality will be low, offering little useful scaffolding. Stage 2 degenerates into something similar to Stage 1 (generating from scratch with a poor-quality prefix), providing minimal refinement signal.

This is a self-organized curriculum. The model spends its training time on the problems where it has the most to learn: those where it can sometimes succeed (providing good drafts for Stage 2) but often fails in specific, correctable ways. Problems that are trivially easy (always correct, nothing to refine) or impossibly hard (never correct, no useful scaffolding) receive weaker learning signals, so training resources are implicitly concentrated on the regime where refinement provides the most value. The paper never explicitly designs this curriculum — it emerges from the interaction between the two-stage structure and the bootstrapping dynamic of Proposition 3.1.

Evidence. The entropy analysis in Figure 3 provides indirect evidence for this curriculum effect. GRPO's rapid entropy collapse (from 2.45 nats to 0.60 at 10% training, 0.42 by 30%) suggests the policy quickly converges to a narrow set of generation patterns, effectively ceasing to explore after the early phases of training. iGRPO's more gradual entropy decay (0.80 at 15%, 0.48 at 30%, remaining elevated through mid-training) is consistent with a model that continues to encounter diverse training scenarios — sometimes refining correct drafts, sometimes correcting erroneous ones — which preserves entropy by requiring the policy to maintain multiple behavioral modes. The convergence of both methods to similar final entropy (0.44 vs. 0.42 at 100%) indicates that iGRPO does not produce a fundamentally different final policy distribution, but rather resists premature collapse during the critical mid-training period when the most learning occurs.

This is a diagnostic insight, not just a performance result. The entropy curve suggests that standard GRPO's training dynamics are suboptimal not because the objective is wrong, but because the data distribution seen by the optimizer collapses too quickly — the model converges to a low-entropy policy that mostly succeeds on easy-to-moderate problems, and then has no mechanism to recover from the residual errors it still makes. iGRPO's self-scaffolding mechanism forces the model to continue confronting its own mistakes (via incorrect drafts in Stage 1) even as overall performance improves, which sustains the learning signal longer.

The distinction from prior work is sharp: methods like DAPO and GSPO (Yu et al., 2025; Zheng et al., 2025) attempt to stabilize or improve the optimization objective itself (better advantage estimation, sequence-level clipping). iGRPO instead changes what data the optimizer sees — the augmented prompts in Stage 2 are fundamentally different training instances than the original prompts, even though they derive from the same underlying problems. The "curriculum" is in the data, not in the objective. This is a different axis of innovation from the GRPO-variant literature, which is why the self-feedback wrapper applies successfully to DAPO and GSPO (Table 2) — those methods improve the optimizer, and iGRPO improves the data, so the benefits are complementary and additive.

5. Experimental Analysis

Evaluation Methodology

  • Dataset. The primary training data is the MATH dataset (Hendrycks et al., 2021), consisting of 7,500 step-by-step competition problems. For the state-of-the-art results, the paper uses AceReason-Math (Chen et al., 2025b), containing 9,400 problems. For the large-scale OpenMath-Nemotron-14B experiment (Appendix B), 94,000 examples from OpenR1-Math-220k (R1, 2024) are used. All datasets provide ground-truth reference answers for rule-based reward computation.

  • Base model(s). The paper evaluates across three model families at multiple scales: (1) Nemotron-H-8B-Base-8K (8B parameters, a generalist model), (2) DeepSeek-R1-Distill-Qwen at 7B and 14B scales (distilled reasoning models), (3) OpenMath-Nemotron at 7B and 14B scales (math-specialized models). The state-of-the-art experiment starts from OpenReasoning-Nemotron-7B (NVIDIA, 2025), a stronger initialization trained on AceReason-Math. This diversity — generalist vs. distilled vs. specialized, 7B to 14B — is chosen to demonstrate that iGRPO's benefits are not specific to a single model family or scale.

  • Metrics. The primary metric is Pass@1 accuracy (the fraction of test problems where the model's single sampled answer matches the ground truth). For AIME24 and AIME25, accuracy is averaged over 64 independent runs to ensure robustness given the small test set size (30 problems each). For all other benchmarks (MATH500, AMC23, GSM8K, Minerva Math), accuracy is averaged over 8 runs. The paper also reports a macro-average across all six benchmarks for each model, averaging the per-benchmark Pass@1 scores. All accuracy values are expressed as percentages.

  • Baselines. The paper compares against five methods: (1) Vanilla GRPO (Shao et al., 2024) — the standard single-stage group relative policy optimization, trained under identical hyperparameters. (2) Self-Verification (Zhang et al., 2025a) — unifies problem-solving and generative verification within a single RL framework, using the model's own verification scores to reweight or aggregate sampled solutions. (3) Critique-GRPO (Zhang et al., 2025b) — augments GRPO with natural-language critiques by generating critique-conditioned refinements and optimizing over both initial answers and their refinements in an online RL loop. (4) DAPO (Yu et al., 2025) — a GRPO variant targeting long chain-of-thought training through dynamic sampling, decoupled clipping, and reward shaping. (5) GSPO (Zheng et al., 2025) — operates at the sequence level, redefining importance ratios and applying sequence-level clipping for improved stability. The base (untrained) models are also reported as reference points.

  • Generation budget / compute accounting. The central constraint is a matched rollout budget: every method uses exactly 8 completions per prompt total. For GRPO, this means sampling 8 independent completions from the original prompt. For iGRPO, the 8 completions are split as N=4 Stage 1 drafts and G=4 Stage 2 refinements, so the total number of sampled completions is identical. All methods use the same rule-based reward function (Equation 6) and the same training hyperparameters (learning rate, batch size, epoch count). Differences in Table 1 therefore reflect the effect of conditioning on the best draft rather than the effect of more sampling. For the throughput and GPU-hour measurements in Appendix D, the paper measures both peak GPU memory (GB) and samples processed per second, confirming that iGRPO's memory usage (54.9349 GB) is nearly identical to GRPO's (54.9286 GB) while throughput is modestly lower (0.34 vs. 0.41 samples/sec) due to the additional round of Stage 1 generation.

  • Cross-validation / statistical protocol. The AIME24 and AIME25 benchmarks use 64-run averaging to stabilize the small test sets (30 problems each). Other benchmarks use 8-run averaging. The paper does not describe a held-out validation set strategy for hyperparameter selection — all hyperparameters (learning rate, batch size, N=4, G=4) appear to be fixed a priori rather than tuned. The ablation in Table S.4 systematically varies individual hyperparameters (KL coefficient β, total completions) post-hoc to validate the chosen settings. There is no cross-validation or statistical significance testing reported.

Main Quantitative Results

Controlled Study with Matched Sampling Budget (Table 1)

The core experimental claim is that iGRPO consistently outperforms GRPO and self-improvement baselines across model families and scales when the total sampling budget is held constant. Table 1 presents the full comparison across 7B, 8B, and 14B models.

Generalist 8B model (Nemotron-H-8B-Base-8K). The base model achieves a 29.65% macro-average across all six benchmarks. Vanilla GRPO improves this to 41.08% (+11.43 points), demonstrating that standard RL fine-tuning provides substantial gains on this model. Self-Verification and Critique-GRPO further improve to 42.86% and 43.39% respectively. iGRPO achieves 45.04%, which is +3.96 points over GRPO and +1.65 points over the strongest baseline (Critique-GRPO). The per-benchmark breakdown shows iGRPO leading on all six benchmarks: AIME25 (9.17% vs. 8.50% for the next best), AIME24 (9.56% vs. 9.25%), MATH500 (78.80% vs. 76.05%), AMC (48.75% vs. 46.80%), GSM8K (91.26% vs. 88.40%), and Minerva (32.72% vs. 31.50%). The largest absolute gains are on GSM8K (+2.86 over Critique-GRPO) and MATH500 (+2.75), while the relative gains are most salient on the AIME benchmarks where even small absolute improvements reflect solving additional hard competition problems.

Stronger 7B distilled reasoner (DeepSeek-R1-Distill-Qwen-7B). The base model is already strong at 61.93% macro-average. GRPO raises this to 68.29% (+6.36 points), a substantial gain but smaller in absolute terms than for the 8B generalist. Self-Verification and Critique-GRPO reach 69.08% and 69.14%. iGRPO achieves 69.87%, which is +1.58 points over GRPO and +0.73 over the strongest baseline. The gains concentrate on benchmarks requiring multi-step reasoning with correction of small errors: AMC (95.00% vs. 92.80% for Critique-GRPO, a +2.20 gap), AIME24 (56.30% vs. 55.80%), and Minerva (41.54% vs. 41.10%). On benchmarks where the model already performs near ceiling — MATH500 (93.80% vs. 93.50%) and GSM8K (92.42% vs. 92.25%) — the margins are small (0.30 and 0.17 points respectively), consistent with limited headroom for improvement.

Math-specialized 7B model (OpenMath-Nemotron-7B). The base model starts at 74.83% macro-average, already much stronger than the generalist 8B or distilled 7B models. GRPO yields only a 0.19-point improvement to 75.02%, suggesting that standard one-shot RL optimization offers minimal additional benefit when the model already performs near the ceiling of what single-attempt generation can achieve. iGRPO raises performance to 76.07%, a +1.05 point improvement over GRPO and a +1.24 point improvement over the base model. The gains are most visible on AIME24 (+1.51 points, from 73.28% to 74.79%) and AMC (+2.50 points, from 95.00% to 97.50%), where the refinement mechanism systematically addresses residual errors that single-shot RL cannot correct. On GSM8K, the improvement is marginal (+0.23 over the base, +0.15 over GRPO), reflecting the model's already-high 90.52% base performance on this relatively simpler benchmark.

Scaling to 14B parameters. For DeepSeek-R1-Distill-Qwen-14B, GRPO achieves 71.29% macro-average (from a 70.47% base). iGRPO reaches 73.02%, a +1.73 point gain over GRPO. The largest single-benchmark improvement is AIME24: 64.06% for iGRPO vs. 60.26% for GRPO, a +3.80 point gain — the largest absolute improvement for any model-benchmark pair in Table 1. AIME25 improves by +1.82 points (45.52% vs. 43.70%). For OpenMath-Nemotron-14B, GRPO reaches 76.73% (from 75.41% base), and iGRPO achieves 78.00%, a +1.27 point gain. The AIME improvements are again notable: AIME24 rises from 74.79% (GRPO) to 76.72% (+1.93), and AIME25 from 64.53% to 65.57% (+1.04). The consistent pattern across both model families at 14B — with gains that are actually larger in absolute terms than at 7B for DeepSeek-R1 (AIME24: +3.80 at 14B vs. +1.30 at 7B) — suggests that iGRPO's refinement mechanism scales favorably with model size rather than diminishing as models become stronger.

Comparison with self-improvement baselines. Across all five model configurations where Self-Verification and Critique-GRPO are reported (Nemotron-H-8B, DeepSeek-R1-Distill-Qwen-7B, and the corresponding base models), iGRPO achieves the highest macro-average in every case. The gap over Critique-GRPO ranges from +0.73 (DeepSeek-R1-Distill-Qwen-7B) to +1.65 (Nemotron-H-8B). The consistency of this margin — always positive, never reversed — is the primary empirical evidence for the claim that conditioning on an externally selected best draft is more effective than requiring the model to generate its own feedback (critiques or verification rationales).

Generalization to a Stronger Base and Harder Dataset (Figure 2)

The controlled study trains all models on the 7,500-problem MATH set. To test whether iGRPO remains effective with stronger initialization and more challenging training data, the authors start from OpenReasoning-Nemotron-7B and train on AceReason-Math (9,400 problems), keeping the iGRPO configuration identical to the main experiments.

The base OpenReasoning-Nemotron-7B achieves strong performance: AIME24 at 84.1%, AIME25 at 77.9%, MATH500 at 96.2%, GSM8K at 91.7%, MMLU-Pro at 71.9%, and GPQA at 61.1%, with an average of 80.5%. iGRPO training improves performance on all six benchmarks: AIME24 rises to 85.62% (+1.52), AIME25 to 79.64% (+1.78), MATH500 to 97.0% (+0.80), GSM8K to 92.3% (+0.60), MMLU-Pro to 72.8% (+0.91), and GPQA to 62.9% (+1.84). The overall average improves from 80.5% to 81.7% (+1.23).

Two findings are notable. First, the largest absolute gains appear on GPQA (+1.84) and MMLU-Pro (+0.91) — benchmarks that test general reasoning and scientific knowledge, not purely mathematical problem-solving. Since iGRPO is trained exclusively on math problems (AceReason-Math), these gains represent transfer: the refinement behaviors learned during RL training on math generalize to adjacent reasoning domains. Second, the AIME25 gain (+1.78) is larger than the AIME24 gain (+1.52), which is the opposite of what one might expect if harder problems benefit less from refinement (since drafts are less likely to be correct). AIME25 is the more challenging benchmark for this model (base performance is 77.9% vs. 84.1% for AIME24), suggesting that the refinement mechanism is actually more valuable when the base model has more room for improvement — consistent with the hypothesis that iGRPO helps most on problems within the model's capability range but not yet solved reliably.

These results establish new state-of-the-art scores of 85.62% on AIME24 and 79.64% on AIME25 for the 7B parameter class, as claimed in the abstract. The paper does not provide context for how these numbers compare to larger models or other approaches, but within the scope of the experiments reported, they represent the best performance achieved.

Ablation Studies and Robustness Checks

Self-feedback as a reusable refinement wrapper (Table 2): The paper tests whether the two-stage refinement mechanism is tied to GRPO specifically or generalizes to other group-based PPO variants. Applying the same iGRPO wrapper (Stage 1 draft selection + Stage 2 conditioned refinement with matched rollout budgets) to DAPO and GSPO on DeepSeek-R1-Distill-Qwen-7B trained on MATH yields consistent improvements: DAPO base achieves 69.74% average, rising to 70.93% with iGRPO (+1.19); GSPO base achieves 69.20%, rising to 70.31% with iGRPO (+1.11). Both gains are similar in magnitude to the iGRPO-over-GRPO improvement on the same model (+1.58 from Table 1), suggesting that the refinement wrapper provides orthogonal benefits to the specific choice of group-based PPO objective. This is evidence for the paper's claim that the gains "primarily stem from the refinement interface rather than GRPO-specific details," though it should be noted that Table 2 reports only the macro-average (not per-benchmark scores) and does not include error estimates or statistical tests.

Generative judge as reward replacement (Table 3): To test whether iGRPO's binary reward constraint is necessary or whether richer reward signals provide additional benefit, the paper replaces the rule-based binary checker (Equation 6) with a GPT-5 judge that scores each solution on a [0,1] scale. On DeepSeek-R1-Distill-Qwen-7B trained on MATH, the GPT-5 judge improves Pass@1 on all six benchmarks: AIME25 rises from 40.16% to 41.12% (+0.96), AIME24 from 56.30% to 57.45% (+1.15), MATH500 from 93.80% to 94.20% (+0.40), AMC from 95.00% to 96.25% (+1.25), GSM8K from 92.42% to 92.95% (+0.53), and Minerva from 41.54% to 42.88% (+1.34). The macro-average improves from 69.87% to 70.81% (+0.94). The largest gains on AIME24/25 and Minerva are attributed to partial credit for near-miss reasoning traces: under the rule-based reward, a solution that makes a single algebraic error at the end of an otherwise correct chain-of-thought receives zero reward and is indistinguishable from a nonsensical response in Stage 1 draft selection. The GPT-5 judge assigns non-zero scores to such solutions, allowing them to be selected as the best draft and subsequently refined into correct answers in Stage 2. This demonstrates modularity — iGRPO only requires a scalar reward for draft ranking and advantage computation, and can benefit from improvements in reward modeling without any algorithmic changes.

Entropy dynamics during training (Figure 3): To probe how self-feedback alters learning dynamics, the paper tracks per-token Shannon entropy of the policy during RL training on DeepSeek-R1-Distill-Qwen-7B trained on MATH. Both GRPO and iGRPO start at 2.45 nats. GRPO collapses rapidly: 0.60 nats at 10% of training steps, 0.42 nats by 30%, then remains essentially flat through the end. iGRPO decays more gradually: 0.80 nats at 15% of training, 0.48 nats at 30%, and stays slightly above GRPO through mid-training (0.46 nats at 60% vs. GRPO's ~0.43 nats), before converging to similar values near the end (0.44 vs. 0.42 nats at 100%). The interpretation is that "iGRPO delays premature mode collapse": conditioning on the best draft encourages the policy to explore around a strong scaffold rather than immediately converging to a narrow high-reward mode. Since the final entropies are close, the paper argues that the performance gains are better explained by sustained mid-training exploration (which allows recovery from near-miss traces) rather than higher randomness at convergence. A limitation of this analysis is that it reports only a single training run — there are no error bars, and the extent to which the entropy difference at mid-training (0.03–0.04 nats) is practically meaningful is not quantified.

Effect of KL divergence coefficient β (Table S.4, top): Varying the KL penalty coefficient β on DeepSeek-R1-Distill-Qwen-7B trained on MATH yields: β=0 gives 69.87%, β=0.0001 gives 70.23%, β=0.001 gives 69.31%, β=0.01 gives 69.91%. The maximum gain from the non-zero setting is +0.36 points at β=0.0001, and all values cluster within a 0.92-point range. The paper concludes that "the difference among all settings is relatively small" and uses β=0 for simplicity. This is a practically useful result — it eliminates a hyperparameter — but it also raises questions about why the KL penalty has so little effect. Standard PPO implementations typically require KL regularization to prevent the policy from diverging too far from the reference model and losing general capabilities. The paper speculates (Section 3.4) that training for only one epoch on math-specific data limits the degree of distribution shift, making KL regularization less critical. This hypothesis is not explicitly tested.

Effect of total number of completions (Table S.4, bottom): Varying the total completions per prompt in iGRPO (split evenly across stages) shows: 4 completions yield 67.79%, 8 completions yield 69.87% (+2.08), 16 completions yield 70.17% (+0.30), and 32 completions yield 70.33% (+0.16). The jump from 4 to 8 completions is clearly beneficial, but gains beyond 8 are modest (±0.30 for doubling from 8 to 16, ±0.16 for doubling again to 32). The paper uses 8 completions as the default in all main experiments, consistent with this diminishing returns pattern. This ablation addresses the question of whether iGRPO's gains over GRPO (which uses 8 completions in its own configuration) could be matched by simply giving GRPO more samples: the fact that iGRPO at 8 completions (69.87%) outperforms GRPO at the paper's reported configuration (68.29% for DeepSeek-R1-Distill-Qwen-7B in Table 1), and that increasing iGRPO to 16 or 32 completions yields only modest additional gains, suggests that the two-stage structure provides a qualitatively different training signal that more samples in a single-stage setup cannot easily replicate. However, the paper does not report what GRPO performance would be at 16 or 32 completions for a direct comparison.

Training dynamics: average reward and response length (Figure S.2): The paper compares average training rewards for iGRPO and GRPO across checkpoints (Figure S.2a), showing that iGRPO "consistently maintains a higher reward throughout training." The reward curves diverge early and the gap persists, consistent with the bootstrapping hypothesis. Figure S.2b shows that response lengths are nearly identical between the two methods, with GRPO producing slightly longer outputs on average. The paper argues that iGRPO's gains arise "more from improved response quality than from verbosity." No numerical values are reported for the reward curves or response length differences, so the magnitude of the effect is unclear.

Scaling OpenMath-Nemotron-14B with iGRPO on large-scale data (Table S.1): Training OpenMath-Nemotron-14B with iGRPO on 94,000 examples from OpenR1-Math-220k (rather than the ~7,500-problem MATH set) yields significant improvements over the base model: AIME25 rises from 61.18% to 66.04% (+4.86), AIME24 from 73.28% to 76.61% (+3.33), MATH500 from 95.55% to 96.90% (+1.35), AMC from 95.00% to 97.50% (+2.50), GSM8K from 94.01% to 94.16% (+0.15), and Minerva from 33.46% to 38.24% (+4.78). The macro-average improves from 75.41% to 78.24% (+2.83). These gains are substantially larger than the improvements on the same model when trained on MATH alone (Table 1: +1.27 over GRPO, +2.59 over the base), suggesting that the iGRPO mechanism scales favorably with training data quantity — more diverse training problems provide more opportunities for the model to encounter and learn from near-miss reasoning traces. However, Table S.1 only reports the base model and the iGRPO-trained model, making it unclear how much of the gain is due to iGRPO versus the larger training set alone (GRPO on the same 94,000 examples is not reported).

Pass@N analysis on AIME benchmarks (Figure S.1): The paper evaluates iOpenMath-Nemotron-14B (the model from Appendix B) at various pass@N values on AIME24 and AIME25. On AIME24, pass@1 is 76.61% and rises rapidly: pass@2 = 83.33%, pass@4 = 86.67%, pass@8 = 90.00%, and plateaus at pass@16 = 93.33%, with no further improvement through pass@256. On AIME25, pass@1 is 66.04% and improves more gradually: pass@8 = 86.67%, pass@16 = 90.00%, pass@32 = 90.00% (plateau), pass@64 = 90.00%, pass@128 = 90.00%, and pass@256 = 96.67%. The key finding is that AIME25, the harder benchmark, continues to benefit from more samples at very high N (96.67% at N=256), while AIME24 saturates by N=16. The paper suggests that "certain problem distributions... may benefit from a larger number of sampled attempts, whereas others... can be adequately solved with fewer solution attempts." This analysis uses only the iGRPO-trained model — it does not compare pass@N curves for GRPO-trained vs. iGRPO-trained models, which would indicate whether iGRPO training improves the model's ability to benefit from multiple samples at inference time.

Critical Assessment

Does the paper demonstrate that iGRPO consistently outperforms GRPO under matched rollout budgets?

Yes, with qualifications. Table 1 provides comprehensive evidence across 5 model configurations (Nemotron-H-8B, DeepSeek-R1-Distill-Qwen-7B/14B, OpenMath-Nemotron-7B/14B) and 6 benchmarks, showing iGRPO achieves a higher macro-average than GRPO in every case. The gains range from +0.69 points (OpenMath-Nemotron-7B: 76.07% vs. 75.02% for GRPO) to +3.96 points (Nemotron-H-8B: 45.04% vs. 41.08%). However, the evidence has several limitations:

  • No statistical significance or confidence intervals are reported. For benchmarks with small test sets (AIME24 and AIME25: 30 problems each), the 64-run averaging provides some stability, but without error estimates it is impossible to determine whether the gains (e.g., +1.30 points on AIME24 for DeepSeek-R1-Distill-Qwen-7B) are statistically reliable or within noise. The 500-question MATH500 and the ~1,300-question GSM8K provide more samples, but still no variance estimates are provided.

  • The matched budget is small. At 8 completions per prompt total, the allocation of N=4, G=4 is a specific design choice that is not ablated. The paper does not test whether other splits (e.g., N=2, G=6 or N=6, G=2) at the same total budget produce different results. The sensitivity to this allocation is unknown, and it is possible that the equal split is suboptimal for some models or benchmarks.

  • Training is for only one epoch. The paper does not explore whether the relative advantage of iGRPO over GRPO changes with longer training. Given that iGRPO's core mechanism (delayed entropy collapse, bootstrapped draft improvement) operates across iterations, it is possible that the gap would widen (if GRPO plateaus earlier) or narrow (if GRPO eventually catches up) with more training. The single-epoch design means the results are conditional on this specific training duration.

  • The 13% training time overhead is documented but not controlled. Appendix D reports that iGRPO uses 94.1 GPU hours vs. GRPO's 83.3 GPU hours. In a FLOPs-matched comparison (analogous to Section 7 of the reference example), one would give GRPO a proportionally larger sampling budget to equalize wall-clock time and see if the gap persists. This experiment is not performed. The ablation on total completions (Table S.4, bottom) partially addresses this by showing that iGRPO at 8 completions (69.87%) still outperforms what we can estimate GRPO at 8 completions would achieve (~68.29% from Table 1), and that giving iGRPO 16 completions (70.17%) yields only modest additional gains. But a direct GRPO-at-16-completions comparison is absent.

Does the paper demonstrate that iGRPO outperforms self-improvement baselines (Self-Verification, Critique-GRPO)?

Partially. The baselines are only evaluated on two model configurations: Nemotron-H-8B-Base-8K and DeepSeek-R1-Distill-Qwen-7B. They are absent for DeepSeek-R1-Distill-Qwen-14B, OpenMath-Nemotron-7B, and OpenMath-Nemotron-14B, meaning we cannot assess whether iGRPO's advantage over these baselines persists at larger scales or with specialized models. The improvement over Critique-GRPO is consistent but small (0.73–1.65 points). Given the complexity of implementing critique generation within an RL loop, a 0.73-point gain on an already-strong 7B distilled model may be within the range of hyperparameter sensitivity — the paper does not explore whether Critique-GRPO with different critique prompt templates or generation budgets would close the gap.

Additionally, the baselines are evaluated only under the matched 8-completion budget. Self-Verification's mechanism — using the model's own verification scores to reweight solutions — is designed to improve with more samples at inference time. Testing it with only 8 completions per prompt during training may not reflect its full potential, since the verification model benefits from more training data to calibrate its scores. The paper's conclusion that iGRPO is "more effective" would be strengthened by showing that the advantage persists at larger training budgets for the baselines.

Does the evidence support the claim that iGRPO creates a bootstrapping dynamic where conditioning quality improves as the policy improves?

Indirectly. Proposition 3.1 provides a theoretical guarantee that the expected reward of the selected draft increases monotonically with V_θ(q) under binary rewards. This is mathematically correct for the stated assumptions. However, the empirical evidence for the bootstrapping dynamic is limited to:

  • Training reward curves (Figure S.2a). The paper states that iGRPO "consistently maintains a higher reward throughout training," which is consistent with bootstrapping (better drafts → better refinements → higher rewards → better policies) but could also be explained by other factors, such as iGRPO simply having a more favorable optimization landscape due to the easier prompts in Stage 2.

  • Entropy dynamics (Figure 3). The delayed entropy collapse is suggestive but does not directly measure draft quality or refinement success. A more direct test would track the probability that the best Stage 1 draft is correct over training iterations (which Proposition 3.1 predicts should increase), or the probability that a Stage 2 refinement corrects an incorrect Stage 1 draft. Neither metric is reported.

  • No ablation removing the bootstrapping. The paper does not test a variant where Stage 1 drafts are sampled from a fixed reference model (not the current snapshot) to break the bootstrapping loop. Such an experiment would isolate whether the dynamic adaptation of draft quality across iterations is necessary for iGRPO's gains, or whether any reasonable draft — even a static one — would provide similar benefits.

In summary, the bootstrapping claim is theoretically grounded but empirically underexplored. The evidence is consistent with bootstrapping but does not rule out simpler explanations (e.g., that conditioning on any high-quality draft is beneficial regardless of whether it co-evolves with the policy).

Does the paper demonstrate that iGRPO's gains transfer beyond math?

Yes, but only in one experiment. Figure 2 shows that training OpenReasoning-Nemotron-7B with iGRPO on AceReason-Math (a math dataset) improves performance on GPQA (+1.84) and MMLU-Pro (+0.91), which test general and scientific reasoning. This is evidence for generalization of the learned refinement behaviors beyond the training domain. However, the evidence is limited to a single model (the strongest tested) and a single training dataset. We do not know whether the same transfer would occur for smaller models, different model families, or different training datasets. The paper also does not report performance on a math-unrelated benchmark where refinement might be harmful (e.g., a task where the model should not second-guess its initial answer), which would help characterize the boundaries of generalization.

Does the paper convincingly establish new state-of-the-art results?

Within its reported scope, yes. The AIME24 score of 85.62% and AIME25 score of 79.64% for a 7B model are the highest values claimed in the paper. However, the paper does not provide a leaderboard or comparison to other published results at the time of writing, so "state-of-the-art" is an assertion without external validation. The scores should be understood as the best among the models and methods the authors tested, not necessarily as verified records on public leaderboards.

Missing experiments that would strengthen the paper:

  • Ablation of the Stage 1 / Stage 2 allocation ratio. The N=4, G=4 split is used throughout but never compared to alternatives (e.g., N=2/G=6, N=6/G=2). It is plausible that the optimal split depends on problem difficulty (harder problems might need more Stage 1 drafts to find a correct scaffold) or model capability (weaker models might benefit from more Stage 1 exploration). The absence of this ablation makes it impossible to assess whether the reported gains are near-optimal or could be substantially improved.

  • Ablation of the conditioning prompt template. Appendix C includes explicit instructions for how to handle the draft in Stage 2 (e.g., "treat it as guidance, not as ground truth," "do not repeat the draft verbatim"). These instructions could be responsible for a significant portion of the improvement by explicitly discouraging the model from simply copying the draft. An ablation removing or varying these instructions would help disentangle the effect of the prompt engineering from the effect of the two-stage RL structure.

  • GRPO with a matched computational budget (not just sample budget). Since iGRPO takes 13% more GPU hours, a FLOPs-matched comparison would give GRPO ~9 completions per prompt (13% more samples) and test whether the gap persists. This is the standard for fair comparison in the scaling literature and its absence weakens the claim of superior efficiency.

  • Performance on hard subset analysis. The paper claims (Section 3.2.3) that the bootstrapping dynamic is most effective when the model can sometimes produce correct drafts. This predicts that gains should be concentrated on problems of intermediate difficulty and minimal on very hard problems (where no draft is correct and the conditioning provides no signal). The benchmarks used (MATH500, AIME) have heterogeneous difficulty, but the paper does not break out performance by difficulty tier. Such an analysis would provide direct evidence for or against the theoretical mechanism.

  • Statistical significance tests or confidence intervals. For all reported results, the absence of variance estimates makes it difficult to assess whether the reported gains — particularly small ones (e.g., +0.69 on OpenMath-Nemotron-7B) — are reliable or within sampling error.

Where the claims hold conditionally:

  • iGRPO outperforms GRPO under matched rollout budgets when training for one epoch on the MATH dataset with the specified hyperparameters (N=4, G=4, temperature 0.7, no KL penalty). The evidence strongly supports this conditional claim across multiple model families and scales. Whether the advantage holds under different training configurations (more epochs, different sampling budgets, different temperature, with KL penalty, different N/G splits) is not tested.

  • iGRPO's refinement wrapper generalizes to other group-based PPO variants (DAPO, GSPO) when tested on a single model (DeepSeek-R1-Distill-Qwen-7B) with the macro-average metric. Table 2 supports this, but the evidence is limited to one model and one metric (no per-benchmark results reported). Generalization to other models and the per-benchmark pattern is unknown.

  • iGRPO benefits from richer reward signals (GPT-5 judge) without algorithmic modification. Table 3 supports this for one model, with consistent gains across all benchmarks. The modularity claim is well-supported within this scope but has not been tested with other reward models or on other base models.

  • iGRPO delays entropy collapse during RL training. Figure 3 shows this for one training run on one model. The reproducibility of this pattern (across seeds, models, datasets) and its causal connection to performance gains are not established. The entropy difference in mid-training is small (~0.03 nats), and its practical significance is unclear.

  • The "new state-of-the-art" claim holds for the specific setup of OpenReasoning-Nemotron-7B trained on AceReason-Math with iGRPO, evaluated with temperature 0.6, top-p 0.95, and averaged over 64 runs (AIME) or 8 runs (other benchmarks). Without external leaderboard comparisons or standardized evaluation protocols, the absolute status of these scores as "state-of-the-art" cannot be independently verified from the paper alone.

6. Limitations and Trade-offs

The Difficulty Estimation Gap: Bootstrapping Only Works When the Model Can Produce Good Drafts

The assumption or constraint. The bootstrapping dynamic formalized in Proposition 3.1 — that the expected quality of Stage 1 drafts improves monotonically as the policy improves — fundamentally requires that the model can produce correct drafts at a non-trivial rate. If the base policy's success probability $V_\theta(q)$ is near zero for a given prompt, then $\mathbb{E}[R_\varphi(\hat{d}_\theta(q))] = 1 - (1 - V_\theta(q))^N \approx 0$ regardless of $N$. The selected draft will be incorrect, Stage 2 will condition on a wrong scaffold, and the refinement signal degenerates into something approximating unconditional generation — providing no bootstrapping benefit. The paper acknowledges this implicitly in Section 3.2.3, noting that "if optimization increases $V_\theta(q)$" then conditioning quality improves, but does not address the converse: what happens when $V_\theta(q)$ remains stubbornly low throughout training for certain problems.

The consequence. iGRPO offers no path forward for problems that are fundamentally outside the base model's capability range — problems where the model almost never produces a correct answer even after standard RL fine-tuning. On such problems, Stage 1 cannot provide useful scaffolding (every draft is wrong), Stage 2 learns to refine incorrect attempts into other incorrect attempts (at best) or to blindly copy drafts (at worst, if the model learns that drafts are unreliable), and the method confers no advantage over standard GRPO. This is a hard capability bound: test-time compute (in the form of RL training with self-feedback) can amplify existing reasoning capability but cannot create it from nothing.

What evidence exists in the paper. While the paper does not report per-difficulty breakdowns, the pattern is visible across benchmarks. On GSM8K, where all models already perform at 90%+ (base or GRPO-trained), iGRPO's gains are consistently the smallest in both absolute and relative terms — for example, only +0.23 points over the base for OpenMath-Nemotron-7B, and +0.15 over GRPO on the same model (Table 1). This is not the failure case (performance is already high), but it demonstrates the saturation of the refinement mechanism when there is little to refine. The converse — problems where the model never succeeds — is not explicitly measured, but the theoretical prediction is clear: on the hardest MATH contest problems where even the strongest 14B models get near-zero accuracy, iGRPO should provide no benefit. The paper does not include a difficulty-stratified analysis that would directly confirm this.

Mitigation status. The paper does not address this limitation directly. It does not propose difficulty estimation, adaptive draft allocation (assigning more Stage 1 drafts to harder problems), or alternative mechanisms for problems where bootstrapping fails. The discussion of Proposition 3.1 focuses exclusively on the positive case (policy improvement → better drafts → more improvement) without characterizing the failure regime. A practitioner deploying iGRPO needs to know that the method cannot expand the set of solvable problems — it can only improve performance on problems the model already handles at some non-trivial rate — but this boundary condition is not stated as an explicit design limitation.

Difficulty Estimation Is Implicit and Uniform; No Adaptation Per Problem

The assumption or constraint. iGRPO uses a fixed allocation of $N=4$ Stage 1 drafts and $G=4$ Stage 2 refinements for every training prompt, regardless of the prompt's difficulty or the model's current capability on that specific prompt. The bootstrapping dynamic described in Proposition 3.1 predicts that the optimal allocation should depend on $V_\theta(q)$: when $V_\theta(q)$ is very low, more Stage 1 drafts are needed to increase the probability of finding a correct scaffold (since the expected number of correct drafts is $N \cdot V_\theta(q)$); when $V_\theta(q)$ is moderate, a balanced allocation provides both sufficient exploration and sufficient refinement signal; when $V_\theta(q)$ is high, fewer Stage 1 drafts are needed because a correct draft is almost guaranteed even with small $N$. The uniform allocation ignores this dependency entirely.

The consequence. Training resources are allocated suboptimally across the prompt distribution. On very hard problems, $N=4$ may be insufficient to ever find a correct draft (if $V_\theta(q) \approx 0.05$, the probability of at least one correct draft among 4 is only $1 - 0.95^4 \approx 0.19$), meaning the model rarely receives useful conditioning and may never learn to solve these problems. On very easy problems where $V_\theta(q) \approx 0.95$, $N=4$ is excessive — a correct draft is almost guaranteed from $N=1$ or $N=2$, and the extra Stage 1 generations are wasted compute that could be redirected to Stage 2 refinement. This inefficiency means iGRPO may be leaving performance on the table, particularly in the intermediate-difficulty regime where the method is theoretically most valuable.

What evidence exists in the paper. The paper does not ablate the $N$ vs. $G$ allocation ratio at a fixed total budget. Table S.4 varies the total number of completions (4, 8, 16, 32) but keeps the split even in all cases. There is no experiment comparing, for example, $N=2, G=6$ vs. $N=6, G=2$ vs. $N=4, G=4$ at a fixed total of 8. The diminishing-returns pattern in Table S.4 (67.79% → 69.87% → 70.17% → 70.33%) suggests that the current allocation is not grossly suboptimal — the method works — but does not indicate whether it is near-optimal. A difficulty-stratified analysis of optimal allocation would be a natural extension, analogous to the compute-optimal test-time scaling framework described in the reference example, but it is entirely absent here.

Mitigation status. Not addressed. The uniform allocation is a simplifying assumption that makes the method easy to implement, but the paper neither justifies it theoretically (e.g., by showing that $N=G$ is optimal under some reasonable model of the problem) nor tests alternatives empirically. The absence of this ablation is a significant gap because it means the reported gains — already modest on strong models (+0.69 to +1.58 over GRPO at 7B) — may be substantially improvable with better allocation, or conversely, that the gains are sensitive to an arbitrary choice that may not transfer to other datasets or model families. A practitioner cannot know whether the $N=4, G=4$ split generalizes or requires tuning.

Single Benchmark Family and Reward Type; No Evidence of Generalization Beyond Verifiable Math

The assumption or constraint. All training experiments use math problem datasets (MATH, AceReason-Math, OpenR1-Math-220k) with a binary rule-based reward function (exact answer match). The evaluation benchmarks are also math-focused (AIME24/25, MATH500, AMC, GSM8K, Minerva Math), with only two non-math benchmarks (MMLU-Pro and GPQA) tested in a single generalization experiment (Figure 2). The verifiable reward structure — a deterministic, objective correctness signal computed by string matching — is essential to iGRPO's design: it enables reliable Stage 1 draft selection via $\arg\max$ over $R_\varphi$, and it ensures that the reward signal used for optimization matches the signal used for draft ranking. The paper does not study any domain where correctness is ambiguous, multi-dimensional, subjective, or requires human evaluation.

The consequence. It is unknown whether iGRPO's mechanism transfers to tasks without clean verifiability. In open-ended generation (creative writing, dialogue, summarization), code generation (where correctness is testable but reward is not binary — there are partial-credit signals from test pass rates), or reasoning tasks requiring factual knowledge rather than deductive logic, the $\arg\max$ draft selection over a binary or scalar reward may behave very differently. If the reward signal is noisy, subjective, or model-generated (as the GPT-5 judge experiment in Table 3 partially explores), the bootstrapping guarantee of Proposition 3.1 weakens: the selected draft may not actually be the best in terms of ground-truth quality, and conditioning on a spuriously high-scoring draft could reinforce errors rather than correct them. The paper's positive result with the GPT-5 judge (+0.94 macro-average) is encouraging for scalar rewards beyond binary, but the judge is still applied to math problems where "correctness" is well-defined and the judge is presumably well-calibrated. The behavior with unreliable or adversarially noisy rewards is entirely uncharacterized.

What evidence exists in the paper. The generalization experiment in Figure 2 shows that refining on AceReason-Math (math) transfers modest gains to MMLU-Pro (+0.91) and GPQA (+1.84). This is evidence that refinement behaviors learned during RL training on math generalize to adjacent reasoning domains, but it does not test iGRPO training directly on those domains. We do not know whether training iGRPO on, say, MMLU-Pro with a verifiable subset of questions (or a learned reward model) would be equally effective, or whether the mechanism fundamentally depends on the structure of math problem-solving (step-by-step deductive chains where errors are localized and correctable) rather than the type of reasoning required for GPQA (scientific knowledge with plausible distractors).

Mitigation status. The paper is transparent about its scope — all experiments are on math reasoning — but does not frame this as a limitation. The abstract claims the method advances "verifiable mathematical reasoning," which appropriately bounds the contribution, but the broader framing in Section 1 about "how humans naturally solve problems" and the citations to general self-improvement work imply a generality that is not tested. The single generalization experiment (Figure 2) is a step toward addressing this gap, but it only tests transfer, not direct training. A practitioner deploying iGRPO for non-math reasoning tasks has no guidance on whether the method will work, whether the reward structure needs modification, or what failure modes to expect.

The 13% Training Time Overhead Is Not Matched in the Main Comparisons

The assumption or constraint. The headline comparisons in Table 1 use a matched sampling budget (8 completions per prompt for all methods) but do not match wall-clock training time or total GPU hours. Appendix D documents that iGRPO training uses 94.1 GPU hours vs. 83.3 GPU hours for GRPO — a ~13% increase — due to the sequential Stage 1 and Stage 2 decoding. The main experiments in Table 1 compare iGRPO (8 completions, 94.1 GPU hours) against GRPO (8 completions, 83.3 GPU hours) and report that iGRPO outperforms GRPO. This is a confounded comparison: iGRPO receives 13% more compute, and part of the performance gain may be attributable to the additional training time rather than the algorithmic structure.

The consequence. The claim that iGRPO is more compute-efficient than GRPO is not supported by the primary evidence. To make this claim, one would need to compare iGRPO against GRPO with a proportionally larger sampling budget (to equalize GPU hours) — for example, GRPO with ~9 completions per prompt vs. iGRPO with 8. The paper never performs this comparison. It is possible that simply giving GRPO 13% more samples (or training it for 13% longer) would close some or all of the gap reported in Table 1. The ablation in Table S.4 (bottom) partially addresses the related question of whether increasing the iGRPO budget yields gains equivalent to the iGRPO-over-GRPO gap, but it does not provide the critical GRPO-at-higher-budget baseline.

What evidence exists in the paper. Appendix D provides the GPU-hour measurements (83.3 for GRPO, 94.1 for iGRPO) but does not compute a FLOPs-matched or time-matched baseline. Section 3.2.5 argues that "iGRPO can be run at essentially the same dominant generation cost as GRPO" by matching the sampling budget, which is true for per-iteration generation cost but false for total training time due to the serialization overhead. The paper's framing — "iGRPO redistributes the same number of rollouts across Stage 1 and Stage 2 rather than increasing them" — is accurate regarding sampling cost per optimization step but misleading regarding total training cost.

The Table S.4 ablation on total completions provides indirect evidence. iGRPO with 8 completions achieves 69.87% macro-average on DeepSeek-R1-Distill-Qwen-7B, while GRPO with 8 completions achieves 68.29% (Table 1) — a +1.58 gap. iGRPO with 16 completions achieves 70.17%, a gain of only +0.30 over iGRPO at 8 completions. If GRPO exhibited similar returns to additional samples, giving it ~9 completions (to match the 13% time increase) would yield at most a fraction of the +1.58 gap, meaning iGRPO's structural advantage would persist after time-normalization. However, this is speculative — GRPO's sample-efficiency curve at higher budgets is not measured, so the extent of confounding is unknown.

Mitigation status. Partially addressed in Appendix D, where the authors characterize the overhead and frame the tradeoff explicitly: "Given that this modest time increase delivers several-point gains on AIME24 and AIME25... we view the tradeoff... as a favorable and practical value proposition." This is a candid acknowledgment, but it does not replace a controlled comparison. The paper treats the 13% overhead as an acceptable price for better accuracy, which is a reasonable position, but the efficiency claims ("under matched rollout budgets") should be interpreted as matched per-iteration sampling cost, not matched total training cost. A practitioner deciding whether to adopt iGRPO needs to weigh the 13% longer training time against the accuracy gains, and the paper's data supports doing so — it simply does not control for this variable in the main comparisons.

No Within-Chain Selection or Corrector Mechanism; Refinement Can Degrade Correct Drafts

The assumption or constraint. The iGRPO training procedure assumes that conditioning on the best Stage 1 draft and training Stage 2 refinements will teach the model to produce answers that are strictly better than the draft. However, there is no mechanism during training to prevent the model from learning to produce answers that are worse than the draft, or from learning to "refine" correct drafts into incorrect ones. The prompt template in Appendix C instructs the model to "produce a strictly improved final answer" and "do not repeat the draft verbatim," but these are natural-language instructions, not enforced constraints. The reward function evaluates Stage 2 completions against the ground-truth answer, not against the draft quality. If a Stage 2 completion is incorrect but still receives a positive advantage (because other Stage 2 completions were even worse), the model receives a positive gradient for generating an incorrect refinement of a potentially correct draft.

The consequence. The model may learn a "corrector" behavior that is not reliably correctness-improving. During inference, the trained model is used in single-shot mode (generating directly from the original prompt without any draft), so the refinement skill learned during training must generalize to first-shot generation. The paper's results show that this generalization does occur — iGRPO outperforms GRPO across benchmarks (Table 1) — but the mechanism by which training-time refinement translates to inference-time accuracy is not characterized. It is possible that the model learns beneficial reasoning patterns from conditioning on drafts (e.g., more thorough verification of intermediate steps, better error-checking) that transfer to single-shot generation, rather than learning a specific "given X, produce better-than-X" function. The paper does not distinguish between these possibilities, and the absence of a direct refinement evaluation (e.g., measuring the probability that a Stage 2 refinement corrects an incorrect Stage 1 draft) makes it impossible to assess the quality of the learned refinement behavior itself.

What evidence exists in the paper. The paper does not report any direct measure of refinement quality, such as: the fraction of Stage 2 completions that are correct when the Stage 1 draft was incorrect (the "correction rate"), or the fraction of Stage 2 completions that are incorrect when the Stage 1 draft was correct (the "degradation rate"). These metrics would directly quantify whether the model learns to reliably improve upon drafts. The entropy analysis (Figure 3) and training reward curves (Figure S.2a) provide indirect evidence that something beneficial is happening during training, but they do not isolate the refinement behavior.

The response length comparison (Figure S.2b) shows that GRPO and iGRPO produce nearly identical completion lengths, ruling out the hypothesis that iGRPO simply produces longer, more verbose answers. The generalization experiment (Figure 2) shows that training-time refinement transfers to improved single-shot performance on non-math benchmarks, suggesting that the learned capability is not purely a "copy with minor edits" strategy.

Mitigation status. Not addressed. The paper does not include a "correct-to-incorrect reversion" analysis analogous to what the reference example describes for revision models (~38% of correct answers get converted back to incorrect ones). Such an analysis would be straightforward to conduct: for a held-out set of problems, sample Stage 1 drafts, identify which are correct and which are incorrect, then measure Stage 2 refinement outcomes conditioned on correct and incorrect drafts separately. The absence of this analysis means we cannot assess whether iGRPO's refinement mechanism faces the same degradation problem documented in prior work on iterative self-correction (Huang et al., 2023, cited in the reference example), nor whether the gains reported in Table 1 arise despite a high degradation rate (masked by other benefits) or because the degradation rate is low. A practitioner considering iGRPO for a domain where the cost of incorrect refinements is high (e.g., code generation where a "refined" version could introduce subtle bugs) has no data on this failure mode.

No Combination of Draft Quality Levels; Training Only Conditions on the Best Draft

The assumption or constraint. iGRPO's Stage 2 conditioning uses exclusively the best Stage 1 draft, selected via $\arg\max$ over the reward function. Drafts that receive lower rewards — including drafts that are structurally close to correct but fail on a minor detail — are discarded and never used as conditioning context. The model never learns to refine intermediate-quality drafts or to distinguish between drafts of varying quality. This is a deliberate design choice (Section 3.2.2): "we identify the highest-scoring draft... This stage performs implicit curriculum generation: early in training, $\hat{d}$ may be a weak solution, but as the policy improves, $\hat{d}$ increasingly represents a high-quality attempt." The assumption is that conditioning only on the best draft is sufficient to teach refinement, because the quality of that draft naturally varies across prompts and training iterations, providing a range of conditioning difficulty.

The consequence. The model may not learn robust refinement strategies that generalize to drafts of arbitrary quality. During training, the model almost always conditions on a draft that is either correct (when at least one Stage 1 attempt succeeds) or uniformly incorrect (when all fail, the $\arg\max$ selects an arbitrary incorrect draft). It rarely conditions on "mostly correct" drafts that require targeted error correction — the precise scenario where refinement is most valuable. Consider a math solution that contains a 10-step chain-of-thought with a single arithmetic error at step 7. Under binary rewards, this draft receives reward 0. If another draft in the same batch also receives reward 0 but is complete nonsense, the $\arg\max$ over binary rewards is indifferent between them — they are tied at 0. The model may end up conditioning on the nonsensical draft (if it happens to be selected by the arbitrary tie-breaking) and never see the high-quality-but-flawed draft as a scaffold.

This is partially mitigated by the fact that as training progresses and $V_\theta(q)$ increases, the probability of at least one correct draft in Stage 1 grows, so the model increasingly conditions on correct drafts. But this means the model spends most of its training either (a) conditioning on correct drafts (learning to replicate or slightly vary correct solutions — valuable but not teaching error correction) or (b) conditioning on random incorrect drafts that provide no useful scaffolding (teaching the model to ignore the draft, which may contribute to the eventual single-shot performance by encouraging the model to treat conditioning as optional). The intermediate regime — conditioning on high-quality incorrect drafts to learn targeted error correction — is systematically underrepresented.

What evidence exists in the paper. The paper's comparison with Critique-GRPO (Table 1) is relevant here. Critique-GRPO explicitly generates natural-language critiques of initial answers and conditions refinements on those critiques, which means the model's Stage 2 conditioning context contains diagnostic information about what went wrong. iGRPO outperforms Critique-GRPO despite providing only the draft (not a critique) as conditioning, suggesting that the draft alone may be sufficient for the model to infer what needs improvement — or that the critique signal in Critique-GRPO is not as informative as intended. However, this comparison does not directly test whether conditioning on a mix of quality levels (best, median, worst) would improve over conditioning only on the best.

The GPT-5 judge experiment (Table 3) is also relevant. By replacing binary rewards with a $[0,1]$ scalar score, the $\arg\max$ can distinguish between different incorrect drafts — a draft scoring 0.8 (mostly correct) will be selected over one scoring 0.1 (nonsense). The +0.94 macro-average improvement with the GPT-5 judge is consistent with the hypothesis that better draft selection (distinguishing near-misses from poor attempts) improves refinement learning, but it does not test conditioning on multiple quality levels.

Mitigation status. Not addressed. The paper does not ablate the selection criterion (e.g., conditioning on the median draft, a random draft, or the worst draft) or test multi-draft conditioning (e.g., appending both the best and worst drafts with annotations). The $\arg\max$ design is presented as a natural choice without justification or comparison. Given the centrality of draft selection to the method — it is the sole mechanism by which Stage 1 influences Stage 2 — the absence of any ablation on the selection rule is a significant gap. A practitioner might reasonably ask: would selecting the median draft (to ensure conditioning on typical rather than extreme quality) work better? Would conditioning on both a correct and incorrect draft (contrastive learning) accelerate error-correction learning? The paper provides no evidence to answer these questions.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not introduce a new optimizer, a new reward function, or a new model architecture. It introduces a training protocol — a specific way of structuring the data that the optimizer sees — and demonstrates that this protocol alone, applied to existing algorithms (GRPO, DAPO, GSPO), yields consistent improvements on verifiable reasoning tasks. The conceptual shift is from optimizing "what the model generates" to optimizing "what the model generates conditioned on its own best prior attempt." This is not a paradigm shift on the scale of PPO itself or the discovery of chain-of-thought reasoning. It is better characterized as a reframing of the RL-for-reasoning problem: self-improvement is recast from an auxiliary task (learning to critique, verify, or self-reward) into a data-augmentation strategy (using the model's own outputs as in-context scaffolding during training).

What changes. Prior work on self-improvement for LLM reasoning — Self-Verification, Critique-GRPO, SPIN, Self-Rewarding LMs — all ask the model to learn an additional capability alongside the primary task: how to evaluate, critique, or score its own outputs. These methods couple two learning problems (generation and evaluation) that may conflict, dilute the training signal, or require careful balancing of multiple objectives. iGRPO demonstrates that this coupling is unnecessary. By decoupling feedback generation (handled by the external reward function in Stage 1 draft selection) from feedback utilization (handled by the policy in Stage 2 conditioned refinement), iGRPO achieves better performance (Table 1: +0.73 to +1.65 over Critique-GRPO and Self-Verification across model configurations) with a simpler system — no auxiliary objectives, no learned verifiers, no dual-role training. The paper provides a constructive existence proof that self-improvement in RL training does not require the model to be its own critic. This is a genuinely surprising result: the natural intuition, reflected in the design of prior methods, is that self-improvement requires self-evaluation. iGRPO shows that external evaluation plus in-context scaffolding is not only sufficient, but empirically better.

What becomes more attractive. The success of the refinement wrapper — which applies identically to GRPO, DAPO, and GSPO (Table 2) — suggests that data-augmentation through self-generated context is a general design principle orthogonal to the choice of RL optimizer. This opens a new axis of research: rather than developing ever-more-sophisticated advantage estimators or clipping schemes (the GRPO-variant literature), researchers can investigate how to structure the prompts that the optimizer sees during training. The paper's finding that the gains "primarily stem from the refinement interface rather than GRPO-specific details" (Section 5) implies that investment in better conditioning strategies — what draft to select, how to format it, whether to include multiple drafts, whether to annotate errors — may yield larger returns than investment in better optimization algorithms, at least in the verifiable reasoning domain. This is a direct parallel to the lesson from the LLM scaling literature: data quality and diversity often matter more than architectural innovations.

What becomes less attractive. The paper weakens the case for self-critique methods that require the model to generate auxiliary evaluation content (verification scores, natural-language critiques) as part of the RL training loop. If conditioning on an externally selected draft outperforms conditioning on a model-generated critique (Table 1), and if this benefit persists when the external evaluation is a learned model (GPT-5 judge, Table 3), then the additional complexity of training the model to be both solver and critic may not be justified. This does not mean self-critique methods are obsolete — they may be necessary in domains without external verifiability, or when the goal is to produce interpretable critiques for human consumption — but for the specific setting of verifiable reasoning with an available reward function, iGRPO establishes a simpler and stronger baseline that future self-critique methods must surpass.

Reconciling contradictions. The paper resolves a tension between two observations in the self-improvement literature. On one hand, iterative refinement at inference time (Self-Refine, Reflexion) can improve outputs, suggesting that models possess latent refinement capability. On the other hand, training models to self-correct during RL often yields disappointing results (Huang et al., 2023, finding that "LLMs cannot self-correct reasoning yet"). iGRPO reconciles these findings by showing that the critical variable is not whether refinement happens, but how the refinement signal is constructed during training. When the model must both generate and evaluate its own refinements (as in self-critique methods), the learning problem is hard because the evaluation signal is noisy and the model's capacity is split. When the evaluation is handled externally and the model only needs to learn to refine, the learning problem simplifies and the refinement capability emerges. The paper's entropy analysis (Figure 3) provides a mechanistic hypothesis: self-critique methods may cause even faster entropy collapse than standard GRPO (because the model converges to a narrow "critique + revise" pattern), while external evaluation preserves exploration by varying the conditioning context.

A new diagnostic tool. Proposition 3.1 — the guarantee that expected draft quality increases monotonically with policy performance under binary rewards — is not just a theoretical justification. It is a diagnostic for predicting when iGRPO will work and when it won't. The formula E[R_φ(d̂)] = 1 - (1 - V_θ)^N tells us that the method's effectiveness depends on the base policy's success rate V_θ. When V_θ is moderate (0.2–0.8), the bootstrapping cycle is active: better policies produce better drafts, which enable better refinements, which produce better policies. When V_θ is near 0, the cycle stalls because drafts are never correct and provide no useful scaffolding. When V_θ is near 1, the cycle saturates because drafts are almost always correct and there is little to refine. This predicts that iGRPO's gains should be concentrated on intermediate-difficulty problems — those where the model sometimes succeeds and sometimes fails — and minimal on trivially easy or impossibly hard problems. The paper does not test this prediction directly (it does not stratify results by difficulty), but the pattern across benchmarks is consistent: largest gains on AIME (hard competition problems where models are in the 40–85% range) and smallest gains on GSM8K (where models are at 90%+). This diagnostic changes how researchers should think about deploying iGRPO: it is not a universal booster, but a tool for the specific regime where models have non-trivial but imperfect capability. Future work applying iGRPO to new domains should first measure V_θ on the target task and predict, using Proposition 3.1, whether the bootstrapping dynamic will activate.

Follow-Up Research This Work Enables

Difficulty-stratified analysis of iGRPO to validate the bootstrapping hypothesis. The paper's central theoretical claim — that the bootstrapping dynamic depends on the base policy's success rate V_θ(q) — generates a clear, testable prediction: iGRPO's gains over GRPO should be largest on problems of intermediate difficulty (where V_θ is moderate, drafts are sometimes correct, and refinement can convert near-misses into successes), and smallest on problems that are trivially easy (V_θ ≈ 1, few errors to correct) or impossibly hard (V_θ ≈ 0, no correct drafts to scaffold from). A strong follow-up would replicate the MATH experiments in Table 1 and report per-quintile accuracy — binning the 7,500 training problems and the MATH500 test problems by the base model's pass@1 rate, then measuring iGRPO's gain over GRPO within each bin. This would directly test whether Proposition 3.1 correctly identifies the effective regime of the method, and would produce a practical deployment heuristic: measure pass@1 on a new task, and only apply iGRPO if the rate falls in the 0.1–0.9 range. A negative result — finding that iGRPO helps equally across all difficulty levels — would falsify the bootstrapping hypothesis and suggest that the benefit comes from a different mechanism (e.g., the prompt template instructions in Appendix C, or simply from training on longer prompts).

Optimal allocation of the Stage 1 vs. Stage 2 budget as a function of problem difficulty. The paper uses a uniform N=4, G=4 split for all prompts at a total budget of 8 completions, without ablating the allocation ratio. Proposition 3.1 implies that the optimal split should depend on V_θ: when V_θ is low, more Stage 1 drafts are needed to find a correct scaffold (since the probability of at least one success is 1 - (1 - V_θ)^N); when V_θ is high, fewer Stage 1 drafts are sufficient and the budget can shift to Stage 2 refinement. A natural experiment would train separate iGRPO policies with splits N=2/G=6, N=4/G=4, N=6/G=2 at a fixed total of 8, and measure accuracy per difficulty quintile. The prediction is that N=6/G=2 (exploration-heavy) should outperform on hard problems, N=2/G=6 (refinement-heavy) should outperform on easy problems, and N=4/G=4 should be best on intermediate problems. This would produce a compute-optimal allocation policy analogous to the difficulty-conditioned test-time scaling strategy in the reference example — a direct extension that the paper's framework naturally enables but does not explore.

Direct measurement of refinement quality: correction rate and degradation rate. The paper never measures whether Stage 2 actually produces answers better than the Stage 1 draft, nor how often it makes correct drafts worse. A simple diagnostic experiment: on a held-out set of problems, generate Stage 1 drafts, label each as correct or incorrect, then generate Stage 2 refinements conditioned on each draft, and measure (a) the correction rate — the fraction of incorrect drafts that are successfully refined into correct answers — and (b) the degradation rate — the fraction of correct drafts that are refined into incorrect answers. These metrics would directly quantify the quality of the learned refinement behavior and reveal whether iGRPO suffers from the correct-to-incorrect reversion problem documented in prior work on iterative self-correction (the reference example reports a ~38% degradation rate for revision models). If the degradation rate is high, it would suggest that iGRPO's gains in Table 1 come despite poor refinement behavior — perhaps because the model learns to ignore low-quality drafts during training, and the benefits accrue from the prompt template or the increased prompt diversity. If the degradation rate is low, it would validate the core claim that iGRPO teaches genuine refinement. A follow-up could compare correction/degradation rates for iGRPO vs. Critique-GRPO and Self-Verification to localize where the advantage comes from.

Training iGRPO directly on non-math reasoning tasks with learned reward models. The paper's generalization experiment (Figure 2) shows that refinement behaviors learned on math transfer to GPQA (+1.84) and MMLU-Pro (+0.91), but it never trains iGRPO directly on non-math tasks. A critical stress-test: train iGRPO on a dataset where the reward function is a learned verifier rather than a rule-based checker — for instance, training on a subset of MMLU or ARC where ground-truth answers exist but require a model-based extractor and grader, or on code generation tasks (HumanEval, MBPP) where the reward is test-case pass rate (a continuous scalar rather than binary). This would test whether iGRPO's mechanism is robust to reward noise. The theoretical prediction from Proposition 3.1 is that noisy rewards weaken the bootstrapping dynamic (a spuriously high-scoring draft may be selected in Stage 1, providing misleading conditioning for Stage 2), but the GPT-5 judge experiment (Table 3) suggests that even learned scalar rewards provide useful signal. A strong follow-up would compare iGRPO and GRPO on a code generation benchmark with varying levels of reward noise (e.g., using 1 test case vs. 10 test cases vs. 100 test cases to compute the reward), measuring whether iGRPO's advantage shrinks as the reward becomes noisier, and at what noise level the advantage disappears.

Combining iGRPO with process reward models (PRMs) for draft selection. The paper uses outcome-based rewards (binary or scalar) for Stage 1 draft selection. This means a nine-step solution with an error only in the final step receives the same reward as a completely nonsensical response, and the argmax cannot distinguish between them. A process reward model — which scores intermediate steps and can identify solutions that are mostly correct but fail at the end — would provide a richer ranking signal for draft selection. The GPT-5 judge experiment (Table 3) provides partial credit and yields +0.94 macro-average improvement, suggesting that better draft ranking helps. A natural extension would train a PRM on the base model's outputs (following the Monte Carlo rollout approach described in the reference example), use the PRM's step-level scores to rank Stage 1 drafts, and measure whether this improves the correction rate and final accuracy relative to binary-reward selection. This combines iGRPO's two-stage training structure with the verifier-guided search insights from the test-time compute scaling literature, and would test whether the two research directions (training-time iteration and verifier-guided selection) are complementary or redundant.

Ablation of the prompt template instructions to isolate their contribution. The Stage 2 prompt template in Appendix C includes explicit natural-language instructions: "treat it as guidance, not as ground truth," "produce a strictly improved final answer," "fix mistakes, fill gaps, strengthen justification, and improve clarity," "do not repeat the draft verbatim," "if the feedback is wrong or incomplete, correct it and proceed." These instructions are an inductive bias that may be responsible for a significant fraction of iGRPO's gains — they tell the model exactly what refinement behavior is desired, supplementing (or even substituting for) what the RL objective would teach on its own. A critical ablation would compare iGRPO with the full prompt template against iGRPO with a neutral template (e.g., simply appending the draft with a separator and no instructions) and against GRPO with the same instructions applied to its single-stage generations. If the instructions account for most of the gain, then iGRPO's contribution is primarily prompt engineering rather than the two-stage RL structure — an important finding that would redirect research toward better prompting strategies rather than multi-stage optimization. If the gain persists without the instructions, it strengthens the claim that the RL structure itself teaches refinement. The paper never isolates this variable, and it is one of the most straightforward experiments that would clarify the mechanism.

Practical Applications and Downstream Use Cases

RL fine-tuning of strong base models where single-shot optimization plateaus. The OpenMath-Nemotron-7B result in Table 1 exemplifies the practical scenario: a math-specialized model starting at 74.83% macro-average improves by only 0.19 points with standard GRPO (to 75.02%), but by 1.24 points with iGRPO (to 76.07%). For organizations that have invested in pretraining or supervised fine-tuning of domain-specific models and find that standard RL yields diminishing returns — a common experience as models approach the ceiling of what single-attempt generation can achieve — iGRPO provides a drop-in replacement that extracts additional performance without requiring more training data, larger models, or changes to the reward function. The concrete benefit is ~1 additional percentage point of macro-average accuracy at 7B scale and ~1.3–1.7 points at 14B scale (DeepSeek-R1-Distill-Qwen-14B: 71.29% → 73.02%; OpenMath-Nemotron-14B: 76.73% → 78.00%), achieved with 13% additional training time (94.1 vs. 83.3 GPU hours). The implementation cost is low — the refinement wrapper is a modification to the data sampling pipeline, not to the optimizer — making this a practical upgrade for existing GRPO training workflows.

Cost-efficient training of reasoning models with limited sampling budgets. The matched-budget design (8 completions per prompt for both iGRPO and GRPO) demonstrates that iGRPO achieves higher accuracy per sampled completion. In Table S.4, iGRPO with 8 completions achieves 69.87% on DeepSeek-R1-Distill-Qwen-7B, while GRPO with 8 completions achieves 68.29% (Table 1) — a +1.58 point gain at the same generation budget. For resource-constrained settings where generation cost dominates the training budget (e.g., academic labs, smaller companies, or very large models where each completion is expensive), iGRPO's ability to redistribute a fixed generation budget across exploration and refinement — rather than simply increasing the number of samples — offers a better accuracy-per-sample tradeoff. The diminishing returns in Table S.4 (70.17% at 16 completions, 70.33% at 32 completions) further suggest that iGRPO at 8 completions captures most of the available gain, making it a sweet spot for efficiency. A practitioner with a fixed GPU budget for RL training can expect higher final accuracy from iGRPO than from GRPO at the same sample count, with the 13% time overhead as the only additional cost.

Scaling to large training sets where diverse problems provide more bootstrapping opportunities. The large-scale experiment in Appendix B (OpenMath-Nemotron-14B trained on 94,000 examples from OpenR1-Math-220k with iGRPO) yields a +2.83 macro-average improvement over the base model (75.41% → 78.24%), with AIME25 rising from 61.18% to 66.04% (+4.86) and Minerva from 33.46% to 38.24% (+4.78). These gains are substantially larger than the improvements on the same model trained on the 7,500-problem MATH set (+2.59 over the base, with AIME25 increasing from 61.18% to 65.57% in Table 1), suggesting that iGRPO's mechanism scales favorably with training data quantity. The interpretation is straightforward: more diverse training problems provide more opportunities for the model to encounter near-miss reasoning traces in Stage 1 — problems where the draft is mostly correct but contains a specific, correctable error — and Stage 2 refinement learns to address these errors. For organizations with access to large-scale problem datasets (e.g., synthetic math problem generators, historical competition archives, or programmatically generated reasoning tasks), iGRPO provides a training protocol that efficiently converts data diversity into refinement capability, without requiring per-problem human feedback or critique annotation.

When to Prefer This Method

The paper positions iGRPO as an alternative to both standard GRPO and self-critique methods (Self-Verification, Critique-GRPO). Based on the evidence presented:

Prefer iGRPO over standard GRPO when:

  • The base model already achieves non-trivial but imperfect accuracy on the target task (approximately 20–80% pass@1), so Stage 1 can sometimes produce correct drafts that provide useful conditioning. The gains are largest in this regime (AIME benchmarks, where models are in the 40–85% range).
  • The training budget is measured in sampled completions rather than wall-clock time, and a 13% time overhead is acceptable in exchange for higher per-sample accuracy.
  • The reward function is reliable and deterministic (rule-based or high-quality model-based), so Stage 1 draft selection via argmax produces genuinely good drafts rather than spuriously high-scoring ones.
  • The model is being fine-tuned from a strong initialization (SFT or distilled model) where single-shot RL has begun to plateau, and additional gains require learning to correct residual errors.

Prefer iGRPO over self-critique methods (Self-Verification, Critique-GRPO) when:

  • A clean external reward function is available (as in verifiable math, code with test cases, or any domain with objective correctness criteria), so there is no need for the model to learn an internal evaluation capability. iGRPO decouples evaluation from generation and achieves better results (+0.73–1.65 points over both baselines in Table 1) with a simpler training loop.
  • Training simplicity is valued — no auxiliary objectives, no multi-task balancing, no learned verifier to maintain. iGRPO modifies only the data pipeline, not the optimization objective.
  • The deployment scenario uses single-shot generation at inference (standard in most production systems), and the model does not need to produce critiques or verification outputs for downstream consumers.

Prefer standard GRPO when:

  • The base model's accuracy on the target task is near zero, meaning Stage 1 will almost never produce a correct draft. iGRPO's bootstrapping dynamic stalls in this regime, and the additional complexity of the two-stage training provides no benefit.
  • Training wall-clock time is the binding constraint and the 13% overhead cannot be tolerated, even if per-sample efficiency is higher.
  • The reward function is extremely noisy or unreliable, making argmax draft selection potentially harmful (selecting a spuriously high-scoring incorrect draft that misleads Stage 2). In this regime, a method that learns to evaluate drafts internally might be more robust, though this hypothesis is untested in the paper.

Prefer self-critique methods (Self-Verification, Critique-GRPO) when:

  • The deployment scenario requires the model to produce interpretable self-evaluations (e.g., a tutoring system where the model explains why an answer is wrong, or a pipeline where critiques are surfaced to users). iGRPO does not train the model to generate critiques or verification rationales, so it cannot serve applications that need these outputs.
  • No external reward function is available (e.g., open-ended generation tasks where correctness is subjective), so the model must learn to evaluate itself. The paper does not test iGRPO in this regime, and the argmax over an unavailable reward is undefined.
  • The goal is to improve inference-time performance through self-verification (e.g., using verification scores to reweight sampled solutions at test time), rather than to improve the single-shot policy. iGRPO is purely a training-time method and provides no inference-time verification capability.