Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing

ArXiv: 2604.02288

🎯 Pitch

Combining reinforcement learning with self-distillation usually crashes during extended training, but routing correct samples to GRPO and failed ones to a stabilized SDPO not only prevents collapse—it delivers both faster early gains and the highest final scores across scientific reasoning benchmarks, all while cutting per-step compute by up to 17%.

---

1. Executive Summary

This paper introduces Sample-Routed Policy Optimization (SRPO), a unified on-policy framework that combines Group Relative Policy Optimization (GRPO) with Self-Distillation Policy Optimization (SDPO) by routing each rollout to the supervision signal best suited to its learning status—correct samples receive GRPO's reward-aligned reinforcement (coarse sequence-level advantage) while incorrect samples with available teacher information receive SDPO's targeted logit-level correction (dense token-level self-distillation from a successful sibling rollout). Evaluated across five benchmarks—Chemistry, Physics, Biology, Materials, and Tool Use—with Qwen3-8B and Qwen3-4B, SRPO raises the five-benchmark average by 3.4% over GRPO and 6.3% over SDPO at the 8B scale while reducing per-step compute time by up to 17.2% over long training horizons, establishing that sample-level routing preserves the rapid early improvement of self-distillation while maintaining the long-horizon stability of reward-driven reinforcement only when distillation targets are confined to failed trajectories and further modulated by an entropy-aware dynamic weighting mechanism that downweights unreliable high-entropy teacher predictions.

2. Context and Motivation

The Core Problem: Post-Training LLMs With Incompatible Supervision Signals

The fundamental tension this paper confronts is deceptively simple: the two dominant paradigms for post-training LLMs with reinforcement learning each have a critical weakness, and these weaknesses are complementary in a way that suggests they could be combined—but doing so naively fails. Understanding why requires stepping back to see what each paradigm contributes and where it breaks down.

When we apply reinforcement learning with verifiable rewards (RLVR) to language models, the policy generates a response, receives a scalar outcome reward (e.g., 1 for a correct answer, 0 for an incorrect one), and the policy is updated to maximize expected reward. The challenge is how to convert that single scalar reward into token-by-token training signals across potentially thousands of generated tokens. This credit assignment problem is the central axis along which GRPO and SDPO differ.

GRPO (Group Relative Policy Optimization) addresses credit assignment through group-relative normalization: for a given prompt, the policy generates multiple rollouts, computes the mean and standard deviation of their outcome rewards, and assigns each rollout a normalized advantage—essentially, how much better or worse that rollout was than the group average. This advantage is a single scalar, and GRPO broadcasts it uniformly across every token in the rollout. If the rollout was correct, every token gets the same positive signal; if it was incorrect, every token gets the same negative one.

This coarse credit assignment has a specific, well-documented failure mode: it cannot localize errors. A rollout that is entirely correct except for one token where the model made a simple arithmetic mistake receives exactly the same uniform penalty as a rollout that was nonsensical from the first token. The policy update lacks the granularity to strengthen the correct tokens while suppressing only the erroneous ones. Prior work has characterized this as causing diluted gradients across causally irrelevant tokens (Khandoga et al., 2026), failing to localize semantic errors in near-correct programs (Kumar et al., 2026), and introducing bias that grows with sequence length (Parthasarathi et al., 2025). The result is poor sample efficiency: the policy needs many examples to disambiguate which parts of a failed rollout are worth preserving and which need correction.

SDPO (Self-Distillation Policy Optimization) takes the opposite approach to credit assignment. Instead of a scalar reward, it constructs a self-teacher by conditioning the same model on privileged context—typically a correct sibling rollout from the same prompt—and uses the self-teacher's token-level probability distribution as a dense, logit-level supervision target. For each token position, SDPO minimizes the divergence between the student's predicted distribution (conditioned on its own partial trajectory) and the self-teacher's distribution (conditioned on both the partial trajectory and the privileged correct-answer context). This provides fine-grained guidance: rather than saying "this entire rollout was wrong," SDPO says "at this specific position, given where you were, you should have assigned higher probability to token X."

This dense supervision gives SDPO a substantial advantage in early training efficiency. On complex domains like scientific reasoning and agentic tool use, SDPO often converges much faster than GRPO, as demonstrated in Figure 1(a) of the paper. The reason is intuitive: when the policy is still weak and making many errors, token-level correction provides much more information per sample than a scalar reward, enabling rapid improvement.

However, SDPO has a catastrophic failure mode: late-stage instability and performance collapse. As training progresses, SDPO's performance saturates early, is overtaken by GRPO, and often degrades substantially—a pattern visible in Figure 1(a) (Chemistry benchmark) and Figure 3(c) (Tool Use benchmark), where SDPO's accuracy actually decreases over time after an initial peak.

Why This Gap Matters: Practical and Theoretical Significance

The gap this paper addresses has immediate practical consequences. RLVR has become the standard post-training pipeline for state-of-the-art reasoning models (OpenAI's o1 series, DeepSeek-R1, Kimi k1.5), and the choice of optimization method directly impacts training cost, final accuracy, and deployment characteristics. If we could combine GRPO's stability with SDPO's sample efficiency, we would get faster training with higher final accuracy and no collapse risk. Conversely, if we deploy SDPO unaware of its collapse mode, we risk catastrophic degradation in production models.

Beyond the practical training cost, there is a theoretical puzzle: why does SDPO collapse? The paper identifies a genuine mystery in the literature. Prior work by Kim et al. (2026) attributed similar instability in math domains to the suppression of epistemic verbalization—roughly, the model stops externalizing uncertainty ("I think...", "Perhaps...") when it is forced to match a confident self-teacher, and this loss of explicit reasoning degrades performance. But the present paper observes collapse patterns that suggest a more fundamental issue in the distillation signal itself, independent of verbalization effects. Resolving this puzzle matters because self-distillation is an increasingly popular technique (Agarwal et al., 2024; Zhao et al., 2026; Ye et al., 2026), and understanding its failure modes is essential for deploying it safely.

Where Prior Approaches Fall Short

The paper identifies two specific, previously undiagnosed failure modes of SDPO that explain its late-stage instability. These are not minor implementation details—they are intrinsic to the self-distillation mechanism as currently formulated.

Failure Mode 1: Self-distillation on already-correct samples introduces optimization ambiguity. In SDPO, the self-teacher is conditioned on a successful sibling rollout to provide dense targets. When the student's own rollout is already correct, forcing it to match a different correct sibling imposes arbitrary logit-level preferences between reward-equivalent reasoning paths. Two rollouts that both produce the correct final answer may arrive at that answer through different intermediate steps, different ordering of reasoning, or different degrees of verbalization—all of which receive the same reward (1.0) but look different at the token level. By minimizing the divergence between these two equally valid trajectories, SDPO pushes the policy toward one arbitrary style over another, creating optimization pressure that is not grounded in the reward objective.

The paper provides direct evidence for this claim in Figure 1(b): restricting SDPO updates to only incorrect samples retains most of the benefit (the model still improves), whereas applying SDPO only to correct samples degrades performance and accelerates collapse. This is a clean ablation: if self-distillation on correct samples were harmless or beneficial, the "correct-only" variant should perform similarly to or better than the "incorrect-only" variant. The fact that it is actively harmful demonstrates that the ambiguity on correct samples is a real source of destructive gradient signal.

This failure mode is subtle because it manifests differently at different training stages. Early in training, most rollouts are incorrect, so the "correct-sample ambiguity" problem is relatively rare—SDPO's rapid early improvement comes almost entirely from correcting failed rollouts, which is genuinely beneficial. As training progresses and more rollouts become correct, a larger fraction of SDPO updates involve this ambiguous, reward-decoupled optimization, and the noise accumulates until it overwhelms the useful signal.

Failure Mode 2: The self-teacher's distillation signal progressively degrades. The self-teacher is not a separate, fixed model—it is the same policy with an exponential moving average (EMA) update, conditioned on privileged context. As training proceeds and the gap between the self-teacher and student narrows (since both are tracking the same policy), the distillation signal necessarily becomes less informative. If the student and teacher distributions are nearly identical, the KL divergence is near zero regardless of correctness, and the gradient provides minimal useful guidance.

The paper provides novel evidence for the quality of this degradation: Figure 1(c) shows that the self-teacher's token-level entropy rises during training. This is crucial because entropy is a direct measure of prediction uncertainty. A low-entropy self-teacher distribution (peaked sharply on a few tokens) provides a confident, clear correction signal: "at this position, the answer should definitely be X." A high-entropy self-teacher distribution (spread across many tokens) provides an ambiguous signal: "at this position, the answer could be any of several things." High-entropy targets are effectively noisy labels—they pull the student in many directions weakly rather than one direction strongly—and are likely to introduce variance without improving signal quality.

Why would the self-teacher's entropy increase? One plausible mechanism is that as the policy improves and produces more correct rollouts, the self-teacher encounters a wider variety of correct solutions during conditioning. When conditioned on different correct siblings, the self-teacher's distribution at a given token position may spread out to accommodate multiple valid continuations, increasing entropy. Another possibility is that the policy's own representations become less calibrated at the token level as it optimizes for sequence-level outcomes, and this miscalibration transfers to the self-teacher via EMA updates.

Complementary Strengths and the Routing Hypothesis

The two failure modes suggest a natural division of labor that motivates SRPO:

• Correct rollouts are already reward-aligned. Applying SDPO to them is harmful (Failure Mode 1). Applying GRPO to them is appropriate: the uniform positive advantage reinforces the entire trajectory, which is exactly what we want for a sequence that produced the correct outcome. There is no need for token-level correction because the rollout is already correct.

• Incorrect rollouts need targeted correction. Applying GRPO to them is inefficient because the uniform penalty cannot localize the specific errors. Applying SDPO to them is beneficial because it provides dense, token-level signals that identify which parts of the trajectory need adjustment. And applying SDPO only to incorrect samples avoids Failure Mode 1 entirely—there is no ambiguity about which correct path to follow because the student's path was incorrect to begin with.

This complementarity is the paper's central insight. It is not merely that GRPO and SDPO have different strengths—it is that each method's weakness aligns precisely with the other method's strength, and the division can be made along a clean, observable criterion: was the rollout correct?

How This Paper Positions Itself

The paper frames itself as addressing a gap between two mature post-training paradigms that have been studied largely in isolation. Prior work on improving credit assignment in RLVR has pursued two independent directions: (1) process supervision and process reward models that provide denser, step-level reward signals but require additional learned reward estimators (Lightman et al., 2023; Setlur et al., 2025; Cui et al., 2025), and (2) on-policy and self-distillation methods that provide dense logit-level guidance without additional reward models but that remove the need for an external teacher (Hübotter et al., 2026; Zhao et al., 2026; Agarwal et al., 2024).

The paper's position is distinctive because it does not propose a new supervision signal. Instead, it proposes a routing mechanism that decides which existing signal to apply to each sample. The two branches—GRPO and DW-SDPO—are essentially unchanged from their standalone implementations; what is novel is the decision rule that routes each rollout to the appropriate branch and the entropy-aware weighting mechanism that improves the reliability of the SDPO branch in later training.

This is conceptually similar to mixture-of-experts routing or adaptive computation strategies, but applied at the level of supervision signals rather than model parameters. The paper explicitly draws the analogy to advantage estimators at different granularities (Section 3.1): both GRPO and SDPO can be viewed as computing an advantage—GRPO's advantage is scalar, reward-derived, and sequence-level; SDPO's advantage is vector-valued, teacher-derived, and logit-level. Sample routing simply selects the more appropriate advantage estimator for each sample based on its learning status.

The paper also positions itself against a naive alternative: advantage-level mixing (linearly combining GRPO and SDPO advantages into a single loss). The ablation in Table 2 explicitly compares sample routing against this approach, showing that advantage mixing provides a small early benefit (+0.7 at 1 hour) but degrades substantially over time (-3.3 at 10 hours). This comparison is important because it demonstrates that the routing decision matters—simply blending the two signals without discriminating between correct and incorrect samples propagates the noise from Failure Mode 1 throughout training, undermining the stability benefit that sample routing provides.

Finally, the paper positions its contribution as complementary to the epistemic verbalization explanation of SDPO collapse proposed by Kim et al. (2026). The authors do not dispute that suppression of verbalized uncertainty may contribute to degradation; rather, they argue that there is an additional, independent cause rooted in the distillation signal itself—one that manifests even when verbalization is not the primary factor. The empirical evidence (Figure 1(b-c)) supports this claim by demonstrating signal degradation patterns that are distinct from verbalization effects, establishing SRPO as addressing a broader set of failure modes.

3. Technical Approach

3.1 Reader Orientation

SRPO is a routing system that sits between the language model's policy and the training objectives, inspecting each generated rollout and deciding which optimization signal to apply based on whether the rollout was correct: correct rollouts go to GRPO for coarse sequence-level reinforcement, while incorrect rollouts that have access to a correct sibling go to an entropy-weighted SDPO branch for targeted token-level correction. The problem it solves is that GRPO and SDPO each have complementary failure modes—GRPO cannot localize specific token errors in failed rollouts (slow convergence), while SDPO introduces optimization ambiguity on correct rollouts and suffers from degrading self-teacher quality (late-stage collapse)—and SRPO resolves both by applying each method only to the samples where it is appropriate, using the rollout's correctness as a clean, observable routing criterion.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components connected in a pipeline:

1. Policy $\pi_\theta$ (the LLM) — a Qwen3 model that generates rollouts from prompts, serves as both the student being optimized and (via EMA updates and privileged conditioning) the self-teacher for distillation.

2. Sampling and Evaluation — for each prompt, the policy generates $G = 8$ rollouts; each rollout is evaluated against an environment-provided verifiable reward function to obtain scalar rewards $r_i \in \{0, 1\}$, determining correctness.

3. Teacher Information Construction — for each rollout, the system checks whether a correct sibling rollout exists in the same group; if so, that sibling's full response text serves as the privileged context $f_i$ for the self-teacher.

4. Sample Router — a binary decision rule inspects each rollout's correctness flag $c_i$ and teacher-availability flag $m_i$, routing correct rollouts and rollouts without teacher information to the GRPO branch, and incorrect rollouts with available teacher information to the Dynamic-Weighted SDPO branch.

5. Two-Branch Loss Computation — the GRPO branch computes a group-relative scalar advantage broadcast uniformly across all tokens; the DW-SDPO branch computes token-level KL divergences between the student and a feedback-conditioned self-teacher, reweighted by an entropy-based confidence measure that downweights unreliable high-entropy teacher predictions.

Information flows as follows: a prompt is sampled → the policy generates $G$ rollouts → rewards are computed → teacher information is constructed from correct siblings → the router assigns each rollout to either the GRPO or DW-SDPO branch → both branches compute their respective token-level losses → losses are summed and normalized by total token count → the policy parameters are updated via gradient descent. As training progresses, more rollouts become correct, so the fraction of samples flowing through the SDPO branch automatically decreases, shifting the effective optimization mix from distillation-heavy (early) to reinforcement-heavy (late) without manual scheduling.

3.3 Roadmap for the Deep Dive

• First, the sample routing rule — the binary decision logic that assigns each rollout to GRPO or DW-SDPO, since this is the central mechanism that distinguishes SRPO from prior work and enables the complementary use of both signals.
• Second, the GRPO branch — what happens to routed samples in terms of advantage computation and policy update, since this branch anchors stable long-horizon optimization and must be understood as the baseline against which the SDPO branch is compared.
• Third, the DW-SDPO branch — the self-teacher construction, the distillation loss, and the entropy-aware dynamic weighting mechanism, since this is where the paper's diagnostic insights (Failure Modes 1 and 2) are operationalized into a concrete optimization procedure.
• Fourth, the combined objective — how the two branches' losses are aggregated and normalized, since this determines the effective mixing ratio between reward-driven and distillation-driven updates at each training step.
• Fifth, the training algorithm end-to-end — the full loop from sampling through routing through gradient update, since this integrates all components into a concrete procedure.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a method paper whose core idea is that GRPO and SDPO can be unified within a single on-policy framework by routing each rollout to the supervision signal that matches its learning status, and that this routing—combined with entropy-based reweighting of the SDPO branch—simultaneously achieves the rapid early convergence of SDPO and the long-horizon stability of GRPO.

---

Sample-Level Routing

The routing mechanism is the central architectural innovation of SRPO. It operates on a per-rollout basis, making binary decisions based on two observable properties of each rollout: whether the rollout is correct and whether teacher information is available.

Routing criteria. For each rollout $y_i$ generated from prompt $x$, the system defines two binary indicators. The correctness flag is:

$c_i = \mathbf{1}[y_i \text{ is correct}]$

where $c_i = 1$ if the rollout's reward satisfies $r_i \geq 0.5$ (the environment returns binary 0/1 rewards, so this is equivalent to $r_i = 1$), and $c_i = 0$ otherwise. The correctness determination comes from the environment's verifiable reward function—for the Science Q&A benchmarks, this checks whether the model's final answer letter (A/B/C/D) matches the ground truth; for Tool Use, it checks whether the structured tool call matches the expected API invocation.

The teacher-availability flag is:

$m_i = \mathbf{1}[\text{teacher information is available for } y_i]$

where $m_i = 1$ if at least one correct rollout exists in the same group excluding $y_i$ itself, and $m_i = 0$ otherwise. The exclusion of the rollout itself prevents a sample from serving as its own teacher, which would create a degenerate self-consistency objective with zero gradient.

What it computes: the teacher-availability flag checks whether there exists a correct sibling rollout $y_j$ (with $j \neq i$ and $r_j \geq 0.5$) in the same group of $G = 8$ rollouts. If at least one such sibling exists, the system selects one (the paper does not specify a selection criterion, implying random selection among correct siblings) and uses its full response text as the privileged feedback $f_i$. If no correct sibling exists (all $G = 8$ rollouts are incorrect), $m_i = 0$ for every rollout in that group.

Why this form: using a correct sibling as teacher information grounds the self-distillation signal in a trajectory that is known to be reward-aligned. Alternative sources of teacher information—such as an external stronger model, human-written solutions, or environment execution traces—would introduce different cost and availability tradeoffs. The paper follows SDPO's original design choice of using sibling rollouts because it requires no external resources beyond what is already generated during on-policy sampling. The constraint that the sibling must be from the same prompt ensures that the teacher's trajectory is directly relevant to the question being answered, avoiding the distribution mismatch that would arise from using solutions to different questions.

Routing decision. Given these two indicators, the routing mask for the SDPO branch is:

$z^{\text{SDPO}}_i = (1 - c_i) \, m_i$

where $z^{\text{SDPO}}_i = 1$ means the rollout is routed to the SDPO branch. The GRPO mask is the complement:

$z^{\text{GRPO}}_i = 1 - z^{\text{SDPO}}_i$

where $z^{\text{GRPO}}_i = 1$ means the rollout is routed to the GRPO branch.

What it computes: this is a logical AND of two conditions: the rollout must be incorrect ($1 - c_i = 1$, which is $c_i = 0$) AND teacher information must be available ($m_i = 1$). The four possible cases are:

• Correct rollout, teacher available ($c_i = 1, m_i = 1$): $z^{\text{GRPO}}_i = 1$, $z^{\text{SDPO}}_i = 0$ → GRPO branch. The rollout is already correct, so GRPO's uniform positive advantage reinforces the entire trajectory without introducing ambiguity from matching a different correct sibling.
• Correct rollout, no teacher available ($c_i = 1, m_i = 0$): $z^{\text{GRPO}}_i = 1$, $z^{\text{SDPO}}_i = 0$ → GRPO branch. This is the same as above; the lack of a teacher doesn't matter because we wouldn't use SDPO on a correct rollout anyway.
• Incorrect rollout, teacher available ($c_i = 0, m_i = 1$): $z^{\text{SDPO}}_i = 1$, $z^{\text{GRPO}}_i = 0$ → SDPO branch. This is the core case: the rollout needs correction, and a correct sibling provides the privileged context for targeted token-level guidance.
• Incorrect rollout, no teacher available ($c_i = 0, m_i = 0$): $z^{\text{GRPO}}_i = 1$, $z^{\text{SDPO}}_i = 0$ → GRPO branch (fallback). When all rollouts in a group are incorrect, there is no sibling to serve as teacher, so the system falls back to GRPO's uniform penalty. This is a graceful degradation: we lose the fine-grained correction that SDPO would provide, but we still get the reward-aligned signal.

Why this form: the routing rule encodes the paper's central diagnostic insight from Figure 1(b) and Section 1: self-distillation on already-correct samples introduces optimization ambiguity (Failure Mode 1), so correct samples must be excluded from the SDPO branch. The AND with $m_i$ is a practical necessity—you cannot apply SDPO without teacher information—but it is not the primary theoretical motivation. The key design choice is the $(1 - c_i)$ term, which operationalizes the finding that restricting SDPO to incorrect samples retains most of its benefit while avoiding the ambiguity-induced collapse.

An important subtlety: the routing rule does NOT send correct rollouts to a "no-update" or "discard" path. Correct rollouts are still updated via GRPO. This is crucial because GRPO's uniform positive advantage serves a different function than SDPO's token-level correction—it anchors the policy to reward-maximizing behavior, providing a stable optimization target that prevents drift. If correct rollouts were simply ignored (as might be suggested by a surface reading of Failure Mode 1), the policy would lose the reinforcement signal that steers it toward the reward objective. The routing rule is not "apply SDPO only to incorrect samples and do nothing for correct ones"—it is "apply SDPO to incorrect samples and GRPO to all other samples."

Adaptive mixing without hyperparameters. A deliberate consequence of this routing design is that the effective ratio of GRPO to SDPO updates is determined entirely by the policy's current accuracy, not by a manually tuned mixing coefficient. Early in training, when the policy is weak and most rollouts are incorrect with available teachers, a large fraction of samples flows through the SDPO branch, providing dense corrective supervision. As training progresses and more rollouts become correct, the SDPO fraction decreases and the GRPO fraction increases correspondingly. Figure 5 in Appendix C quantifies this: at the start of training on Chemistry with Qwen3-8B, approximately 40% of samples route to SDPO and 60% to GRPO; these proportions shift steadily as accuracy improves. This automatic scheduling is a practical advantage over the "Advantage Mix" baseline (Table 2), which uses a fixed mixing coefficient $\lambda = 0.9$ and requires manual tuning that cannot adapt to changing conditions.

---

The GRPO Branch

When a rollout is routed to the GRPO branch, it receives a uniform sequence-level advantage signal. The GRPO loss is computed identically to the standalone GRPO baseline, with the same strengthened implementation described in Section 4.1.

Advantage computation. For a group of $G = 8$ rollouts with rewards $\{r_i\}_{i=1}^G$, the GRPO advantage for rollout $i$ is:

$A^{\text{GRPO}}_i = \frac{r_i - \bar{r}}{\sigma_r + \epsilon}$

where $\bar{r} = \frac{1}{G} \sum_{j=1}^G r_j$ is the group mean reward, $\sigma_r = \sqrt{\frac{1}{G} \sum_{j=1}^G (r_j - \bar{r})^2}$ is the group standard deviation, and $\epsilon$ is a small constant for numerical stability (typically $10^{-8}$, preventing division by zero when all rollouts have identical rewards).

What it computes: this is a z-score normalization of the rollout's reward relative to the group it belongs to. A rollout that is better than the group average receives a positive advantage (scaled by how much better, relative to the group's spread); a rollout that is worse receives a negative advantage. The normalization removes the need for a learned value function (critic) by using the group itself as the reference distribution.

Why this form: group-relative normalization has two key properties. First, it is baseline-subtracted: by subtracting the group mean, the advantage estimator has zero mean across the group, which reduces the variance of the policy gradient estimator (this is the standard REINFORCE with baseline technique, Williams, 1992). Second, it is scale-invariant: dividing by the standard deviation makes the advantage magnitude independent of the reward scale, which prevents one benchmark with larger raw rewards from dominating the optimization and eliminates the need to tune reward scaling per-task. The unbiased advantage normalization variant used in the paper's GRPO implementation (citing Liu et al., 2025) corrects for the bias introduced by using the sample standard deviation rather than the population standard deviation when the group size is small.

This advantage $A^{\text{GRPO}}_i$ is a single scalar—it does not depend on token position $t$. Unlike SDPO, which provides a different signal at each token, GRPO assigns this same scalar to every token in the rollout.

Policy update. The GRPO loss for a single rollout $i$ uses the clipped surrogate objective from PPO (Schulman et al., 2017):

$\mathcal{L}^{\text{GRPO}}_i = -\frac{1}{|y_i|} \sum_{t=1}^{|y_i|} \min\left( \rho_{i,t}(\theta) \, A^{\text{GRPO}}_i, \; \text{clip}(\rho_{i,t}(\theta), 1 - \varepsilon_{\text{low}}, 1 + \varepsilon_{\text{high}}) \, A^{\text{GRPO}}_i \right)$

where $\rho_{i,t}(\theta) = \pi_\theta(y_{i,t} \mid x, y_{i,<t}) / \pi_{\theta_{\text{old}}}(y_{i,t} \mid x, y_{i,<t})$ is the importance-sampling ratio at token $t$, $\pi_{\theta_{\text{old}}}$ is the policy from before the current update (the "old" policy used for sampling), $\varepsilon_{\text{low}} = 0.28$ is the asymmetric lower clip threshold, $\varepsilon_{\text{high}} = 0.28$ is the symmetric upper clip threshold (though the paper's use of asymmetric clipping from Yu et al., 2025 implies these may differ in practice), and $|y_i|$ is the number of valid response tokens in the rollout.

What it computes: for each token, the loss is the negative of the clipped advantage-weighted log-probability ratio. The min operation implements a trust-region constraint: if the importance ratio $\rho$ moves too far from 1, the gradient is clipped, preventing the policy from changing too much in a single update. When the advantage is positive (good rollout), the loss encourages increasing $\rho$ (making the action more likely) but stops encouraging once $\rho$ exceeds $1 + \varepsilon_{\text{high}}$. When the advantage is negative (bad rollout), the loss encourages decreasing $\rho$ (making the action less likely) but stops encouraging once $\rho$ drops below $1 - \varepsilon_{\text{low}}$. The overall loss is the average over all tokens, which means every token in the rollout contributes equally to the gradient.

Why this form: the clipped surrogate is the standard technique for stabilizing policy-gradient updates by bounding the per-step policy change. The asymmetric clipping ($\varepsilon_{\text{low}}$ potentially different from $\varepsilon_{\text{high}}$) allows more aggressive suppression of bad actions than reinforcement of good ones, which recent work (Yu et al., 2025) has shown improves training stability for LLM post-training. The averaging over tokens distributes the sequence-level advantage uniformly: because $A^{\text{GRPO}}_i$ does not vary with $t$, the per-token gradient is proportional to $\nabla_\theta \log \pi_\theta(y_{i,t} \mid x, y_{i,<t})$, scaled by the same advantage. This is precisely the coarse credit assignment that Section 1 identifies as a limitation—but one that is appropriate for correct rollouts, where all tokens are presumed to have contributed to the successful outcome.

Additional GRPO-specific parameters mentioned in Table 3 include a rollout importance-sampling clip $\rho$ of 2 (which clips the importance ratio $\rho_{i,t}$ before it enters the PPO clip, preventing extreme ratios from dominating) and a KL coefficient of 0.0 (indicating no explicit KL penalty against a reference policy—the trust region is enforced solely through clipping).

---

The Dynamic-Weighted SDPO Branch

When a rollout is routed to the SDPO branch, it receives dense, token-level supervision from a feedback-conditioned self-teacher, with per-token weights modulated by the teacher's prediction entropy to suppress unreliable signals.

Self-teacher construction. The self-teacher is the same policy $\pi_\theta$ but conditioned on an enriched context. For a student rollout $y_i$ that has been routed to SDPO, the teacher information $f_i$ is the full response text of a correct sibling rollout from the same group. The teacher prompt is constructed by concatenating the original question, the correct solution text, and a final instruction to "Correctly solve the original question" (see Listing 5 in Appendix B.5). The self-teacher distribution at token position $t$ is:

$q_{i,t}(v) = \pi_\theta(v \mid x, f_i, y_{i,<t})$

where $v$ ranges over the vocabulary $\mathcal{V}$, $x$ is the original prompt, $f_i$ is the privileged feedback context, and $y_{i,<t}$ are the student's own preceding tokens along the trajectory.

What it computes: for each position $t$ in the student's generated response, the self-teacher produces a probability distribution over the entire vocabulary, representing what the model would predict at that position if it had access to the correct answer as context alongside the partial trajectory so far. Crucially, the self-teacher does not generate any new tokens—it only re-scores the existing student trajectory under the enriched conditioning. This is an important efficiency property: we need only one forward pass through the self-teacher to obtain log-probabilities for all positions, rather than autoregressive generation.

Why this form: feeding the correct solution as privileged context gives the self-teacher information about what a successful trajectory looks like for this prompt, creating a discrepancy between the teacher's predictions (conditioned on the correct answer) and the student's predictions (conditioned only on the partial trajectory). This discrepancy is the source of the SDPO learning signal: at positions where the student made an error, the teacher's distribution (informed by the correct answer) will assign high probability to different tokens than the student's distribution, creating a gradient that pushes the student toward the teacher's predictions. The EMA update of the teacher parameters (update rate 0.05 per Table 3) stabilizes the teacher distribution, preventing it from changing too rapidly as the student updates and creating a slowly-moving target that reduces training variance.

Distillation loss. The SDPO loss at a single token position $t$ is the Jensen-Shannon divergence between the student and teacher distributions:

$\ell^{\text{SDPO}}_{i,t} = \text{JS}\left( \pi_\theta(\cdot \mid x, y_{i,<t}) \;\Big\Vert\; \text{stopgrad}\left( \pi_\theta(\cdot \mid x, f_i, y_{i,<t}) \right) \right)$

where stopgrad indicates that gradients do not flow through the teacher—the teacher distribution is treated as a fixed target for the purpose of backpropagation. The Jensen-Shannon divergence between two distributions $P$ and $Q$ is defined as:

$\text{JS}(P \parallel Q) = \frac{1}{2} \text{KL}(P \parallel M) + \frac{1}{2} \text{KL}(Q \parallel M)$

where $M = \frac{1}{2}(P + Q)$ is the mixture distribution, and $\text{KL}$ is the Kullback-Leibler divergence. The divergence is computed over the top-$K$ tokens, with $K = 100$ (Table 3), meaning only the 100 highest-probability tokens in the teacher distribution are considered; all other tokens are masked out of the loss computation.

What it computes: the JS divergence measures the symmetric dissimilarity between the student and teacher distributions at each token position. A value of 0 means the distributions are identical; larger values indicate greater disagreement. The loss encourages the student to bring its predictions into alignment with the teacher's feedback-informed predictions. By using the student's own partial trajectory $y_{i,<t}$ as context, the distillation is performed on-policy—the student is corrected at exactly the states it actually visits, rather than at states from a separate teacher-generated trajectory.

Why this form: the paper uses JS divergence rather than the more common forward KL ($\text{KL}(Q \parallel P)$ where $Q$ is the teacher) or reverse KL ($\text{KL}(P \parallel Q)$). Forward KL is "mean-seeking"—it penalizes the student most heavily when the teacher assigns high probability to a token that the student assigns low probability to, which is appropriate for distillation but can be unstable when the teacher distribution has sharp peaks. Reverse KL is "mode-seeking"—it penalizes the student when it assigns high probability to tokens the teacher assigns low probability to, which can cause the student to collapse to a single mode of the teacher distribution. JS divergence symmetrizes these behaviors, providing a balanced objective that the SDPO authors (Hübotter et al., 2026) found empirically superior in their grid search. The top-K restriction to $K = 100$ prevents the loss from being dominated by tokens in the long tail of the vocabulary, which have near-zero probability under both distributions and contribute noise rather than signal.

The stopgrad operator is essential: without it, gradients would flow through both the student and teacher branches, creating a degenerate objective where the optimal solution is to collapse both distributions to a point mass, rather than the student moving toward the teacher while the teacher remains stable.

Entropy-aware dynamic weighting. The core innovation in the SDPO branch (relative to vanilla SDPO) is the entropy-based reweighting mechanism that modulates each token's contribution to the loss based on the self-teacher's prediction confidence.

For token position $t$ of rollout $i$, the self-teacher's entropy is:

$H_{i,t} = -\sum_{v \in \mathcal{V}} q_{i,t}(v) \log q_{i,t}(v)$

where $q_{i,t}(v) = \pi_\theta(v \mid x, f_i, y_{i,<t})$ is the self-teacher distribution defined above. The sum is over the full vocabulary $\mathcal{V}$, and $\log$ is the natural logarithm.

What it computes: entropy is the standard information-theoretic measure of uncertainty in a probability distribution. A distribution sharply peaked on a few tokens (e.g., "the answer is almost certainly 'B'") has low entropy, indicating the teacher is confident about what should come next. A distribution spread evenly across many tokens (e.g., "it could be A, B, C, or D, with roughly equal probability") has high entropy, indicating the teacher is uncertain. Entropy is measured in nats (since the natural logarithm is used), with typical values ranging from near 0 (very confident) to $\log |\mathcal{V}|$ (completely uniform, maximally uncertain).

Why this form: entropy is chosen as the confidence measure because it captures the shape of the entire distribution, not just the probability of the most-likely token (which would be captured by max probability). A teacher that assigns 90% probability to one token and 10% spread across the rest has lower entropy than one that assigns 50/50 to two tokens even though the max probability is 90%, because the latter distribution is genuinely more uncertain about the relative plausibility of the two top candidates. This matters because SDPO's loss is a divergence over the entire top-K distribution—a teacher uncertain between two plausible continuations provides a noisier gradient signal than one confident in a single continuation.

Weight computation. The unnormalized weight for token $t$ is:

$\tilde{w}_{i,t} = \exp(-\beta \, H_{i,t})$

where $\beta > 0$ is the dynamic-weighting temperature (default value $\beta = 1$, per Section 4.1). This weight is then normalized across all tokens routed to the SDPO branch in the current batch:

$w_{i,t} = \frac{\tilde{w}_{i,t}}{\frac{1}{|\Omega_{\text{sdpo}}|} \sum_{(j,s) \in \Omega_{\text{sdpo}}} \tilde{w}_{j,s}}$

where $\Omega_{\text{sdpo}}$ is the set of all $(\text{rollout\_index}, \text{token\_position})$ pairs that are routed to the SDPO branch, and $|\Omega_{\text{sdpo}}|$ is the total number of such tokens.

What it computes: the exponential $\exp(-\beta H_{i,t})$ maps entropy to a weight in $(0, 1]$. When entropy is low (teacher confident), $-\beta H_{i,t}$ is close to 0, so $\tilde{w}_{i,t} \approx 1$. When entropy is high (teacher uncertain), $-\beta H_{i,t}$ is a large negative number, so $\tilde{w}_{i,t} \approx 0$. The normalization by the mean weight across all SDPO tokens ensures that the average weight is 1.0—the overall scale of the SDPO loss is preserved, but the relative contribution of different tokens is rebalanced: confident tokens receive weights above 1 (amplified), uncertain tokens receive weights below 1 (suppressed).

Why this form: the exponential transformation $\exp(-\beta H)$ is the standard Gibbs/Boltzmann distribution derived from treating entropy as an energy (lower energy = higher probability = higher confidence). It has the property that small differences in entropy produce exponentially scaled differences in weight: a token with entropy 0.5 gets weight $\exp(-0.5) \approx 0.61$, while a token with entropy 2.0 gets weight $\exp(-2.0) \approx 0.14$—roughly a 4.4× difference in contribution. This strong suppression of high-entropy targets is intentional: the paper's diagnostic in Figure 1(c) shows that teacher entropy rises during training, meaning an increasing fraction of SDPO tokens would otherwise be contributing noisy, uncertain gradient signals. The exponential weighting concentrates the SDPO loss on the remaining high-confidence corrections, which are more likely to be genuinely informative.

The normalization by the mean weight ($\frac{1}{|\Omega_{\text{sdpo}}|} \sum \tilde{w}$) rather than the sum weight ($\sum \tilde{w}$) is a subtle but important design choice. Normalizing by the mean makes the reweighting distribution-preserving: it changes which tokens contribute most to the loss while keeping the total loss magnitude roughly constant across batches with different numbers of SDPO tokens. Normalizing by the sum would make the loss scale inversely proportional to the number of SDPO tokens, creating an undesirable coupling between batch composition and effective learning rate.

Why dynamic weighting is necessary. The paper's Failure Mode 2 diagnosis establishes that the self-teacher's signal degrades as entropy rises during training. Without dynamic weighting, SDPO would continue applying equal importance to all teacher predictions, including increasingly noisy high-entropy ones, which would inject variance into the policy update and contribute to late-stage instability. Dynamic weighting directly addresses this by making the SDPO branch self-limiting: as the teacher becomes less reliable, the effective contribution of SDPO decreases not just in volume (fewer samples routed there, since more rollouts become correct) but also in per-token quality (uncertain predictions are downweighted within the remaining SDPO samples). This two-level attenuation—fewer SDPO rollouts overall, and lower weight on uncertain tokens within those rollouts—is what enables SRPO to continue improving beyond the point where pure SDPO collapses.

The temperature parameter $\beta$ controls the sharpness of this attenuation. With $\beta = 0$, all weights are 1.0 (no entropy-based reweighting, equivalent to SRPO without dynamic weighting). With $\beta \to \infty$, only the single lowest-entropy token in each batch would receive non-zero weight (hard thresholding). The default $\beta = 1$ provides a moderate degree of confidence-based filtering that the paper's ablation (Table 2) shows provides an additional 1.8 percentage point gain at 10 hours over SRPO without dynamic weighting.

---

The Combined Objective

The total training loss is the weighted sum of the GRPO and DW-SDPO losses, normalized by the total number of tokens being optimized:

$\mathcal{L}_{\text{final}} = \frac{\sum_{i,t} z^{\text{GRPO}}_i \, \ell^{\text{GRPO}}_{i,t} + \sum_{i,t} z^{\text{SDPO}}_i \, \ell^{\text{DW-SDPO}}_{i,t}}{\sum_{i,t} z^{\text{GRPO}}_i + \sum_{i,t} z^{\text{SDPO}}_i}$

where $\ell^{\text{GRPO}}_{i,t}$ is the per-token GRPO loss (the min-clipped surrogate objective with the sequence-level advantage $A^{\text{GRPO}}_i$ distributed over all response tokens in rollout $i$), and $\ell^{\text{DW-SDPO}}_{i,t} = w_{i,t} \cdot \ell^{\text{SDPO}}_{i,t}$ is the entropy-weighted SDPO loss at token $t$. The summations over $i$ run over all rollouts, and the summations over $t$ run over all valid response tokens within each rollout (padding and prompt tokens are excluded).

What it computes: the numerator sums the token-level losses from both branches, each token contributing either the GRPO loss or the DW-SDPO loss depending on its routing assignment (since $z^{\text{GRPO}}_i$ and $z^{\text{SDPO}}_i$ are mutually exclusive, each token contributes exactly one of the two). The denominator counts the total number of valid response tokens across all rollouts. The result is the average loss per token across the entire batch, with each branch contributing to the final loss in proportion to the number of tokens it covers.

Why this form: the per-token averaging (rather than summing) has three important properties:

1. No manual mixing hyperparameter: the effective weight of GRPO vs. SDPO is determined by the fraction of tokens routed to each branch, which in turn depends on the policy's current accuracy. This eliminates the need for a mixing coefficient $\lambda$ (contrast with the Advantage Mix baseline, which requires manually setting $\lambda = 0.9$), reducing hyperparameter sensitivity and enabling automatic adaptation to changing conditions.

2. Scale invariance to batch composition: whether a batch contains mostly correct rollouts (few SDPO tokens) or mostly incorrect rollouts (many SDPO tokens), the average loss per token remains on a consistent scale. This prevents large swings in effective learning rate as training progresses and the routing composition shifts.

3. Natural early-late transition: early in training, when failures are frequent, many tokens flow through the SDPO branch, giving dense correction a proportionally larger effective weight. As the policy improves and more rollouts succeed, the GRPO branch dominates the denominator, anchoring the update to the reward objective. This automatic rebalancing is visible in the routing statistics (Appendix C, Figure 5): the SDPO fraction decreases from roughly 40% to below 20% over the course of training.

The combined objective does not include any explicit KL penalty against a reference policy (the GRPO KL coefficient is 0.0 per Table 3), relying instead on the PPO clipping in the GRPO branch and the stopgrad target in the SDPO branch to prevent destructive policy updates.

---

Full Training Algorithm

The complete SRPO training loop is specified in Algorithm 1 of the paper. Here we walk through each step with the concrete hyperparameters and design choices that instantiate it.

Initialization. The policy $\pi_\theta$ is initialized from an instruct-tuned Qwen3 checkpoint (4B or 8B). The optimizer is AdamW with learning rate $5 \times 10^{-6}$, weight decay 0.01, and gradient clipping at norm 1.0. The learning rate is chosen to be halfway between the GRPO learning rate ($1 \times 10^{-6}$) and the SDPO learning rate ($1 \times 10^{-5}$), balancing the sensitivity requirements of the reward-driven and distillation-driven signals within a single objective. Training proceeds with 10 warmup steps (linearly increasing the learning rate from 0 to $5 \times 10^{-6}$).

Step 1: Prompt sampling. A batch of 32 prompts is sampled from the training dataset. Each benchmark has its own training split (e.g., 1,890 Chemistry prompts, 720 Physics prompts). The prompt is formatted according to the benchmark's template (Listings 1–4 in Appendix B.3), which includes a system prompt specifying the output format (for Science Q&A: reasoning in <reasoning> tags followed by the answer letter in <answer> tags; for Tool Use: a structured Thought/Action/Action Input format).

Step 2: Rollout generation. For each of the 32 prompts, the policy $\pi_\theta$ generates $G = 8$ rollouts via SGLang inference with temperature 1.0 (no top-p filtering is applied during training rollouts; top-p 0.95 is used only for validation). The maximum prompt length is 2048 tokens and the maximum response length is 8192 tokens. Thinking mode is disabled (Table 3, "Thinking: False"), meaning the model generates responses directly without an internal chain-of-thought within special tags—the reasoning is produced as part of the visible output sequence. This yields 32 × 8 = 256 rollouts per training step.

Step 3: Reward evaluation. Each rollout is evaluated in the benchmark's environment to obtain a scalar reward $r_i$. For Science Q&A benchmarks, the reward is 1 if the extracted answer letter (the content within <answer> tags) matches the ground truth, and 0 otherwise. For Tool Use, the reward is 1 if the structured tool call matches the expected API invocation (function name and parameter values). This reward computation is deterministic and verifiable—no learned reward model or human judgment is involved. The correctness flag $c_i = \mathbf{1}[r_i \geq 0.5]$ is set to 1 for correct rollouts and 0 for incorrect ones.

Step 4: Teacher information construction. For each prompt, the system identifies all correct rollouts in its group of 8. For each rollout $y_i$:

• If at least one correct sibling exists ($\exists j \neq i$ such that $c_j = 1$), select one and use its full response text as $f_i$. Set $m_i = 1$. The teacher prompt is: "{original_question}\nCorrect solution:\n{sibling_response}\nCorrectly solve the original question."
• If no correct sibling exists, set $m_i = 0$ for all rollouts in that group.

The self-teacher parameters are maintained as an EMA of the student parameters with update rate 0.05 (Table 3), meaning after each student update, the teacher parameters move 5% of the way toward the new student parameters: $\theta_{\text{teacher}} \leftarrow 0.95 \cdot \theta_{\text{teacher}} + 0.05 \cdot \theta_{\text{student}}$.

Step 5: Routing and loss computation. For each rollout in the batch:

• If $c_i = 0$ and $m_i = 1$ (incorrect, teacher available): route to DW-SDPO branch.

  • Compute the self-teacher distribution $q_{i,t}(v) = \pi_\theta(v \mid x, f_i, y_{i,<t})$ at each token position via a single forward pass through the EMA teacher.
  • Compute entropy $H_{i,t}$ for each token position.
  • Compute unnormalized weights $\tilde{w}_{i,t} = \exp(-\beta H_{i,t})$ with $\beta = 1$.
  • Normalize weights across all SDPO tokens in the batch: $w_{i,t} = \tilde{w}_{i,t} / \text{mean}(\tilde{w}_{\text{all\_sdpo}})$.
  • Compute base SDPO loss $\ell^{\text{SDPO}}_{i,t}$ as JS divergence with top-K = 100.
  • Apply weighting: $\ell^{\text{DW-SDPO}}_{i,t} = w_{i,t} \cdot \ell^{\text{SDPO}}_{i,t}$.

• Otherwise (correct, or incorrect without teacher): route to GRPO branch.

  • Compute group-relative advantage $A^{\text{GRPO}}_i$ using the rewards from all 8 rollouts in the group.
  • Compute the clipped surrogate loss at each token with asymmetric clipping $\varepsilon = 0.28$ and importance-sampling ratio clipped to $\rho = 2$.

Step 6: Loss aggregation. Sum the per-token losses from both branches (each token contributes exactly one branch's loss, determined by the routing masks) and divide by the total number of valid response tokens across all 256 rollouts.

Step 7: Gradient update. Compute gradients of $\mathcal{L}_{\text{final}}$ with respect to student parameters $\theta$, clip gradients to norm 1.0, and apply the AdamW update. Update the EMA teacher parameters as described above. Increment the training step counter and repeat from Step 1.

Mini-batching. The batch of 32 prompts is further split into mini-batches for gradient computation. The paper uses a mini-batch size of 32 (Table 3), meaning that the 32 prompts are processed in a single mini-batch (the total batch size equals the mini-batch size). This is a simplification relative to the GRPO baseline, which uses a mini-batch size of 8 with a total batch size of 32, requiring 4 gradient accumulation steps.

Validation protocol. During validation (occurring periodically during training), the policy generates 16 rollouts per prompt (rather than 8) at temperature 0.6 and top-p 0.95 (rather than temperature 1.0 with no top-p). The average accuracy across rollouts (avg@16) is reported—the fraction of the 16 rollouts that are correct, averaged over all test prompts. No SDPO or GRPO updates are applied during validation; the policy is evaluated in inference-only mode.

Training duration. Experiments are reported at wall-clock time budgets of 1 hour, 5 hours, and 10 hours (Table 1). All training is conducted on 8 NVIDIA H20 GPUs with 768 GB total VRAM, using FSDP2 for distributed training and SGLang for rollout inference. The training continues until the time budget is exhausted; there is no explicit early stopping criterion based on validation performance.

Why this design over alternatives: the mini-batch size of 32 and rollout count of 8 are inherited from SDPO's original configuration (Hübotter et al., 2026), which selected these values via grid search to maximize validation accuracy. The decision to keep SDPO's batching configuration rather than GRPO's (which uses smaller mini-batches) reflects the fact that the SDPO branch—requiring an additional forward pass through the self-teacher—is the compute bottleneck, and larger mini-batches amortize the fixed cost of the teacher computation more effectively.

4. Key Insights and Innovations

Innovation 1: Diagnosing SDPO's Collapse as a Sample-Dependent Signal Ambiguity Problem, Not Merely a Verbalization Issue

This paper's most intellectually distinctive contribution is not the SRPO algorithm itself, but the diagnostic framework that motivated it. Prior to this work, the dominant explanation for why self-distillation methods degrade during prolonged training came from Kim et al. (2026), who attributed the collapse to the suppression of epistemic verbalization—the model stops expressing uncertainty in its reasoning traces when forced to match a confident self-teacher, and this loss of explicit metacognitive language impairs its ability to reason correctly. This explanation is plausible, fits some domains, and suggests that the fix lies in preserving or encouraging verbalized uncertainty.

The present paper challenges this as the sole cause by identifying two additional, independent failure modes that are intrinsic to the self-distillation mechanism itself, independent of whether the model verbalizes its reasoning. The key diagnostic move is the ablation in Figure 1(b): restricting SDPO updates to only incorrect samples preserves most of the benefit, while applying SDPO only to correct samples actively degrades performance and accelerates collapse. This result cannot be explained by the verbalization hypothesis—if the problem were simply that SDPO suppresses uncertainty expressions, we would expect degradation regardless of which samples SDPO is applied to, since the self-teacher's confident conditioning would suppress verbalization on both correct and incorrect trajectories. Instead, the differential effect implicates a sample-dependent mechanism: self-distillation on correct samples introduces optimization ambiguity (forcing a correct trajectory to match a different correct sibling imposes arbitrary logit-level preferences between reward-equivalent reasoning paths), while self-distillation on incorrect samples provides genuine corrective signal.

This diagnostic framing is significant beyond the specific algorithm it inspired because it reframes the conversation around self-distillation's failure modes from what the model outputs (surface-level verbalization patterns) to what the optimization signal encodes (the information-theoretic quality of the teacher distribution relative to the student's learning status). It suggests that the research community should focus less on preserving specific linguistic patterns and more on ensuring that distillation targets carry genuine, non-ambiguous corrective information. This is a conceptual shift from treating self-distillation as uniformly beneficial (with degradation being a side effect of verbalization loss) to treating it as sample-conditional: beneficial on failures, harmful on successes.

The second piece of this diagnostic framework—Figure 1(c)'s demonstration that the self-teacher's token-level entropy rises during training—provides an independent axis of signal degradation that is similarly independent of verbalization. Rising entropy means the teacher's predictions become increasingly uncertain and therefore less informative as distillation targets, regardless of what the model says. This is a purely information-theoretic failure: even if the model perfectly preserved its verbalized reasoning, a high-entropy teacher would still provide noisy gradients that inject variance into the policy update.

Together, these two diagnostics establish that SDPO's collapse has multiple causes operating at different levels of the optimization process. This is a fundamental advance in understanding self-distillation failure, not an incremental refinement. It doesn't just propose a fix—it provides a diagnostic toolkit (correctness-conditional ablation, entropy tracking) that future work can use to evaluate whether proposed self-distillation variants have resolved these specific failure modes.

Innovation 2: Reframing the GRPO-SDPO Tradeoff as a Credit Assignment Complementarity That Can Be Resolved by Observability, Not Tuning

Before this work, the relationship between GRPO and SDPO was typically framed as a performance tradeoff: GRPO is stable but slow; SDPO is fast but unstable; more advanced methods or careful hyperparameter tuning might find a better balance. This framing implicitly treats the two methods as alternative optimization algorithms competing for the same role—providing a training signal for the policy—and suggests that combining them requires finding the right mixing ratio, loss weight, or schedule.

SRPO introduces a fundamentally different framing: GRPO and SDPO are complementary credit assignment mechanisms whose suitability depends on an observable property of the rollout (whether it is correct). This shifts the problem from "how should we blend these two signals?" (a continuous optimization over mixing coefficients) to "which signal is appropriate for which sample?" (a discrete routing decision based on a condition we can check). The critical insight is that the correctness of the rollout is a sufficient statistic for deciding which credit assignment mechanism to use. Correct rollouts have no identifiable token-level errors to correct (the outcome was right), so token-level distillation is unnecessary and potentially harmful; GRPO's uniform positive reinforcement is sufficient and avoids the ambiguity problem. Incorrect rollouts have specific errors that the uniform GRPO penalty cannot localize, so token-level distillation is needed.

This is a fundamental reframing of the combination problem, not just a new combination method. The key intellectual move is recognizing that the weakness of each method (GRPO's inability to localize errors, SDPO's ambiguity on correct samples) is not a global property of the method but rather a conditional property that manifests only on specific types of samples. Prior work treated these as inherent limitations—"GRPO has coarse credit assignment" and "SDPO suffers from ambiguity"—rather than as limitations that are active only when the method is applied to the wrong type of rollout.

The evidence supporting this reframing is in Table 2's comparison between SRPO and Advantage Mix. Advantage Mix treats the combination as a blending problem: linearly combine GRPO and SDPO advantages with a fixed coefficient λ = 0.9, applied uniformly to all samples regardless of correctness. This approach shows modest early benefit (+0.7 at 1 hour) but degrades substantially over time (−3.3 at 10 hours), because it does not discriminate between samples—the SDPO signal's ambiguity on correct samples contaminates the blended advantage, and as more rollouts become correct during training, this contamination grows. By contrast, SRPO's correctness-conditional routing avoids this contamination entirely by excluding correct samples from the SDPO branch, yielding sustained improvement over time. The comparison demonstrates that how the signals are combined (routing vs. blending) matters more than the specific mixing ratio—a finding that would not emerge from a view of the problem as a simple tradeoff to be tuned.

This reframing has practical significance beyond the SRPO algorithm itself. It suggests that future work on combining supervision signals should focus on identifying routing criteria—observable properties of samples that indicate which signal is appropriate—rather than on more sophisticated blending strategies. It also implies that the appropriate combination strategy may be inherently non-stationary in a way that is automatically handled by routing based on the policy's own accuracy, avoiding the need for manually scheduled mixing ratios that must be re-tuned for different tasks or training regimes.

Innovation 3: Entropy-Aware Dynamic Weighting as a Self-Limiting Mechanism That Attenuates Distillation Influence as Signal Quality Degrades

The dynamic weighting mechanism introduced in the DW-SDPO branch represents a distinct conceptual contribution that is separable from the sample routing framework. While the routing decision addresses which rollouts receive distillation (only incorrect ones), dynamic weighting addresses how much to trust the distillation signal within those rollouts, as a function of the teacher's own uncertainty.

The key insight is that the self-teacher's entropy provides an intrinsic, per-token measure of distillation signal quality that can be used to automatically modulate the influence of self-distillation over the course of training. This is not merely a regularization technique or a heuristic for stabilizing training—it is a principled mechanism for making the SDPO branch self-limiting. As the teacher becomes less reliable (entropy rises, Figure 1(c)), the dynamic weights automatically reduce the effective contribution of distillation, preventing the noise from accumulating and causing collapse.

What makes this intellectually distinctive is the negative feedback loop it creates between teacher quality and distillation influence. In standard SDPO, the distillation loss has a fixed weight regardless of teacher quality—as the teacher degrades, it continues pulling the student with equal force, compounding the noise. In DW-SDPO, degrading teacher quality reduces the force of distillation, creating a natural brake on the degradation process. This means the SDPO branch implicitly "knows when to be quiet"—it contributes strongly when it has confident, informative corrections to make (early training, on clear errors) and attenuates itself when its signal becomes noisy (late training, when the teacher is uncertain). This self-limiting property is what enables SRPO to continue improving beyond the point where pure SDPO collapses, without requiring external monitoring or manual intervention.

The ablation evidence (Table 2, second block) isolates this effect: adding dynamic weighting to sample routing provides an additional gain that grows over time (from +0.4 at 1h to +1.8 at 10h). The widening gap is exactly what we would expect from a mechanism designed to suppress late-stage noise—early in training, when the teacher is still relatively confident, dynamic weighting makes little difference because most tokens would receive weights near 1.0 anyway. Later, when entropy has risen and many teacher predictions have become noisy, dynamic weighting increasingly differentiates between high-confidence and low-confidence targets, concentrating the remaining distillation budget on the most reliable corrections and discarding the rest.

This is a fundamental conceptual advance in how to handle degrading teacher quality in self-distillation, not an incremental trick. Prior approaches to teacher degradation either ignored it (accepting collapse), used early stopping (sacrificing potential further gains), or manually scheduled the distillation weight (requiring task-specific tuning). Dynamic weighting provides a third option: make the distillation objective itself quality-aware, so that it automatically scales its influence based on how reliable its targets are. The mechanism is simple—exponential of negative entropy, normalized across tokens—but the conceptual move from "apply distillation with fixed weight, hope the teacher doesn't degrade too much" to "let the distillation weight track teacher confidence" represents a qualitative shift in how we think about integrating self-distillation into RL pipelines.

The choice of entropy (rather than, say, the maximum probability or the variance of the teacher distribution) as the confidence signal is also notable. Entropy captures the full distribution shape—a teacher that is 50/50 between two plausible continuations has higher entropy (and thus receives lower weight) than one that is 90/10, even though the maximum probability is higher in the latter case. This matters because SDPO's loss is a divergence over the full top-K distribution, not just the argmax—a teacher uncertain between two continuations provides a genuinely noisier gradient signal than one confident in a single continuation, even if both would select the same most-likely token. The entropy metric correctly captures this distinction.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper uses five benchmarks: Chemistry, Physics, Biology, Materials, and Tool Use. The first four are drawn from the reasoning subsets of SciKnowEval (Feng et al., 2024), targeting undergraduate-level scientific reasoning with four-option single-choice questions (e.g., predicting molecular solubility, calculating physical quantities, estimating protein folding stability, computing crystal unit cell volume). Tool Use is drawn from ToolAlpaca (Tang et al., 2023) and evaluates whether the model can map a user request and tool specification to the correct structured tool call. The train/test splits are identical to those used by SDPO (Hübotter et al., 2026): Chemistry (1,890 train / 210 test), Physics (720 / 80), Biology (450 / 50), Materials (841 / 94), and Tool Use (4,046 / 68). All evaluation is in-domain (train and test drawn from the same distribution per benchmark).

• Base model(s). Experiments use instruct-tuned Qwen3 models (Yang et al., 2025) at two scales: Qwen3-4B and Qwen3-8B. The paper chooses Qwen3 because it is a contemporary open-weight family with strong base capabilities, and evaluating at two scales allows checking whether SRPO's behavior generalizes across model sizes. The base instruct checkpoints achieve the following avg@16 accuracies across the five benchmarks: Qwen3-8B achieves 41.1 on Chemistry, 58.7 on Physics, 30.5 on Biology, 59.3 on Materials, and 57.9 on Tool Use; Qwen3-4B achieves 43.6, 59.8, 30.8, 61.2, and 58.8 respectively. Notably, Qwen3-4B slightly outperforms Qwen3-8B on the base checkpoints—an anomalous ordering that the paper attributes to these benchmarks not being explicitly targeted during Qwen3 fine-tuning and to the well-documented phenomenon of nonmonotonic scaling on out-of-distribution downstream tasks (McKenzie et al., 2023; Lourie et al., 2025). Crucially, the larger 8B model still achieves higher post-training performance and larger total training gains, consistent with expected scaling behavior.

• Metrics. The primary metric throughout is avg@16 accuracy (%), which is the fraction of 16 validation rollouts that are correct, averaged over all test prompts in a benchmark. For Science Q&A, correctness is determined by whether the extracted answer letter (from <answer> tags) matches the ground truth; for Tool Use, correctness is whether the structured tool call matches the expected API invocation. Validation rollouts use temperature 0.6 and top-p 0.95 (more conservative decoding than training rollouts, which use temperature 1.0 with no top-p). Results are reported at three wall-clock training budgets (1 hour, 5 hours, 10 hours), with the "highest achieved avg@16 within each budget" serving as the reported number (following SDPO's protocol). For the five-benchmark average, the paper reports the simple mean across the five individual benchmark accuracies. Additional tracked metrics include response length (number of tokens per rollout, measured during training on Chemistry) and per-step compute time (average seconds per training step, averaged across all five benchmarks).

• Baselines. The paper compares against two methods: (1) GRPO, a strengthened implementation of Group Relative Policy Optimization (Shao et al., 2024) incorporating asymmetric clipping (Yu et al., 2025), unbiased advantage normalization (Liu et al., 2025), and off-policy correction for distributed inference (Yao et al., 2025); and (2) SDPO, Self-Distillation Policy Optimization (Hübotter et al., 2026), which replaces reward-only supervision with self-distillation from a feedback-conditioned self-teacher (using successful sibling rollouts as teacher information) and provides dense logit-level guidance. Both baselines use hyperparameters from the original SDPO paper's grid search: GRPO uses learning rate $1 \times 10^{-6}$ and mini-batch size 8; SDPO uses learning rate $1 \times 10^{-5}$ and mini-batch size 32. An additional ablation baseline, Advantage Mix, linearly combines GRPO and SDPO advantages with a fixed coefficient $\lambda = 0.9$ and is used only in the ablation study (Table 2).

• Generation budget / compute accounting. Fair comparison is ensured by equalizing the wall-clock training time (1h, 5h, 10h on identical hardware: 8 NVIDIA H20 GPUs, 768 GB total VRAM, NVLink interconnect). For the main results, all methods use the same batch size (32 prompts), number of rollouts per prompt (G = 8), and maximum response length (8192 tokens). The compute differences arise from per-step overhead: SDPO and SRPO require an additional self-teacher forward pass for routed samples, while GRPO does not. Per-step compute time is tracked explicitly (Figure 4(b)): at 1h, GRPO averages 71.0s per step, SDPO 85.9s, and SRPO 83.4s; at 10h, these shift to 91.5s, 83.7s, and 75.8s respectively. SRPO's decreasing per-step cost results from fewer rollouts requiring the SDPO branch as training progresses (Appendix C, Figure 5) and from generating moderately shorter responses than GRPO (Figure 4(a)), which reduces both inference and forward-pass computation.

• Cross-validation / statistical protocol. The paper does not employ cross-validation for model selection—training continues until the wall-clock budget is exhausted, and the highest avg@16 achieved within each budget window is reported. Training curves (Figure 3) show 5-step rolling means with shaded bands denoting ±1 standard deviation, providing a measure of variance across training steps. The main results table (Table 1) reports point estimates (the peak accuracy within each budget) without confidence intervals, which is consistent with the reporting protocol of the SDPO baseline paper but limits statistical rigor. All experiments use the exact train/test splits from the SDPO repository to ensure comparability.

Main Quantitative Results

Aggregate Performance Across Benchmarks and Budgets

The headline result appears in Table 1: SRPO consistently achieves the highest peak performance across both model scales and at all three training budgets, with particularly large gains at the 10-hour budget. On Qwen3-8B at 10h, SRPO raises the five-benchmark average to 77.4%, compared to 74.0% for GRPO (+3.4 percentage points) and 71.1% for SDPO (+6.3 percentage points). On Qwen3-4B at 10h, the corresponding averages are 74.2% (SRPO), 69.7% (GRPO, +4.5), and 66.7% (SDPO, +7.5).

Several patterns emerge from the per-benchmark breakdown at the 8B scale:

• SRPO dominates at 10h across all benchmarks. On Chemistry, SRPO achieves 83.0 vs. 78.9 (GRPO) and 80.6 (SDPO). On Physics: 78.4 vs. 73.6 (GRPO) and 74.0 (SDPO). On Biology: 72.8 vs. 70.6 (GRPO) and 58.5 (SDPO). On Materials: 81.5 vs. 77.8 (GRPO) and 76.6 (SDPO). On Tool Use: 71.2 vs. 69.0 (GRPO) and 65.7 (SDPO). The improvements over GRPO range from +2.2 (Biology, Tool Use) to +4.8 (Physics); the improvements over SDPO are uniformly larger, ranging from +4.4 (Physics) to +14.3 (Biology), reflecting SDPO's early saturation and collapse.

• SDPO saturates early; GRPO improves steadily; SRPO combines both traits. SDPO's 5h and 10h averages are identical on all benchmarks at both model scales—e.g., on Qwen3-8B Chemistry, SDPO scores 80.6 at both 5h and 10h; on Biology, 58.5 at both. This stagnation or collapse after an initial peak is the late-stage instability the paper diagnoses. GRPO continues improving from 5h to 10h on most benchmarks (e.g., Chemistry: 75.9 → 78.9; Biology: 68.1 → 70.6). SRPO improves substantially from 5h to 10h on all benchmarks (e.g., Chemistry: 81.8 → 83.0; Biology: 68.3 → 72.8), indicating it avoids both SDPO's early saturation and GRPO's slower convergence.

• At 1h, SRPO is competitive with or slightly behind SDPO on some benchmarks. On Chemistry at 1h, SDPO leads (71.6 vs. 69.2 for SRPO), while on Physics (69.5 vs. 67.6) and Biology (55.8 vs. 52.1), SRPO leads. This is consistent with the design: early in training, SDPO's dense correction provides rapid improvement; SRPO's routing initially sends a substantial fraction of samples to SDPO (approximately 40%, per Figure 5), so it largely tracks SDPO's early trajectory. The small initial gap on some benchmarks may reflect the slightly reduced effective SDPO budget (some samples fall back to GRPO) or the learning rate difference (SRPO's $5 \times 10^{-6}$ vs. SDPO's $1 \times 10^{-5}$).

Training Dynamics: Learning Curves on Representative Benchmarks

Figure 3 plots avg@16 against wall-clock training time for Qwen3-8B on three benchmarks: Chemistry (panel a), Biology (panel b), and Tool Use (panel c). These curves complement Table 1 by showing the full trajectory, not just the peak within each budget window.

Chemistry (Figure 3a): SRPO extends SDPO's advantage while avoiding collapse. SDPO rises rapidly, reaching approximately 0.75 avg@16 by ~2 hours and peaking around 0.80. It then oscillates with high variance (wide shaded bands) between roughly 0.72–0.80, consistent with the instability the paper attributes to growing teacher entropy and correct-sample ambiguity. GRPO rises more slowly but steadily, reaching approximately 0.78 by 10h with lower variance. SRPO largely tracks SDPO's early trajectory (overlapping curves through ~2h), then continues to improve while SDPO oscillates, reaching approximately 0.82–0.83 by 10h with variance comparable to GRPO. The key visual pattern: SRPO's curve separates from SDPO's around the point where SDPO's variance widens, suggesting that SRPO's routing and entropy weighting suppress the noise that drives SDPO's oscillations.

Biology (Figure 3b): SRPO leads throughout and widens the gap as SDPO stalls. SRPO achieves the highest 1h result (approximately 0.56 vs. 0.52 for SDPO and 0.47 for GRPO). SDPO plateaus around 0.58 by 5h and shows minimal further improvement, while SRPO continues climbing to approximately 0.73 by 10h. GRPO follows a shallower trajectory, reaching approximately 0.71. The variance bands are notably wider for all methods on Biology than on Chemistry, likely reflecting the smaller training set (450 examples vs. 1,890).

Tool Use (Figure 3c): SRPO remains stable when SDPO degrades. This is the most striking panel for demonstrating SRPO's robustness. SDPO's curve exhibits a clear degradation pattern: after an early peak around 0.66 by ~2h, accuracy steadily declines to approximately 0.62–0.63 by 10h, with widening variance. This is the "catastrophic collapse" pattern the paper warns about. GRPO improves gradually from approximately 0.64 to 0.69 with low variance. SRPO tracks GRPO closely but slightly ahead, reaching approximately 0.71 by 10h with variance comparable to GRPO. Crucially, SRPO shows no sign of the degradation that afflicts SDPO—even on a benchmark where self-distillation is actively harmful in later training, the GRPO branch anchors the update, and SRPO's performance never drops below GRPO's.

These curves collectively demonstrate the paper's central claim: SRPO matches or approaches SDPO's early efficiency (visible on Chemistry and Biology) while maintaining GRPO's stability and continuing to improve beyond SDPO's saturation point (visible on all three). On benchmarks where SDPO would collapse entirely (Tool Use), SRPO falls back to tracking GRPO without degradation.

Response Length and Compute Efficiency

Figure 4 provides complementary metrics that characterize the practical deployment characteristics of SRPO relative to baselines.

Response length (Figure 4a, measured on Chemistry with Qwen3-8B). The three methods produce markedly different response length trajectories. GRPO's responses remain long throughout training, fluctuating between approximately 300–500 tokens with a slight upward trend (the curve appears noisy, with the 5-step rolling mean showing oscillations). SDPO's responses start moderately long (~400 tokens) but drop rapidly within the first ~100 training steps to below 100 tokens and remain extremely short thereafter. SRPO's responses start similarly to SDPO but stabilize at an intermediate length, roughly 150–250 tokens throughout training. The paper interprets SDPO's brevity as a potential contributor to degraded reasoning ("suppression of epistemic verbalization," citing Kim et al., 2026), where the model learns to skip reasoning steps because the self-teacher's confident conditioning discourages verbose uncertainty expression. SRPO's moderate response length suggests that the GRPO branch—which continues to receive uniform positive advantage for correct rollouts regardless of length—counterbalances the brevity pressure from the SDPO branch, maintaining enough verbosity for adequate reasoning while avoiding GRPO's uncontrolled length growth.

Per-step compute time (Figure 4b, averaged across five benchmarks for Qwen3-8B). Measured over three time windows (1h, 5h, 10h), the per-step cost dynamics shift in SRPO's favor:

• At 1h: GRPO 71.0s, SDPO 85.9s, SRPO 83.4s. SRPO incurs a 17.4% overhead relative to GRPO and is marginally (2.9%) faster than SDPO. The overhead comes from the self-teacher forward pass on samples routed to the SDPO branch. SDPO's higher cost (85.9s vs. 83.4s) despite doing similar forward passes may reflect SDPO's longer response lengths early in training (Figure 4a shows SDPO responses are initially long before the rapid drop).

• At 5h: GRPO 82.4s, SDPO 83.9s, SRPO 78.3s. SRPO becomes 4.9% faster than GRPO and 6.7% faster than SDPO. GRPO's cost increases because its responses remain long or grow longer; SDPO's cost decreases slightly as responses shorten; SRPO benefits from both moderate response length and the decreasing fraction of samples routed to SDPO (fewer self-teacher forward passes).

• At 10h: GRPO 91.5s, SDPO 83.7s, SRPO 75.8s. SRPO achieves 17.2% reduction over GRPO and 9.4% over SDPO. GRPO's cost has grown substantially (71.0 → 91.5, a 28.9% increase) consistent with its verbosity trend. SDPO's cost is roughly flat (85.9 → 83.7). SRPO's cost decreases from 83.4 → 75.8 (a 9.1% reduction), reflecting the combined effect of fewer SDPO-routed samples and moderate-length responses that are shorter than GRPO's but longer than SDPO's.

The paper attributes SRPO's decreasing per-step compute time to the dynamic routing composition documented in Appendix C (Figure 5): the fraction of samples routed to the SDPO branch (which requires an additional teacher forward pass and thus extra computation) decreases from approximately 40% to below 20% over training, while the GRPO branch (which has no teacher overhead) grows correspondingly. Additionally, SRPO's response length (Figure 4a) remains substantially shorter than GRPO's, reducing the token count for both the policy forward pass and the loss computation. This compute-time advantage is a practical benefit beyond accuracy: SRPO not only achieves higher final performance but does so with lower per-step cost over long training horizons.

Ablation Studies and Robustness Checks

The ablation study (Table 2) is organized into two blocks, each isolating a specific component of SRPO on Qwen3-8B across all five benchmarks. Results are reported as the five-benchmark average avg@16 at 1h, 5h, and 10h.

Mixing strategy: sample routing vs. advantage-level mixing. The first block compares SRPO without dynamic weighting against the Advantage Mix baseline, which linearly combines GRPO and SDPO advantages with a fixed coefficient $\lambda = 0.9$ applied uniformly to all samples. SRPO w/o dynamic weighting achieves 66.5 at 1h, 74.8 at 5h, and 75.6 at 10h. Advantage Mix achieves 67.2 (+0.7) at 1h, 72.3 (−2.5) at 5h, and 72.3 (−3.3) at 10h. The key pattern: Advantage Mix shows a small early benefit but then plateaus at 72.3 by 5h with no further improvement, while SRPO w/o dynamic weighting continues improving from 5h to 10h (74.8 → 75.6) and maintains a growing lead. The early advantage of Advantage Mix (+0.7 at 1h) is notable—it suggests that when SDPO's signal is still high-quality (early training), blending both advantages helps slightly. However, the late-stage degradation (−3.3 at 10h) confirms that blending propagates SDPO's noise into the GRPO signal on correct samples, contaminating the reward-aligned advantage and limiting long-horizon performance.

Dynamic weighting: additive contribution on top of sample routing. The second block compares full SRPO against SRPO w/o dynamic weighting. Full SRPO achieves 66.9, 75.5, and 77.4 at 1h, 5h, and 10h respectively, while SRPO w/o dynamic weighting achieves 66.5 (−0.4), 74.8 (−0.7), and 75.6 (−1.8). The gain from dynamic weighting grows monotonically with training time: −0.4 at 1h, −0.7 at 5h, −1.8 at 10h. This widening gap supports the paper's claim that entropy-aware weighting matters most in later training, when the self-teacher becomes noisier (as documented in Figure 1(c)). Early in training, when teacher entropy is low and most predictions are confident, the entropy-based weights are close to uniform, so dynamic weighting has minimal effect. As training progresses and teacher entropy rises, the gap between SRPO with and without dynamic weighting grows because dynamic weighting increasingly downweights high-entropy (uncertain) teacher predictions, suppressing the noise that would otherwise accumulate in the SDPO branch. The fact that the gain at 10h (−1.8) is larger than the gap between SRPO and Advantage Mix at 10h (−3.3) by proportion (approximately 55% of the total routing benefit) suggests that dynamic weighting is responsible for roughly half of SRPO's long-horizon advantage over naive mixing.

Diagnostic ablations from Figure 1 (in the main text, Section 1). Beyond Table 2, the paper presents two critical diagnostic ablations in Figure 1 that motivated SRPO's design:

• Figure 1(b): SDPO restricted to incorrect vs. correct samples. Restricting SDPO updates to only incorrect samples retains most of the overall SDPO benefit (the training curve largely overlaps with full SDPO's early trajectory and avoids collapse), while applying SDPO only to correct samples degrades performance and accelerates collapse (the curve drops substantially below full SDPO). This ablation directly supports Failure Mode 1—self-distillation on correct samples is not merely unnecessary but actively harmful.

• Figure 1(c): Self-teacher entropy over training. The self-teacher's token-level entropy rises monotonically during SDPO training (from approximately 1.5 to above 2.5 on Chemistry with Qwen3-8B), indicating that the distillation signal becomes increasingly dominated by uncertain predictions. This supports Failure Mode 2 and motivates the entropy-aware dynamic weighting.

Robustness across model scales. Table 1 includes results for both Qwen3-8B and Qwen3-4B. The pattern of SRPO outperforming both baselines is consistent across scales: at 10h, SRPO improves the Qwen3-4B five-benchmark average by +4.5 over GRPO and +7.5 over SDPO, compared to +3.4 and +6.3 at the 8B scale. The absolute gap is comparable, suggesting that SRPO's benefits are not specific to a particular model capacity. The anomalous base-checkpoint ordering (4B outperforming 8B) is acknowledged but does not affect the relative improvements from post-training.

Routing statistics over time (Appendix C, Figure 5). The fraction of samples routed to the SDPO branch decreases steadily from approximately 40% to below 20% over the course of training on Chemistry with Qwen3-8B, while the GRPO fraction increases correspondingly. The fraction of samples with constructable teacher information remains high (>85%) throughout, indicating that the routing shift is driven by increasing rollout correctness (fewer incorrect samples to route to SDPO), not by teacher unavailability. This dynamic shift is not an ablation per se but validates the intended adaptive behavior described in Section 3.3: SRPO automatically reduces the influence of self-distillation as the policy improves, without manual scheduling.

Response length as an implicit ablation of the SDPO brevity problem. Figure 4(a) demonstrates that SDPO alone produces extremely short responses (falling below 100 tokens), while SRPO maintains moderate length (150–250 tokens). Although the paper does not ablate response length directly (e.g., by adding a length penalty to SDPO), the length trajectories serve as evidence that SRPO's routing prevents the collapse of response verbosity observed in pure SDPO, which the paper links (via Kim et al., 2026) to degraded reasoning from suppressed epistemic verbalization.

Missing ablations. Several potentially informative ablations are not reported: (1) the sensitivity of SRPO to the dynamic-weighting temperature $\beta$ (only $\beta = 1$ is tested); (2) the effect of different teacher information sources (only correct siblings are used—richer feedback such as execution traces is mentioned as future work in Section 5 but not tested); (3) the sensitivity to the number of rollouts per prompt $G$ (fixed at 8); (4) the effect of the learning rate choice (SRPO uses $5 \times 10^{-6}$, halfway between GRPO and SDPO rates—a grid over this parameter would clarify whether the improvement is robust to learning rate tuning); (5) applying dynamic weighting to the SDPO baseline directly (without routing)—this would isolate whether entropy weighting alone can prevent SDPO collapse, or whether routing is strictly necessary.

Critical Assessment

Claim 1 from the executive summary: SRPO raises the five-benchmark average on Qwen3-8B by 3.4% over GRPO and 6.3% over SDPO.

This claim is directly supported by Table 1 at the 10h budget, where Qwen3-8B achieves 77.4 (SRPO), 74.0 (GRPO), and 71.1 (SDPO). The per-benchmark breakdown shows SRPO outperforms GRPO on all five benchmarks (Chemistry: 83.0 vs. 78.9; Physics: 78.4 vs. 73.6; Biology: 72.8 vs. 70.6; Materials: 81.5 vs. 77.8; Tool Use: 71.2 vs. 69.0) and SDPO on all five (by margins ranging from +4.4 to +14.3). The 10h budget is a fair comparison point because it represents the longest training horizon, where both baselines' trajectories have largely plateaued (SDPO saturates by 5h; GRPO's improvement from 5h to 10h is modest, e.g., 75.9 → 78.9 on Chemistry). The 3.4% and 6.3% figures are simple arithmetic: 77.4 − 74.0 = 3.4, 77.4 − 71.1 = 6.3.

However, several qualifications are warranted. First, the 10h budget represents different numbers of training steps for each method (since per-step time varies—Figure 4(b) shows SRPO at 75.8s/step vs. GRPO at 91.5s/step at 10h, meaning SRPO completes approximately 20% more steps in the same wall-clock time). The paper's choice of wall-clock time rather than step count as the equalization criterion is defensible (practitioners care about training time, not step count) but means part of SRPO's advantage comes from faster per-step computation, not purely from better sample efficiency per gradient step. A step-matched comparison would isolate the algorithmic benefit from the throughput benefit.

Second, the results are reported as "highest achieved avg@16 within each budget" without confidence intervals or standard errors. The five-benchmark average is computed over five tasks with test sets ranging from 50 (Biology) to 210 (Chemistry) examples. A 3.4 percentage point gap in the average could be driven by large gains on one or two benchmarks with smaller improvements on others—and indeed, the gains are uneven: +4.1 on Chemistry, +4.8 on Physics, +2.2 on Biology, +3.7 on Materials, +2.2 on Tool Use. The smallest test set (Biology, 50 examples) shows the smallest gain, which could indicate higher variance in that estimate. Without uncertainty quantification, it is difficult to assess whether the 3.4% average gain is statistically robust.

Third, the baseline implementations may not be equally optimized. The GRPO baseline is described as a "strengthened implementation" with multiple enhancements (asymmetric clipping, unbiased advantage normalization, off-policy correction), while SDPO uses the original configuration from Hübotter et al. (2026). If SDPO's collapse could be mitigated by similar enhancements (e.g., different divergence measure, different EMA schedule, different teacher construction), the 6.3% gap over SDPO might overstate SRPO's advantage over the best possible self-distillation approach. The paper does not ablate improvements to the SDPO baseline itself.

Claim 2: SRPO achieves rapid early improvement (matching SDPO) and long-horizon stability (matching or exceeding GRPO).

The learning curves in Figure 3 provide mixed support for this claim, depending on the benchmark. On Chemistry (Figure 3a), SRPO's early trajectory (0–2h) largely overlaps with SDPO's—both rise rapidly to approximately 0.75 by 2h, ahead of GRPO's approximately 0.70. SRPO then continues improving while SDPO oscillates, reaching approximately 0.82–0.83 vs. SDPO's 0.72–0.80 by 10h. This is a textbook example of the claimed behavior. On Biology (Figure 3b), SRPO actually leads from the start (approximately 0.56 at 1h vs. 0.52 for SDPO), so it exceeds rather than matches SDPO's early performance. On Tool Use (Figure 3c), SRPO's early trajectory is not clearly faster than GRPO's—both start around 0.64 and improve at similar rates—but SRPO avoids the degradation that SDPO suffers. So the claim that SRPO "matches SDPO's early improvement" holds on some benchmarks (Chemistry) but not universally; on others, SRPO either exceeds SDPO early (Biology) or shows no early advantage over GRPO (Tool Use).

The long-horizon stability claim is more uniformly supported. On all three benchmarks in Figure 3, SRPO's 10h accuracy equals or exceeds GRPO's. On Chemistry and Biology, SRPO's late-training variance (shaded bands) is comparable to or lower than GRPO's, indicating stability. On Tool Use, SRPO tracks GRPO closely with similar variance. The claim that SRPO "maintains long-horizon stability" is therefore well-supported: there is no benchmark where SRPO degrades or diverges in later training.

Claim 3: SRPO reduces per-step compute time by up to 17.2% over long training horizons.

This claim is supported by Figure 4(b) at the 10h window: SRPO averages 75.8s/step vs. 91.5s for GRPO (17.2% reduction) and 83.7s for SDPO (9.4% reduction). However, the "up to 17.2%" framing should be contextualized: the reduction is relative to GRPO, which is the slowest method at 10h due to its long response lengths. SRPO's per-step time is actually faster than SDPO's by a more modest 9.4%. And at the 1h window, SRPO is 17.4% slower than GRPO (83.4s vs. 71.0s). The compute-time advantage only materializes later in training when GRPO's response length has grown substantially and fewer SRPO samples require the SDPO branch. For a practitioner with a 1-hour budget, SRPO would be slower, not faster, than GRPO—so the compute-time advantage is conditional on training long enough for the dynamic routing shift to take effect.

Additionally, the per-step time measurement averages over all five benchmarks, but response length dynamics likely differ across benchmarks (Science Q&A may have different verbosity pressures than Tool Use). The paper does not report per-benchmark compute-time breakdowns.

Claim 4: SRPO yields moderate response lengths, avoiding both GRPO's verbosity and SDPO's excessive brevity.

Figure 4(a) supports this claim on Chemistry: GRPO responses fluctuate around 300–500 tokens, SDPO drops to below 100 tokens, and SRPO stabilizes at 150–250 tokens. The "moderate" characterization is accurate relative to the two extremes. However, the paper's linkage of SDPO brevity to "degraded epistemic reasoning" (Kim et al., 2026) is asserted rather than demonstrated—the paper does not analyze whether SRPO's responses actually contain more verbalized reasoning than SDPO's, or whether the length difference correlates with accuracy. The claim that moderate length is beneficial relies on the implicit assumption that SDPO's brevity is harmful (supported by the accuracy degradation) and GRPO's verbosity is wasteful (supported by the higher compute cost), but the optimal response length is not established.

Missing experiments that would strengthen the paper:

• Ablation of $\beta$ (dynamic-weighting temperature). Only $\beta = 1$ is tested. The sensitivity of results to this parameter determines whether dynamic weighting provides a robust benefit or requires careful tuning. A sweep over $\beta \in \{0, 0.5, 1, 2, 5\}$ would clarify this.
• Step-matched comparison. Wall-clock time is a practical metric, but a step-matched comparison would isolate the algorithmic contribution of SRPO from its throughput advantage.
• Application of dynamic weighting to standalone SDPO. This would test whether entropy-based reweighting alone can prevent SDPO collapse without sample routing, clarifying whether routing is strictly necessary or whether reweighting is sufficient.
• Larger-scale experiments. Both tested models are relatively small (4B, 8B). Whether the patterns hold at 70B+ scales—where self-distillation dynamics and verbosity pressures may differ—is unknown.
• Statistical significance testing. Point estimates without error bars or p-values, while consistent with the SDPO baseline paper's reporting, limit the strength of comparative claims.
• Analysis of what SRPO's routing actually does to token-level gradients. The paper claims that routing avoids ambiguity on correct samples and provides targeted correction on incorrect ones, but provides no direct evidence of improved credit assignment (e.g., gradient attribution analysis showing that SRPO's updates better localize to error tokens compared to GRPO).
• Evaluation on a broader task distribution. All five benchmarks are from the same sources (SciKnowEval and ToolAlpaca). Testing on math reasoning (GSM8K, MATH), code generation (HumanEval, MBPP), or general knowledge tasks would establish whether SRPO's benefits generalize beyond scientific Q&A and tool use.

What the experiments genuinely demonstrate vs. what they claim:

The experiments genuinely demonstrate that on five scientific-reasoning and tool-use benchmarks from SciKnowEval and ToolAlpaca, training Qwen3-4B and Qwen3-8B with SRPO achieves higher peak accuracy than training with GRPO or SDPO alone under a fixed wall-clock budget of up to 10 hours on 8×H20 GPUs. This is a narrower claim than "SRPO unifies GRPO and SDPO to achieve the best of both," which implies broader generality. The experiments also convincingly demonstrate the two diagnosed SDPO failure modes (Figure 1(b-c)) and the adaptive routing behavior (Figure 5), establishing that the mechanism works as designed on these benchmarks.

What the experiments do not demonstrate is: (1) that SRPO works on fundamentally different task types (mathematical reasoning, code generation, open-ended generation); (2) that SRPO scales to larger models; (3) that the results are statistically robust (no confidence intervals); (4) that the specific design choices (JS divergence, EMA rate 0.05, top-K=100, $\beta=1$, learning rate $5 \times 10^{-6}$) are optimal rather than accidental artifacts of the experimental configuration; (5) that SRPO would outperform a version of SDPO with its own entropy-based weighting and no routing (this ablation is missing). The paper's claims about general superiority should be tempered by these gaps.

6. Limitations and Trade-offs

6.1 The Routing Mechanism Depends on Verifiable Binary Rewards — No Clear Path to Tasks Without Deterministic Correctness

SRPO's routing rule is built on the correctness flag $c_i = \mathbf{1}[r_i \geq 0.5]$, which requires a verifiable binary reward signal that unambiguously determines whether a rollout is correct. The Science Q&A benchmarks provide this through exact string matching of answer letters to ground truth, and Tool Use provides this through exact API call matching. The paper does not evaluate SRPO on any task that lacks such clean, deterministic correctness signals.

The consequence is that SRPO's central mechanism — distinguishing correct from incorrect rollouts to decide which optimization branch to use — has no straightforward extension to tasks where correctness is ambiguous, multi-dimensional, subjective, or otherwise not cleanly binarized. Examples include open-ended generation (dialogue, creative writing, summarization), multi-step planning where partial correctness matters, or tasks where reward is a learned model rather than a verifiable function. In such settings, the routing condition $c_i$ becomes either unavailable or unreliable. If correctness is determined by a learned reward model (e.g., a preference model in RLHF), routing could amplify reward model errors: a rollout incorrectly classified as correct would be routed to GRPO and reinforced, while one incorrectly classified as incorrect would receive SDPO's token-level correction — potentially correcting parts of the trajectory that were actually fine. The paper does not discuss this failure mode, nor does it propose any mechanism for extending routing to non-binary reward settings.

The evidence that this limitation is consequential is the total absence of such tasks in the evaluation. All five benchmarks — Chemistry, Physics, Biology, Materials, and Tool Use — share the property of having deterministic, verifiable ground-truth answers. Section 8 only briefly gestures toward "environments with richer feedback" as future work without specifying how the routing criterion would adapt when rewards are non-binary, noisy, or learned. The paper's claim that correctness is "an observable property of the rollout" (Section 4, Innovation 2) is valid for the evaluated benchmarks but is not a general property of RLVR tasks. This limitation is not addressed or mitigated — it is a scope constraint that bounds SRPO's applicability to tasks structurally similar to the evaluation suite.

6.2 The Self-Teacher Requires a Correct Sibling Rollout — When All Rollouts Are Incorrect, the SDPO Branch Is Silent

SRPO routes incorrect rollouts to the SDPO branch only when a correct sibling rollout exists in the same group to serve as teacher information ($m_i = 1$). When all $G = 8$ rollouts for a prompt are incorrect, $m_i = 0$ for every rollout, and all fall back to GRPO's uniform penalty — the SDPO branch provides no corrective signal whatsoever. This is a fundamental design constraint: the self-teacher requires privileged context (a correct answer) to produce informative logit-level targets, and when no such context is available, the method degenerates to pure GRPO.

The consequence is most severe for the hardest problems — precisely where dense token-level correction would be most valuable. On problems where the base policy's pass@1 is near zero, most or all rollouts in every group will be incorrect. On such problems, SRPO provides no benefit over GRPO: the SDPO branch never activates, and the policy receives only the coarse, uniform penalty that GRPO applies to incorrect rollouts. This means SRPO's advantage over GRPO is concentrated on problems of moderate difficulty — where at least one rollout per group is usually correct, providing teacher information, but many rollouts are incorrect, providing opportunities for SDPO's targeted correction. On very easy problems (most or all rollouts correct), SRPO also provides limited benefit because there are few incorrect rollouts to route to SDPO (most samples go to GRPO), though here the accuracy is already high so the absolute gain is small.

The paper provides indirect evidence for this limitation via the difficulty-dependent routing statistics. Figure 5 in Appendix C shows that on Chemistry with Qwen3-8B, the fraction of samples with constructable teacher information starts at approximately 90% and stays high. This reflects the fact that Chemistry has moderate base accuracy (41.1% base, rising to ~80%+ after training), so almost every group contains at least one correct rollout. But the paper does not report routing statistics on harder benchmarks or early in training when accuracy is very low. If the base policy had, for example, 5% accuracy on a task, a group of 8 rollouts would have all-incorrect groups $(1-0.05)^8 \approx 66\%$ of the time, meaning the SDPO branch is silent for two-thirds of all prompts — and SRPO collapses to GRPO on those prompts. The paper does not measure SRPO's performance in this regime, so the degradation behavior as difficulty increases (and teacher availability decreases) is uncharacterized.

This limitation is inherent to the design — using correct siblings as the only teacher source — and the paper acknowledges it only indirectly by noting that all-incorrect groups trigger GRPO fallback (Table 7). There is no proposed mitigation (e.g., using an external teacher model, storing past correct solutions across training steps, or using partial teacher information from rollout prefixes), and Section 5's mention of "environments with richer feedback" as future work suggests the authors view this as a fundamental constraint of the current teacher construction approach rather than a solvable sub-problem within the existing framework.

6.3 The Difficulty Estimation Cost Is Explicitly Excluded From the Paper's Compute Budget, Making the Reported Efficiency Gains Potentially Overstated

SRPO requires no explicit difficulty estimation — the routing decision uses correctness and teacher availability, both of which are byproducts of the on-policy sampling already being performed. However, the paper's comparison against baselines uses wall-clock training time as the equalization criterion (Table 1 and Figure 3), and SRPO's per-step compute time advantage (Figure 4(b)) is a significant contributor to its headline performance gains. This creates a subtle but important issue: SRPO's compute-time advantage depends on specific properties of the training dynamics (decreasing SDPO fraction, moderate response length) that are themselves consequences of the routing design, and the paper does not account for the computational cost of the mechanisms that enable this advantage.

Specifically, SRPO's per-step time is lower than GRPO's at 10h (75.8s vs. 91.5s) partly because SRPO generates shorter responses (Figure 4(a)). But the paper does not control for response length in the comparison — it is an emergent property of the training process, not a design parameter. If response length is a consequence of SRPO's routing (the GRPO branch produces long responses, the SDPO branch produces short ones, and routing balances them), then SRPO's compute advantage is not guaranteed to transfer to settings where response length is constrained differently (e.g., with a length penalty, minimum length requirement, or different prompt formatting that changes verbosity pressure). A practitioner who applies a length penalty to GRPO to control verbosity might close the compute gap without changing the optimization method.

More importantly, the paper does not measure how much of SRPO's accuracy advantage comes from the algorithmic improvement (better credit assignment) versus the throughput advantage (more training steps in the same wall-clock time). At 10h, SRPO completes approximately $10 \times 3600 / 75.8 \approx 475$ steps, while GRPO completes approximately $10 \times 3600 / 91.5 \approx 393$ steps — SRPO sees roughly 21% more training data. A step-matched comparison would isolate the algorithmic contribution: if SRPO still outperforms GRPO at equal step counts, the routing and dynamic weighting are genuinely improving sample efficiency; if not, part of the headline gain is simply "SRPO trains faster because it generates shorter responses." The paper does not provide this comparison.

The mitigation status is that this is an unaddressed confound in the experimental design. The paper reports per-step compute time (Figure 4(b)) and acknowledges the throughput difference, but does not disentangle the algorithmic and throughput contributions to the final accuracy gap. The 3.4% improvement over GRPO at 10h (Table 1) should be interpreted with this confound in mind: it is an upper bound on the pure algorithmic gain, and the true algorithmic contribution may be smaller.

6.4 Evaluation Is Confined to Two Small-Scale Qwen3 Models on Five Benchmarks From Two Sources — Generalization to Other Model Families, Scales, and Task Types Is Unestablished

SRPO is evaluated exclusively on Qwen3-4B and Qwen3-8B, and all five benchmarks come from two sources: SciKnowEval (Feng et al., 2024) for the four science Q&A tasks and ToolAlpaca (Tang et al., 2023) for the tool-use task. The benchmarks share structural properties — all have deterministic verifiable rewards, four are four-option multiple-choice, one is structured tool calling — that differ from major RLVR evaluation categories such as mathematical reasoning (GSM8K, MATH), code generation (HumanEval, MBPP, LiveCodeBench), or general chat alignment (AlpacaEval, MT-Bench). The largest model tested (8B parameters) is significantly smaller than the models typically used in production RLVR pipelines (70B+).

The consequence is that the paper's core claims about SRPO's behavior — rapid early improvement, long-horizon stability, moderate response length, lower per-step cost — may not transfer to other settings. Several specific concerns arise. First, the self-teacher's entropy dynamics (Figure 1(c)), which motivate dynamic weighting, may be model-specific: larger models with better calibration might not exhibit the same entropy increase during training, reducing the need for and benefit from dynamic weighting. Second, response length dynamics (Figure 4(a)) are known to be sensitive to prompt formatting, task type, and model family — SDPO's extreme brevity on Chemistry may not occur on math reasoning tasks where step-by-step solutions are longer and more structured. Third, the routing composition (Figure 5) depends on the base policy's accuracy, which varies across tasks and model scales — on tasks where base accuracy is very different from the 40–60% range of the evaluated benchmarks, the SDPO-to-GRPO ratio throughout training would be different, potentially changing SRPO's behavior qualitatively.

The paper provides some evidence for robustness across model scales (Qwen3-4B and Qwen3-8B show similar patterns in Table 1), but this is within a single model family at adjacent scales. The anomalous base-checkpoint ordering (4B outperforming 8B) that the paper notes in the footnote to Table 1 raises additional concerns about whether these specific Qwen3 checkpoints have idiosyncratic properties that affect the results. The paper provides no evidence about robustness to different:

• Model families (Llama, DeepSeek, Mistral)
• Model scales (the largest tested is 8B; 70B+ behavior is unknown)
• Task types (math, code, general reasoning, open-ended chat)
• Prompt formats or system instructions
• Reward sparsity or noisiness levels

This limitation is unmitigated — the paper does not address it beyond acknowledging the model scale in the experimental setup (Section 4.1) and suggesting "extension to environments with richer feedback" as future work (Section 5). The breadth of the evaluation is sufficient to establish feasibility and demonstrate the mechanism on the tested benchmarks, but insufficient to support claims about SRPO as a general post-training framework.

6.5 The Dynamic Weighting Temperature $\beta$ Is Set to 1 Without Any Sensitivity Analysis — the Robustness of This Critical Hyperparameter Is Unknown

The entropy-aware dynamic weighting mechanism uses a temperature parameter $\beta$ (default value 1) that controls how sharply weights decay with teacher entropy: $\tilde{w}_{i,t} = \exp(-\beta H_{i,t})$. With $\beta = 0$, dynamic weighting is disabled (all weights are 1, equivalent to SRPO without dynamic weighting). With $\beta \to \infty$, only the single lowest-entropy token receives non-zero weight (hard thresholding). The paper tests only $\beta = 1$ and reports no sensitivity analysis over this parameter.

The consequence is that a practitioner implementing SRPO on a new task or model has no guidance on how to set $\beta$ , and the reported gains from dynamic weighting (+1.8 at 10h, Table 2) may not generalize to settings with different teacher entropy distributions. If the teacher's entropy on a new task is systematically higher or lower than on the evaluated benchmarks, $\beta = 1$ might either over-suppress (throwing away useful, moderately-entropy corrections) or under-suppress (failing to filter noisy high-entropy targets). A task with very long responses (e.g., math proofs) might have different per-token entropy distributions than four-option science Q&A; a larger model with better calibration might have lower entropy overall, changing the effective sensitivity to $\beta$.

The ablation in Table 2 shows that dynamic weighting provides a growing benefit over time (+0.4 at 1h, +0.7 at 5h, +1.8 at 10h), indicating it is genuinely helpful — but at only one value of $\beta$. This single-datapoint evaluation means we cannot distinguish between two scenarios:

• Scenario A: Dynamic weighting is robust — any $\beta$ in a reasonable range (e.g., 0.5–2) provides similar benefits, and $\beta = 1$ is a safe default.
• Scenario B: Dynamic weighting is sensitive — $\beta = 1$ is near-optimal for these specific benchmarks and models, but different settings require careful tuning.

The evidence does not distinguish between these scenarios. The ablation compares SRPO (with $\beta = 1$) to SRPO without dynamic weighting (implicitly $\beta = 0$), which only tells us that the mechanism helps at this specific temperature setting. A sweep over $\beta$ would also reveal whether SRPO without dynamic weighting already captures most of the benefit (suggesting routing is the primary driver and weighting is a small additive improvement) or whether the gap grows substantially at larger $\beta$ values (suggesting aggressive entropy filtering provides even larger gains).

This limitation is unaddressed — the paper does not discuss the sensitivity of results to $\beta$, does not provide guidance for selecting it on new tasks, and does not report experiments varying this parameter. The hyperparameter table (Table 3) lists $\beta = 1$ as the only value for the dynamic weighting temperature. This is a practical gap for reproducibility and deployment.

6.6 The Test Sets Are Small (50–210 Examples) and Reported Without Confidence Intervals — Statistical Reliability of the Comparisons Is Uncertain

The five benchmarks have modest test set sizes: Chemistry (210 examples), Physics (80), Biology (50), Materials (94), and Tool Use (68). The five-benchmark average is computed over a total of 502 test examples, but individual benchmark comparisons rely on as few as 50 examples (Biology). The paper reports peak avg@16 accuracy within each wall-clock budget as point estimates without any measure of uncertainty — no confidence intervals, standard errors, or statistical tests are provided for any result in Table 1, Figure 3, or Figure 4.

The consequence is that the reliability of pairwise comparisons between methods is unclear, particularly on small benchmarks. On Biology (50 test examples, avg@16 is over 16 × 50 = 800 individual rollouts), the 10h comparison is SRPO 72.8 vs. GRPO 70.6 vs. SDPO 58.5. The gap between SRPO and GRPO (2.2 percentage points) on 50 test questions corresponds to approximately 1.1 additional correct questions on average — a difference that could plausibly fall within sampling error. Without confidence intervals, we cannot assess whether SRPO vs. GRPO on Biology is a reliable difference or noise. The larger gaps (e.g., SRPO 83.0 vs. GRPO 78.9 on Chemistry, 210 test examples) are more likely robust, but the five-benchmark average (which weights all benchmarks equally regardless of test set size) is influenced by potentially unreliable estimates on small benchmarks.

This limitation is particularly relevant for the paper's headline claim of a 3.4% average improvement over GRPO at 10h on Qwen3-8B. If the improvement is concentrated on benchmarks with larger test sets (Chemistry: +4.1, Materials: +3.7, Physics: +4.8) and smaller or negligible on benchmarks with smaller test sets (Biology: +2.2, Tool Use: +2.2), the average is driven by the more reliable estimates — but the paper does not provide the analysis to confirm or refute this pattern.

The reporting protocol is inherited from the SDPO baseline paper (Hübotter et al., 2026), which also reports point estimates without confidence intervals, but this does not mitigate the limitation — it means the entire line of comparison inherits the same statistical weakness. The paper does not discuss this limitation, does not provide uncertainty quantification, and does not perform any statistical testing. This is an unaddressed gap in the experimental methodology.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a routing-based reframing of how to combine supervision signals in RLVR, shifting the problem from "how should we blend GRPO and SDPO?" (a continuous mixing problem) to "which signal is appropriate for which sample?" (a discrete routing decision based on an observable property). This is not a paradigm shift—GRPO and SDPO remain the workhorse algorithms—but it is more than an incremental refinement. It resolves a specific, previously undiagnosed tension in the literature: why self-distillation methods sometimes accelerate training and sometimes cause catastrophic collapse, depending on the task and training duration. By identifying two intrinsic failure modes of SDPO (optimization ambiguity on correct samples, degrading teacher signal quality) and showing that they can be circumvented by conditioning the distillation update on rollout correctness, the paper provides a diagnostic framework that explains when self-distillation helps and when it hurts.

This reframing has several downstream effects on the research landscape:

It redirects attention from mixing strategies to routing criteria. Prior to this work, the natural approach to combining GRPO and SDPO was advantage-level blending (as in the Advantage Mix baseline). The paper's ablation (Table 2) demonstrates that routing (correctness-conditional branch selection) substantially outperforms blending (fixed-coefficient mixing) over long horizons, with a growing gap (−3.3 at 10h for blending vs. sustained improvement for routing). This suggests that future work on multi-signal optimization should prioritize discovering routing criteria—observable properties of samples that indicate which signal is appropriate—over developing more sophisticated blending or scheduling strategies. The key question shifts from "what mixing weight should we use?" to "what sample property predicts which signal will be most beneficial?"

It reconciles conflicting evidence about self-distillation effectiveness. Prior work presented contradictory findings: SDPO (Hübotter et al., 2026) showed rapid early improvement on scientific reasoning and tool use, while concurrent work (Kim et al., 2026) showed self-distillation degrading reasoning capability in math domains. The paper's diagnostic framework suggests a unifying explanation: self-distillation is beneficial primarily on failed rollouts (where it provides genuine corrective signal) and harmful primarily on successful ones (where it imposes arbitrary preferences between reward-equivalent paths). The net effect on a given benchmark depends on the proportion of correct vs. incorrect rollouts during training, which in turn depends on the base policy's accuracy on that benchmark and the training duration. Benchmarks where the policy quickly achieves high accuracy would see self-distillation's harmful effects dominate earlier, explaining the degradation Kim et al. observed. This reconciliation converts a confusing contradiction into a coherent picture with clear boundary conditions.

It establishes the self-teacher's entropy as a practical signal-quality metric. Figure 1(c) demonstrates that the self-teacher's token-level entropy rises during training, indicating degrading signal quality—a phenomenon that, to the paper's credit, is measured rather than merely hypothesized. The dynamic weighting mechanism operationalizes this insight by making the SDPO branch self-limiting: as teacher entropy rises, uncertain predictions are automatically downweighted, creating a negative feedback loop that prevents noise accumulation. This makes entropy tracking a concrete diagnostic tool that future work can use to monitor self-distillation health during training, rather than relying on post-hoc accuracy curves to detect collapse after it has occurred.

It makes verifier-free, reward-only RLVR pipelines more attractive relative to process supervision approaches. A growing body of work pursues process reward models (PRMs) and step-level supervision (Lightman et al., 2023; Setlur et al., 2025; Cui et al., 2025) to provide denser credit assignment than outcome rewards. SRPO demonstrates that comparable or better densification can be achieved through on-policy self-distillation on failed rollouts only—without training a separate reward model or collecting step-level human labels. This does not eliminate the value of PRMs (which provide supervision even when all rollouts are incorrect, a regime where SRPO's SDPO branch falls silent), but it narrows the gap and provides a lower-cost alternative for the moderate-difficulty regime where PRM training data is hardest to collect.

It de-emphasizes sophisticated RL algorithms for post-training in favor of credit assignment improvements. The paper's GRPO baseline is already a strengthened implementation with multiple enhancements (asymmetric clipping, unbiased normalization, off-policy correction), yet SRPO's routing and dynamic weighting provide an additional 3.4% average improvement without changing the underlying policy-gradient algorithm. This suggests that improvements in how the supervision signal is constructed and applied to individual samples may yield larger gains than further refinements to the RL optimizer itself—a finding that parallels the observation in pretraining that data quality and allocation often matter more than architectural innovations.

Follow-Up Research This Work Enables

1. Applying entropy-aware dynamic weighting to standalone SDPO to isolate whether routing or reweighting is the primary driver of collapse prevention. The paper demonstrates that SRPO (routing + reweighting) outperforms both SRPO without reweighting and Advantage Mix, but never tests whether dynamic weighting applied to vanilla SDPO (without routing) prevents collapse. This is a critical ablation: if entropy-weighted SDPO alone avoids collapse, then the routing mechanism may be unnecessary—the entire benefit would come from suppressing noisy teacher targets. Conversely, if entropy-weighted SDPO still collapses, then routing (excluding correct samples from distillation) is the essential mechanism. A concrete experiment would train SDPO + DW (same β=1, same teacher construction, same divergence, but applied to all rollouts regardless of correctness) on the same five benchmarks with Qwen3-8B, plotting learning curves against vanilla SDPO and SRPO. The key measurement is whether SDPO + DW matches SRPO's 10h accuracy or plateaus like SDPO. This experiment would resolve whether Failure Mode 1 (correct-sample ambiguity) is independently harmful or merely a consequence of high-entropy teacher targets on correct samples.

2. Stress-testing SRPO on tasks where base policy accuracy is very low (≤10%), to map the degradation behavior when the SDPO branch is frequently silent. The paper's routing statistics (Figure 5) show that on Chemistry with 41.1% base accuracy, teacher information is constructable for >85% of samples throughout training. But on tasks where the base policy rarely produces correct rollouts, all-incorrect groups become common: at 5% base accuracy, approximately 66% of groups of 8 have no correct sibling. In this regime, SRPO degenerates to pure GRPO on two-thirds of prompts, providing no token-level correction where it is arguably most needed. A concrete experiment would evaluate SRPO on a task with deliberately low base accuracy—for example, MATH (Hendrycks et al., 2021) with the base Qwen3-8B instruct model, which is unlikely to have high accuracy on competition-level math without specialized math training—and compare against both GRPO and a version of SRPO augmented with an external teacher (e.g., a stronger model's solutions) that can provide teacher information even when no correct sibling exists. The key measurement is whether SRPO's advantage over GRPO shrinks or vanishes as base accuracy decreases, characterizing the difficulty horizon beyond which SRPO provides no benefit.

3. Investigating the interaction between SRPO and response length, with controlled experiments that disentangle algorithmic improvement from the length-driven throughput advantage. At 10h, SRPO completes approximately 21% more training steps than GRPO (475 vs. 393 steps on Chemistry, based on per-step times in Figure 4(b)) while generating substantially shorter responses (150–250 vs. 300–500 tokens, Figure 4(a)). The paper's 3.4% accuracy advantage over GRPO conflates the algorithmic benefit (better credit assignment) with the throughput benefit (more training steps in the same wall-clock time). A step-matched comparison—training GRPO and SRPO for the same number of gradient steps, ignoring wall-clock time—would isolate the algorithmic contribution. Additionally, training GRPO with an explicit length penalty or maximum response length constraint that brings GRPO's response lengths down to SRPO's range would test whether GRPO's longer responses are wasteful (generating tokens that don't improve accuracy) or genuinely useful (providing more reasoning that helps). A concrete experiment would run: (a) SRPO vs. GRPO at equal step counts, (b) GRPO with a length bonus tuned to match SRPO's average response length vs. SRPO, and (c) SRPO with the length penalty removed vs. unconstrained GRPO. The key measurements are accuracy at matched steps and accuracy at matched response lengths, which would reveal how much of SRPO's headline gain is algorithmic vs. throughput-driven vs. length-driven.

4. Sensitivity analysis of the dynamic-weighting temperature β across a range of values and task types, to establish whether β=1 is a safe default or requires per-task tuning. The paper tests only β=1 and reports that dynamic weighting provides a 1.8 percentage point improvement at 10h (Table 2). But the self-teacher's entropy distribution is likely task-dependent—tasks with longer, more diverse responses will have systematically higher per-token entropy than four-option multiple-choice questions, changing the effective sensitivity to β. A concrete experiment would sweep β ∈ {0, 0.1, 0.5, 1, 2, 5, 10} on at least two benchmarks with different response-length characteristics (e.g., Chemistry with short multiple-choice answers vs. a code generation task with longer structured outputs) and measure both final accuracy and training stability (variance of the learning curve). The key finding would be whether the accuracy-vs-β curve is flat in a neighborhood of β=1 (suggesting robustness and a safe default) or sharply peaked (suggesting careful tuning is necessary, which would complicate deployment). An additional measurement would be the effective number of tokens receiving weight above some threshold (e.g., 0.5) as β varies, to characterize how aggressively dynamic weighting filters teacher predictions.

5. Extending SRPO to tasks with learned or non-binary reward signals, replacing the hard correctness threshold with a soft routing criterion. SRPO's routing rule requires a binary correctness flag ci = 1[ri ≥ 0.5], which is only available when the environment provides deterministic verifiable rewards. Many important RLVR applications use learned reward models (RLHF with preference models), partial-credit rewards (e.g., code generation with test-case pass rates), or multi-dimensional rewards. A natural extension would replace the hard routing rule with a soft, reward-gated mechanism: route samples to SDPO with probability proportional to 1 − r_i (the probability of being incorrect) and to GRPO otherwise, or use the reward magnitude directly as a mixing weight between the two branch losses for each sample. A concrete experiment would test this on a code generation benchmark (e.g., MBPP or HumanEval) where the reward is the fraction of test cases passed (a continuous value in [0, 1]), comparing: (a) hard routing with threshold 0.5 (rollouts passing >50% of tests are "correct"), (b) soft routing where each rollout's loss is a reward-weighted mixture of GRPO and SDPO losses, (c) pure GRPO, and (d) pure SDPO. The key measurement is whether soft routing outperforms hard routing when rewards are continuous, establishing whether the binary routing assumption is a necessary condition or merely a convenient simplification for the benchmarks studied.

6. Replicating the diagnostic analysis (correctness-conditional ablation, entropy tracking) on larger models and different model families to establish the generality of the two diagnosed failure modes. The paper's central diagnostic claim—that SDPO fails due to correct-sample ambiguity and degrading teacher signal quality—is supported on Qwen3-8B on the evaluated benchmarks. But the mechanisms may be model-specific: larger models with better calibration might not exhibit the entropy increase shown in Figure 1(c), and the optimization ambiguity on correct samples might be less severe if the model's correct trajectories are more consistent (reducing the divergence between different correct siblings). A concrete experiment would replicate Figure 1(b) and 1(c) on a 70B-class model (e.g., Llama-3-70B-Instruct) on the same Chemistry benchmark, measuring: (a) whether SDPO still collapses, (b) whether restricting SDPO to incorrect samples still preserves most of the benefit, and (c) whether teacher entropy still rises during training. The key finding would be whether the paper's diagnostic framework generalizes beyond the specific model scale tested, or whether larger models exhibit qualitatively different self-distillation dynamics that would require different mitigation strategies. This experiment is important because the paper's recommendations for deploying SRPO depend on the assumption that the failure modes it diagnoses are universal properties of self-distillation, not artifacts of the 4B–8B parameter scale.

Practical Applications and Downstream Use Cases

Deploying RLVR post-training pipelines for domain-specific reasoning models with limited compute budgets. The paper demonstrates that on the five-benchmark average with Qwen3-8B, SRPO achieves higher accuracy at every wall-clock budget (1h, 5h, 10h) than either GRPO or SDPO alone, with the gap growing over time (Table 1). For an organization fine-tuning a small-to-medium LLM (4B–8B parameters) on a domain-specific reasoning task—scientific Q&A, technical support, regulatory compliance checking, or any setting with verifiable ground-truth answers—SRPO provides a drop-in replacement for GRPO that yields higher final accuracy (3.4% average improvement at 10h) while reducing long-horizon compute cost (17.2% lower per-step time at 10h, Figure 4(b)). The practical recipe is straightforward: take the existing GRPO training loop, add the SDPO branch with entropy-aware weighting, implement the correctness-conditional routing rule, and adjust the learning rate to halfway between the GRPO and SDPO defaults. The paper's hyperparameter configuration (Table 3) provides concrete defaults that worked on the evaluated benchmarks, reducing the need for extensive tuning on similar tasks.

Mitigating verbosity explosion in GRPO-trained models without explicit length penalties. GRPO's training dynamics produce consistently long responses (300–500 tokens on Chemistry, Figure 4(a)) that inflate inference cost without necessarily improving accuracy. The paper shows that SRPO naturally produces moderate-length responses (150–250 tokens) by routing a fraction of samples through the SDPO branch, which exerts brevity pressure (SDPO alone produces responses below 100 tokens), while the GRPO branch anchors correct rollouts and prevents the extreme brevity and degraded reasoning associated with pure SDPO. For deployments where per-query inference latency or cost is a concern—such as real-time chatbots, on-device inference, or high-throughput API services—SRPO provides a training-time mechanism to control response length without adding explicit length penalties, reward shaping, or post-training compression. The 17.2% per-step compute reduction at 10h (Figure 4(b)) is partly attributable to these shorter responses, meaning the length benefit compounds: training is faster and the resulting model is cheaper to serve.

Training on scientific or technical benchmarks where base model accuracy is moderate (30–60%) but not high, making token-level error correction most impactful. The paper's routing statistics (Figure 5) show that on Chemistry with 41.1% base accuracy, approximately 40% of rollouts are routed to SDPO early in training, and the SDPO fraction remains above 20% even in later training—the regime where SDPO's targeted correction is most active. This suggests SRPO is most beneficial on tasks where the base policy is neither completely incompetent (where no correct rollouts exist to serve as teachers) nor highly proficient (where most rollouts are correct and SDPO has little to correct). Concretely, for a benchmark like Chemistry with mid-range base accuracy, SRPO's 10h accuracy of 83.0 represents a +4.1 improvement over GRPO's 78.9 (Table 1)—a 5.2% relative error reduction on the 21.1% error rate. For an organization deploying a model on a technical domain where base accuracy is in this 30–60% sweet spot, SRPO offers the largest proportional gains. Outside this range (very low base accuracy where teachers are unavailable, or very high base accuracy where few rollouts need correction), the advantage over GRPO would likely shrink, and the paper's results on Biology (30.5% base accuracy, +2.2 gain) and Materials (59.3%, +3.7 gain) are consistent with this pattern.

When to Prefer This Method

The paper explicitly positions SRPO against GRPO and SDPO as alternatives, with clear conditions under which each is preferable. Based on the empirical results:

• Prefer SRPO over standalone GRPO when: the base policy has moderate accuracy on the target task (30–60% range, ensuring teacher availability while leaving many rollouts to correct), training will continue for more than a few hours (the compute-time advantage only materializes after ~5h, Figure 4(b)), and response verbosity is a concern (SRPO produces naturally shorter responses, Figure 4(a)). The 3.4% average accuracy gain at 10h on Qwen3-8B and the 4.5% gain on Qwen3-4B (Table 1) represent the expected benefit in this regime.

• Prefer SRPO over standalone SDPO when: training will extend beyond the point where SDPO saturates (SDPO's 5h and 10h averages are identical on all benchmarks, Table 1), the task requires sustained improvement over long horizons, or the benchmark exhibits SDPO collapse (as on Tool Use, Figure 3(c), where SDPO degrades from ~0.66 to ~0.62 while SRPO reaches 0.71). The +6.3% average gain at 10h on Qwen3-8B represents the expected benefit of avoiding SDPO's late-stage instability.

• Prefer standalone SDPO over SRPO when: the training budget is very short (≤1 hour) AND the benchmark is one where SDPO shows strong early advantage (e.g., Chemistry, where SDPO leads SRPO 71.6 vs. 69.2 at 1h, Table 1). In this narrow regime, SDPO's slightly faster early convergence may outweigh the benefits of routing and dynamic weighting.

• Prefer GRPO over SRPO when: the base policy's accuracy is very low (<10–15%), making teacher information frequently unavailable and causing SRPO to degenerate to GRPO on most prompts while still incurring the implementation complexity of maintaining both branches. In this regime, SRPO provides little benefit over GRPO but adds engineering overhead. However, the paper does not evaluate SRPO in this regime, so this recommendation is extrapolated from the mechanism rather than empirically grounded.

• Prefer neither SRPO nor SDPO when: the task lacks verifiable binary rewards—SRPO's routing rule depends on a correctness flag that requires deterministic outcome evaluation, and the paper provides no mechanism for extending the routing criterion to learned, noisy, or continuous rewards. For RLHF with preference models or tasks with partial-credit scoring, GRPO (or another reward-only method) remains the safer choice until SRPO is extended to soft routing.