ArXiv: 2605.10781

🎯 Pitch

This paper reveals a surprising failure mode in self-distillation for reasoning: when a student already succeeded, pulling it toward the teacher actively erases its original reasoning path. It proposes simply reversing that signal—reinforcing tokens where the student diverged from the teacher on correct rollouts—turning self-distillation into a powerful exploration mechanism that boosts Qwen3 models by up to 18% on math benchmarks.

1. Executive Summary

This paper proposes RLRT (RLVR with Reversed Teacher), a new algorithm that inverts the standard self-distillation signal in reinforcement learning with verifiable rewards — rather than pulling the student toward a teacher conditioned on privileged context, RLRT amplifies tokens where the student diverged from the teacher on correct rollouts, treating this teacher–student gap as a signal of self-driven reasoning (exploration tokens like "wait," "another," "consider" rather than exploit tokens like "conclude," "final," "correct"). Across Qwen3-4B/8B-Base, Qwen3-4B-Instruct, and Qwen3-8B models trained on DAPO-Math-17k and evaluated on six math benchmarks (AIME24/25/26, HMMT26, AMC23, MATH500), RLRT yields an average gain of 8.9% over self-distillation baselines, with gains as large as 18.0% on Qwen3-4B-Base and consistent pass@k dominance over exploration-based methods like entropy bonuses and DIVER. Through causal reflection-injection experiments and distribution-shift analysis, the paper establishes that information asymmetry between teacher and student views serves as a principled exploration signal — but only when gated on correct rollouts and applied in reverse, since removing the correctness gate causes training to collapse via unbounded entropy growth.

2. Context and Motivation

The Core Problem: RLVR's Sparse Reward and the Collapse of Reasoning Diversity

The central tension this paper addresses is both practical and theoretical: reinforcement learning with verifiable rewards (RLVR) has become the dominant post-training paradigm for reasoning LLMs, yet it operates with only a single scalar correctness signal at the end of each trajectory. This credit-assignment bottleneck means the model receives no guidance about which tokens or reasoning steps led to success or failure — only that the final answer was right or wrong. The consequence, well-documented in recent work the paper cites (Yue et al., 2026; Nguyen et al., 2025), is reasoning boundary collapse: the policy concentrates probability mass on a narrow set of high-reward strategies it already prefers, rather than genuinely expanding its reasoning repertoire.

This is not a minor efficiency concern. When an RLVR-trained model collapses to its base model's preferred reasoning paths, the reinforcement learning process effectively becomes an expensive way to sharpen existing behaviors rather than discover new ones. The model may improve on problems it could already solve, but fails to develop the novel reasoning strategies that post-training is supposed to unlock. The paper's Figure 1(b) provides concrete evidence of this phenomenon: across four model backbones, standard GRPO provides only modest gains over the base model (e.g., 33.3% → 40.0% pass@16 on AIME24 for Qwen3-4B-Base), leaving substantial room for improvement that existing methods do not capture.

Self-Distillation as a Response — and Its Hidden Failure Mode

Self-distillation has emerged as a powerful response to the credit-assignment bottleneck. The unifying mechanism across recent work (Zhao et al., 2026; Hübotter et al., 2026; Yang et al., 2026; Li et al., 2026) is information asymmetry: the same model serves as both teacher and student, but the teacher is conditioned on privileged context hidden from the student — a successful rollout, ground-truth solution, textual feedback, or corrective critique. By distilling the teacher's token-level predictions into the student, these methods convert the sparse scalar reward into dense supervision.

The paper acknowledges this lineage as productive but identifies a critical blind spot. In all prior self-distillation work, the teacher–student gap is treated as an alignment target: the student should move toward the teacher, regardless of whether the student's own trajectory was already successful. This design choice masks a subtle failure mode that the paper brings into focus:

"On already-successful trajectories, the same mechanism inverts its role. Even when the student already reached the correct answer, distilling toward the teacher overwrites the student's choices with the teacher's, a problem recently identified as optimization ambiguity in self-distillation [12]. Rather than being corrected, the student is forced to imitate a path it had already solved its own way, undermining the independent reasoning that produced the success."

In other words, self-distillation on correct rollouts is a form of self-suppression. When the student arrives at the correct answer through its own reasoning, the teacher's distribution — conditioned on privileged context the student did not have — will differ from the student's choices at precisely the tokens where the student exercised independent judgment. Standard self-distillation treats these differences as errors to be corrected, pulling the student toward the teacher's preferred path and away from the self-driven reasoning that actually succeeded. The paper's key insight is that this suppression is not a bug to be tolerated for the sake of correcting failures, but a design flaw that can be inverted to positive effect.

Where Prior Approaches Fall Short

The paper positions RLRT against two distinct lines of prior work, each of which addresses a different aspect of the problem but leaves a critical gap.

Self-distillation methods conflate correction with suppression. RLSD (Yang et al., 2026) uses the teacher–student probability ratio as a magnitude-only credit signal, up-weighting teacher-favored tokens on correct rollouts and down-weighting them on incorrect ones. SDPO (Hübotter et al., 2026) applies a logit-level KL loss pulling the student toward the teacher on all rollouts regardless of correctness. SRPO (Li et al., 2026) recognizes the optimization ambiguity problem but resolves it by routing: SDPO-style distillation on failed rollouts, standard GRPO on correct ones. While SRPO's routing is a step in the right direction, the paper finds it performs even worse than full self-distillation on math reasoning (Figure 5, Qwen3-8B panel), suggesting that simply turning off distillation on correct rollouts loses useful signal. The paper conjectures that "self-distillation and GRPO promote different reasoning styles, leading to conflicting gradients" (Section 6.1).

The deeper issue is that none of these methods recognize the teacher–student gap on correct rollouts as evidence of the student's reasoning capability. They either suppress it (RLSD, SDPO), ignore it (SRPO), or treat it as noise. RLRT's contribution is to read this same signal in the opposite direction: tokens where PS>PTP_S > P_T on a correct rollout are precisely where the student exercised self-driven reasoning that succeeded against the teacher's implicit prediction.

Exploration methods pursue diversity without direction. A separate line of work addresses reasoning collapse by injecting diversity into the policy — token-level entropy bonuses (Cheng et al., 2026; Cui et al., 2025), sequence-level diversity objectives (Hu et al., 2025; Wan et al., 2026), or outcome-based exploration (Song et al., 2025). These methods broaden the sampling distribution in the hope that wider exploration surfaces correct paths the model might otherwise miss.

The paper's critique of this approach is nuanced but fundamental. Diversity-for-its-own-sake pushes the model to try different things, but not necessarily better things. It treats all deviation from the current policy as equally valuable, relying on the sparse RL signal to retrospectively sort useful exploration from noise. The paper argues this is inefficient:

"Rather than encouraging diversity for its own sake, we identify, within the rollouts the model has already produced, tokens that are simultaneously self-driven (departing from the conditioned teacher) and verified (occurring on correct trajectories), and upweight them during training."

This distinction — what the paper terms valuable exploration — is the conceptual anchor of RLRT. The exploration signal is not uniform perturbation but targeted amplification of reasoning choices the model already made and were already verified as leading to success, yet which depart from what a conditioned teacher would have done. This is diversity grounded in demonstrated competence rather than stochastic noise.

How This Paper Positions Itself

The paper's positioning rests on a single conceptual move: inverting the reading of the teacher–student gap on correct rollouts. Rather than interpreting PS>PTP_S > P_T as error to be corrected, RLRT interprets it as self-driven reasoning to be strengthened. This inversion is not merely a sign flip — it redefines what self-distillation is for. Instead of a mechanism for transmitting privileged knowledge from teacher to student, it becomes a mechanism for the student to recognize and reinforce its own successful departures from the teacher's expectations.

The paper formalizes this through the lens of information asymmetry (Section 4). The token-level log-ratio D^t=log⁡PS(yt)−log⁡PT(yt)\hat{D}_t = \log P_S(y_t) - \log P_T(y_t) and its position-level expectation Dˉt=KL(PS∥PT)\bar{D}_t = \text{KL}(P_S \| P_T) provide a theoretically grounded decomposition: Dˉt\bar{D}_t identifies where token choice matters causally for correctness (Theorem 2 proves this via a Pinsker-type bound), while the sign of D^t\hat{D}_t marks which direction to push — toward the teacher (exploit) for D^t<0\hat{D}_t < 0, away from the teacher (explore) for D^t>0\hat{D}_t > 0. Figure 3 provides linguistic validation: explore-leaning tokens are reasoning-openers ("wait," "another," "consider"), while exploit-leaning tokens are reasoning-closers ("conclude," "final," "correct").

The paper positions RLRT as the first method to operationalize this reversed reading. Algorithmically, it is a minimal modification to GRPO: the per-token advantage weight becomes wt=(PS/PT)sign(A)w_t = (P_S / P_T)^{\text{sign}(A)}, gated to apply only on correct rollouts (Algorithm 1). This simplicity is deliberate — it isolates the effect of the reversed signal from confounding factors. The theoretical apparatus (Lemma 1, Theorem 2) provides justification for why this signal works, while the ablation studies (Figure 9) demonstrate that both the reversal and the correctness gate are necessary: removing the gate causes training to collapse via unbounded entropy growth (Section 6.5), confirming that self-driven tokens on failed rollouts are not valuable exploration but spurious divergence.

The broader claim the paper makes — reflected in its third listed contribution — is that information asymmetry is a principled, intrinsic source of exploration in RLVR, establishing a new design axis beyond uniform diversity injection. This positions RLRT not just as a specific algorithm but as an instance of a more general framework that future work can extend through different teacher configurations, different forms of privileged context, and hybrid routing between teacher-guided and self-driven updates.

3. Technical Approach

3.1 Reader Orientation

RLRT is a training algorithm — a modified version of GRPO — that changes how a language model receives credit for individual tokens during reinforcement learning with verifiable rewards. The core problem it solves is that standard self-distillation methods suppress a model's own successful reasoning patterns by pulling them toward a teacher's preferred tokens, even on rollouts that already reached the correct answer; RLRT inverts this by amplifying tokens where the student diverged from the teacher on correct trajectories, treating that divergence as evidence of self-driven reasoning worth strengthening.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four main components, all operating within a single model that serves in two roles:

  1. Student Policy ($\pi_\theta$): The model being trained, which generates rollouts conditioned only on the prompt and prefix history.

  2. Teacher Policy (same parameters $\theta$, but different conditioning): The same model run with privileged context — specifically, a known correct solution — appended to the input. This creates an information asymmetry: the teacher sees what a successful trajectory looks like, while the student does not.

  3. Token-Level Information Asymmetry Signal ($\hat{D}_t$, $\bar{D}_t$): A per-token computation that compares the student's and teacher's next-token probability distributions, producing a signed quantity that reveals where and in which direction the student departed from the teacher.

  4. RLRT Advantage Reweighting (Algorithm 1): A wrapper around GRPO that multiplies each token's advantage by a weight derived from $\hat{D}_t$, gated to apply only when the rollout was correct ($r=1$), and clipped to prevent unbounded updates.

Information flows as follows: a prompt enters → the student generates $K$ rollouts → each rollout receives a verifiable reward and a GRPO group-standardized advantage → for correct rollouts only, each token's advantage is multiplied by a reverse weight $w_t = (P_S/P_T)^{\text{sign}(A)}$ → the standard GRPO surrogate loss is computed with these modified advantages → the model updates. The teacher is used only to compute $P_T$ for the reverse weight; it is never updated toward.

3.3 Roadmap for the Deep Dive

  • First, the unified GRPO surrogate with per-token weighting, which establishes the interface that all methods (GRPO, RLSD, RLRT) share and where RLRT makes its modification.
  • Second, the information asymmetry quantities $\hat{D}_t$ and $\bar{D}_t$, since they are the mathematical objects that RLRT operationalizes — what they measure, how they are computed, and why $\bar{D}_t$ identifies causally critical positions (Theorem 2).
  • Third, the reverse weight $w_t$ construction, which shows exactly how the sign of $\hat{D}_t$ and the GRPO advantage interact to produce directional, reward-gated credit assignment.
  • Fourth, the complete RLRT algorithm (Algorithm 1), which integrates the reverse weight into the GRPO loop with the critical correctness gate and clipping mechanism.
  • Fifth, the conceptual decomposition of teacher–student divergence into explore and exploit directions (Section 4.3, Figure 3), which validates that the signal RLRT amplifies actually corresponds to reasoning-openers rather than reasoning-closers.
  • Sixth, the theoretical justification (Section 4.2, Lemma 1, Theorem 2), which connects $\bar{D}_t$ to causal influence on correctness and establishes why the information asymmetry signal is grounded rather than heuristic.
  • Seventh, the design choices and their justifications, comparing RLRT against the alternatives it deliberately differs from.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a method paper whose core idea is that the teacher–student gap in self-distillation can be read in reverse — as a signal of the student's self-driven reasoning on correct rollouts — and that amplifying those tokens rather than suppressing them yields systematic improvements in RLVR training.

The Shared GRPO Surrogate with Per-Token Weighting

All methods in the paper — GRPO, RLSD, SDPO, SRPO, and RLRT — share the same outer loop: a group of rollouts is sampled, rewards are computed, and group-standardized advantages are assigned. The methods differ only in how they weight each token's advantage contribution to the policy gradient. The paper formalizes this common interface in Appendix G.1, Table 4, which is essential for understanding where RLRT fits.

For a prompt $x$ and a group of $K$ rollouts $\{y^{(k)}\}_{k=1}^K$, each rollout $y^{(k)} = (y_1^{(k)}, \ldots, y_T^{(k)})$ receives a verifiable reward $r(y^{(k)}) \in \{0, 1\}$. Following the DAPO recipe (Yu et al., 2025), the group-standardized advantage for rollout $k$ is:

A(k)=r(y(k))−μgroupσgroupA^{(k)} = \frac{r(y^{(k)}) - \mu_{\text{group}}}{\sigma_{\text{group}}}

where $\mu_{\text{group}}$ is the mean reward across the $K$ rollouts and $\sigma_{\text{group}}$ is the standard deviation (though the paper notes in Table 5 that for GRPO and SRPO, standard deviation normalization is disabled following Liu et al., 2025, while for RLSD and RLRT it is enabled).

What this computes: a signed scalar per rollout indicating how much better or worse that rollout's reward is compared to the average in its group. Positive advantages go to above-average rollouts (typically correct ones when most rollouts are incorrect), negative advantages to below-average ones.

Why this form: group-relative advantages remove the need for a learned value function (as in PPO) by treating the sampling batch as the reference distribution. This is the standard GRPO design from Shao et al. (2024).

Each method then defines a per-token weight $w_t$ that multiplies $A^{(k)}$ at each position $t$, producing the token-level advantage:

At(k)=A(k)⋅[(1−λ)+λ⋅clip(wt,1−εw,1+εw)]A^{(k)}_t = A^{(k)} \cdot \left[(1 - \lambda) + \lambda \cdot \text{clip}\left(w_t, 1 - \varepsilon_w, 1 + \varepsilon_w\right)\right]

where $\lambda \in [0, 1]$ is a mixing coefficient controlling the strength of the per-token modulation ($\lambda=0$ recovers vanilla GRPO with $w_t=1$ everywhere), and $\varepsilon_w$ bounds the maximum deviation from unity to prevent extreme gradient magnitudes.

The policy is then updated with the standard GRPO surrogate using these modified advantages $A^{(k)}_t$ rather than the uniform $A^{(k)}$. The paper uses the DAPO configuration: asymmetric clipping $\varepsilon_{\text{low}} = 0.2$ and $\varepsilon_{\text{high}} = 0.28$, token-level loss aggregation, and no KL penalty (Table 5).

The key insight: every self-distillation method instantiates a different $w_t$ in this shared framework. RLSD uses $w_t = (P_T/P_S)^{\text{sign}(A)}$; RLRT uses $w_t = (P_S/P_T)^{\text{sign}(A)}$, which is $1/w_t^{\text{RLSD}}$. SDPO replaces this multiplicative weight entirely with a separate logit-level KL divergence term. This interface clarifies that RLRT's innovation is entirely in the choice of $w_t$ and the correctness gate — the rollout, reward, and trust-region machinery is unchanged from GRPO.

The Token-Level Information Asymmetry Signal: $\hat{D}_t$ and $\bar{D}_t$

The central mathematical object RLRT operates on is the token-level information asymmetry between the student and teacher views. The student and teacher are the same model $\pi_\theta$ but differ in their conditioning context.

Student view: conditions only on the prefix history $h_t = (x, y_{<t})$, producing the next-token distribution:

PSt(⋅):=πθ(⋅∣ht)P_S^t(\cdot) := \pi_\theta(\cdot \mid h_t)

Teacher view: conditions on the same prefix history plus a privileged context $c$ (a known correct solution), producing:

PTt(⋅):=πθ(⋅∣ht,c)P_T^t(\cdot) := \pi_\theta(\cdot \mid h_t, c)

The token-level information asymmetry at a specific sampled token $y_t$ is defined as:

D^t(yt):=log⁡PSt(yt)−log⁡PTt(yt)=log⁡PSt(yt)PTt(yt)\hat{D}_t(y_t) := \log P_S^t(y_t) - \log P_T^t(y_t) = \log \frac{P_S^t(y_t)}{P_T^t(y_t)}

where $P_S^t(y_t)$ is the student's probability of the token that was actually sampled, $P_T^t(y_t)$ is the teacher's probability of that same token given the privileged context, and $\text{sg}(\cdot)$ denotes stop-gradient (the teacher's probability is treated as a fixed signal, not differentiated through).

What it computes: a signed scalar per token measuring how much more or less likely the student was to choose $y_t$ compared to what the teacher would have predicted. When $\hat{D}_t > 0$, the student favored this token against the teacher's prediction — the token is self-driven. When $\hat{D}_t < 0$, the teacher favored this token more than the student — the token is teacher-aligned.

Why this form: the log-ratio is natural because it centers at zero (student and teacher agree), is symmetric in its treatment of the two directions (unlike a difference of probabilities, which would depend on absolute magnitudes), and appears in the exponent of the per-token advantage weight, directly controlling whether a token is amplified or suppressed. This log-ratio form is the same as the one used in RLSD (Yang et al., 2026), but RLRT reads its sign in the opposite direction.

The position-level information asymmetry $\bar{D}_t$ is the expected value of $|\hat{D}_t|$ under the student's distribution:

Dˉt:=Ev∼PSt[D^t(v)]=Ev∼PSt[log⁡PSt(v)PTt(v)]=KL(PSt ∥ PTt)\bar{D}_t := \mathbb{E}_{v \sim P_S^t}\left[\hat{D}_t(v)\right] = \mathbb{E}_{v \sim P_S^t}\left[\log \frac{P_S^t(v)}{P_T^t(v)}\right] = \text{KL}\left(P_S^t \,\|\, P_T^t\right)

where $\text{KL}(P \| Q) = \sum_v P(v) \log \frac{P(v)}{Q(v)}$ is the Kullback-Leibler divergence.

What it computes: the expected divergence between the student's and teacher's next-token distributions at position $t$, measured in nats. A large $\bar{D}_t$ means the teacher disagrees substantially with the student about the next token; a small $\bar{D}_t$ means they largely agree.

Why this form: KL divergence captures the average surprise between the two distributions, not just the divergence at the sampled token. This matters because the student cannot condition on the privileged context, so the KL measures how much the teacher's additional information changes the model's beliefs in expectation, independent of which specific token happened to be sampled. Theorem 2 below shows that this expected divergence is directly related to the causal influence of token choice at that position.

The paper claims (and Section 4.2 proves) that $\bar{D}_t$ is large precisely at positions where the choice of token meaningfully affects the probability of a correct outcome, and small at positions where token choice is inconsequential. This property is what makes $\hat{D}_t$ a principled signal: it is not merely tracking surface disagreement but localizing causally important decision points.

The RLRT Reverse Weight Construction

Given the information asymmetry signal, RLRT defines its per-token reweighting as:

wtRLRT=exp⁡(sign(A)⋅D^t)=(PSt(yt)PTt(yt))sign(A)w_t^{\text{RLRT}} = \exp\left(\text{sign}(A) \cdot \hat{D}_t\right) = \left(\frac{P_S^t(y_t)}{P_T^t(y_t)}\right)^{\text{sign}(A)}

where $\text{sign}(A) \in \{-1, 0, +1\}$ is the sign of the group-standardized advantage for the rollout containing this token, and $\hat{D}_t = \log(P_S^t(y_t) / P_T^t(y_t))$ as defined above.

What it computes: a per-token multiplier on the advantage that amplifies or attenuates the gradient at $y_t$ based on both (a) whether the rollout was above or below the group average (via $\text{sign}(A)$), and (b) whether the student diverged from the teacher at this token (via $\hat{D}_t$).

Operationally, there are four regimes:

  • Correct rollout ($\text{sign}(A) > 0$), self-driven token ($\hat{D}_t > 0$): Then $w_t = (P_S/P_T)^{+1} > 1$. The token is amplified — the student chose against the teacher and succeeded, so this token gets extra positive credit.

  • Correct rollout, teacher-aligned token ($\hat{D}_t < 0$): Then $w_t = (P_S/P_T)^{+1} < 1$. The token is attenuated — the student followed the teacher's preference on a successful trajectory, so this token receives less credit than the self-driven tokens (but still positive, since $\text{sign}(A) > 0$).

  • Incorrect rollout ($\text{sign}(A) < 0$), self-driven token ($\hat{D}_t > 0$): Then $w_t = (P_S/P_T)^{-1} < 1$. The token is suppressed — the student diverged from the teacher and failed, so this divergence was harmful and receives extra negative credit.

  • Incorrect rollout, teacher-aligned token ($\hat{D}_t < 0$): Then $w_t = (P_S/P_T)^{-1} > 1$. The token is attenuated in its negative penalty — the student followed the teacher and still failed, so the blame is partially mitigated.

However, RLRT applies this weight only on correct rollouts (Equation 7). On incorrect rollouts, $A_t^{\text{RLRT},(k)} = A^{(k)}$ — vanilla GRPO, no reweighting. This correctness gate is the critical design choice that distinguishes RLRT from RLSD, which applies its weight (with the opposite exponent) to all rollouts regardless of correctness.

Why this form: the exponent $\text{sign}(A)$ ensures that the weight always pushes in the direction consistent with the outcome, while the base $P_S/P_T$ ensures that the magnitude of the push scales with how strongly the student and teacher disagreed. This couples the binary reward signal (which fixes the update direction) with the continuous information asymmetry signal (which modulates magnitude). The alternative — using the raw $\hat{D}_t$ without the advantage sign — would conflate exploration on correct rollouts with exploration on incorrect ones. The correctness gate is essential because, as the ablation in Section 6.5 (Figure 9a) demonstrates, applying the reverse weight to incorrect rollouts causes the policy to amplify self-driven tokens on failed trajectories, conflating valuable exploration with harmful divergence and leading to unbounded entropy growth and training collapse.

The clipping wrapper $\text{clip}(w_t, 1-\varepsilon_w, 1+\varepsilon_w)$ bounds how far the weight can deviate from unity. The paper sweeps $\varepsilon_w \in \{0.2, 0.5, 1.0\}$ and finds that looser clipping ($\varepsilon_w = 1.0$) yields the strongest training score on base models (Figure 9b), confirming that the reweighting mechanism itself — not just the correctness filter — drives the gains. The mixing coefficient $\lambda$ interpolates between the reweighted advantage and the vanilla GRPO advantage; $\lambda_{\text{init}} = 0.5$ was used in all experiments, with no decay for base models and 30-step linear decay for instruction-tuned and thinking-tuned models (Table 5).

The Complete RLRT Algorithm (Algorithm 1)

Algorithm 1 in Appendix B specifies the full RLRT update. In prose:

Step 1: Rollout generation. For a prompt $x$, the student policy $\pi_\theta$ generates $K=8$ complete rollouts $\{y^{(k)}\}_{k=1}^K$ (temperature 1.0, max length 20,480 tokens, using vLLM for efficient inference as specified in Table 5).

Step 2: Reward computation and advantage standardization. Each rollout receives a verifiable binary reward $r(y^{(k)}) \in \{0, 1\}$. The group mean $\mu$ and standard deviation $\sigma$ are computed across the $K$ rollouts, and the group-standardized advantage $A^{(k)} = (r^{(k)} - \mu)/\sigma$ is assigned to each rollout. For RLRT, standard deviation normalization is enabled (Table 5).

Step 3: Per-token reverse weight computation (correct rollouts only). For each rollout $k$ where $r(y^{(k)}) = 1$:

  • For each token position $t = 1, \ldots, |y^{(k)}|$:
    • Compute $P_S^t(y_t^{(k)}) = \pi_\theta(y_t^{(k)} \mid x, y_{<t}^{(k)})$: the student's probability of the sampled token given only the prefix.
    • Compute $P_T^t(y_t^{(k)}) = \pi_\theta(y_t^{(k)} \mid x, y_{<t}^{(k)}, c)$: the teacher's probability of the same token given the prefix plus the privileged context $c$ (a correct solution to the prompt).
    • Compute $\hat{D}_t = \log P_S^t(y_t^{(k)}) - \log P_T^t(y_t^{(k)})$.
    • Compute $w_t^{\text{RLRT}} = \exp(\text{sign}(A^{(k)}) \cdot \hat{D}_t) = (P_S^t / P_T^t)^{\text{sign}(A^{(k)})}$.
    • Set $A_t^{\text{RLRT},(k)} = A^{(k)} \cdot [(1-\lambda) + \lambda \cdot \text{clip}(w_t^{\text{RLRT}}, 1-\varepsilon_w, 1+\varepsilon_w)]$.

For rollouts where $r(y^{(k)}) = 0$, set $A_t^{\text{RLRT},(k)} = A^{(k)}$ for all $t$ — no reweighting, identical to vanilla GRPO.

Step 4: Policy update. Update $\theta$ using the standard GRPO surrogate loss with token-level advantages $A_t^{\text{RLRT},(k)}$, using the DAPO hyperparameters: asymmetric clipping $\varepsilon_{\text{low}}=0.2$, $\varepsilon_{\text{high}}=0.28$, token-level loss aggregation, AdamW optimizer with learning rate $1 \times 10^{-6}$, weight decay 0.01, gradient clip norm 1.0, 10 warmup steps, and mini-batch size 128 within a question batch size of 256 (Table 5).

The algorithm is structurally minimal: the only changes relative to GRPO are (i) the computation of $P_T$ for correct rollouts (which requires one additional forward pass per correct rollout with the privileged context appended), and (ii) the per-token advantage reweighting in Step 3. The rollout, reward, and trust-region machinery is otherwise identical.

The Explore/Exploit Decomposition of Information Asymmetry (Section 4.3 and Figure 3)

To validate that $\hat{D}_t$ captures meaningful reasoning distinctions rather than surface variation, Section 4.3 and Appendix D provide a linguistic analysis of which tokens the student systematically chooses against the teacher versus aligns with. This is not part of the algorithm but serves as supporting evidence for the claim that amplifying $\hat{D}_t > 0$ tokens amplifies exploration-oriented reasoning.

The procedure (Appendix D): Starting from 8 rollouts of Qwen3-8B on 100 DAPO-Math-17k problems, one correct and one incorrect trajectory per problem are retained (200 trajectories total). At each position $t$, two tokens from the full vocabulary $\mathcal{V}$ are identified: $\arg\max_{v \in \mathcal{V}} \hat{D}_t(v)$ (the token most favored by the student over the teacher, termed explore corpus) and $\arg\min_{v \in \mathcal{V}} \hat{D}_t(v)$ (the token most favored by the teacher over the student, termed exploit corpus). These are not the tokens actually sampled — they are the vocabulary entries where the student–teacher divergence is most extreme in each direction. Tokens appearing at least 30 times across the combined corpora, restricted to ASCII alphabetic tokens of length 3–15 characters, are scored using the smoothed log-odds z-score of Monroe et al. (2008):

zv=δvVar^(δv),δv=log⁡ev+αE−ev+α−log⁡xv+αX−xv+αz_v = \frac{\delta_v}{\sqrt{\widehat{\text{Var}}(\delta_v)}}, \quad \delta_v = \log\frac{e_v + \alpha}{E - e_v + \alpha} - \log\frac{x_v + \alpha}{X - x_v + \alpha}

where $e_v$ and $x_v$ are the counts of token $v$ in the explore and exploit corpora respectively, $E$ and $X$ are the total corpus sizes, and $\alpha = 0.5$ is a smoothing parameter.

What this computes: a normalized measure of how reliably each token type leans toward the explore or exploit side, accounting for both the raw frequency difference and the statistical uncertainty due to finite counts. $z_v \gg 0$ marks tokens that consistently appear among the student's strongest divergences from the teacher; $z_v \ll 0$ marks tokens that consistently appear among the teacher's strongest divergences from the student.

Why this form: the log-odds ratio with smoothing prevents rare tokens with extreme ratios from dominating, while the z-score normalization accounts for the variance of the estimator, making the rankings statistically meaningful rather than driven by sampling noise.

After filtering with $|z_v| \geq 3$ and removing stopwords (both NLTK English stopwords and a domain-specific list of ~400 math vocabulary items, LaTeX fragments, and tokenizer artifacts), 38 explore-side and 61 exploit-side markers remain (Table 2). The categorized results in Figure 3b show:

  • Explore-leaning tokens ($\hat{D}_t > 0$, student favored): Reflection markers ("wait," "hmm"), deliberation verbs ("let," "consider," "analyze," "think"), metacognitive verbs ("check," "recall"), alternative markers ("another," "different," "alternatively"), and epistemic hedges ("maybe," "perhaps"). These are reasoning-openers — tokens that precede attempts at new solution strategies.

  • Exploit-leaning tokens ($\hat{D}_t < 0$, teacher favored): Decision verbs ("determine," "conclude," "define," "verify"), verdict adjectives ("valid," "incorrect," "correct," "right"), consequential connectives ("thus," "however"), and finalization markers ("final," "finally," "eventually"). These are reasoning-closers — tokens that terminate reasoning and assert conclusions.

This linguistic decomposition provides empirical grounding for the claim that the sign of $\hat{D}_t$ corresponds to a meaningful explore/exploit dimension in mathematical reasoning, not merely a statistical artifact of the teacher–student computation.

Theoretical Justification: $\bar{D}_t$ and Causal Influence (Section 4.2)

The paper provides a formal argument that $\bar{D}_t = \text{KL}(P_S^t \| P_T^t)$ is not merely a measure of teacher–student disagreement but is directly connected to the causal influence of token choice at position $t$ on the probability of a correct outcome. This theoretical grounding is what elevates the information asymmetry signal from a heuristic to a principled quantity.

The Bayesian teacher model. The analysis begins by modeling the teacher as the student conditioned on the event $R=1$ (success):

PTt(⋅):=πθ(⋅∣ht,R=1)P_T^t(\cdot) := \pi_\theta(\cdot \mid h_t, R = 1)

This is an idealized teacher that uses knowledge of the outcome to reweight tokens. In practice, the paper instantiates this by feeding a known correct solution $c$ as conditioning context, which serves as a proxy for conditioning on $R=1$.

For each token $v \in \mathcal{V}$, define the per-token correctness probability $f(v)$ as the probability that choosing $v$ at position $t$ leads to a correct final answer:

f(v):=Pr⁡Y∼PS[R=1∣ht,yt=v]f(v) := \Pr_{Y \sim P_S}[R = 1 \mid h_t, y_t = v]

Define the student-mean correctness $\bar{f}_S^t$ as the expected value of $f(v)$ under the student's distribution:

fˉSt:=Ev∼PSt[f(v)]\bar{f}_S^t := \mathbb{E}_{v \sim P_S^t}[f(v)]

Lemma 1 (Bayesian teacher). At each step $t$:

PTt(v)=PSt(v)f(v)fˉSt  ⟺  D^t(v)=log⁡fˉSt−log⁡f(v)P_T^t(v) = P_S^t(v) \frac{f(v)}{\bar{f}_S^t} \iff \hat{D}_t(v) = \log \bar{f}_S^t - \log f(v)

where the forward direction follows from Bayes' rule and the reverse follows by taking logarithms and applying the definition $\hat{D}_t(v) = \log P_S^t(v) - \log P_T^t(v)$.

What this proves: the teacher distribution is exactly the student distribution tilted toward tokens with higher $f(v)$ — tokens that are more likely to lead to correctness. Equivalently, $\hat{D}_t(v)$ measures how far $f(v)$ falls below the student-mean $\bar{f}_S^t$. When $\hat{D}_t(v) > 0$, then $f(v) < \bar{f}_S^t$ — the token $v$ is less likely to lead to correctness than the average student choice, yet the student chose it anyway. On a correct rollout, this is precisely the situation of interest: the student succeeded despite choosing a token the teacher (conditioned on $R=1$) would have downweighted.

Why this form matters: Lemma 1 is the key link between the computational quantity $\hat{D}_t$ (which can be computed from model probabilities) and the unobservable quantity $f(v)$ (which requires knowing the outcome distribution). It shows that $\hat{D}_t$ is a valid proxy for the correctness-tilt of tokens, with $\hat{D}_t > 0$ corresponding to tokens the teacher would steer away from.

Position-level influence. The paper defines the influence of position $t$ as the expected absolute deviation of $f(v)$ from its mean:

InfS(t):=Ev∼PSt[∣f(v)−fˉSt∣]\text{Inf}_S(t) := \mathbb{E}_{v \sim P_S^t}\left[|f(v) - \bar{f}_S^t|\right]

What this computes: the average amount by which the token choice at position $t$ changes the probability of a correct outcome, measured in probability units. A position is critical when $\text{Inf}_S(t)$ is large — different token choices lead to substantially different success probabilities. A position is inert when $\text{Inf}_S(t) \approx 0$ — all token choices are roughly equivalent for the outcome.

Why this form: the absolute deviation captures the magnitude of correctness-variation without regard to direction, which is the appropriate measure for identifying positions where any token choice matters. Using signed deviation would miss positions where some tokens are better and some worse because the positive and negative deviations would cancel.

Theorem 2 ($\bar{D}_t$ controls $\text{Inf}_S(t)$). At every step $t$:

InfS(t)2≤2Dˉt\text{Inf}_S(t)^2 \leq 2 \bar{D}_t

The proof (Appendix C.2) proceeds in two steps:

  1. Bound by total variation. Using Lemma 1, $f(v) - \bar{f}_S^t = \frac{\bar{f}_S^t}{P_S^t(v)}(P_T^t(v) - P_S^t(v))$. Substituting into the definition of $\text{Inf}_S(t)$ yields:

InfS(t)=fˉSt∑v∈V∣PTt(v)−PSt(v)∣=2fˉSt⋅TV(PSt,PTt)≤2⋅TV(PSt,PTt)\text{Inf}_S(t) = \bar{f}_S^t \sum_{v \in \mathcal{V}} |P_T^t(v) - P_S^t(v)| = 2\bar{f}_S^t \cdot \text{TV}(P_S^t, P_T^t) \leq 2 \cdot \text{TV}(P_S^t, P_T^t)

where $\text{TV}(P, Q) = \frac{1}{2}\sum_v |P(v) - Q(v)|$ is the total variation distance and the final inequality uses $\bar{f}_S^t \in [0,1]$.

  1. Apply Pinsker's inequality. Pinsker's inequality states $\text{TV}(P, Q) \leq \sqrt{\frac{1}{2}\text{KL}(P \| Q)}$. Applying this with $P = P_S^t$ and $Q = P_T^t$, and recalling $\bar{D}_t = \text{KL}(P_S^t \| P_T^t)$:

TV(PSt,PTt)≤12Dˉt\text{TV}(P_S^t, P_T^t) \leq \sqrt{\frac{1}{2}\bar{D}_t}

Squaring the bound from Step 1 and substituting:

InfS(t)2≤4⋅TV(PSt,PTt)2≤4⋅12Dˉt=2Dˉt\text{Inf}_S(t)^2 \leq 4 \cdot \text{TV}(P_S^t, P_T^t)^2 \leq 4 \cdot \frac{1}{2}\bar{D}_t = 2\bar{D}_t

What this proves: small $\bar{D}_t$ guarantees small $\text{Inf}_S(t)$. By contrapositive: if $\bar{D}_t \approx 0$, then $\text{Inf}_S(t) \approx 0$ — the position is nearly inert, and token choice there cannot meaningfully affect correctness. Conversely, positions where $\bar{D}_t$ is large must be positions where the token distribution matters causally for the outcome.

Why this matters for RLRT: Theorem 2 justifies using $\bar{D}_t$ to identify critical positions. The reverse weight $w_t$ applies at every token position, but its effect is naturally concentrated at positions where $\bar{D}_t$ is large, because at inert positions $P_S^t \approx P_T^t$, making $\hat{D}_t \approx 0$ and $w_t \approx 1$. The weighting is adaptive — it automatically focuses gradient modification where it matters most. The reflection injection experiment (Section 6.2, Figure 6) provides causal validation: injecting a reflection prompt at $\arg\max_t \bar{D}_t$ flips incorrect to correct at roughly twice the rate of injection at random or low-$\bar{D}_t$ positions on the untrained checkpoint, confirming the theoretical claim empirically.

Design Choices and Their Justifications

Why reverse the teacher signal rather than align to it? The paper's central design decision is to amplify $\hat{D}_t > 0$ tokens on correct rollouts rather than suppress them. The justification is that these tokens represent the student's self-driven reasoning — choices made against the teacher's implicit prediction that nonetheless led to a correct answer. Aligning to the teacher on correct rollouts (as RLSD does) would overwrite these reasoning patterns, undermining the independent competence that produced the success. The paper demonstrates this through the distribution-shift analysis (Section 6.3, Figure 7): RLRT produces a fundamentally different kind of policy change compared to GRPO and RLSD, pulling tokens from the base model's probability tail into top positions rather than merely re-weighting existing top candidates.

Why gate on correctness? The ablation in Section 6.5 (Figure 9a) shows that removing the $r=1$ gate causes training collapse: response length and entropy grow unbounded, and the reward crashes. The explanation is that $\hat{D}_t > 0$ tokens on failed rollouts are not valuable exploration but spurious divergence — the student departed from the teacher and failed, so amplifying those departures reinforces harmful behavior. The correctness gate ensures that only verified self-driven reasoning is reinforced.

Why use the same model as both teacher and student rather than a separate teacher? The self-distillation setup keeps the teacher–student gap "on-policy" — both views are drawn from the current model's own distribution, avoiding distribution shift from an external teacher. This is computationally efficient (no separate model to load) and theoretically clean (the gap reflects only the effect of the privileged context, not model mismatch). The paper acknowledges (Section A) that extending RLRT to off-policy teachers (stronger or weaker models) is a natural direction.

Why not apply the reverse weight to incorrect rollouts with reversed sign? The paper deliberately restricts the reverse weight to correct rollouts only (incorrect rollouts get vanilla GRPO). This is not because $\hat{D}_t < 0$ tokens on incorrect trajectories are uninformative — in principle, tokens where the student diverged from the teacher and failed should be suppressed. The paper does not explore this symmetric treatment, likely because the magnitude of negative advantages is already small when most rollouts are incorrect (group standardization compresses negative advantages toward zero), and because mixing self-driven suppression with self-driven amplification might create conflicting gradient signals. The ablation (RLRT without reward gate) confirms that naively extending the reversal to all rollouts is catastrophic.

Why the exponential form $w_t = (P_S/P_T)^{\text{sign}(A)}$ rather than a linear weighting? The exponential form arises naturally from the log-ratio $\hat{D}_t$. A linear weighting like $w_t = 1 + \alpha \cdot \hat{D}_t$ would not have the multiplicative property that $w_t > 1$ exactly when $\hat{D}_t$ and $\text{sign}(A)$ have the same sign. The exponential form also connects to the RLSD weight ($w_t^{\text{RLRT}} = 1/w_t^{\text{RLSD}}$), making the comparison between the two methods clean: they are exact inverses on correct rollouts.

Why $\lambda < 1$ (mixing with vanilla GRPO)? The mixing coefficient $\lambda$ interpolates between the reweighted advantage and the uniform GRPO advantage. $\lambda = 0.5$ (the initial value used in all experiments) means that half the gradient comes from the reverse-weighted signal and half from standard GRPO. This prevents the reverse weight from completely dominating the update, which could lead to overfitting to the teacher–student gap at the expense of the reward signal. The paper finds that decaying $\lambda$ (30 steps for instruction-tuned and thinking-tuned models) or keeping it constant (base models) works depending on the backbone, suggesting that the optimal mixing schedule is model-dependent.

Why asymmetric clipping $\varepsilon_{\text{high}} = 0.28$, $\varepsilon_{\text{low}} = 0.2$? The paper adopts the DAPO clipping scheme, which clips more aggressively for negative advantages ($\varepsilon_{\text{low}} = 0.2$ means ratio can deviate by at most 20% below 1) than positive ones ($\varepsilon_{\text{high}} = 0.28$ means ratio can deviate by at most 28% above 1). This asymmetry is motivated by the observation that over-optimism (ratio too large) is less harmful than excessive pessimism (ratio too small) because the latter can cause the policy to abandon promising strategies too quickly. For RLRT, the additional clip $\varepsilon_w$ on the per-token weight operates independently of the PPO ratio clip, bounding how extreme the reverse weight modulation can be.

Why $\varepsilon_w = 1.0$ on base models but smaller on tuned models? The paper sweeps $\varepsilon_w \in \{0.2, 0.5, 1.0\}$ and finds that base models benefit from the largest weight range ($\varepsilon_w = 1.0$), while instruction-tuned models prefer $\varepsilon_w = 0.5$ and thinking-tuned models prefer $\varepsilon_w = 0.2$. The interpretation (Section 6.1) is that base models have not yet concentrated their policy through instruction tuning, so they have more room for the reverse weight to reshape the distribution. Instruction-tuned and thinking-tuned models have already been optimized to prefer certain reasoning styles, making large per-token weight deviations more likely to destabilize training.

Why disable standard deviation normalization for GRPO but enable it for RLRT? Following Liu et al. (2025), the paper disables advantage standard deviation normalization for GRPO and SRPO to "preserve relative signal strength across groups" (Table 5). For RLSD and RLRT, the normalization is enabled following RLSD (Yang et al., 2026), likely because the per-token weight already modulates the advantage magnitude, and standard deviation normalization prevents the combined modulation from producing extreme gradient magnitudes. This is an implementation detail the paper does not ablate, but it reflects the fact that per-token reweighting interacts with advantage normalization in ways that require different defaults.

4. Key Insights and Innovations

Innovation 1: Inverting Self-Distillation on Correct Rollouts — Turning Suppression into Exploration

The paper's most conceptually distinctive move is not a new loss function or architecture, but a reinterpretation of an existing signal: the teacher–student probability gap on correct rollouts. Every prior self-distillation method — SDPO (Hübotter et al., 2026), RLSD (Yang et al., 2026), OPSD (Zhao et al., 2026), SRPO (Li et al., 2026) — reads this gap as an alignment target. The teacher conditions on privileged context the student lacks (a successful rollout, ground-truth answer, or corrective feedback), so the gap represents knowledge the student should absorb. Distillation pulls the student toward the teacher, implicitly treating any deviation as error.

RLRT's inversion is to ask: what if the deviation is not error but evidence of the student's independent reasoning capability? When the student reaches the correct answer through a path the teacher would not have predicted ($P_S > P_T$ at certain tokens), those tokens are not mistakes to be corrected — they are the student's own reasoning, verified by the outcome. The paper's core conceptual move is to read the teacher–student gap in reverse, treating $\hat{D}_t > 0$ tokens on correct trajectories as a signal to amplify, not suppress.

This is not an incremental refinement. It fundamentally changes what self-distillation is for. Under prior approaches, the teacher is an authority whose superior knowledge the student must acquire. Under RLRT, the teacher is a contrastive reference point whose disagreement with the student reveals where the student is exercising independent judgment. The teacher's role shifts from oracle to foil — its value lies not in being followed, but in being departed from on successful trajectories.

The significance extends beyond the specific algorithm. This inversion suggests that self-distillation's value may lie as much in what it reveals about the student as in what it transmits from the teacher. The paper's own linguistic analysis (Figure 3) supports this: the tokens with $\hat{D}_t > 0$ are systematically different from those with $\hat{D}_t < 0$ — exploration markers ("wait," "consider," "another") versus exploitation markers ("conclude," "final," "correct") — confirming that the gap captures a meaningful cognitive dimension, not surface noise. The distribution-shift analysis (Section 6.3, Figure 7c) provides further evidence: RLRT's policy promotes tokens the base model assigned near-zero probability to top-1 positions over 10× more often than GRPO or RLSD, indicating that amplifying the reversed signal genuinely changes what the model considers, not just how it weights existing preferences.

Prior work partially anticipated pieces of this insight without fully committing to it. SRPO (Li et al., 2026) recognized the "optimization ambiguity" problem — that distillation on correct rollouts overwrites the student's choices — but its solution was to simply stop distilling on correct rollouts, reverting to vanilla GRPO. This discards the signal entirely rather than inverting it. RLRT's contribution is to show that the signal on correct rollouts is not merely noise to be ignored, but valuable information about successful reasoning when read in the opposite direction. The fact that SRPO underperforms full self-distillation on math (Figure 5, Qwen3-8B panel) while RLRT substantially outperforms both confirms that the reversal — not just the routing — is the active ingredient.

Innovation 2: Information Asymmetry as a Principled Exploration Signal — Beyond Uniform Diversity

A second conceptual contribution is the reframing of exploration in RLVR from a uniformity problem to an information-asymmetry problem. The dominant paradigm for addressing reasoning boundary collapse treats diversity as a target to be maximized: add entropy bonuses (Cheng et al., 2026; Cui et al., 2025), incentivize semantic variation across rollouts (Hu et al., 2025; Wan et al., 2026), or reward outcome diversity (Song et al., 2025). Under this view, the problem is that the model does not try enough different things, and the solution is to push it toward broader sampling.

RLRT's framing is fundamentally different. The problem is not lack of diversity per se — it is that the model does not know which of its deviations from its default behavior are worth keeping. The model already produces diverse rollouts (temperature 1.0 sampling generates variation naturally); the bottleneck is that the sparse RL signal cannot distinguish productive exploration from unproductive noise. What RLRT provides is a grounding mechanism: it identifies, within the diversity the model already generates, the specific tokens where the student departed from a conditioned teacher and succeeded, then amplifies those tokens. This is what the paper terms valuable exploration — diversity anchored to demonstrated competence rather than stochastic perturbation.

This distinction has implications beyond the specific method. It suggests that the exploration problem in RLVR is better understood as a credit-assignment problem for diversity rather than a sampling problem. The model has the capacity to explore; what it lacks is a signal about where exploration paid off. The teacher–student gap provides exactly this signal: $\hat{D}_t > 0$ on correct rollouts localizes the tokens where exploration was rewarded by the outcome, and amplifies them directly in the gradient. This is more efficient than uniform entropy bonuses, which push the model to explore everywhere and rely on the sparse reward to retrospectively sort good exploration from bad. The comparison against GRPO+entropy bonus and DIVER (Section 6.4, Figure 8) supports this: GRPO+entropy bonus barely deviates from vanilla GRPO across the entire pass@k curve (entropy adds noise but not directed exploration), while RLRT dominates from pass@1 through pass@256, reflecting broader coverage of reasoning modes rather than stochastic variation within one.

The theoretical machinery in Section 4.2 reinforces this reframing. Theorem 2 proves that $\bar{D}_t$ (the expected information asymmetry at a position) directly bounds the causal influence of token choice on correctness. This means the positions where RLRT's reweighting has the largest effect are precisely the positions where exploration is most consequential — not a heuristic choice, but a principled consequence of the Bayesian teacher model (Lemma 1). The reflection injection experiment (Section 6.2, Figure 6) provides causal validation: injecting a reflection prompt at $\arg\max_t \bar{D}_t$ produces roughly double the incorrect-to-correct flip rate compared to random positions on the base checkpoint, and RLRT amplifies this advantage from ~18% to over 40% by step 100 while GRPO lets it collapse to the random baseline.

This is a fundamental reframing, not an incremental improvement on existing exploration methods. It introduces a new axis for exploration design — what signal do you use to identify valuable exploration? — that is orthogonal to the how much diversity axis that prior work focused on. Future work could explore different teacher configurations (stronger models, weaker models, different forms of privileged context) or combine this directional exploration with existing diversity objectives, but the core insight — that information asymmetry within a single model provides an intrinsic, grounded exploration signal — is new.

Innovation 3: The Theoretical Grounding of $\bar{D}_t$ as a Causal Importance Measure

The paper provides what is, to my knowledge, the first formal connection between KL divergence in a self-distillation setup and the causal influence of token choice on outcome probability. Lemma 1 and Theorem 2 in Section 4.2 are not just mathematical decoration — they establish that a quantity computable entirely from model probabilities ($\bar{D}_t = \text{KL}(P_S \| P_T)$) serves as a certificate of causal importance.

Before this work, the use of teacher–student divergence in self-distillation was primarily justified by intuition: the teacher has more information, so its predictions should be better, so the student should move toward them. This is a correlation argument — the teacher is correlated with success — but it does not explain why the magnitude of divergence matters, or what guarantees exist that positions with large divergence are actually important for the outcome.

Theorem 2 provides exactly this guarantee. It proves that $\text{Inf}_S(t)^2 \leq 2\bar{D}_t$, where $\text{Inf}_S(t)$ is the expected absolute deviation in correctness probability across token choices at position $t$. The contrapositive is powerful: if $\bar{D}_t$ is small, the position is certifiably inert — no token choice there can substantially change the probability of a correct outcome. This means that any position where $\bar{D}_t$ is large must be a position where token choice matters causally. The bound is not tight (the constant 2 comes from Pinsker's inequality, which is generally loose), but it provides a monotonic guarantee: larger $\bar{D}_t$ implies a higher ceiling on causal influence, and near-zero $\bar{D}_t$ implies near-zero influence.

This is a theoretical advance in its own right, independent of RLRT's algorithmic performance. It provides a principled justification for token-level credit assignment based on information asymmetry, showing that the signal RLRT amplifies is not merely correlated with success but is causally connected to it through the Bayesian teacher model. Prior work on token-level credit in RLVR (RLSD, for instance) used the teacher–student ratio as a heuristic confidence signal — "the teacher is more confident, so this token matters" — without connecting it to outcome probability. Lemma 1 makes this connection explicit: $\hat{D}_t(v) = \log \bar{f}_S^t - \log f(v)$, meaning the log-ratio directly measures how a token's correctness probability compares to the student's average.

The reflection injection experiment (Section 6.2) provides empirical validation that this theoretical connection holds in practice. On the untrained base checkpoint (step 0), injecting a reflection prompt at $\arg\max_t \bar{D}_t$ produces approximately double the incorrect-to-correct flip rate compared to injection at random or low-$\bar{D}_t$ positions — exactly what Theorem 2 predicts: high-$\bar{D}_t$ positions are causally important. The absence of a comparable wrong-to-right flip advantage at low-$\bar{D}_t$ positions confirms the contrapositive: inert positions cannot meaningfully affect outcomes.

This theoretical contribution is fundamental rather than incremental. It transforms the information asymmetry signal from a useful heuristic to a mathematically grounded quantity with causal semantics, opening the door for future work to design more sophisticated credit-assignment schemes built on this guarantee — for example, using $\bar{D}_t$ to decide not just how much to reweight but whether to apply a different update rule entirely at critical versus inert positions.

Innovation 4: The Correctness Gate as a Necessary Condition — A Diagnostic Negative Result

The paper's ablation on removing the reward gate (Section 6.5, Figure 9a) provides a diagnostic negative result that is intellectually significant beyond confirming a design choice. RLRT without the $r=1$ gate — applying the reverse weight to all rollouts regardless of correctness — causes training to collapse: response length and actor entropy grow unbounded, and the training reward crashes around step 40. This is not merely that performance degrades; the training destabilizes fundamentally.

This negative result establishes a sharp boundary condition on when the reversed teacher signal is beneficial versus harmful. The teacher–student gap $\hat{D}_t$ is agnostic to outcome — it measures divergence, not whether the divergence was productive. On correct rollouts, $\hat{D}_t > 0$ marks self-driven reasoning that succeeded; on incorrect rollouts, the same positive value marks self-driven reasoning that failed. Amplifying both conflates valuable exploration with harmful divergence, and the unbounded entropy growth in Figure 9a shows the consequence: the policy chases its own tail, reinforcing divergence for its own sake until the reward signal is overwhelmed.

This finding has implications for the broader self-distillation literature. It suggests that the value of any teacher–student signal is conditional on outcome — a lesson that prior work has not fully internalized. RLSD applies its per-token weighting to all rollouts (both correct and incorrect), using $\text{sign}(A)$ to flip the direction of the weight. While this prevents the kind of unbounded collapse seen in RLRT-all, it still means that on incorrect rollouts, RLSD down-weights tokens where $\hat{D}_t > 0$ (student diverged from teacher) and up-weights tokens where $\hat{D}_t < 0$ (student aligned with teacher). This is the opposite of RLRT's treatment on correct rollouts, but applied to failed trajectories — it suppresses the student's self-driven tokens when they led to failure and reinforces teacher-aligned tokens even though the trajectory failed. Whether this mixed treatment is coherent or whether it creates conflicting gradient signals (the same token type gets opposite treatment depending on outcome) is an open question the paper implicitly raises but does not answer.

The negative result also clarifies why RLRT's gains are largest on base models and smallest on thinking-tuned models (18.0% vs. 2.2% average improvement, Table 1 and Figure 1b). Base models have not yet had their policy concentrated by instruction tuning or RL, so they produce a wider range of $\hat{D}_t > 0$ tokens on correct rollouts — more self-driven reasoning to amplify. Thinking-tuned models have already been optimized to produce specific reasoning patterns, so the student and teacher distributions are more aligned, leaving less room for the reversed signal to identify genuinely new reasoning. This is consistent with the $\varepsilon_w$ sweep results: base models benefit from the loosest clipping ($\varepsilon_w = 1.0$), while thinking-tuned models require the tightest ($\varepsilon_w = 0.2$), suggesting that the raw $\hat{D}_t$ magnitudes are larger and more varied in untuned models.

This is an incremental contribution in the sense that it emerges from an ablation rather than a novel theoretical claim, but it is diagnostically fundamental: it identifies the correctness gate not as a tuning knob but as a necessary structural condition for the reversed signal to be beneficial. Future extensions of RLRT to hybrid or off-policy settings will need to respect this boundary: self-driven tokens are only valuable when verified by success.

5. Experimental Analysis

Evaluation Methodology

  • Dataset. All training uses DAPO-Math-17k (Yu et al., 2025), a 17,000-problem mathematical reasoning dataset. Evaluation uses six separate math benchmarks: AIME24, AIME25, AIME26 (American Invitational Mathematics Examination, three recent years), HMMT26 (Harvard-MIT Mathematics Tournament, February 2026), AMC23 (American Mathematics Competition, 2023), and MATH500 (a 500-problem subset of the MATH benchmark, Hendrycks et al., 2021). The AIME and HMMT benchmarks are competition-level and substantially harder than MATH500; this range tests both standard and challenging mathematical reasoning.

  • Base model(s). The paper evaluates across four qualitatively distinct checkpoints from the Qwen3 family: Qwen3-4B-Base, Qwen3-8B-Base (pretrained only, no instruction tuning), Qwen3-4B-Instruct (instruction-tuned), and Qwen3-8B (thinking-tuned, with reasoning capabilities already baked in through mid-training). The paper explicitly justifies this diversity by noting that "post-training dynamics depend strongly on the pretrained checkpoint's inductive biases" (Section 6.1), citing Zhao et al. (2025) and Zhang et al. (2025). This four-backbone sweep is a strength: it tests whether RLRT's mechanism is robust across qualitatively different starting policies, from untrained base models to already-reasoning-capable checkpoints.

  • Metrics. The paper reports two standard metrics for math reasoning evaluation:

    • avg@16: For each problem, 16 responses are sampled; the metric is the average number of correct answers among those 16 (a continuous score between 0 and 16, typically reported as a percentage). This measures the model's expected correctness under 16-sample decoding.
    • pass@16: The fraction of problems for which at least one of the 16 sampled responses is correct. This measures the model's coverage — whether it can produce a correct answer at all within 16 attempts.

    For the exploration comparison (Section 6.4), pass@k curves are reported for k in {1, 2, 4, 8, 16, 32, 64, 128, 256}, computed using the unbiased estimator of Chen et al. (2021).

  • Baselines. The paper compares against:

    • GRPO (Shao et al., 2024; Yu et al., 2025): The DAPO recipe — asymmetric clipping (ε_low=0.2, ε_high=0.28), token-level loss aggregation, no KL penalty, no advantage standard deviation normalization.
    • SDPO (Hübotter et al., 2026): Self-distillation via logit-level KL divergence pulling the student toward a teacher conditioned on a correct rollout, applied to all rollouts regardless of correctness.
    • SRPO (Li et al., 2026): Routes correct rollouts to GRPO and incorrect rollouts to SDPO-style distillation, with entropy-aware dynamic weighting.
    • RLSD (Yang et al., 2026): Per-token advantage weighting w_t = (P_T/P_S)^{sign(A)} applied to all rollouts, with teacher conditioned on ground-truth answer.

    For the exploration comparison (Section 6.4), two additional baselines are introduced on Qwen3-8B-Base:

    • GRPO+EB (GRPO with entropy bonus, Cheng et al., 2026): Token-level entropy regularization added to the GRPO loss.
    • DIVER (Hu et al., 2025): Sequence-level diversity objective that incentivizes semantically distinct reasoning traces.

    SDPO and SRPO collapsed early on base models (Appendix F.2, Figure 11): training reward crashed to zero within 20 steps while response length diverged. The paper attributes this to excessive suppression of hedging/reflective tokens like "wait" and "hmm" that are critical for robust reasoning (citing Kim et al., 2026). Consequently, SDPO and SRPO are excluded from base model comparisons in the main results.

  • Generation budget / compute accounting. All training runs use the same sampling budget: K=8 rollouts per prompt per training step, with a maximum response length of 20,480 tokens, temperature 1.0, and a question batch size of 256 (Table 5). This means each training step processes 256 prompts × 8 rollouts = 2,048 trajectories. RLRT requires one additional forward pass per correct rollout to compute teacher probabilities with the privileged context appended; the paper notes that training Qwen3-4B/8B-Base "takes approximately one day, whereas Qwen3-4B-Instruct and Qwen3-8B require 2–3 days" on 2×B200 GPUs (Appendix G), implying that the additional teacher forward pass is a modest overhead. At evaluation time, all models use temperature 0.7, top-p=0.8, top-K=20, and a maximum response length of 38,912 tokens (Table 5, evaluation hyperparameters).

  • Cross-validation / statistical protocol. No cross-validation is reported. Each training run produces a single checkpoint (the final step or the best-performing step based on training score). Evaluation on the six benchmarks uses these trained checkpoints without any held-out validation set for model selection. The paper does not report confidence intervals, standard deviations, or statistical significance tests for benchmark results. This is a standard but notable practice in RLVR benchmarking; the main defense is that gains are consistent across four model backbones and six benchmarks, making the pattern robust even without formal statistical testing at the individual benchmark level. The hyperparameter sweep for ε_w ∈ {0.2, 0.5, 1.0} is done on training reward (Figure 9b), not on downstream benchmark performance, so there is potential for mild overfitting to the evaluation benchmarks through hyperparameter selection — though the consistency across backbones mitigates this concern.

Main Quantitative Results

Benchmark Performance: RLRT vs. Self-Distillation Baselines (Table 1, Table 3, Figure 5)

The central quantitative results are in Table 1 (base and thinking-tuned models) and Table 3 (instruction-tuned model), with training dynamics shown in Figure 5.

Qwen3-4B-Base (Table 1, top section). On the base 4B model, where the starting policy is untrained on reasoning tasks, RLRT produces the largest gains:

  • AIME24 avg@16: RLRT achieves 22.5% vs. GRPO's 15.0% and RLSD's 13.3% — a gain of Δ=+7.5% over the best baseline.
  • AIME25 avg@16: 18.5% (RLRT) vs. 14.4% (GRPO) and 11.2% (RLSD) — Δ=+4.1%.
  • AIME26 avg@16: 19.8% vs. 12.3% vs. 9.0% — Δ=+7.5%.
  • HMMT26 avg@16: 15.9% vs. 10.0% vs. 6.2% — Δ=+5.9%.
  • AMC23 avg@16: 63.9% vs. 58.3% vs. 55.2% — Δ=+5.6%.
  • MATH500 avg@16: 83.8% vs. 80.2% vs. 77.9% — Δ=+3.6%.
  • Mean across six benchmarks: Figure 1b reports an 18.0% average improvement over baselines (in the bar chart, which computes relative improvement, not absolute percentage points; Table 1 numbers confirm consistent absolute gains of 3.6–7.5 percentage points per benchmark).

The pass@16 numbers mirror this pattern: RLRT achieves 50.0% pass@16 on AIME24 (vs. 40.0% for GRPO), 36.7% on AIME25, 40.0% on AIME26, and 33.3% on HMMT26 — consistently 3–10 percentage points higher than the best baseline. On MATH500, pass@16 is 94.2% for RLRT, statistically tied with GRPO's 94.2% (Δ=0.0), suggesting ceiling effects on the easiest benchmark.

Qwen3-8B-Base (Table 1, middle section). The 8B base model shows a similar pattern with slightly smaller but still substantial gains:

  • Mean across six benchmarks: Figure 1b reports 12.0% improvement. Absolute gains range from 0.8 pp (MATH500) to 8.1 pp (AIME24).
  • AIME24 avg@16: 27.9% (RLRT) vs. 19.8% (GRPO) vs. 17.3% (RLSD) — Δ=+8.1 pp. Pass@16 jumps to 63.3% vs. 40.0% — the largest single-benchmark pass@16 gain in the table (Δ=+23.3 pp).
  • AIME26 pass@16: 53.3% vs. 36.7% (Δ=+16.6 pp), indicating RLRT substantially expands the fraction of problems where the model can produce at least one correct answer.
  • A notable pattern: RLSD underperforms GRPO on most benchmarks for the 8B-Base (e.g., 17.3% vs. 19.8% on AIME24 avg@16), while on the 4B-Base the gap is even larger (13.3% vs. 15.0%). This suggests that standard self-distillation (RLSD's teacher-aligned weighting) is actively harmful on base models, consistent with the paper's claim that pulling the student toward the teacher suppresses self-driven reasoning that base models rely on.

Qwen3-4B-Instruct (Table 3, Appendix F.1). On the instruction-tuned 4B model, the base performance is much higher (e.g., AIME24 avg@16: 64.4% for the untuned instruct model vs. 9.6% for the base model), and RLRT's gains are correspondingly smaller:

  • Mean across six benchmarks: Figure 1b reports 3.4% improvement.
  • AIME25 avg@16: 62.9% (RLRT) vs. 58.3% (GRPO) and 51.9% (RLSD) — Δ=+4.6 pp.
  • AIME26 avg@16: 67.9% vs. 62.7% vs. 58.1% — Δ=+5.2 pp.
  • HMMT26 avg@16: 40.2% vs. 37.9% vs. 37.1% — Δ=+2.3 pp.
  • MATH500 avg@16: 94.8% vs. 94.4% — nearly at ceiling.
  • AIME24 avg@16: 70.4% vs. 69.4% (GRPO) — Δ=+1.0 pp, but pass@16 is actually lower (90.0% vs. 93.3%, Δ=−3.3 pp), the only negative pass@16 delta in all benchmarks.
  • SDPO performance is notably poor: 53.1% on AIME24 avg@16 vs. 70.4% for RLRT and 69.4% for GRPO. SDPO degrades the instruct-tuned model's reasoning, consistent with the collapse pattern observed on base models (Appendix F.2).

Qwen3-8B (Thinking-tuned, Table 1, bottom section). The thinking-tuned 8B model starts with strong reasoning capabilities (e.g., AIME24 avg@16: 25.2% base, 70.2% after GRPO) and shows the smallest RLRT gains:

  • Mean across six benchmarks: Figure 1b reports 2.2% improvement (the paper says 2.3% in the figure but 2.2% in the text; this is a minor discrepancy).
  • AIME24 avg@16: 70.6% (RLRT) vs. 70.2% (GRPO) — Δ=+0.4 pp, essentially tied.
  • AIME24 pass@16: 93.3% vs. 86.7% — Δ=+6.6 pp, indicating RLRT improves coverage without much changing average accuracy.
  • AIME25 avg@16: 62.9% vs. 59.4% — Δ=+3.5 pp.
  • AIME26 avg@16: 65.0% vs. 62.9% — Δ=+2.1 pp.
  • HMMT26 avg@16: 43.2% vs. 41.7% — Δ=+1.5 pp.
  • SRPO collapses dramatically on this model: 15.4% AIME24 avg@16 and 9.8% AIME25 — far worse than GRPO or even the untrained base model. This is the most extreme failure of any baseline, confirming the paper's conjecture that routing correct rollouts to GRPO and incorrect to self-distillation creates "conflicting gradients" (Section 6.1). SDPO (26.9% AIME24) also underperforms the base model (25.2%), adding evidence that self-distillation is fundamentally misaligned with reasoning on this checkpoint.
  • RLSD performs competitively (65.4% AIME24) but consistently below RLRT across all benchmarks.

Training dynamics (Figure 5). Across all four backbones, RLRT's training reward curve rises faster and higher than all baselines. On Qwen3-4B-Base, RLRT reaches approximately 0.4 training score by step 100, while GRPO plateaus around 0.32 and RLSD around 0.28. On Qwen3-8B, RLRT tracks GRPO closely but with a slight edge at most steps. The thinking-tuned Qwen3-8B panel shows SRPO collapsing early (reward drops to near zero before step 20) and SDPO declining from step 40 onward, while RLRT and GRPO both rise steadily, with RLRT maintaining a small lead throughout. This training-speed advantage — reaching higher reward with the same number of steps — is consistent with the paper's claim that the reversed teacher signal provides more efficient credit assignment, though it does not directly prove that the learned policy is better (only that it achieves higher in-distribution reward).

Exploration Comparison: Pass@k Curves on AIME (Figure 8, Section 6.4)

To test whether RLRT induces more effective exploration patterns than prior diversity-focused methods, Section 6.4 compares pass@k curves on Qwen3-8B-Base for RLRT, GRPO, GRPO+EB (entropy bonus), and DIVER (sequence-level diversity). The key findings from Figure 8:

  • GRPO+EB tracks GRPO almost exactly across the full pass@k curve on both AIME24 and AIME26, even falling slightly below GRPO at small k (k=1, 2). The entropy bonus injects local stochasticity at individual decision points, but this translates to neither broader coverage (pass@k at high k is nearly identical) nor better single-sample quality (pass@1 is the same or slightly worse). This is a notable negative result: token-level entropy regularization — a widely used exploration technique — provides essentially zero benefit on these math reasoning benchmarks under the GRPO framework.

  • DIVER shows modest improvement over GRPO, most visible at large k: on AIME24 at k=256, DIVER reaches approximately 0.75 pass@k vs. GRPO's ~0.65. However, the margin narrows at small k, and the improvement is incremental rather than transformative. The paper interprets this as evidence that DIVER's semantic-level diversity heuristic "broadens exploration only modestly" — it encourages the model to produce varied reasoning traces, but many of those traces are not actually correct, so the benefit manifests primarily at very high sample counts where at least one trajectory per problem happens to be right.

  • RLRT dominates the entire pass@k spectrum. On AIME24: at k=1, RLRT reaches approximately 0.28 vs. GRPO's ~0.22; at k=16 (the evaluation point), RLRT is at roughly 0.48 vs. ~0.40 for DIVER and ~0.37 for GRPO; at k=256, RLRT reaches approximately 0.85 vs. ~0.75 for DIVER and ~0.65 for GRPO. On AIME26, the pattern is similar but with a larger gap at low k: RLRT pass@1 is roughly 0.18 vs. GRPO's ~0.12, and the advantage is maintained through k=256.

The takeaway is that RLRT's pass@k curve is stochastically dominant: at every sample budget, RLRT finds more correct answers than any baseline. This is stronger evidence for effective exploration than a single-point avg@16 metric — it shows that RLRT is not just more accurate on average but covers more of the solution space, finding correct answers on problems that other methods miss entirely. The paper attributes this to RLRT's "genuinely broader coverage across reasoning modes rather than within one" (Section 6.4), contrasting with GRPO+EB's within-mode stochasticity and DIVER's semantic variation that often explores unproductive directions.

Causal Validation: Reflection Injection Experiment (Figure 6, Section 6.2)

To test the paper's theoretical claim that \bar{D}_t identifies causally critical positions, Section 6.2 conducts a reflection injection experiment. On 100 DAPO-Math-17k problems with 8 rollouts each from Qwen3-8B checkpoints (step 0 through step 100 under both RLRT and GRPO), the reflection prompt "Wait, let me reconsider." is injected at three types of positions:

  • max_kl: \arg\max_t \bar{D}_t — the position where teacher–student divergence is largest.
  • random: A uniformly random token position.
  • min_kl: \arg\min_t \bar{D}_t — the position where teacher–student divergence is smallest.

On a hard subset (problems with 0, 1, or 2 correct answers out of 8 under the base policy), flip→R (wrong answer becomes correct after reflection injection) is measured. On an easy subset (5, 6, or 7 correct out of 8), flip→W (correct answer becomes wrong) is measured. Results are in Figure 6.

On the untrained checkpoint (step 0, black squares):

  • flip→R at max_kl is approximately 18%, compared to ~9% at random and ~9% at min_kl — roughly double the flip rate. This confirms the theoretical prediction: positions with high \bar{D}_t are causally important for correctness, and intervening there has outsized impact.
  • flip→W on the easy subset shows a much smaller absolute rate (~10% at max_kl vs. ~6% at random) and a less dramatic ratio (less than 2×), which the paper attributes to the reflection prompt being "biased toward correcting errors" (Section 6.2) — it is designed to trigger re-evaluation, which is more likely to fix mistakes than to introduce them.

With training (step 0 → 100):

  • RLRT amplifies the max_kl advantage: flip→R at max_kl rises from ~18% at step 0 to over 40% by step 100 (green line with squares), while random and min_kl flip rates remain roughly flat around 10–15%.
  • GRPO lets the advantage collapse: Under GRPO (orange line), flip→R at max_kl starts at ~18% (same as RLRT, since step 0 is the base checkpoint) but declines to ~12–15% by step 100, converging toward the random and min_kl baselines.
  • flip→W declines similarly for both RLRT and GRPO (from ~10% to ~5% at max_kl), indicating that RLRT's increased flip→R does not come at the cost of fragility on already-correct rollouts.

This experiment provides causal evidence for the paper's central mechanism: \bar{D}_t genuinely marks causally consequential positions, and RLRT's training concentrates exploration credit on these positions (explaining the rising flip→R rate), while GRPO's uniform credit assignment causes the model to treat critical and inert positions similarly (explaining the converging flip→R rates). It is one of the strongest pieces of evidence in the paper because it goes beyond correlation (RLRT performs better) to mechanism (RLRT improves because it focuses on causally important tokens).

Distribution-Shift Analysis (Figure 7, Section 6.3)

To characterize how RLRT's trained policy differs from the base model compared to GRPO and RLSD, Section 6.3 analyzes the token-level distributional shift on 30 hard prompts (n_correct ∈ {0, 1, 2} out of 8 under the base policy) from DAPO-Math-17k using Qwen3-8B-Base checkpoints. At each token position along a fine-tuned rollout, the Jensen-Shannon divergence JS(π_ft ∥ π_base) is computed, and positions with JS > 0.1 are termed "high-divergence." Three panels in Figure 7 answer distinct questions:

Figure 7a: How often does the policy diverge from the base? The complementary CDF of JS divergence shows:

  • GRPO and RLSD track each other closely, with only ~5% of positions exceeding JS=0.1 and the distribution dropping off rapidly (fewer than 10⁻⁴ of positions exceed JS=0.2).
  • RLRT places substantially more positions in the high-divergence regime: roughly 15% exceed JS=0.1, and the tail extends to JS≈0.6 (compared to GRPO/RLSD's maximum of ~0.3).

This means RLRT's policy is genuinely different from the base model at more positions, not just slightly perturbed. The paper terms this "reorganizes the candidate set" vs. GRPO/RLSD's "sharpen what π_base already prefers."

Figure 7b: When it diverges, are new tokens entering the top candidates, or are existing ones re-ranked? At high-divergence positions (JS > 0.1), the overlap |top-k(π_ft) ∩ top-k(π_base)| / k is measured for k=1 to 20:

  • GRPO and RLSD retain ~80% overlap even at k=20 — the fine-tuned model's top candidates are largely the same tokens the base model already preferred, just re-weighted.
  • RLRT's overlap drops to ~50% at k=20 — half of the top candidates are tokens the base model did not surface in its top-20. This indicates that RLRT is introducing genuinely new token choices into the top positions, not merely re-ranking the existing set.

Figure 7c: How extreme are the new top-1 candidates? The fraction of high-divergence positions where the fine-tuned model's new top-1 token had base probability below a threshold is reported:

  • For the strictest threshold (base probability < 10⁻⁴): RLRT promotes such extreme-tail tokens to top-1 at approximately 12% of high-divergence positions, compared to ~1% for GRPO and ~0.5% for RLSD — over 10× as often.
  • Across all thresholds (10⁻⁴, 10⁻³, 10⁻², 10⁻¹), RLRT promotes tail tokens more frequently than either baseline, with the gap widening as the threshold tightens.

This triple analysis paints a consistent picture: GRPO and RLSD primarily re-weight the base model's existing preferences, making confident predictions more confident without changing what the model considers. RLRT restructures the candidate set itself, pulling tokens the base model assigned near-zero probability into top positions. This is direct quantitative evidence that RLRT induces exploration that is not just "more of the same with noise" but genuinely different reasoning behavior. The paper's claim that RLRT produces "genuinely new behavior" (Section 6.3) is thus grounded in distributional measurements, not just downstream accuracy.

The analysis uses only 30 hard prompts from the training distribution, so it characterizes in-distribution shift rather than generalization. It also focuses on a single Qwen3-8B-Base checkpoint, so the degree to which this pattern generalizes across model sizes and types is not established. Nevertheless, the stark qualitative difference — RLRT's distribution-shift pattern is fundamentally unlike GRPO's or RLSD's — provides strong corroboration that the reversed teacher signal is doing something categorically different from standard credit assignment or self-distillation.

Ablation Studies and Robustness Checks

RLRT without reward gating (RLRT-all): Removing the r=1 gate — applying the reverse weight to all rollouts regardless of correctness — causes training to collapse. Figure 9a shows this on Qwen3-4B-Instruct: RLRT-all initially tracks RLRT in training score before diverging around step 40, while response length and actor entropy grow unbounded. The paper diagnoses this as the reverse weight "reinforcing teacher-divergent tokens on failed rollouts, conflating valuable exploration with spurious divergence" (Section 6.5). This ablation establishes that the correctness gate is not an optimization detail but a necessary structural condition: self-driven tokens are only valuable when verified by success. Without the gate, the policy chases its own divergence from the teacher regardless of outcome, destabilizing training.

Clipping range ε_w: The sweep over ε_w ∈ {0.2, 0.5, 1.0} on Qwen3-4B-Base and Qwen3-8B-Base (Figure 9b) shows:

  • Tighter clipping (ε_w=0.2) keeps the reverse weight near unity and produces training scores close to GRPO, confirming that the reweighting mechanism — not just the correctness filter — drives RLRT's gains.
  • Looser clipping (ε_w=1.0) yields the highest training scores on both backbones, with ε_w=1.0 > 0.5 > 0.2 throughout training.
  • The optimal ε_w varies by model type (Section 6.5 and Table 5): 1.0 for base models, 0.5 for instruction-tuned, 0.2 for thinking-tuned. The paper attributes this to base models having "more room for the reverse weight to reshape the distribution" before destabilization, while tuned models are more sensitive to large per-token weight deviations.

Mixing coefficient λ schedule: The paper uses λ_init=0.5 for all methods and sweeps decay schedules (Table 5): no decay for base models, linear decay over 30 steps for instruction-tuned and thinking-tuned models. This choice is not directly ablated, but the fact that decay is needed for tuned models suggests that the reverse weight is most impactful early in training when the policy is still adjusting, and maintaining it at full strength can be destabilizing for already-tuned policies.

Entropy and response length monitoring (Figure 9a): The training curves include actor entropy and response length as diagnostic metrics, not just training reward. RLRT's entropy and length remain stable and comparable to GRPO's throughout training (the paper does not report the exact values, but Figure 9a shows RLRT tracking closely with GRPO on these metrics). This is an important sanity check: RLRT's performance gains are not driven by trivial length exploitation (producing longer correct responses that happen to match the verifier) or entropy collapse (overfitting to a narrow set of tokens).

SDPO/SRPO collapse on base models (Appendix F.2, Figure 11): The paper documents that SDPO collapses on Qwen3-4B/8B-Base: training reward drops to zero within 20 steps while response length diverges compared to GRPO. Figure 11 shows this pattern on Qwen3-8B-Base. The paper conjectures that SDPO's excessive suppression of hedging and reflective tokens ("wait," "hmm") is the cause, citing Kim et al. (2026). This negative result provides context for why SDPO and SRPO are excluded from base model comparisons — their failure mode is diagnostic of a fundamental misalignment between uniform self-distillation and mathematical reasoning in untuned models.

Critical Assessment

The experiments provide strong evidence for RLRT's core mechanism — that amplifying self-driven tokens on correct rollouts improves RLVR training — but several qualifications are necessary to interpret the claims precisely.

Claim: "RLRT substantially outperforms self-distillation and exploration-based baselines." This is the most robust finding in the paper. The gains are consistent across four backbones (Tables 1 and 3), six benchmarks (different difficulty levels from MATH500 to AIME), and two evaluation metrics (avg@16 and pass@16). The pass@k curves (Figure 8) extend this to a broader range of sample budgets, showing stochastic dominance over exploration baselines. The training reward curves (Figure 5) show consistent speed-of-learning advantages. The distribution-shift analysis (Figure 7) provides mechanistic evidence that RLRT's policy is qualitatively different — not just a better-tuned version of GRPO. This multi-angle evidence makes the overall performance claim strong.

However: the gains vary substantially by backbone. On Qwen3-4B-Base, the improvement is dramatic (18.0% mean improvement); on the thinking-tuned Qwen3-8B, it is modest (2.2%). The paper's interpretation — that RLRT's exploration signal is most valuable when the policy is not yet concentrated — is plausible but implies that RLRT's practical value is largest for base model training, where pretraining is the dominant cost anyway. For already-capable models, the marginal benefit is smaller, and the additional forward pass per correct rollout (for teacher computation) may not justify the gain in production pipelines where inference cost dominates.

Claim: "Information asymmetry is a principled, intrinsic source of valuable exploration." The theoretical apparatus (Lemma 1, Theorem 2) provides a clean justification for why \bar{D}_t identifies causally critical positions, and the reflection injection experiment (Figure 6) provides causal validation. The distribution-shift analysis (Figure 7) shows that amplifying this signal produces policy changes that are structurally different from GRPO's — introducing new candidates rather than re-weighting existing ones.

However: the theoretical justification relies on modeling the teacher as π_θ(· | h_t, R=1) — conditioning on the event of success. In practice, the teacher is instantiated by appending a known correct solution c to the context, which is only an approximation to conditioning on R=1. The quality of this approximation is not analyzed: how well does π_θ(· | h_t, c) actually approximate π_θ(· | h_t, R=1)? If the teacher's distribution is dominated by surface-level patterns in the provided solution rather than the abstract fact of success, then \hat{D}_t may capture stylistic divergence rather than causally meaningful reasoning differences. The linguistic marker analysis (Figure 3) partially addresses this by showing that the explore/exploit tokens are semantically meaningful, but it does not quantify the gap between the Bayesian teacher ideal and the practical instantiation. This is an important theoretical loose end.

Claim: "RLRT induces exploration that is qualitatively different from diversity-based methods." The pass@k comparison against GRPO+EB and DIVER (Figure 8) supports that RLRT explores more effectively, but the diversity methods are tested in limited configurations:

  • Only one entropy bonus coefficient is shown (the paper does not specify which value; it simply says "GRPO+EB"). Entropy bonus strength is a critical hyperparameter — if too low, it has no effect (as shown); if too high, it can destabilize training. Without a sweep, the claim that token-level entropy is ineffective is based on a single data point.
  • DIVER is run with the authors' default hyperparameters on DAPO-Math-17k, but DIVER was originally designed and tuned for different benchmarks. Performance may be sensitive to its diversity coefficient and semantic embedding model.
  • Neither GRPO+EB nor DIVER was tested on the instruction-tuned or thinking-tuned backbones, so the exploration comparison is restricted to Qwen3-8B-Base. It is possible that diversity-based methods are more effective on already-tuned models where the base policy already has strong reasoning patterns.

Claim: "Self-distillation baselines degrade or collapse on mathematical reasoning." The paper provides clear evidence that SDPO collapses on base models (Figure 11) and that SRPO underperforms GRPO on the thinking-tuned model (Table 1, bottom). However, the interpretation — that this reflects fundamental misalignment between self-distillation and math reasoning — is confounded by potential implementation issues:

  • SDPO uses a specific logit-level KL divergence implementation and an EMA update rate of 0.0 (Table 5). The original SDPO paper may recommend different settings; the paper does not discuss whether the collapse is specific to these hyperparameters or robust across configurations.
  • SRPO's entropy-aware dynamic weighting coefficient β was swept over {0, 0.5, 1.0} (Table 5), but this is a narrow sweep. The original SRPO paper may use different β values or schedules.
  • RLSD, the closest comparator to RLRT, is competitive but consistently below RLRT. However, RLSD uses the ground-truth answer as teacher context, while RLRT uses a correct rollout — these are different forms of privileged information, making the comparison not purely about the reversal direction but also about the teacher's information content. A more controlled comparison would use the same teacher context (correct rollout) for RLSD, which the paper does not report.

Missing experiments that would strengthen the paper:

  1. Difficulty-stratified results. The paper does not break down benchmark performance by problem difficulty. Given that RLRT's mechanism targets exploration on correct rollouts, its benefit should depend on how often the model already succeeds (base pass@1). On very hard problems where the base model rarely succeeds, RLRT has few correct rollouts to apply its reverse weight to, and the method may reduce to GRPO. On very easy problems where the model almost always succeeds, the reverse weight may have little room to amplify (since the base model already produces correct answers). The paper's finding that gains are largest on base models (which have lower base accuracy) and smaller on thinking-tuned models (higher base accuracy) partially reflects this, but a direct difficulty analysis would clarify the operating regime.

  2. Comparison with stronger exploration methods at matched budgets. DR. GRPO (Liu et al., 2025) and R1-S (Zelikman et al., 2022) are not included. These are more established exploration baselines than DIVER (which was published in 2025 and may not yet be widely validated). Including them would provide a stronger test of RLRT's exploration claim.

  3. Varying the teacher context. RLRT uses a correct rollout from the student's own sampling batch as teacher context. What if the teacher context is a ground-truth solution (as in RLSD), or a solution from a stronger model, or a partial hint? The paper's theoretical framework is agnostic to the form of privileged context, so testing robustness to this choice would strengthen the claim that information asymmetry per se — not the specific instantiation — is the active mechanism.

  4. Scaling the group size K. All experiments use K=8 rollouts per prompt. RLRT's reverse weight is most informative when some rollouts are correct and some are incorrect (so that the advantage sign separates them). If K is very small (e.g., 2), most groups will be all-correct or all-incorrect, and the group-standardized advantage may behave differently. If K is very large, the teacher forward pass cost per rollout grows. How sensitive is RLRT to K? This is not ablated.

  5. Wall-clock and FLOP overhead. The paper mentions that RLRT requires one additional teacher forward pass per correct rollout and that training takes "approximately one day" for base models on 2×B200 GPUs (Appendix G), but does not report the exact overhead relative to GRPO. If the teacher pass adds 50% to training time, the pass@k curves should be compared at matched wall-clock time, not matched training steps. The current comparison is at matched steps, which is generous to RLRT.

  6. Confidence intervals or statistical testing. None of the benchmark results include error bars, standard deviations, or significance tests. With 500 test questions for MATH500 and 30 each for AIME (approximate — the paper doesn't specify the per-benchmark sizes), the per-benchmark differences of 1–5 percentage points may fall within sampling noise for the smaller benchmarks. The consistency across six benchmarks mitigates this but does not eliminate it. Reporting standard errors or running bootstrap tests would provide necessary statistical rigor.

Overall assessment. The experiments strongly support the paper's central claim that reversing the teacher signal on correct rollouts improves RLVR training across diverse model configurations. The evidence is multi-dimensional — downstream benchmarks, pass@k curves, training dynamics, causal interventions, and distribution-shift analysis — and internally consistent. The theoretical framework (Lemma 1, Theorem 2) provides a principled justification that elevates RLRT beyond a heuristic trick. The most convincing single result is the reflection injection experiment (Figure 6), which provides direct causal evidence that \bar{D}_t identifies causally important positions and that RLRT's training amplifies their effect. The most important qualification is that RLRT's benefit is largest on base models (where it matters most for pretraining-cost reduction) and modest on already-capable thinking-tuned models, suggesting that the method is best understood as an exploration mechanism for models that have not yet converged to strong reasoning policies — which is precisely the regime where effective exploration is most valuable.

6. Limitations and Trade-offs

The Theoretical Guarantee Depends on an Unvalidated Approximation

The paper's formal justification for why \bar{D}_t identifies causally important positions rests on modeling the teacher as π_θ(· | h_t, R=1) — the student conditioned on the abstract event of success (Section 4.2, Lemma 1). In practice, the teacher is instantiated by appending a concrete correct solution c to the context, so that P_T^t(·) = π_θ(· | h_t, c). The paper acknowledges this gap implicitly (Section 4.3):

"While the analysis above defines the teacher through the abstract event R = 1, this event cannot be conditioned on directly. In practice, we realize the teacher by feeding a known correct solution c as the conditioning context, so that P_T^t(·) = π_θ(· | h_t, c) serves as one instantiation of π_θ(· | h_t, R=1)."

The consequence: The quality of this approximation is never quantified. The teacher conditioned on a specific correct solution may diverge from the idealized Bayesian teacher in two ways. First, the provided solution c may follow a particular reasoning style (e.g., algebraic rather than geometric, or direct rather than case-analysis), causing the teacher to favor tokens that reflect that specific style rather than correctness in general. In that case, \hat{D}_t > 0 could capture stylistic divergence (the student used a different but equally valid approach) rather than self-driven reasoning that departed from what a correctness-conditioned model would predict. Second, surface-level patterns in c (specific variable names, notational conventions, LaTeX formatting) could dominate the teacher's token-level predictions, making \bar{D}_t large at positions that are not causally consequential but merely stylistically divergent. Theorem 2's guarantee — that \bar{D}_t ≈ 0 implies Inf_S(t) ≈ 0 — only holds under the Bayesian teacher model; if the practical teacher deviates substantially, the contrapositive (large \bar{D}_t → causally important position) weakens or breaks.

What evidence exists: The paper provides indirect validation through the linguistic marker analysis (Figure 3, Appendix D) and the reflection injection experiment (Figure 6), both of which show that \hat{D}_t captures semantically meaningful distinctions and that \bar{D}_t correlates with causal importance. However, neither experiment quantifies the gap between the idealized Bayesian teacher and the practical rollout-conditioned teacher. The linguistic analysis (Figure 3) shows that explore-leaning tokens are reasoning-openers ("wait," "consider") and exploit-leaning tokens are reasoning-closers ("conclude," "final") — this is consistent with \hat{D}_t capturing something meaningful, but it does not establish that the practical teacher approximates π_θ(· | R=1) closely enough for Theorem 2's bound to be tight in practice. The reflection injection experiment (Figure 6) shows that \bar{D}_t correlates with causal influence on the base checkpoint (flip→R at max_kl is ~18% vs. ~9% at random), confirming that the correlation exists, but not whether a better teacher instantiation would yield a stronger or more reliable signal.

Mitigation status: The paper does not address this gap. There is no ablation varying the form of teacher context (e.g., ground-truth solution vs. successful rollout from a different model vs. partial hint), no measurement of how closely π_θ(· | h_t, c) matches the idealized π_θ(· | h_t, R=1), and no analysis of whether RLRT's gains are sensitive to the specific rollout chosen as teacher context. The theoretical machinery relies on an equivalence that is assumed rather than verified. The authors acknowledge in Appendix A that "varying the form of privileged information given to the teacher" is a natural direction for future work, but the current results do not establish robustness to this choice.

Difficulty Estimation Is Implicit, Unmeasured, and Potentially Circular

RLRT's mechanism depends on having correct rollouts to apply its reverse weight — the correctness gate r=1 is the necessary structural condition identified in Section 6.5. This means RLRT's benefit is fundamentally tied to the base model's success rate on the training distribution: if the base model rarely produces correct rollouts for a class of problems, RLRT has few trajectories to extract its self-driven reasoning signal from, and the method reduces to vanilla GRPO on those problems.

The consequence: This creates an implicit difficulty-dependence that the paper does not characterize. On very hard problems where the base model's pass@1 is near zero, RLRT offers essentially no advantage over GRPO because there are almost no correct rollouts to apply the reverse weight to. On very easy problems where the base model almost always succeeds, RLRT's reverse weight has ample signal — but the base model already succeeds, so the marginal value of further improvement is limited. The operating regime where RLRT provides the largest gain is the intermediate zone: problems where the model sometimes succeeds and sometimes fails, so that there are enough correct rollouts to identify self-driven reasoning patterns, but enough failures that improving the success rate matters. This regime-dependence is structurally baked into the method but never quantified.

What evidence exists: The paper's results provide indirect but consistent evidence. The gains are largest on base models (18.0% mean improvement on Qwen3-4B-Base, 12.0% on Qwen3-8B-Base) where the starting pass@1 is low (e.g., AIME24 pass@1 ≈ 9.6% for Qwen3-4B-Base, Table 1) but not zero — the model generates some correct rollouts to learn from. Gains shrink on instruction-tuned models (3.4% on Qwen3-4B-Instruct) where baseline performance is much higher (AIME24 avg@16 = 64.4%), and shrink further on thinking-tuned models (2.2% on Qwen3-8B) where baseline performance is highest (AIME24 avg@16 = 70.2% after GRPO). The paper interprets this pattern through the lens of policy concentration — "RLRT's exploration signal is most effective when the policy has not yet been concentrated by instruction tuning" (Section 6.1) — but an equally valid interpretation is that base models sit in the sweet spot of moderate success rate where the reverse weight has both signal (some correct rollouts exist) and room for improvement (many rollouts are still incorrect).

The paper does not report difficulty-stratified benchmark results or training dynamics (e.g., reward curves split by problem difficulty). We do not know whether RLRT's gain on AIME24 comes primarily from problems the base model could sometimes solve (intermediate difficulty) or from problems it almost never solved (hard), or whether the benefit is uniform across the difficulty spectrum. The absence of this analysis is a gap given that the method's theoretical mechanism predicts differential effectiveness by difficulty.

Mitigation status: The paper does not address difficulty dependence explicitly. The four-backbone sweep (Section 6.1) implicitly varies the base success rate, but this conflates difficulty with model type (base vs. instruct vs. thinking-tuned) and model scale (4B vs. 8B). A direct difficulty analysis — splitting DAPO-Math-17k or the evaluation benchmarks into difficulty tiers and reporting RLRT's gain per tier — would clarify the operating regime. The paper's broader implication that "information asymmetry is a principled source of valuable exploration" would be strengthened by knowing the boundary conditions where that signal is available versus absent.

The Computational Overhead Is Unquantified, and the Headline Gains Are at Matched Steps, Not Matched Compute

RLRT requires an additional forward pass through the model for every correct rollout to compute teacher probabilities P_T with privileged context appended. The paper does not report the exact overhead relative to GRPO, making it impossible to assess whether the reported gains persist under matched-compute comparison.

Training a model with 8 rollouts per prompt: if, say, 3 rollouts are correct on average, RLRT requires 8 + 3 = 11 forward passes per prompt vs. GRPO's 8 — a ~38% increase in training FLOPs per step. The paper mentions in Appendix G that training base models "takes approximately one day, whereas Qwen3-4B-Instruct and Qwen3-8B require 2–3 days" on 2×B200 GPUs, but this is an aggregate wall-clock estimate that does not separate RLRT's overhead from GRPO's baseline training time. Since instruction-tuned and thinking-tuned models likely have higher base success rates (more correct rollouts), their teacher-pass overhead would be proportionally larger — which may partially offset RLRT's smaller gains on those backbones.

The consequence: The headline benchmark comparisons (Table 1, Figure 8) and training curves (Figure 5) are all at matched training steps, not matched FLOPs or wall-clock time. If RLRT requires 30–50% more compute per step due to teacher forward passes, then a fair comparison would give GRPO proportionally more training steps. Whether RLRT would still outperform GRPO under matched-compute is unknown. For the pass@k exploration comparison (Section 6.4, Figure 8), this matters even more: RLRT's stochastic dominance over DIVER and GRPO+EB is reported at matched steps, but if those methods had the extra compute budget RLRT consumes, their pass@k curves might shift upward.

The training curves in Figure 5 show RLRT achieving higher reward faster (by step 100, RLRT is ahead of GRPO on all four backbones), which partially mitigates the concern — RLRT might reach a given performance level in fewer steps, even if each step is more expensive. But the paper never reports the step at which RLRT matches GRPO's final performance, which is the relevant metric for compute-efficiency: does RLRT reach GRPO's step-100 reward at step 70? Step 50? Without this, the speed-of-learning advantage is suggestive but not dispositive.

What evidence exists: The paper provides no direct measurement of FLOP overhead or wall-clock comparison. The only relevant data point is the aggregate training time statement in Appendix G, which does not decompose RLRT's cost. The teacher-pass cost depends on the proportion of correct rollouts, which changes over training (as the policy improves, more rollouts become correct, increasing the overhead). This dynamic pattern is not reported.

Mitigation status: The paper does not address this limitation. There is no FLOP-matched comparison, no measurement of per-step computation for RLRT vs. GRPO, and no analysis of whether the same trends hold under matched compute. This omission is significant because RLRT's conceptual contribution — that information asymmetry is a valuable signal — is independent of the computational cost, but the practical claim that RLRT "substantially outperforms" baselines is incomplete without a cost accounting. A practitioner deciding whether to adopt RLRT needs to know: does the extra teacher forward pass pay for itself in reduced training time to reach a target performance? The current results cannot answer this.

The Exploration Baseline Comparison Is Narrow and Lacks Sensitivity Analysis

Section 6.4 compares RLRT against two exploration methods — GRPO+entropy bonus and DIVER — on Qwen3-8B-Base using pass@k curves on AIME24 and AIME26. From this, the paper concludes that RLRT induces "genuinely broader coverage across reasoning modes rather than within one" (Section 6.4) and that diversity-based methods are less effective. This conclusion overreaches the experimental evidence in three respects.

First, hyperparameter sensitivity is unexplored. GRPO+entropy bonus is tested with a single, unspecified entropy coefficient. The paper states only "GRPO with an entropy bonus (GRPO+EB) [3]" without reporting the coefficient value or whether any sweep was conducted. Entropy bonus strength is the critical hyperparameter for this method: too low, and the bonus has negligible effect (as Figure 8 shows — GRPO+EB tracks GRPO almost exactly); too high, and it can destabilize training or force the policy toward uniform distributions. Without a sweep, the conclusion that entropy bonuses are ineffective is based on what could be a poorly tuned instance of the method. DIVER is tested with its default hyperparameters on DAPO-Math-17k, but DIVER was originally designed and tuned for a different set of benchmarks and models; its diversity coefficient and semantic embedding model may not transfer optimally.

Second, the exploration baselines are tested on only one backbone. GRPO+EB and DIVER are evaluated only on Qwen3-8B-Base (Section 6.4, Figure 8). The instruction-tuned and thinking-tuned Qwen3 checkpoints are not included in the exploration comparison. This matters because the paper's own results show that base models have the most room for exploration (RLRT's gains are largest there), while tuned models are already concentrated. It is possible that diversity-based methods are more competitive on instruction-tuned or thinking-tuned models, where the base policy already has strong reasoning and exploration needs only to broaden rather than discover. Without testing on these backbones, the paper's claim about exploration effectiveness is restricted to one model type.

Third, the comparison set is thin. The paper frames its exploration contribution as a new paradigm — "valuable exploration" grounded in information asymmetry rather than uniform diversity — but compares against only two diversity methods, one of which (GRPO+EB) is arguably the weakest possible exploration baseline (a scalar bonus on entropy that is known to provide marginal benefits in RLVR). Stronger exploration methods from the literature — such as DR. GRPO (Liu et al., 2025), which uses KL regularization to prevent policy collapse, or R1-S (Zhao et al., 2025), which employs rejection sampling from a stronger model — are not included. The paper also does not compare against variants that combine diversity objectives with outcome filtering, which would be the closest conceptual analogue to RLRT's approach of amplifying verified self-driven tokens.

The consequence: The paper's claim to establish a new exploration paradigm requires showing superiority over representative methods from the existing paradigm. The current evidence shows RLRT > one trivial baseline (entropy bonus) and one recent method (DIVER) on one backbone — this is suggestive but not definitive. A practitioner comparing exploration methods would want to see RLRT against the strongest available diversity-based approach, ideally with hyperparameter sweeps for fairness.

Mitigation status: The paper does not acknowledge these limitations of the exploration comparison. The text in Section 6.4 presents the pass@k dominance as evidence that RLRT's exploration is categorically better without the qualifications above. Appendix G.1 notes that DIVER was run with "the same DAPO-Math-17k corpus and hyperparameters as the other baselines," but there is no discussion of whether those hyperparameters are optimal for DIVER or whether the comparison is fair. The paper does suggest future work on "adaptive routing between teacher-guided and self-driven updates" (Appendix A) but does not address the narrowness of the exploration baseline set as a limitation of the current study.

Self-Distillation Baselines May Not Be Optimally Configured, Confounding the Comparison

The paper compares RLRT against three self-distillation methods — SDPO, SRPO, and RLSD — and finds that RLRT consistently outperforms them. However, several baselines exhibit pathological behavior that may reflect suboptimal hyperparameter configurations rather than fundamental weaknesses of the methods.

SDPO and SRPO collapse on base models. Appendix F.2 (Figure 11) shows SDPO's training reward crashing to zero within 20 steps on Qwen3-8B-Base while response length diverges. The paper attributes this to "excessive suppression of hedging and reflective tokens" (Section 6.1), citing Kim et al. (2026). SRPO, which routes correct rollouts to GRPO and incorrect rollouts to SDPO-style distillation, performs "even worse than full self-distillation on math" on the thinking-tuned Qwen3-8B (Table 1, bottom: SRPO avg@16 = 15.4% on AIME24 vs. SDPO's 26.9% and GRPO's 70.2%). The paper conjectures that "self-distillation and GRPO promote different reasoning styles, leading to conflicting gradients" (Section 6.1).

However, SDPO was originally demonstrated to work effectively on reasoning tasks (Hübotter et al., 2026), and SRPO was specifically designed to address optimization ambiguity in self-distillation (Li et al., 2026). Their collapse in this paper's experiments might reflect hyperparameter choices rather than fundamental incompatibility with mathematical reasoning. The paper uses specific settings for SDPO: Jensen-Shannon divergence (α=0.5), top-K distillation of 100, EMA update rate of 0.0, and learning rate 1×10⁻⁵ (Table 5). The original SDPO paper may recommend different configurations — the EMA update rate being zero, in particular, means the teacher distribution is not stabilized by exponential moving averaging, which could contribute to training instability. The paper does not report whether alternative SDPO configurations were attempted on the base models.

RLSD uses a different teacher context than RLRT. RLSD conditions its teacher on the ground-truth answer, while RLRT conditions its teacher on a correct rollout from the student's own sampling batch (Table 4). These are fundamentally different forms of privileged information. The ground-truth answer provides the correct final result but no reasoning trace; the correct rollout provides reasoning steps and intermediate conclusions. The comparison between RLRT (teacher = correct rollout, weight = (P_S/P_T)^{sign(A)}) and RLSD (teacher = ground-truth answer, weight = (P_T/P_S)^{sign(A)}) thus confounds two differences: (1) the direction of the weight (reversed vs. standard) and (2) the form of teacher context. It is possible that giving RLSD the same correct-rollout teacher context (and keeping its standard weight direction) would close or reverse the gap. The paper does not report this controlled comparison.

RLSD hyperparameters are set at their best (ε_w = 0.2), but this may be suboptimal for different teacher contexts. The paper sweeps ε_w ∈ {0.2, 0.5, 1.0} for RLSD and finds that ε_w = 0.2 is consistently best, "with larger values degrading performance below GRPO" (Appendix G.2). This is the setting used in all RLSD comparisons. However, this sweep is only reported in aggregate — we do not know whether ε_w = 0.2 is optimal across all four backbones or whether the optimal ε_w for RLSD varies by model type as it does for RLRT (1.0 for base, 0.5 for instruct, 0.2 for thinking-tuned, per Table 5). If RLSD's optimal ε_w varies by backbone but was clamped to 0.2 throughout, the comparison is biased in RLRT's favor on base models.

The consequence: The strong claim that "RLRT substantially outperforms self-distillation baselines" is supported by the raw numbers but weakened by the possibility that those baselines are not operating at their best. The pathological collapse of SDPO and SRPO on base models means those comparisons are largely uninformative — they test methods that are clearly broken in this configuration. The RLSD comparison is the most relevant (it shares the same per-token weighting interface), but the confounded teacher context prevents attributing the performance gap specifically to the signal reversal rather than to the different information available to the teacher.

Mitigation status: The paper acknowledges SDPO's collapse in Appendix F.2 and attributes it to token suppression, but does not discuss whether alternative hyperparameters might rescue the method. The teacher context difference between RLSD and RLRT is documented in Table 4 but not discussed as a confound. The paper does not attempt the controlled comparison (RLSD with correct-rollout teacher, or RLRT with ground-truth answer teacher) that would isolate the reversal effect. This limits the strength of the conclusion that the reversal — as opposed to the richer teacher context — is the active ingredient.

Only Mathematical Reasoning Is Tested, and the Signal May Not Transfer

All experiments are conducted on mathematical reasoning: training uses DAPO-Math-17k, and evaluation uses six math competition benchmarks (AIME, HMMT, AMC, MATH). The paper does not test RLRT on code generation, logical reasoning, scientific QA, or any non-math domain.

The consequence: Mathematical reasoning has properties that may make the reversed teacher signal particularly informative: solutions have clear right/wrong answers (clean verifiable reward), follow structured multi-step derivations (making the explore/exploit distinction at step transitions meaningful), and the teacher's privileged context (a correct solution) provides step-by-step reasoning that strongly shapes token-level predictions at causally important junctures. In domains where these properties are weaker — open-ended generation with no clean correctness signal, tasks where reasoning is implicit rather than explicit, or problems where the teacher context provides only the final answer without intermediate steps — the information asymmetry signal may be less informative or even misleading. The linguistic marker analysis (Figure 3) identifies explore-leaning tokens like "wait," "consider," "another" — these are discourse markers specific to chain-of-thought mathematical reasoning. In code generation, the analogous exploration tokens might be different (e.g., "refactor," "alternatively," "try-except" patterns), and the paper provides no evidence that \hat{D}_t captures them.

The paper's broader claim — that "information asymmetry is a principled, intrinsic source of valuable exploration" establishing "a new design axis for RLVR" — implicitly assumes domain-generality. If the signal works only for mathematical reasoning (or only for tasks with explicit step-by-step derivations), this claim overstates the contribution.

What evidence exists: None outside math. The paper acknowledges this in Appendix A: "the experiments are restricted to mathematical reasoning," listing extension to other domains as future work. There is no discussion of what properties of mathematical reasoning might make the signal particularly informative, and no analysis of whether the explore/exploit decomposition (Figure 3) would look different in a non-math domain.

Mitigation status: The paper is transparent about the domain restriction in Appendix A and frames it as a limitation, which is appropriate. However, the main text's claims about "valuable exploration" and "new design axis" are not qualified by domain, creating a mismatch between the strength of the conceptual claims and the narrowness of the empirical support. A more conservative framing would characterize RLRT as a method demonstrated on mathematical reasoning, with the extension to other domains as an open question rather than an assumed property of the framework.

7. Implications and Future Directions

How This Work Changes the Landscape

RLRT introduces a conceptual reframing rather than a new architecture or loss function. Its core move — reading the teacher–student probability gap in reverse on correct rollouts — does not require new infrastructure, new data, or even new model components. It requires only that practitioners stop interpreting $P_S > P_T$ as error and start interpreting it as evidence of the student's own successful reasoning. This is a shift in how we think about self-distillation, not just how we implement it.

The magnitude of the reframing is substantial but bounded. It does not overturn self-distillation as a paradigm — RLRT still uses the same teacher–student setup, the same privileged context, the same per-token reweighting interface. What changes is the teleology: the teacher is no longer an authority to be imitated but a contrastive reference point whose value lies in being departed from on successful trajectories. This is a genuinely new perspective on what the teacher–student gap means, and it has consequences beyond RLRT itself. Any self-distillation method that applies uniform alignment (SDPO pulling $P_S \to P_T$ on all rollouts; RLSD up-weighting teacher-favored tokens everywhere) is now vulnerable to the critique that it suppresses self-driven reasoning on correct trajectories. The paper's demonstration that SRPO — which routes correct rollouts away from distillation — performs worse than full self-distillation (Figure 5, Qwen3-8B panel; Table 1, SRPO 15.4% vs. SDPO 26.9% on AIME24) while RLRT's reversal improves performance establishes that the active ingredient is not merely avoiding suppression but actively amplifying self-driven tokens. This is a sharper claim than "distillation on correct rollouts is harmful" — it asserts that the signal on correct rollouts is directionally valuable when read in reverse.

A second reframing concerns exploration in RLVR. Prior work treats reasoning boundary collapse as a sampling problem: the model does not generate enough diverse candidates, so inject entropy or diversity. RLRT's implicit claim is that the problem is better understood as a credit-assignment problem for diversity. The model already samples at temperature 1.0 and produces varied rollouts; what it lacks is not diversity but a signal about which deviations from its default behavior were productive. The teacher–student gap, gated on correctness, provides exactly this signal — it localizes the tokens where the student departed from the teacher and succeeded, and amplifies them directly in the gradient. This reframes exploration from a uniformity target (make everything more diverse) to a grounding problem (identify and reinforce the specific diversity that worked). The consequence is that uniform exploration methods lose their rationale: if the bottleneck is not sampling breadth but credit assignment, then entropy bonuses and diversity objectives are pushing in the wrong place. Figure 8 provides evidence for this interpretation — GRPO+entropy bonus barely deviates from vanilla GRPO on pass@k curves, while RLRT's targeted amplification yields a stochastically dominant curve. This does not mean diversity methods are useless in all contexts, but it suggests that in math RLVR specifically, directed exploration grounded in verified success is more compute-efficient than undirected diversity.

The paper also reconciles contradictory signals in the self-distillation literature. Prior work was split: some studies found self-distillation beneficial (Zhao et al., 2026; Hübotter et al., 2026; Yang et al., 2026), while others found it degraded reasoning (Li et al., 2026; Kim et al., 2026). RLRT's framework provides a unified explanation: self-distillation does provide useful corrective signal on failed rollouts (where the teacher points toward solutions the student could not reach), but also suppresses self-driven reasoning on correct rollouts. Whether the net effect is positive or negative depends on the relative weight of these two forces, which in turn depends on the base model's reasoning style, the difficulty distribution, and the specific distillation implementation. Base models with weak reasoning rely more on self-driven exploration and are thus more harmed by suppression (explaining SDPO's collapse in Figure 11); thinking-tuned models already have crystallized reasoning patterns and are less affected (explaining SDPO's smaller but still negative effect in Table 1, bottom). RLRT's reversal resolves the contradiction by turning the previously harmful signal (suppression on correct rollouts) into a beneficial one (amplification of self-driven tokens), aligning the direction of update with the value of the divergence.

A specific research direction that becomes less attractive in light of this work is the development of ever-finer-grained entropy or diversity objectives for RLVR. If the exploration bottleneck is not sampling breadth but credit assignment — and RLRT's pass@k dominance over entropy bonuses and DIVER (Figure 8) suggests it is — then incremental improvements to diversity injection (new divergence measures, new semantic similarity metrics, new temperature schedules) are unlikely to yield transformative gains. The more promising axis, which RLRT opens, is developing better signals for identifying valuable exploration — different teacher configurations, different forms of privileged context, different gating mechanisms, or adaptive policies that switch between self-driven amplification and teacher-guided correction depending on context. This shifts research attention from "how do we make the model try more things?" to "how do we recognize which of the things it tries are worth keeping?"

Follow-Up Research This Work Enables

Stress-testing the Bayesian teacher idealization by varying teacher context quality. Lemma 1 and Theorem 2 model the teacher as $\pi_\theta(\cdot | h_t, R=1)$ — conditioning on the abstract event of success. In practice, RLRT instantiates this by appending a correct rollout as privileged context. The gap between the idealized Bayesian teacher and the practical rollout-conditioned teacher is unquantified. A natural experiment: vary the teacher context along a spectrum of informativeness about correctness — (a) a correct rollout from the student's own batch (current RLRT), (b) a ground-truth solution with full reasoning steps, (c) only the final correct answer without reasoning, (d) a correct solution from a stronger model (e.g., Qwen3-32B), (e) a partial hint (e.g., "use case analysis"). If RLRT's gains are driven by the theoretical property that $\bar{D}_t$ bounds causal influence (Theorem 2), then more informative teacher contexts should yield stronger $\bar{D}_t$ at causally important positions and larger RLRT gains. If the gains are driven by surface-level divergence from the specific rollout text, then teacher context quality should have weak or inconsistent effects, and the theoretical framing is less empirically grounded than claimed. A strong follow-up would measure both downstream benchmark performance and the correlation between $\bar{D}_t$ and $\text{Inf}_S(t)$ (via reflection injection or similar causal probe) across teacher context variants, establishing whether the Bayesian teacher model is a useful approximation or merely a post-hoc justification.

Combining RLRT's reversed signal with RLSD's standard signal via adaptive routing. The paper compares RLRT (reverse weight, correct rollouts only) against RLSD (standard weight, all rollouts), but the two signals are not mutually exclusive. On correct rollouts, RLRT's reversal amplifies self-driven tokens; on incorrect rollouts, RLSD's standard weighting suppresses self-driven tokens and reinforces teacher-aligned ones (since $w_t^{\text{RLSD}} = (P_T/P_S)^{\text{sign}(A)}$ with $\text{sign}(A) < 0$ means tokens where $P_S > P_T$ get $w_t < 1$, attenuating the negative advantage). A natural hybrid: apply RLRT on correct rollouts and RLSD on incorrect rollouts as a unified algorithm that uses the teacher–student gap directionally — amplify self-driven on success, suppress self-driven on failure. This would treat the gap symmetrically, using its sign to route the update direction based on outcome. The paper's SRPO baseline attempted a cruder routing (GRPO on success, SDPO-style KL on failure) and collapsed, but that used a different loss entirely. A unified routing within the per-token weighting framework would be cleaner. A strong experiment would compare the hybrid against pure RLRT and pure RLSD across the four-backbone sweep, measuring whether the incorrect-rollout suppression provides additional gains or whether it introduces conflicting gradients (since the same token type — self-driven — gets amplified on correct rollouts and suppressed on incorrect ones, creating opposing signals that may cancel). The paper's negativity about SDPO/SRPO and the correctness gate ablation (Figure 9a, showing unbounded entropy when the gate is removed) suggests the hybrid might be unstable, but testing this directly would clarify the boundary between RLRT's successful asymmetry and the failure of uniform self-distillation.

Measuring and mitigating the computational overhead of teacher forward passes. RLRT's practical adoption depends on whether its teacher-pass overhead is justified by accelerated learning. The paper does not report FLOPs or wall-clock comparisons. A follow-up study should: (a) measure the exact per-step FLOP overhead of RLRT vs. GRPO as a function of the correctness rate (which changes over training, since improving policy → more correct rollouts → more teacher passes → higher overhead), (b) run matched-FLOP comparisons by giving GRPO proportionally more training steps, and (c) report the step at which RLRT reaches GRPO's final performance (the "crossover step"). If RLRT reaches GRPO step-100 reward at step 60 but each step costs 1.4× more, the net speedup is ~1.2× — a modest but real gain. If the crossover step is 90 with the same 1.4× overhead, RLRT is actually slower to reach a given performance despite higher per-step reward. The paper already has the training curves (Figure 5); what is missing is the x-axis rescaling to matched compute. A related experiment: use a smaller group size K (e.g., 4 instead of 8) for RLRT matched to a larger K (8) for GRPO, equalizing total forward passes per prompt. This would test whether RLRT's signal quality compensates for reduced sampling breadth.

Difficulty-stratified analysis of RLRT's mechanism. The paper's correctness gate means RLRT's signal is only active when the model produces correct rollouts. This creates an implicit difficulty-dependence: on problems where the base model rarely succeeds, RLRT reduces to GRPO. The paper never quantifies this. A straightforward follow-up: split DAPO-Math-17k and the evaluation benchmarks into difficulty tiers (by base model pass@1, as in Snell et al., 2024) and report RLRT's gain over GRPO per tier. The theoretical prediction is an inverted-U shape: zero gain on very hard problems (no correct rollouts to learn from), maximum gain on intermediate problems (some correct rollouts with room for improvement), and diminishing gain on very easy problems (already near-ceiling). This prediction is partially borne out by the backbone sweep — largest gains on base models (low starting accuracy) and smallest on thinking-tuned models (high starting accuracy) — but conflates difficulty with model type. A within-model difficulty analysis would cleanly test the mechanism. If RLRT shows gains on hard problems where the base model barely succeeds, that would suggest the reversed teacher signal extracts value from rare correct rollouts more efficiently than GRPO's uniform credit. If gains are concentrated in the intermediate tier, RLRT is best understood as an efficiency mechanism for the model's current capability range rather than a way to push the capability frontier.

Extending RLRT to off-policy teachers. The paper uses the same model as both student and teacher, making the teacher–student gap purely a function of the privileged context. This is clean but limiting: the teacher can only provide guidance that is latent in the student's own parameters. A natural extension is to use a separate, stronger teacher model (e.g., a larger Qwen3 variant or a model fine-tuned on high-quality reasoning data) and apply the same reversed signal: on correct student rollouts, amplify tokens where the student diverged from the stronger teacher. This tests whether RLRT's mechanism generalizes when the teacher–student gap reflects not just context asymmetry but genuine capability differences. A stronger teacher would produce larger $\hat{D}_t$ at the student's self-driven tokens (since the stronger model's predictions would diverge more from the weaker student's), potentially amplifying the exploration signal — but also increasing the risk of amplifying stylistic divergence rather than reasoning divergence. A converse experiment: use a weaker teacher and apply RLRT. If the student is stronger, the student's choices should generally be better than the teacher's, so $\hat{D}_t > 0$ tokens on correct rollouts represent the student outperforming the teacher. Amplifying these tokens should further improve the student, creating a self-reinforcing loop. This connects to the paper's suggestion of "iterative self-improvement loops" (Appendix A) and could be tested with a smaller Qwen3 variant as teacher for a larger one as student.

Diagnostic replication on non-math reasoning domains. The paper's claims about "valuable exploration" and the explore/exploit decomposition (Figure 3) are demonstrated exclusively on mathematical reasoning. Math has properties that may make $\hat{D}_t$ particularly informative: explicit multi-step derivations, clear step boundaries, and discourse markers that cleanly separate exploration ("wait, let's try another approach") from exploitation ("therefore, the answer is"). A critical stress-test: replicate RLRT on code generation (HumanEval, MBPP, or LiveCodeBench) and logical reasoning (FOLIO, ARC). In code, the explore/exploit distinction might manifest differently — exploration tokens might be "refactor," "alternatively," or "try-except"; exploitation tokens might be "return," "output," or "assert." The key question is not just whether RLRT improves downstream metrics (pass@k on HumanEval) but whether the $\hat{D}_t$ signal captures the same semantically meaningful explore/exploit distinction in a different domain. A negative result — RLRT shows no gain on code despite gains on math — would bound the domain-generality of the approach and suggest that RLRT's mechanism depends on the specific discourse structure of chain-of-thought mathematical reasoning. This would not invalidate the math results but would significantly qualify the paper's broader framing of information asymmetry as a general exploration principle.

Practical Applications and Downstream Use Cases

Post-training base models for math reasoning with improved sample efficiency. The clearest practical takeaway from this paper is that when fine-tuning a base LLM on mathematical reasoning with RLVR, replacing standard GRPO's uniform token-level credit with RLRT's reversed-teacher reweighting yields consistent improvements at no additional data cost and modest additional compute cost. The gains are largest precisely where they matter most — on untrained base models that lack reasoning capabilities. A team training a math-specialized model from a base checkpoint (e.g., Qwen3-4B-Base or a comparable model) can expect RLRT to reach a given MATH/AIME performance level in fewer training steps than GRPO, based on the faster training reward growth in Figure 5. The 18.0% mean improvement on Qwen3-4B-Base (Figure 1b) translates to, for example, AIME24 avg@16 improving from 15.0% to 22.5% (Table 1) — a 50% relative improvement — using the same DAPO-Math-17k training data and the same 8-rollout-per-prompt sampling budget. The implementation cost is low: RLRT requires only an additional forward pass per correct rollout to compute teacher probabilities, and the algorithm (Appendix B, Algorithm 1) is a minimal modification to any existing GRPO codebase. For organizations training math reasoning models on a budget, RLRT offers a drop-in improvement that does not require new data, new reward models, or new infrastructure.

Data generation for self-improvement pipelines with targeted credit assignment. When LLMs are used to generate their own training data — as in ReST-style pipelines (Singh et al., 2024) or rejection sampling fine-tuning — the quality and diversity of generated solutions matter. RLRT's mechanism provides a way to generate training trajectories that are not just correct but diverse in causally meaningful ways. The distribution-shift analysis (Figure 7) shows that RLRT-trained models pull tokens from the base model's probability tail into top positions, producing solutions that differ structurally from what the base model would generate. If these RLRT-generated solutions are used as fine-tuning data for the next iteration of self-improvement, they may provide more varied reasoning patterns than GRPO-generated solutions (which primarily sharpen existing preferences, as Figure 7b shows). A concrete pipeline: (1) train with RLRT on DAPO-Math-17k, (2) use the RLRT-trained model to generate solutions on a larger math corpus, (3) filter for correctness, (4) fine-tune the base model on these solutions, (5) repeat. The hypothesis — suggested but not tested by the paper — is that RLRT-generated data would yield better self-improvement than GRPO-generated data because it contains more novel reasoning strategies, not just more confident versions of base model strategies. The paper's brief negative result with ReST(^EM) (Appendix K in the main paper's Appendix, not included here) suggests that naive self-improvement can backfire, making a careful comparison with RLRT-generated data a high-value experiment.

Efficient exploration for small-model deployment on math tutoring or competition prep. For applications where a small model (4B–8B parameters) is deployed for interactive math problem-solving — math tutoring systems, competition training tools, or automated grading assistants — RLRT-trained models offer better coverage of correct solution paths. The pass@k dominance (Figure 8) means that at a given sample budget (e.g., generating 16 candidate solutions per problem), an RLRT-trained Qwen3-8B-Base finds correct answers on more problems than a GRPO-trained equivalent. For a tutoring system that needs to present students with diverse correct solution strategies, RLRT's broader reasoning coverage (reflected in Figure 7b's reduced top-k overlap with the base model) means the system can surface qualitatively different solution approaches — algebraic, geometric, case-analysis — rather than minor variations on the same approach. The specific numbers: on AIME24 at k=16, RLRT pass@k is approximately 0.48 vs. GRPO's ~0.37 (Figure 8, approximate from the plotted curves) — a ~30% relative improvement in the fraction of problems where at least one correct answer is found within 16 samples. For a system that needs to provide solutions to hard competition problems, this coverage improvement directly translates to solving problems that would otherwise require human intervention.