Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation

ArXiv: 2602.12125

🎯 Pitch

Standard on-policy distillation (OPD) can merge expert models or transfer knowledge, but a unified student usually still falls short of its domain-specialist teachers. By reframing OPD as constrained RL and amplifying the reward signal, the student sur­passes all individual teachers in both multi‑teacher and strong‑to‑weak settings—a result that conventional OPD and weight‑extrapolation baselines never achieve.

---

1. Executive Summary

This paper proposes a Generalized On-Policy Distillation (G-OPD) framework that extends standard on-policy distillation (OPD) by introducing a reward scaling factor that controls the relative weighting between the implicit reward term and KL regularization, along with a flexible choice of reference model. Through experiments on math reasoning (AIME24, AIME25, HMMT25) and code generation (HumanEval+, MBPP+, LiveCodeBench) benchmarks using Qwen3-4B-Non-Thinking as the base model, the authors introduce ExOPD, the reward extrapolation variant where λ > 1 (amplifying the log-probability shift between teacher and reference beyond standard OPD's λ = 1), and demonstrate that ExOPD consistently enables the distilled student to surpass domain teachers in both single-teacher and multi-teacher settings — producing a unified student that outperforms all domain-specific teachers on all benchmarks, a result that weight-extrapolation (ExPO) and standard OPD fail to achieve. In the strong-to-weak distillation setting, ExOPD with reward correction (replacing the student's base model as reference with the teacher's pre-RL variant) yields further improvements — for instance, boosting the Qwen3-1.7B student's average math accuracy from 23.1% (OPD) to 25.4% (ExOPD) when distilling from Qwen3-30B-A3B-Instruct-2507 — establishing that reward extrapolation enables learning beyond the teacher's capability boundary only when appropriate λ values are chosen and excessive extrapolation (λ = 1.5) is avoided due to implicit reward hacking.

2. Context and Motivation

The Core Problem: OPD Works, But We Don't Understand Why — and We're Leaving Performance on the Table

On-policy distillation (OPD) has emerged as an empirically effective post-training paradigm. It enables a student model to learn from a teacher's supervision on the student's own generated tokens, rather than passively imitating teacher-generated trajectories as in traditional off-policy distillation. Prior work has shown that OPD can (1) near-losslessly merge capabilities from different domain-specific RL variants back into a single base model (Xiao et al., 2026) and (2) efficiently distill knowledge from larger teachers into smaller students (Gu et al., 2024; Yang et al., 2025a). These are practically valuable capabilities: the first enables a multi-task post-training workflow where domain experts are independently fine-tuned with RL and then consolidated, while the second makes deployment-efficient small models more capable.

However, the paper identifies a fundamental gap: the field lacks a mechanistic understanding of what OPD actually does under the hood. The standard OPD objective minimizes the reverse KL divergence between the student policy $\pi_{\bm{\theta}}$ and the teacher policy $\pi^*$ on student-generated trajectories:

$\mathcal{J}_{\text{OPD}}(\bm{\theta}) = \min_{\bm{\theta}} \; \mathbb{E}_{\bm{x}\sim D, \bm{y}\sim \pi_{\bm{\theta}}(\cdot|\bm{x})} \left[ \mathcal{D}_{\mathrm{KL}}\!\left(\pi_{\bm{\theta}}(\bm{y}|\bm{x}) \,\big\|\, \pi^*(\bm{y}|\bm{x})\right) \right]$

This looks simple, but its relationship to reinforcement learning — the dominant paradigm for post-training LLMs — has been unexplored. The paper argues (Section 1) that "a mechanistic understanding of OPD remains limited in the field, leaving its full potential under-explored." This is not an academic nuance; it has direct practical consequences. Without understanding OPD's connection to RL, we cannot systematically improve it. The hyperparameter space is essentially frozen: standard OPD fixes the relative weighting between the implicit reward signal and the KL regularizer to 1:1, and the reference model for computing that implicit reward is typically chosen as the student's initial policy by default. No prior work had asked whether these fixed choices are optimal, or what happens when we break them.

Why This Problem Matters: The Gap Between Being Good and Being Optimal

The practical stakes are significant. OPD already outperforms off-policy distillation (SFT) by a substantial margin — but as the paper shows in Table 2, standard OPD's performance ceiling is typically bounded by the domain teachers. In multi-teacher distillation, OPD produces a student that approaches teacher-level performance in each domain but does not surpass the teachers. This is a ceiling, not a law of nature. The observation that OPD is bounded by teachers suggests we are leaving capability gains unexploited — and that tweaking the optimization objective might push performance beyond what the teacher has achieved.

The theoretical stakes are equally high. OPD provides dense token-level credit assignment through an implicit reward function:

$r_t^{\text{OPD}} = \log\frac{\pi^*(y_t|\bm{x}, \bm{y}_{<t})}{\pi_{\text{ref}}(y_t|\bm{x}, \bm{y}_{<t})}, \quad t=1,\cdots,T$

This is in stark contrast to standard RL (e.g., GRPO), where the reward signal is sparse — typically delivered only at the final token when the answer is verified. The dense nature of OPD's reward explains why it can be more efficient than RL, but it also raises questions: the implicit reward $\log\frac{\pi^*}{\pi_{\text{ref}}}$ takes essentially the same form as the implicit reward derived from the closed-form solution of KL-constrained RL in DPO (Rafailov et al., 2023), yet OPD places no requirement that $\pi^*$ be obtained by applying RL starting from $\pi_{\text{ref}}$. This means the reward signal may contain noise or biases — particularly when the teacher and reference models are of fundamentally different capacities (as in strong-to-weak distillation). Understanding when this implicit reward is well-behaved versus noisy is crucial for knowing when OPD can be pushed harder.

Where Prior Approaches Fall Short

Off-policy distillation (SFT) is cheap but suboptimal. The dominant paradigm for knowledge transfer — supervised fine-tuning on teacher-generated trajectories (Taori et al., 2023; Zhou et al., 2023; Guha et al., 2025) — has the critical drawback of being off-policy: the student learns to imitate the teacher's behavior but never learns from reward signals induced by its own actions. As stated in Section 3.1, the student "may fail to adapt and generalize from its own experience at test time, when faced with similar problems." The paper's results confirm this limitation: in Table 2, SFT produces a multi-teacher student that underperforms the domain teachers, with an average math accuracy of 44.3% versus the math teacher's 46.0%.

Standard OPD is bounded by the teacher. While OPD addresses the off-policy problem by letting the student generate its own trajectories and learn from the teacher's token-level feedback on those trajectories, it suffers from a fixed 1:1 weighting between the reward term and KL regularization. This is the critical constraint the paper identifies: by always weighting the implicit reward equally with the KL penalty, standard OPD has no mechanism to amplify the reward signal. The student learns to match the teacher — but cannot learn beyond the teacher.

Weight extrapolation (ExPO) is training-free but brittle. Zheng et al. (2025) proposed a method that extrapolates model weights against the base model: merge domain teachers' weights, then extrapolate away from the student's initial weights using an extrapolation factor $\alpha$. While this is computationally cheap (no training required), the paper shows it "cannot ensure that the weight-extrapolated student consistently surpasses all domain teachers, lacking good controllability" (Section 4.1.3). In Table 2, ExPO's multi-teacher student achieves 45.0% average math accuracy versus 46.0% for the math teacher — actually degrading relative to the single-teacher ExPO baseline.

RL with sparse rewards is inefficient. Standard outcome-based RL (e.g., GRPO with a final-answer correctness reward of 1.0 or 0.0) provides supervision only at the final token. As the paper notes in Section 3.1, this "may make optimization inefficient and ineffective" — a known challenge that has motivated work on process rewards and dense credit assignment (Cui et al., 2025; Yuan et al., 2024). OPD already solves the sparsity problem through its token-level implicit reward, but the connection between OPD's dense reward structure and RL's sparse reward framework had never been formalized.

How This Paper Positions Itself

The paper's positioning is fundamentally about generalization through theoretical insight. Rather than proposing a new distillation method from scratch, it asks: what happens if we re-derive the OPD objective through the lens of KL-constrained RL, and then relax the constraints that standard OPD implicitly imposes?

The key move is in Section 3.2, where the authors rewrite the OPD objective as a dense RL problem with a flexible reference model:

$\mathcal{J}_{\text{OPD}}(\bm{\theta}) = \max_{\bm{\theta}} \; \mathbb{E}_{\bm{x}\sim D, \bm{y}\sim \pi_{\bm{\theta}}(\cdot|\bm{x})} \left[ \log\frac{\pi^*(\bm{y}|\bm{x})}{{\pi_{\mathrm{ref}}}(\bm{y}|\bm{x})} - \mathcal{D}_{\mathrm{KL}}\!\big(\pi_{\bm{\theta}}(\bm{y}|\bm{x}) \,\big\|\, {\pi_{\mathrm{ref}}}(\bm{y}|\bm{x})\big) \right]$

This reveals that OPD is already doing dense RL — the token-level reward is $\log\frac{\pi^*}{\pi_{\text{ref}}}$, and the KL regularization is against $\pi_{\text{ref}}$. But the weighting is locked at 1:1. The paper's core contribution is the introduction of a reward scaling factor $\lambda$ that breaks this lock:

$\mathcal{J}_{\text{G-OPD}}(\bm{\theta}) = \max_{\bm{\theta}} \; \mathbb{E}_{\bm{x}\sim D, \bm{y}\sim \pi_{\bm{\theta}}(\cdot|\bm{x})} \left[ {\lambda}\log\frac{\pi^*(\bm{y}|\bm{x})}{{\pi_{\mathrm{ref}}}(\bm{y}|\bm{x})} - \mathcal{D}_{\mathrm{KL}}\!\big(\pi_{\bm{\theta}}(\bm{y}|\bm{x}) \,\big\|\, {\pi_{\mathrm{ref}}}(\bm{y}|\bm{x})\big) \right]$

This seemingly simple modification — a single scalar $\lambda$ — fundamentally changes the optimization landscape. When $0 < \lambda < 1$, the student's log-probability distribution becomes a linear interpolation between the teacher and reference models (reward interpolation). When $\lambda > 1$, the student is encouraged to go beyond matching the teacher by fitting an additional shift term $(\lambda-1)(\log\pi^* - \log\pi_{\text{ref}})$ — what the paper terms reward extrapolation. The optimal solution to G-OPD (derived in Section 3.2) reveals this explicitly:

$\log\pi_{\bm{\theta}}(\bm{y}|\bm{x}) = \lambda\log\pi^*(\bm{y}|\bm{x}) + (1-\lambda)\log\pi_{\mathrm{ref}}(\bm{y}|\bm{x})$

The paper positions this as a unified framework that subsumes both OPD ($\lambda=1$) and the base reference model ($\lambda=0$) as special cases, while enabling two new regimes: controllable budget-sensitive reasoning through interpolation ($0 < \lambda < 1$), and learning beyond the teacher through extrapolation ($\lambda > 1$). The framework also formalizes the role of the reference model choice ($\pi_{\text{ref}}$), which becomes critically important when $\lambda \neq 1$ — a dimension that standard OPD's $\lambda=1$ setting renders invisible because the reference model term cancels out of the original OPD objective.

The paper's empirical positioning is to validate this theoretical framework across three increasingly challenging settings: (1) same-sized single-teacher distillation (Section 4.1.2), where it establishes that $\lambda=1.25$ consistently outperforms $\lambda=1$; (2) multi-teacher distillation (Section 4.1.3), where ExOPD is shown to be "the only method that yields a unified student that consistently outperforms all domain teachers" (Figure 1, caption); and (3) strong-to-weak distillation (Section 4.2), where additional reference model engineering (reward correction) is introduced to handle the fundamental knowledge gap between teacher and student.

3. Technical Approach

3.1 Reader Orientation

This is primarily a theoretical-analysis-and-empirical-validation paper whose core idea is that standard on-policy distillation (OPD) is secretly a restricted form of dense KL-constrained reinforcement learning, and that by relaxing one key constraint—the rigid 1:1 weighting between the implicit reward signal and the KL regularizer—we can create a generalized framework (G-OPD) where students can interpolate between the base model and teacher, or extrapolate beyond the teacher's capability boundary (ExOPD).

3.2 Big-Picture Architecture (Diagram in Words)

The G-OPD framework consists of four major components that interact during training:

1. Student policy model (π_θ) — the LLM being trained. It generates its own trajectories (responses to prompts) autoregressively, then receives token-level feedback on those trajectories.
2. Teacher policy model (π*) — the domain expert whose knowledge is being transferred. This can be either a same-sized model fine-tuned with domain-specific RL (same-sized setting, Section 4.1) or a larger, more capable model (strong-to-weak setting, Section 4.2).
3. Reference policy model (π_ref) — the anchor distribution used to compute the implicit reward signal (log(π\*/π_ref)) and against which the KL penalty is applied. By default, this is the student's initial checkpoint (π_base^student), but in the reward correction variant (Section 3.2, "Reward correction in strong-to-weak distillation"), it is replaced with the teacher's pre-RL base model (π_base^teacher).
4. Reward scaling factor (λ) — a single scalar hyperparameter that controls the relative weight of the implicit reward term versus the KL regularization. When λ = 1, G-OPD reduces to standard OPD. When 0 < λ < 1, the student interpolates between the reference and teacher (reward interpolation). When λ > 1, the student is pushed to amplify the log-probability shift beyond what the teacher exhibits—this is ExOPD (reward extrapolation).

Information flows through the system during a single training step as follows:

• Step 1 (Sampling): The student policy π_θ generates a batch of responses y ~ π_θ(·|x) for prompts x drawn from the training distribution D. Temperature is set to 1.0 with top-p = 1.0.
• Step 2 (Scoring): For each token y_t in each generated response, three log-probabilities are computed: (a) log π_θ(y_t|...) from the current student, (b) log π\*(y_t|...) from the teacher, and (c) log π_ref(y_t|...) from the reference model.
• Step 3 (Advantage computation): The token-level advantage A_t^{G-OPD} is computed as a combination of the standard OPD advantage and an extrapolation term controlled by λ: $A_t^{\text{G-OPD}} = (\log\pi_{\bm{\theta}}(y_t|...) - \log\pi^*(y_t|...)) + (\lambda-1)(\log\pi_{\text{ref}}(y_t|...) - \log\pi^*(y_t|...))$
• Step 4 (Policy gradient update): The student's parameters are updated using a policy gradient where each token's contribution is weighted by A_t^{G-OPD}. In practice, a discount factor of 0 is used (focusing on next-token optimization only), making this equivalent to a token-level REINFORCE update with dense reward.

The key insight is that changing λ from 1.0 to 1.25 or 0.5 changes the direction and magnitude of the gradient update, not just its scale—enabling qualitatively different learning behaviors from the same teacher-student pair.

3.3 Roadmap for the Deep Dive

• First, I'll derive the connection between OPD and dense RL from first principles (Eq. 7), showing exactly how the standard OPD objective is algebraically equivalent to a KL-constrained RL problem with a specific fixed weighting. This is the theoretical foundation for everything that follows.
• Second, I'll explain the G-OPD generalization (Eq. 11) — what λ does geometrically to the optimization landscape, why the choice of reference model matters when λ ≠ 1, and how the optimal solution (Eq. 12) reveals the interpolation/extrapolation behavior.
• Third, I'll cover the gradient computation (Eq. 14) in detail, explaining how the token-level advantage is constructed, why a discount factor of 0 is used in practice, and how the gradient connects to standard policy gradient.
• Fourth, I'll walk through the training pipeline — how teachers are created via GRPO, how G-OPD training is configured (batch sizes, rollout settings, learning rates), and the key design choices like token-level rollout correction and prompt-size tuning.
• Fifth, I'll detail the reward correction technique for strong-to-weak distillation, explaining why π_base^teacher provides a more accurate reward signal than π_base^student when the teacher and student have fundamentally different capacities.
• Sixth, I'll cover the experimental setup for the three distillation settings (single-teacher same-sized, multi-teacher, strong-to-weak), including evaluation benchmarks, sampling configurations, and the specific λ values swept.

3.4 Detailed, Sentence-Based Technical Breakdown

3.4.1 The Central Theoretical Insight: OPD IS Dense RL

The paper's foundational contribution is showing through algebraic manipulation that the standard OPD objective (Eq. 4) is mathematically equivalent to a KL-constrained dense reinforcement learning problem. This is not just a metaphorical connection — it is exact, and it reveals two constraints that standard OPD implicitly imposes: (1) the reward term and KL penalty are always weighted 1:1, and (2) the reference model can be chosen arbitrarily without affecting the original objective.

The derivation begins with the standard OPD objective, which minimizes the reverse KL divergence from student to teacher on student-generated trajectories:

$\mathcal{J}_{\text{OPD}}(\bm{\theta}) = \min_{\bm{\theta}} \; \mathbb{E}_{\bm{x}\sim D, \bm{y}\sim \pi_{\bm{\theta}}(\cdot|\bm{x})} \left[ \mathcal{D}_{\mathrm{KL}}\!\left(\pi_{\bm{\theta}}(\bm{y}|\bm{x}) \,\big\|\, \pi^*(\bm{y}|\bm{x})\right) \right]$

where $\pi_{\bm{\theta}}$ is the student policy being optimized, $\pi^*$ is the fixed teacher policy, and $D$ is the input distribution of prompts. The expectation over $\bm{y} \sim \pi_{\bm{\theta}}$ is what makes this on-policy: the student evaluates its own generated outputs against the teacher's distribution.

The paper then expands the KL divergence into its definition:

$\mathcal{J}_{\text{OPD}}(\bm{\theta}) = \min_{\bm{\theta}} \; \mathbb{E}_{\bm{x}\sim D, \bm{y}\sim \pi_{\bm{\theta}}(\cdot|\bm{x})} \left[ \log\pi_{\bm{\theta}}(\bm{y}|\bm{x}) - \log\pi^*(\bm{y}|\bm{x}) \right]$

What this computes: For each student-generated token sequence, it computes the log-probability difference between the student's current policy and the teacher's policy. Minimizing this objective pushes the student to assign higher probability to tokens the teacher considers likely — but does so on the student's own distribution, not the teacher's.

The critical algebraic manipulation comes next. The paper introduces an arbitrary reference model $\pi_{\mathrm{ref}}$ by adding and subtracting $\log\pi_{\mathrm{ref}}(\bm{y}|\bm{x})$:

$\mathcal{J}_{\text{OPD}}(\bm{\theta}) = \max_{\bm{\theta}} \; \mathbb{E}_{\bm{x}\sim D, \bm{y}\sim \pi_{\bm{\theta}}(\cdot|\bm{x})} \left[ (\log\pi^*(\bm{y}|\bm{x}) - \log\pi_{\mathrm{ref}}(\bm{y}|\bm{x})) - (\log\pi_{\bm{\theta}}(\bm{y}|\bm{x}) - \log\pi_{\mathrm{ref}}(\bm{y}|\bm{x})) \right]$

This can be rewritten as:

$\mathcal{J}_{\text{OPD}}(\bm{\theta}) = \max_{\bm{\theta}} \; \mathbb{E}_{\bm{x}\sim D, \bm{y}\sim \pi_{\bm{\theta}}(\cdot|\bm{x})} \left[ \log\frac{\pi^*(\bm{y}|\bm{x})}{\pi_{\mathrm{ref}}(\bm{y}|\bm{x})} - \mathcal{D}_{\mathrm{KL}}\!\big(\pi_{\bm{\theta}}(\bm{y}|\bm{x}) \,\big\|\, \pi_{\mathrm{ref}}(\bm{y}|\bm{x})\big) \right]$

Why this form matters: Compare this to the standard RL objective (Eq. 2):

$\mathcal{J}_{\text{RL}}(\bm{\theta}) = \max_{\bm{\theta}} \; \mathbb{E}_{\bm{x}\sim D, \bm{y}\sim \pi_{\bm{\theta}}(\cdot|x)} \left[ r(\bm{x},\bm{y}) - \beta\mathcal{D}_{\mathrm{KL}}(\pi_{\bm{\theta}} \|\ \pi_{\mathrm{ref}}) \right]$

The OPD objective has exactly the same structure: a reward term minus a KL penalty. The "reward" in OPD is $\log\frac{\pi^*}{\pi_{\mathrm{ref}}}$ — the implicit reward from DPO (Rafailov et al., 2023) — and the KL coefficient $\beta$ is implicitly fixed to 1. This is the key revelation: OPD is RL with $\beta = 1$, a dense token-level reward function derived from the teacher's log-probability shift relative to a reference, and trajectories sampled on-policy from the student.

The paper emphasizes three differences from standard RL (Section 3.2, "From the above remark"):

• (1) Dense rewards: In standard RL, rewards are sparse — typically zero for all tokens except the final one: $r_t^{RL} = \begin{cases} 0 & t=1,\cdots,T-1 \\ \text{Outcome Reward} & t=T \end{cases}$ In OPD, every token $y_t$ receives an immediate reward: $r_t^{OPD} = \log\frac{\pi^*(y_t|\bm{x}, \bm{y}_{<t})}{\pi_{\mathrm{ref}}(y_t|\bm{x}, \bm{y}_{<t})}, \quad t=1,\cdots,T$ This is the token-level implicit reward — it captures how much more likely the teacher finds token $y_t$ compared to the reference model, given the context so far. This dense signal enables more efficient credit assignment than sparse outcome rewards.

• (2) Fixed weighting: In OPD, the reward term and KL regularization are always weighted equally ($\beta = 1$). This is the constraint that G-OPD explicitly relaxes by introducing $\lambda$.

• (3) Flexible reference model: Unlike RL where $\pi_{\mathrm{ref}}$ is typically the initial policy checkpoint, OPD's derivation allows $\pi_{\mathrm{ref}}$ to be any model — the algebra works because the $\log\pi_{\mathrm{ref}}$ terms cancel when $\lambda = 1$. However, when $\lambda \neq 1$, the choice of $\pi_{\mathrm{ref}}$ does affect the objective, making it a design dimension worth optimizing.

3.4.2 The G-OPD Generalization: Breaking the 1:1 Lock

Building on the RL connection, the paper introduces the Generalized On-Policy Distillation (G-OPD) objective by inserting a reward scaling factor $\lambda$ in front of the implicit reward term:

$\mathcal{J}_{\text{G-OPD}}(\bm{\theta}) = \max_{\bm{\theta}} \; \mathbb{E}_{\bm{x}\sim D, \bm{y}\sim \pi_{\bm{\theta}}(\cdot|\bm{x})} \left[ {\lambda}\log\frac{\pi^*(\bm{y}|\bm{x})}{{\pi_{\mathrm{ref}}}(\bm{y}|\bm{x})} - \mathcal{D}_{\mathrm{KL}}\!\big(\pi_{\bm{\theta}}(\bm{y}|\bm{x}) \,\big\|\, {\pi_{\mathrm{ref}}}(\bm{y}|\bm{x})\big) \right]$

where $\lambda$ controls the relative weight of the reward term against the KL regularization, and is essentially $1/\beta$ in the standard RL notation. $\pi_{\mathrm{ref}}$ is the reference model for both the implicit reward computation and the KL penalty.

What this computes: For each student-generated sequence, the objective maximizes $\lambda$ times the log-ratio $\pi^*/\pi_{\mathrm{ref}}$ (the cumulative implicit reward), minus the KL divergence from the student to $\pi_{\mathrm{ref}}$. When $\lambda = 1$, this recovers standard OPD exactly. When $\lambda \neq 1$, it changes the optimization landscape by reweighting the reward term relative to the KL constraint.

Why this form: The RL formulation makes it natural to ask: what if we change $\beta$ (or equivalently $\lambda$)? In RL, $\beta$ controls the trust region — how far the policy can deviate from the reference. Setting $\beta < 1$ ($\lambda > 1$) means the reward signal is amplified relative to the KL penalty, encouraging the policy to move more aggressively toward high-reward regions. Setting $\beta > 1$ ($\lambda < 1$) means the KL penalty dominates, keeping the policy closer to the reference. This parameter is routinely tuned in RL but had never been explored in OPD because the connection wasn't explicit.

The paper derives the optimal solution to G-OPD (the policy that maximizes the objective at equilibrium):

$\log\pi_{\bm{\theta}}(\bm{y}|\bm{x}) = \lambda\log\pi^*(\bm{y}|\bm{x}) + (1-\lambda)\log\pi_{\mathrm{ref}}(\bm{y}|\bm{x})$

What this equation reveals: The optimal student's log-probability distribution is a linear combination of the teacher's and the reference model's log-probability distributions, with weights $\lambda$ and $(1-\lambda)$ respectively. This is what makes the framework interpretable:

• When $\lambda = 1$ (standard OPD): $\log\pi_{\bm{\theta}} = \log\pi^*$ — the student converges exactly to the teacher's distribution.
• When $\lambda = 0$: $\log\pi_{\bm{\theta}} = \log\pi_{\mathrm{ref}}$ — the student remains at the reference distribution (no distillation occurs).
• When $0 < \lambda < 1$ (reward interpolation): $\log\pi_{\bm{\theta}}$ is a convex combination — the student's behavior lies between the reference and teacher. Both performance and response length are expected to fall between those of $\pi_{\mathrm{ref}}$ and $\pi^*$.
• When $\lambda > 1$ (reward extrapolation): $\log\pi_{\bm{\theta}} = \log\pi^* + (\lambda-1)(\log\pi^* - \log\pi_{\mathrm{ref}})$ — the student goes beyond the teacher by amplifying the log-probability shift $\log\pi^* - \log\pi_{\mathrm{ref}}$. This is the ExOPD variant.

An alternative interpretation (from Eq. 12, second form) is that the student's log-probability equals the teacher's log-probability plus an extra shift term $(\lambda-1)(\log\pi^* - \log\pi_{\mathrm{ref}})$. From a reward perspective, G-OPD with $\lambda > 1$ performs an extrapolation of the reward function's weight in the original OPD objective.

The paper also provides an equivalent form of G-OPD that makes the role of $\lambda$ clearer (Eq. 13):

$\mathcal{J}_{\text{G-OPD}}(\bm{\theta}) = \max_{\bm{\theta}} \; \mathbb{E}_{\bm{x}\sim D, \bm{y}\sim \pi_{\bm{\theta}}(\cdot|\bm{x})} \left[ (\lambda-1)\log\frac{\pi^*(\bm{y}|\bm{x})}{\pi_{\mathrm{ref}}(\bm{y}|\bm{x})} - \mathcal{D}_{\mathrm{KL}}\!\big(\pi_{\bm{\theta}}(\bm{y}|\bm{x}) \,\big\|\, \pi^*(\bm{y}|\bm{x})\big) \right]$

What this form shows: When $\lambda > 1$, the first term $(\lambda-1)\log\frac{\pi^*}{\pi_{\mathrm{ref}}}$ becomes an additional reward bonus on top of the standard OPD objective (which is the KL to the teacher). The optimization is now: minimize divergence to the teacher (standard OPD) plus maximize the log-ratio $\pi^*/\pi_{\mathrm{ref}}$ weighted by $(\lambda-1)$. This is what pushes the student beyond the teacher — it's rewarded not just for matching the teacher, but for exceeding the teacher's probability ratio relative to the reference.

3.4.3 The G-OPD Gradient: Token-Level Advantage with Extrapolation

The practical implementation of G-OPD uses an approximated gradient with a discount factor of 0 (next-token optimization only), following the practice established in prior OPD work (Lu and Lab, 2025; Xiao et al., 2026). The gradient is:

$\nabla_{\bm{\theta}}\mathcal{J}_{\text{G-OPD}}(\bm{\theta}) = \mathbb{E}_{\bm{x}\sim D, \bm{y}\sim \pi_{\bm{\theta}}(\cdot|\bm{x})} \left[ \sum_{t=1}^{T} A_t^{\text{G-OPD}} \, \nabla_{\bm{\theta}}\log\pi_{\bm{\theta}}(y_t|\bm{x}, \bm{y}_{<t}) \right]$

where $A_t^{\text{G-OPD}}$ is the token-level advantage:

$A_t^{\text{G-OPD}} = (\log\pi_{\bm{\theta}}(y_t|\bm{x}, \bm{y}_{<t}) - \log\pi^*(y_t|\bm{x}, \bm{y}_{<t})) + (\lambda-1)(\log\pi_{\text{ref}}(y_t|\bm{x}, \bm{y}_{<t}) - \log\pi^*(y_t|\bm{x}, \bm{y}_{<t}))$

What each term means:

• $\log\pi_{\bm{\theta}}(y_t|...) - \log\pi^*(y_t|...)$ is the standard OPD advantage: if the student assigns higher probability to a token than the teacher, this term is positive (encouraging the student to reduce that probability); if the teacher assigns higher probability, this term is negative (encouraging the student to increase that probability). This is the part that pulls the student toward the teacher.
• $\log\pi_{\text{ref}}(y_t|...) - \log\pi^*(y_t|...)$ is the extrapolation term: it captures the log-probability difference between the reference and the teacher. When $\lambda > 1$, this term is weighted positively by $(\lambda-1)$, adding an extra push in the direction away from the reference and beyond the teacher.

What this gradient computes operationally: For each token in each student-generated sequence, compute three log-probabilities (student, teacher, reference), combine them into a scalar advantage $A_t$, and use that advantage to weight the policy gradient update. A positive $A_t$ means "increase probability of this token"; a negative $A_t$ means "decrease probability of this token." The magnitude of $A_t$ determines the strength of the update.

Why a discount factor of 0: The full OPD gradient derived in Appendix A (Eq. 20) includes a sum over future tokens $\sum_{t'=t}^{T}(\log\pi_{\bm{\theta}}(y_{t'}|...) - \log\pi^*(y_{t'}|...))$ as the advantage for token $y_t$. This captures the intuition that a token's quality should be judged by its downstream consequences. However, the paper notes that "current studies use a discount factor of 0 (focus on next-token optimization only)" and approximate the gradient with only the immediate token's log-probability difference. This simplification is justified by the mathematical property shown in Eq. 19: when $t' < t$, the expected gradient contribution from future tokens conditioned on the current token is zero because $\mathbb{E}_{y_t}[\nabla_{\bm{\theta}}\log\pi_{\bm{\theta}}(y_t|...)] = \nabla_{\bm{\theta}}\sum_{y_t}\pi_{\bm{\theta}}(y_t|...) = \nabla_{\bm{\theta}}1 = 0$. This means the full gradient reduces to a form where only future tokens matter for each current token, and the discount-0 approximation drops the dependency on even further future tokens — effectively assuming that the immediate next-token advantage captures sufficient signal.

3.4.4 The Training Pipeline: GRPO → G-OPD

The experimental workflow proceeds in two phases:

Phase 1: Teacher Creation via GRPO. Domain teachers are obtained by applying Group Relative Policy Optimization (GRPO) (Shao et al., 2024) to the base model Qwen3-4B-Non-Thinking on domain-specific data:

• Math teacher (Qwen3-4B-Non-Thinking-RL-Math): Trained on 57K samples from the DeepMath dataset (He et al., 2025), filtered to difficulty level ≥ 6. The reward is 1.0 when the final answer is correct (verified by Math-Verify as a rule-based verifier), and 0.0 otherwise. Training hyperparameters (Table 4): train batch size 128, micro batch size 128, rollout n = 8 (8 responses generated per prompt for GRPO's group-based advantage computation), maximum prompt length 2048 tokens, maximum response length 16,384 tokens, temperature 1.0, top-p 1.0, learning rate $1 \times 10^{-6}$, 500 optimization steps, KL coefficient 0.0 (no KL penalty during GRPO training — the model is allowed to deviate freely from the base policy during RL).

• Code teacher (Qwen3-4B-Non-Thinking-RL-Code): Trained on 25K samples from the Eurus-RL-Code dataset (Cui et al., 2025). The reward is 1.0 when all unit tests pass, 0.0 otherwise. Training hyperparameters (Table 5): same as math RL except maximum response length is 8,192 tokens (shorter than math because code solutions tend to be more concise), and 300 optimization steps.

Phase 2: Distillation via G-OPD. The original student model (Qwen3-4B-Non-Thinking) is distilled from the domain teacher(s) using the G-OPD objective with different λ values: $\lambda \in \{0.0, 0.25, 0.5, 0.75, 1.0, 1.25, 1.5\}$. Note that $\lambda = 0$ corresponds to no distillation (the student remains at the initial state), and $\lambda = 1.0$ corresponds to standard OPD.

Key training configurations (Table 6):

• Batch size: 1024 (significantly larger than the GRPO batch size of 128)
• Rollout n: 1 (only one response is generated per prompt, unlike GRPO's n=8 — this is a notable difference that the paper attributes to smoother convergence under larger prompt sizes, stating: "we find that under the same prompt size ∗ rollout n conditions, setting a larger prompt size leads to smoother convergence")
• Maximum prompt length: 2048 tokens
• Maximum response length: 16,384 tokens
• Temperature: 1.0, top-p: 1.0
• Learning rate: $1 \times 10^{-5}$ (10× higher than the GRPO learning rate of $1 \times 10^{-6}$)
• Optimization steps: 50 for same-sized teacher-student experiments (Section 4.1), 100 for strong-to-weak experiments (Section 4.2). The paper states: "We find that further increasing the number of distillation steps may degrade generalization performance due to overfitting."

Both GRPO and G-OPD implement token-level rollout correction (Liu et al., 2025b) to mitigate training-inference mismatch — a technique that adjusts for the distribution shift between training-time rollouts (generated under the training policy) and inference-time rollouts.

The reference model $\pi_{\mathrm{ref}}$ is naturally fixed as Qwen3-4B-Non-Thinking (the student's initial checkpoint) for all same-sized experiments, since the teachers are obtained by applying RL starting from this exact base model.

All experiments are conducted using the verl framework (Sheng et al., 2024).

3.4.5 Reward Correction for Strong-to-Weak Distillation

In the strong-to-weak distillation setting (Section 4.2), a smaller student (e.g., Qwen3-1.7B or Qwen3-4B) is distilled from a larger, more capable teacher (e.g., Qwen3-30B-A3B-Instruct-2507). The default G-OPD with $\pi_{\mathrm{ref}} = \pi_{\mathrm{base}}^{\mathrm{student}}$ uses the implicit reward $\log\frac{\pi^*}{\pi_{\mathrm{base}}^{\mathrm{student}}}$.

However, the paper argues that this reward signal can be noisy because "there exists fundamental gap between the internal knowledge and capacity of teacher and student base models" (Section 3.2, "Reward correction in strong-to-weak distillation"). The implicit reward is well-defined in DPO theory only when $\pi^*$ is obtained by applying RL starting from $\pi_{\mathrm{ref}}$ — but in strong-to-weak distillation, the large teacher was trained from its own base model, not from the small student's base model. The log-ratio $\log\frac{\pi^*}{\pi_{\mathrm{base}}^{\mathrm{student}}}$ conflates the teacher's RL-learned improvements with the fundamental capacity gap between the models, introducing noise.

The proposed reward correction technique addresses this by replacing the reference model with the teacher's pre-RL base model (if available):

• Default reward: $\log\frac{\pi^*}{\pi_{\mathrm{base}}^{\mathrm{student}}}$
• Corrected reward: $\log\frac{\pi^*}{\pi_{\mathrm{base}}^{\mathrm{teacher}}}$ (obtained by adding $\log\frac{\pi_{\mathrm{base}}^{\mathrm{student}}}{\pi_{\mathrm{base}}^{\mathrm{teacher}}}$ to the default reward)

The corrected reward $\log\frac{\pi^*}{\pi_{\mathrm{base}}^{\mathrm{teacher}}}$ corresponds exactly to the implicit reward induced by the teacher's RL post-training — it captures the log-probability shift that occurred during the teacher's RL phase, independent of the capacity gap between the student and teacher base models. This provides a cleaner signal for the student to learn from.

The limitations of this approach, explicitly acknowledged by the paper, are: (1) it requires access to $\pi_{\mathrm{base}}^{\mathrm{teacher}}$, an additional model that may not be available (e.g., the paper couldn't get the pre-RL variant of Qwen3-30B-A3B-Instruct-2507, so they used their own Qwen3-4B-Non-Thinking-RL-Math/Code as teachers with Qwen3-4B-Non-Thinking as the pre-RL variant for the reward correction experiments in Section 4.2.3); and (2) it incurs higher computational cost because computing $\log\pi_{\mathrm{base}}^{\mathrm{teacher}}$ requires forward passes through a larger model than computing $\log\pi_{\mathrm{base}}^{\mathrm{student}}$.

3.4.6 Multi-Teacher Distillation Setup

In the multi-teacher setting (Section 4.1.3), the goal is to merge the capabilities of multiple domain experts back into a single unified student. The domain teachers are Qwen3-4B-Non-Thinking-RL-Math and Qwen3-4B-Non-Thinking-RL-Code — both obtained by applying domain-specific RL to the same base model Qwen3-4B-Non-Thinking.

The distillation data is the combined RL training data from both domains. For fair comparison, the paper downweights the sample size of the math RL data to match that of the code RL data (25K each), ensuring balanced domain representation.

The reference model is the original base model Qwen3-4B-Non-Thinking for all methods. ExOPD uses $\lambda = 1.25$, selected based on the single-teacher preliminary experiments (Section 4.1.2) where this value "consistently leads to better performance than OPD." No further tuning is performed.

Baselines compared against (Table 2):

• SFT (off-policy distillation): Supervised fine-tuning on teacher-generated trajectories using cross-entropy loss. The number of trajectories per problem generated by the teacher is matched to that generated by the student in OPD/ExOPD for fairness. SFT hyperparameters (Table 7): batch size 1024, maximum sequence length 32,768, warm-up ratio 0.05, learning rate $1 \times 10^{-5}$. The number of optimization steps is kept consistent with the corresponding G-OPD experiment.
• ExPO (weight extrapolation): First averages the weights of all domain teachers (simple weight averaging), then extrapolates the averaged weights against the student model using an extrapolation factor $\alpha$, tuned from $\{0.25, 0.5\}$ following the recommendations from Zheng et al. (2025). No training is required.

3.4.7 Strong-to-Weak Distillation Setup

In the strong-to-weak setting (Section 4.2.1), the teacher is Qwen3-30B-A3B-Instruct-2507 — a much larger model — and the students are Qwen3-1.7B-Non-Thinking and Qwen3-4B-Non-Thinking. Experiments are conducted in the math reasoning domain only, using the same training and evaluation datasets as in Section 4.1.

The default ExOPD uses $\pi_{\mathrm{ref}} = \pi_{\mathrm{base}}^{\mathrm{student}}$ (the student's initial checkpoint), assuming only two models are available: the student base model and the stronger teacher. The training hyperparameters are the same as in the same-sized experiments except for 100 optimization steps instead of 50.

The reward correction experiments (Section 4.2.3) use Qwen3-4B-Non-Thinking-RL-Math/Code as teachers (trained in Section 4.1), Qwen3-4B-Non-Thinking as the pre-RL variant serving as $\pi_{\mathrm{base}}^{\mathrm{teacher}}$, and Qwen3-1.7B-Non-Thinking as the student. This setup is a compromise because the pre-RL variant of Qwen3-30B-A3B-Instruct-2507 is unavailable.

3.4.8 Evaluation Methodology

Benchmarks: Math reasoning is evaluated on four competition-level benchmarks:

• AIME24: The 2024 American Invitational Mathematics Examination (AI-MO, 2024)
• AIME25: The 2025 AIME (OpenCompass, 2025)
• HMMT25 (February): Harvard-MIT Mathematics Tournament, February 2025 edition (Balunović et al., 2025)
• HMMT25 (November): HMMT, November 2025 edition

Code generation is evaluated on three benchmarks:

• HumanEval+ and MBPP+: Enhanced versions of standard code benchmarks with rigorous test case evaluation (Liu et al., 2023)
• LiveCodeBench (v6 only, February 2025 ∼ May 2025): A contamination-free live benchmark for code generation (Jain et al., 2024)

Sampling configuration: For all evaluations, temperature is set to 1.0, top-p to 1.0, and maximum generation length to 16,384 tokens. On math reasoning benchmarks, 32 solutions are sampled per problem; on code generation benchmarks, 4 solutions per problem. The paper reports "the average accuracy of each model on each benchmark" — meaning pass@k where k = 32 for math and k = 4 for code (the fraction of problems where at least one of the k sampled solutions is correct).

Answer verification: Math-Verify, a rule-based verifier, is used to validate answer correctness for math reasoning benchmarks. For code, unit test pass/fail serves as the correctness signal.

3.4.9 Design Choices and Their Justifications

• λ = 1.25 chosen over tuning per experiment: Based on single-teacher preliminary results (Figures 2 and 3) where λ = 1.25 consistently outperforms λ = 1.0 across all math reasoning and code generation benchmarks. The paper explicitly states: "In all subsequent experiments, we fix λ = 1.25 for ExOPD without any further specific tuning." This is a deliberate choice to demonstrate that the method works without expensive per-task hyperparameter search.

• Batch size of 1024 with rollout n = 1 in G-OPD versus batch size 128 with rollout n = 8 in GRPO: The paper finds that "under the same prompt size ∗ rollout n conditions, setting a larger prompt size leads to smoother convergence." This means G-OPD benefits more from seeing diverse prompts (1024 unique prompts) with single rollouts each, rather than multiple rollouts per prompt.

• Token-level rollout correction applied: This mitigates the distribution mismatch between the policy that generates training rollouts and the policy those rollouts are evaluated under, a known source of instability in on-policy training.

• Matching trajectory counts between SFT and OPD: For the SFT baseline, the paper ensures that "the number of trajectories to each problem generated by the teacher in SFT is consistent with that generated by the student in OPD and ExOPD." This controls for the amount of supervision data, making the on-policy vs. off-policy comparison fair.

• Downweighting math data in multi-teacher experiments: To prevent the math domain from dominating due to its larger dataset (57K vs. 25K samples), the paper downweights math data to match code data size, ensuring "each domain has the same sample size."

• 50 vs. 100 optimization steps: Determined empirically — the paper finds that "further increasing the number of distillation steps may degrade generalization performance due to overfitting," so the smaller number (50) is used for same-sized distillation and the larger number (100) for the more challenging strong-to-weak setting where the capacity gap may require more steps to bridge.

• Discount factor of 0 for gradient approximation: This follows established practice in prior OPD work (Lu and Lab, 2025; Xiao et al., 2026) and is justified theoretically by the property that expected gradients from past tokens conditioned on the current token vanish (Eq. 19 in Appendix A). The practical benefit is computational efficiency — only immediate token-level log-probabilities are needed, not accumulated future advantage estimates.

4. Key Insights and Innovations

Innovation 1: Reframing On-Policy Distillation as an Uninhibited Form of Dense RL

The paper's foundational conceptual move is not the G-OPD equation itself — it is the diagnostic reframing that reveals standard OPD has been doing dense reinforcement learning all along, just with its hands tied behind its back. Prior work treated OPD as a distillation technique — a way to match a student's distribution to a teacher's, with the on-policy sampling being an engineering choice that improved data efficiency. The dominant mental model was "the student imitates the teacher, but on its own outputs rather than the teacher's." This framing made OPD seem like a clever variant of knowledge distillation, not a fundamentally different optimization paradigm.

What this paper shows — through the algebraic derivation in Eq. 7 that is rigorous but deliberately simple — is that OPD is precisely equivalent to a KL-constrained RL problem where the reward function is log(π*/π_ref), the KL penalty is against π_ref, and the reward-to-KL weighting is locked at exactly 1:1. This is not an analogy or a loose connection. It is exact. The paper surfaces this by doing something that seems almost trivial in retrospect: introducing an arbitrary reference model π_ref into the OPD objective through addition and subtraction, then regrouping the terms to expose the RL structure that was always latent there.

Why this reframing matters intellectually: it unlocks the entire RL hyperparameter toolkit for distillation. In RL, adjusting the KL coefficient β (or equivalently the reward scaling factor λ) is standard practice — it controls the trust region, the exploration-exploitation tradeoff, and how aggressively the policy pursues reward signals. But because the field viewed OPD through a distillation lens rather than an RL lens, no one had asked whether changing this implicit β from its fixed value of 1 would help. The paper doesn't just propose a new hyperparameter — it explains why that hyperparameter should exist by revealing that OPD already has the structure of an RL problem that normally has it.

This is a fundamental reframing, not an incremental addition. It changes the question from "how can we improve distillation?" to "given that OPD is RL under the hood, what RL insights have we been neglecting?" The paper's entire empirical program — interpolation for budget control, extrapolation for exceeding the teacher — flows directly from this conceptual pivot, not from any novel algorithmic machinery.

Innovation 2: Extrapolation as a New Axis for Knowledge Transfer — Learning Beyond the Teacher

The field's implicit assumption in knowledge distillation has always been that the teacher represents a performance ceiling. Off-policy SFT can at best approach the teacher's output distribution. Standard OPD matches the teacher's distribution on the student's own trajectories. Even weight-extrapolation methods like ExPO (Zheng et al., 2025) average teacher weights and extrapolate from the base model, but the paper shows this is brittle — ExPO's multi-teacher student underperforms the domain teachers on math reasoning (45.0% vs. 46.0% in Table 2, a −2.2% relative drop from the teacher average).

ExOPD breaks this ceiling explicitly. By setting λ > 1, the optimization objective ceases to be about matching the teacher. Instead, as the optimal solution in Eq. 12 shows, the student's log-probability becomes log π_θ = log π\* + (λ−1)(log π\* − log π_ref) — it amplifies the direction in log-probability space that the teacher moved relative to the reference. The student is not just learning what the teacher knows; it is learning to extrapolate the teacher's trajectory of improvement beyond where the teacher stopped.

This is genuinely novel at the conceptual level because it redefines the teacher's role. In standard distillation, the teacher is an oracle whose outputs define the target. In ExOPD, the teacher is a vector — its value lies not just in its absolute log-probability distribution, but in the difference log π\* − log π_ref, which encodes the direction and magnitude of improvement the teacher achieved through RL. The student is then free to travel further along that same direction, proportionally to λ−1. The teacher becomes a compass, not a destination.

The empirical evidence that this works is striking: in the single-teacher same-sized setting (Table 2, top), ExOPD with λ = 1.25 achieves +4.7 points on AIME24, +1.5 points on AIME25, +1.4 points on HMMT25 (Feb.), and +0.4 points on HMMT25 (Nov.) over the domain teacher — a model that was itself trained with RL on the exact same data distribution. This is not a case of the teacher being undertrained: Table 1 shows that giving the math teacher an additional 100 steps of RL (roughly 2× the ExOPD training budget of 50 steps) yields smaller improvements (+2.9, +0.5, +0.3, −0.5) than ExOPD's extrapolation (+4.7, +1.5, +1.4, +0.4). The mechanism is qualitatively different from simply training longer.

The multi-teacher result is perhaps even more compelling: ExOPD is "the only method that yields a unified student that consistently outperforms all domain teachers" (Figure 1, caption). Standard OPD produces a student that approaches but doesn't consistently exceed teacher performance — the math average is 46.4% vs. the math teacher's 46.0% (Table 2), while the code average drops to 60.6% vs. the code teacher's 61.2%. ExOPD's unified student achieves 47.7% on math (+1.7 over the math teacher) and 62.0% on code (+0.8 over the code teacher). This is not a tradeoff — it improves on both domains simultaneously.

This innovation is fundamental rather than incremental because it defines a new capability that no prior distillation method possessed. The paper doesn't just improve distillation efficiency — it changes what distillation can achieve.

Innovation 3: The Implicit Reward as a Diagnosable, Correctable Signal — Not a Black Box

A recurring theme in the paper is that the implicit reward log(π*/π_ref) is not a magical quantity — it has definable properties, failure modes, and a dependence on the choice of π_ref that can be exploited or corrected. Prior work that used implicit rewards (DPO, process reward models) largely treated them as a derived quantity from the RL objective — something that emerges from the math but isn't directly manipulated. This paper treats the implicit reward as a first-class design object whose properties change depending on the distillation setting.

The key diagnostic insight comes from the strong-to-weak distillation analysis. The paper argues that when π_ref = π_base^student, the reward log(π*/π_base^student) conflates two distinct signals: (1) the improvements the teacher achieved through RL (which the student should learn), and (2) the fundamental knowledge and capacity gap between a large teacher and a small student (which the student cannot learn). The ratio of a 30B model's log-probabilities to a 1.7B model's log-probabilities contains information about model scale, not just about task-specific improvements. This noise is invisible if you treat the implicit reward as a black box — you'd just observe that OPD works but might not work as well as it could.

The paper's reward correction technique — replacing π_base^student with π_base^teacher as the reference — is conceptually elegant because it isolates the signal the student can actually use: the shift that occurred during the teacher's RL phase, independent of the base capacity gap. When the teacher's pre-RL variant is available (which the paper acknowledges is not always the case), the corrected reward log(π*/π_base^teacher) corresponds exactly to what DPO theory says an implicit reward should represent — the improvement from a specific optimization process starting from a specific reference.

The empirical validation (Figure 14a, Section 4.2.3) shows that reward correction consistently boosts ExOPD performance in strong-to-weak distillation. This is not just a performance gain — it's evidence that the diagnosis was correct. The default reward was noisier, and cleaning it up helped.

This innovation is a conceptual advance with practical implications rather than a fundamental paradigm shift. It doesn't change the framework — it deepens our understanding of one of its components. But it's significant because it provides a principled way to think about when and why implicit rewards might fail, rather than treating them as universally reliable. The paper's explicit acknowledgment of the limitations (needing access to π_base^teacher, incurring extra computational cost) also models good practice — it tells the reader exactly when this correction applies and when it doesn't, rather than overselling it.

Innovation 4: Reward Interpolation as a Mechanism for Controllable, Budget-Aware Reasoning

While ExOPD (λ > 1) is the paper's flashiest result, the reward interpolation regime (0 < λ < 1) provides a conceptually distinct contribution: it shows that G-OPD can produce a continuous spectrum of models whose performance and response length vary monotonically with λ. This is visible in Figure 9a (Section 4.1.1), where the accuracy and average token count of distilled models trace smooth curves from the base model (λ = 0) to the teacher (λ = 1), with intermediate λ values producing intermediate behaviors.

This matters because it addresses a practical deployment challenge that the field has largely ignored: reasoning models (those trained with RL on verifiable tasks) tend to produce longer responses — they "think more" before answering. This is generally good for accuracy but bad for inference cost and latency. An organization deploying such a model faces a binary choice: either pay the cost of the full RL-trained model's reasoning chains, or use the base model and sacrifice accuracy. There is no knob to turn.

G-OPD's interpolation regime provides that knob. By choosing λ = 0.5 instead of λ = 1.0, you get a model whose accuracy and response length fall midway between the base and teacher — a concrete point on a Pareto frontier of cost vs. capability. The paper connects this explicitly to "budget-controlled reasoning" (citing Yang et al., 2025e; Liang et al., 2026), but what G-OPD offers is distinct: it's not a test-time technique (like adaptive compute allocation) but a training-time technique that produces a single model with a specific cost-capability profile. You could distill multiple students at different λ values and route queries to them based on difficulty or budget constraints.

This innovation is incremental in mechanism but significant in implication because it transforms G-OPD from a single-purpose tool (make the student as good as possible) into a framework for cost-aware capability engineering. The paper doesn't fully explore this direction (there are no experiments on routing or multi-budget deployment), but the framework provides the theoretical foundation for doing so — λ is no longer just a hyperparameter to be optimized for maximum accuracy, but a design parameter that expresses a preference along the cost-capability spectrum.

Innovation 5: Verifier Over-Optimization Through the Lens of Implicit Reward Hacking

The paper's finding that excessive reward extrapolation (λ = 1.5) degrades performance is not just a hyperparameter sensitivity observation — it provides a new diagnostic lens on reward hacking in distillation. The standard narrative around reward hacking comes from RL, where a policy exploits imperfections in a learned reward model to achieve high scores without genuinely solving the task. In G-OPD, the "reward model" is the implicit reward log(π*/π_ref), and the paper argues that at high λ, the student "aggressively fit[s] the peak of the log ratio, even if some tokens have excessively large log ratios due to bias" (Section 4.1.2).

What makes this interesting is that the implicit reward is not a neural network trained on preference data — it's derived directly from the teacher's and reference model's log-probabilities. The "bias" that enables hacking comes from structural properties of the log-probability ratio itself, not from annotation errors or distribution shift in preference data. The paper specifically identifies a length bias: as λ increases beyond 1.25, response lengths continue to grow (visible in Figure 9a and the training dynamics in Figure 12a), suggesting that the implicit reward systematically favors longer sequences. The student learns that it can increase its reward by generating more tokens, even if those tokens don't improve correctness — a direct analog of the "verbosity bias" known in RLHF but manifesting through a completely different mechanism (log-ratio amplification rather than reward model exploitation).

This innovation is a negative result with diagnostic value rather than a positive contribution. It tells future practitioners: reward extrapolation works, but only up to a point, and the failure mode is implicit reward hacking with a specific signature (increasing response length, higher entropy, eventual accuracy degradation). It also suggests a research direction the paper doesn't pursue: if the implicit reward's biases (length, specific token patterns) could be characterized and corrected, perhaps higher λ values could be used safely, pushing the extrapolation envelope further. The connection to the broader reward hacking literature — and the fact that even a "clean" signal like log-probability ratios can be exploited — is a conceptually valuable cautionary tale.

5. Experimental Analysis

Evaluation Methodology

Dataset. The primary training data consists of 57K math samples from DeepMath (He et al., 2025), filtered to difficulty level ≥ 6, and 25K code samples from Eurus-RL-Code (Cui et al., 2025). For distillation, the training data is identical to the RL data used to create domain teachers — the same prompts are used for teacher RL training and for student G-OPD training. In multi-teacher experiments, the math data is downweighted to match the code data size (25K each) for balanced domain representation.

Base model(s). The primary base model is Qwen3-4B-Non-Thinking (Yang et al., 2025a), chosen as a representative mid-scale model where RL fine-tuning produces meaningful capability gains without saturating benchmark performance. Domain teachers are obtained by applying GRPO to this same base model on domain-specific data (producing Qwen3-4B-Non-Thinking-RL-Math and Qwen3-4B-Non-Thinking-RL-Code). For strong-to-weak experiments, the teacher is Qwen3-30B-A3B-Instruct-2507, and students are Qwen3-1.7B-Non-Thinking and Qwen3-4B-Non-Thinking. The reward correction experiments use Qwen3-4B-Non-Thinking-RL-Math/Code as teachers with Qwen3-1.7B-Non-Thinking as the student.

Metrics. The primary metric is average accuracy — the fraction of problems for which at least one sampled solution is correct. For math reasoning, correctness is verified using Math-Verify, a rule-based verifier that checks final answers. For code generation, correctness is determined by whether all unit tests pass. On math benchmarks, 32 solutions are sampled per problem; on code benchmarks, 4 solutions per problem. All evaluations use temperature 1.0, top-p 1.0, and maximum generation length 16,384 tokens. The paper also reports average response length (in tokens) and training dynamics (reward, response entropy) as secondary metrics for analyzing model behavior.

Baselines. Compared methods include:

• SFT (off-policy distillation): Supervised fine-tuning of the student on trajectories generated by the teacher, using standard cross-entropy loss. The number of teacher-generated trajectories per problem is matched to the number of student-generated trajectories in OPD/ExOPD for fair comparison.
• Standard OPD (λ = 1.0): The baseline on-policy distillation approach from Agarwal et al. (2024), Gu et al. (2024), and Lu and Lab (2025), which corresponds to G-OPD with λ = 1.0.
• ExPO (Zheng et al., 2025): A training-free weight extrapolation method that averages domain teachers' weights and extrapolates against the student model using an extrapolation factor α, tuned from {0.25, 0.5}.
• Base student model (λ = 0.0): The original Qwen3-4B-Non-Thinking without any distillation, representing the lower bound.
• Domain teachers: The RL-trained models themselves (Qwen3-4B-Non-Thinking-RL-Math and Qwen3-4B-Non-Thinking-RL-Code), representing the performance ceiling that standard OPD is expected to approach but not exceed.

Generation budget / compute accounting. Compute is measured in optimization steps rather than FLOPs or GPU-hours. GRPO teacher training uses 500 steps for math and 300 steps for code, each with batch size 128 and rollout n=8 (1024 responses per step total). G-OPD distillation uses 50 optimization steps for same-sized experiments and 100 steps for strong-to-weak experiments, each with batch size 1024 and rollout n=1 (1024 responses per step total). The per-step compute for G-OPD is higher than for GRPO because three models (student, teacher, reference) must perform forward passes to compute token-level log-probabilities, versus GRPO's single-policy evaluation plus reward verification. The paper does not explicitly account for the cost of difficulty estimation or reference model inference in its compute comparisons.

Cross-validation / statistical protocol. The paper does not employ cross-validation or statistical significance testing. The λ sweep in Section 4.1.2 serves as the primary hyperparameter validation: λ ∈ {0.0, 0.25, 0.5, 0.75, 1.0, 1.25, 1.5} is tested on single-teacher distillation, and λ = 1.25 is selected based on consistent superiority across all benchmarks in both math and code domains. This fixed λ = 1.25 is then used for all subsequent experiments (multi-teacher and strong-to-weak) without further tuning, which serves as a form of validation through generalization. No separate validation split is used — the training data is the same as the RL data, and evaluation is on held-out benchmarks that were not part of the training distribution.

---

Main Quantitative Results

Single-Teacher Same-Sized Distillation: λ Sweep and ExOPD Emergence

The paper's foundational empirical result is the systematic sweep over reward scaling factors λ in the same-sized single-teacher setting (Section 4.1.2, Figures 2, 3, and Figure 9a), which establishes three clear regimes and identifies λ = 1.25 as the consistent winner.

Standard OPD (λ = 1.0) recovers teacher performance near-losslessly. Across all four math benchmarks, the λ = 1.0 student closely matches the domain teacher's accuracy (Figure 2): on AIME24, OPD achieves 60.7% versus the teacher's 58.0% (already a +2.7 point gain, shown in Table 2); on AIME25, 55.0% versus 54.6% (+0.4); on HMMT25 (Feb.), 32.4% versus 32.5% (−0.1); and on HMMT25 (Nov.), 37.9% versus 38.9% (−1.0). The average math accuracy of 46.5% tracks the teacher's 46.0% closely. Response lengths in Figure 9a show a parallel pattern: the λ = 1.0 student's average token count essentially matches the teacher's, confirming that OPD reproduces not just the accuracy but the behavioral profile of the teacher.

Reward interpolation (0 < λ < 1) produces a monotonic accuracy-length continuum. As λ increases from 0.0 to 1.0, both accuracy and response length increase monotonically (Figure 9a). The λ = 0.5 student generates responses whose average token count falls roughly midway between the base model and the teacher, and whose accuracy on each benchmark interpolates between the two extremes. This is not simply a consequence of seeing less training signal — it reflects the optimal solution of G-OPD, where the student's log-probability is a convex combination of teacher and reference distributions (Eq. 12). The monotonicity is consistent across all benchmarks (Figures 2 and 3), with no visible threshold effects or discontinuities.

Reward extrapolation (λ > 1) enables exceeding the teacher, with λ = 1.25 as the sweet spot. The headline result from Table 2 (top section, single-teacher distillation): ExOPD with λ = 1.25 achieves 62.7% on AIME24 (+4.7 over the teacher), 56.1% on AIME25 (+1.5), 33.9% on HMMT25 (Feb.) (+1.4), and 39.3% on HMMT25 (Nov.) (+0.4), for an average of 48.0% (+2.0 over the teacher). On code generation, ExOPD achieves 86.9% on HumanEval+ (+0.9 over the teacher), 70.7% on MBPP+ (+0.5), and 28.6% on LiveCodeBench (+1.3), for an average of 62.1% (+0.9). The gains are consistent across all seven benchmarks — no benchmark shows a regression.

Excessive extrapolation (λ = 1.5) degrades performance. Figures 2 and 3 show that λ = 1.5 students perform worse than λ = 1.25 students across most benchmarks. On AIME24, the λ = 1.5 accuracy drops noticeably below the λ = 1.25 peak (visible in Figure 2, though exact numbers aren't in Table 2). On code generation (Figure 3), the λ = 1.5 degradation is more pronounced, with LiveCodeBench performance dropping substantially — likely due to the implicit reward's length bias being exploited more aggressively at higher λ. Figure 9a confirms that response length continues to increase at λ = 1.5 even as accuracy plateaus or declines, consistent with the reward hacking interpretation.

ExOPD's advantage is not explained by teacher undertraining. Table 1 provides the critical control experiment: giving the math teacher an additional 100 steps of GRPO training (doubling or more its original 500-step budget) yields improvements of +2.9, +0.5, +0.3, and −0.5 points on the four math benchmarks (average +0.9). ExOPD with only 50 distillation steps achieves +4.7, +1.5, +1.4, and +0.4 (average +2.0). The mechanism is qualitatively different: continued RL training provides diminishing returns (and even a regression on HMMT25 Nov.), while extrapolation from the same teacher produces larger and more consistent gains with less compute. This is direct evidence that ExOPD is not simply doing "more of the same" optimization — it is finding a different, better solution by amplifying the teacher's learned improvement direction rather than continuing to climb the original RL gradient.

Multi-Teacher Distillation: ExOPD Unifies Without Compromise

The multi-teacher setting (Section 4.1.3, Table 2 bottom section) tests whether ExOPD can merge two domain-specific experts into a single student that performs well on both domains simultaneously. This is a harder test than single-teacher distillation because the optimization must balance conflicting gradients — what improves math performance might not improve code performance, and vice versa.

Standard OPD produces a student bounded by teachers with domain tradeoffs. The multi-teacher OPD student achieves 60.6% on AIME24 (+2.6 over the math teacher), 54.1% on AIME25 (−0.5), 32.5% on HMMT25 Feb. (0.0), and 38.3% on HMMT25 Nov. (−0.6), for a math average of 46.4% (+0.4 over the math teacher's 46.0%). On code, it achieves 84.6% on HumanEval+ (−1.4 vs. the code teacher), 69.5% on MBPP+ (−0.7), and 27.6% on LiveCodeBench (+0.3), for a code average of 60.6% (−0.6 vs. the code teacher's 61.2%). The pattern is clear: OPD approaches but does not consistently exceed either teacher, and the code performance actually degrades slightly relative to single-teacher OPD (which achieved 60.8% code average).

ExOPD (λ = 1.25) creates a unified student that surpasses all domain teachers. The multi-teacher ExOPD student achieves 61.0% on AIME24 (+3.0 over the math teacher), 56.0% on AIME25 (+1.4), 34.4% on HMMT25 Feb. (+1.9), and 39.2% on HMMT25 Nov. (+0.3), for a math average of 47.7% (+1.7 over the math teacher). On code, it achieves 86.3% on HumanEval+ (+0.3 over the code teacher), 70.6% on MBPP+ (+0.4), and 29.0% on LiveCodeBench (+1.7), for a code average of 62.0% (+0.8 over the code teacher). Compared to single-teacher ExOPD (48.0% math, 62.1% code), the multi-teacher variant shows only minor degradation (−0.3 math, −0.1 code) while gaining both capabilities — a near-lossless merger that actually improves on three of seven benchmarks (HMMT25 Feb., MBPP+, LiveCodeBench).

SFT (off-policy distillation) is substantially suboptimal. The multi-teacher SFT student achieves only 44.3% average math accuracy (−1.7 vs. the math teacher) and 60.8% average code accuracy (−0.4 vs. the code teacher). On math, SFT underperforms OPD by 2.1 points (46.4% vs. 44.3%) and ExOPD by 3.4 points (47.7% vs. 44.3%). The gap is particularly large on HMMT25 benchmarks (Feb.: 30.7% SFT vs. 34.4% ExOPD; Nov.: 34.8% SFT vs. 39.2% ExOPD), suggesting that off-policy training on teacher-generated trajectories fails to capture the reasoning patterns needed for harder competition problems.

ExPO (weight extrapolation) is brittle and cannot guarantee teacher-surpassing performance. The multi-teacher ExPO student achieves 45.0% average math accuracy (−1.0 vs. the math teacher) — actually worse than the single-teacher ExPO variant (45.8%). On code, ExPO does better: 62.6% average (+1.4 over the code teacher), and on two code benchmarks it achieves the best performance (MBPP+ at 72.0%, LiveCodeBench at 29.0%). However, ExPO's results are inconsistent: it outperforms ExOPD on code (62.6% vs. 62.0%) but significantly underperforms on math (45.0% vs. 47.7%). The paper's claim that ExOPD is "the only method that yields a unified student that consistently outperforms all domain teachers" (Figure 1 caption) is supported by the data: ExPO fails to surpass the math teacher (45.0% vs. 46.0%), while ExOPD surpasses both (47.7% math, 62.0% code).

Training dynamics reveal ExOPD's mechanism. Figure 12a compares OPD and ExOPD during multi-teacher training. ExOPD achieves higher training rewards throughout — this is expected because λ = 1.25 amplifies the reward term. More revealing is the response length: ExOPD students generate progressively longer responses over training, consistent with the extrapolation pushing beyond the teacher's distribution. Response entropy is also higher for ExOPD than OPD, which the paper attributes to longer responses increasing diversity — an interesting observation that suggests the extrapolation doesn't simply produce more confident (lower-entropy) outputs but explores a wider distribution.

Strong-to-Weak Distillation: ExOPD and Reward Correction

The strong-to-weak setting (Section 4.2, Table 3) tests whether ExOPD benefits transfer when distilling from a much larger teacher (Qwen3-30B-A3B-Instruct-2507, with 59.7% average math accuracy) into smaller students (Qwen3-1.7B and Qwen3-4B).

ExOPD provides significant gains over standard OPD in default strong-to-weak setting. For the Qwen3-1.7B student (Table 3, top section), standard OPD achieves 23.1% average math accuracy, while ExOPD achieves 25.4% (+2.3). The absolute gains are largest on the most challenging benchmarks: +4.3 on AIME24 (33.0% → 37.3%) and +2.8 on AIME25 (28.7% → 31.5%). For the Qwen3-4B student, OPD achieves 42.6% average, while ExOPD achieves 45.3% (+2.7). Again, the largest absolute gains are on AIME24 (+3.7) and HMMT25 Feb. (+3.2). The consistency of these improvements — positive gains across all four benchmarks for both student sizes — confirms that reward extrapolation works even when the implicit reward log(π*/π_base^student) contains cross-model-capacity noise.

SFT substantially underperforms OPD and ExOPD in strong-to-weak setting. For Qwen3-1.7B, SFT achieves only 13.5% average math accuracy — less than 60% of ExOPD's 25.4%. The gap is especially stark on competition problems: 18.1% vs. 37.3% on AIME24, 20.5% vs. 31.5% on AIME25. For Qwen3-4B, SFT achieves 35.1% average versus ExOPD's 45.3%. These results quantify the cost of off-policy training: when distilling from a large teacher to a small student, learning from the teacher's outputs directly (SFT) captures substantially less knowledge than on-policy learning with token-level teacher feedback.

Reward correction further improves ExOPD performance. Figure 14a (Section 4.2.3) shows the effect of switching the reference model from π_base^student (Qwen3-1.7B-Non-Thinking) to π_base^teacher (Qwen3-4B-Non-Thinking, the pre-RL variant of the teacher). The corrected reward yields consistent improvements over default ExOPD across all benchmarks, though exact numbers depend on the teacher-student pair (here, Qwen3-4B teachers distilling into Qwen3-1.7B student). The paper does not report a full strong-to-weak table with reward correction — Figure 14a is a bar chart — but the trend is clear and consistent, validating the theoretical argument that log(π*/π_base^teacher) provides a cleaner signal by isolating the teacher's RL improvements from the base capacity gap.

---

Ablation Studies and Robustness Checks

Teacher continued training vs. ExOPD (Table 1): Giving the math teacher an additional 100 steps of GRPO training (beyond its original 500 steps) yields an average improvement of +0.9 points across four benchmarks, with one benchmark showing a regression (−0.5 on HMMT25 Nov.). ExOPD with 50 distillation steps yields +2.0 average improvement. This demonstrates that ExOPD's gains are not explained by the teacher being undertrained — extrapolation captures something that continued RL along the original gradient does not.

λ sweep across the full range (Figures 2, 3, and Figure 9a): The sweep covers λ ∈ {0.0, 0.25, 0.5, 0.75, 1.0, 1.25, 1.5} in both math and code domains, with each λ value producing a separately trained model. The smooth monotonic trends from λ = 0 to λ = 1 confirm the interpolation property predicted by Eq. 12. The peak at λ = 1.25 and subsequent degradation at λ = 1.5 is consistent across all seven evaluation benchmarks, establishing that the extrapolation sweet spot is not benchmark-specific.

Multi-teacher baseline suite (Table 2, bottom): Four methods (SFT, ExPO, OPD, ExOPD) are compared on the same multi-teacher task, using matched data quantities and fair compute budgets. ExPO's α parameter is tuned from {0.25, 0.5} following the original paper's recommendations, so the comparison is fair to the baseline. The consistent superiority of ExOPD over ExPO on math (47.7% vs. 45.0%) despite ExPO's advantage on code (62.6% vs. 62.0%) highlights that the methods have different strengths, but only ExOPD guarantees teacher-surpassing performance on both domains.

Data downweighting for domain balance (Section 4.1.3): The math RL dataset (57K samples) is downweighted to match the code RL dataset (25K samples) in multi-teacher experiments, preventing domain imbalance from confounding the results. This is not presented as a formal ablation but is a control worth noting — without it, math performance might be artificially inflated relative to code simply due to more training data.

Student scale variation in strong-to-weak (Table 3): Two student sizes (Qwen3-1.7B and Qwen3-4B) are tested with the same Qwen3-30B teacher. The consistent pattern of ExOPD > OPD > SFT across both student scales suggests the extrapolation benefit is robust to the student-teacher capacity gap, though the absolute gap between methods narrows for the larger student (42.6% → 45.3%, a +2.7 gain for Qwen3-4B vs. 23.1% → 25.4%, a +2.3 gain for Qwen3-1.7B).

Discount factor of 0 for gradient computation (Appendix A, Eq. 19-21): The paper acknowledges that the full OPD gradient includes future-token contributions but that prior work and this paper use a discount-0 approximation. The mathematical justification (Eq. 19 shows past tokens' expected gradients vanish) is provided in the appendix but is not experimentally ablated — no comparison between discount-0 and full-gradient OPD is reported.

Token-level rollout correction (Section 4.1.1): Applied to both GRPO and G-OPD training to mitigate training-inference mismatch, following Liu et al. (2025b). This is not ablated — there are no experiments showing performance with and without rollout correction. Its effect on G-OPD specifically is unknown from the reported results.

Optimization step sensitivity (Section 4.1.1, Appendix B): The paper states that "further increasing the number of distillation steps may degrade generalization performance due to overfitting," but provides no learning curves or step-count ablations to quantify this. The 50-step and 100-step choices are empirical but unexplained in detail.

---

Critical Assessment

Claim 1: "ExOPD consistently enables the distilled student to surpass domain teachers in both single-teacher and multi-teacher settings." This claim is supported by the data in Table 2, but with important nuance. In single-teacher distillation, ExOPD (λ = 1.25) surpasses the teacher on all seven benchmarks — a clean result. In multi-teacher distillation, ExOPD surpasses the math teacher by +1.7 points average (47.7% vs. 46.0%) and the code teacher by +0.8 points average (62.0% vs. 61.2%). However, the code margin is small: on two of three code benchmarks, ExOPD's improvement is less than 1 point (+0.3 on HumanEval+, +0.4 on MBPP+), which raises the question of statistical significance given the small test sets (HumanEval+ has 164 problems, MBPP+ has ~400, LiveCodeBench v6 has an unspecified but likely small number of problems within the February–May 2025 window). The paper reports no confidence intervals or standard errors, so whether a +0.3 or +0.4 point improvement is distinguishable from noise is unclear. The claim holds most robustly for math reasoning, where margins are larger and more consistent.

Claim 2: "ExOPD is the only method that yields a unified student that consistently outperforms all domain teachers." This claim is supported by Table 2's multi-teacher section but depends on the definition of "consistently." ExPO outperforms the code teacher by +1.4 points average (62.6% vs. 61.2%) — a larger margin than ExOPD's +0.8 — but fails to surpass the math teacher (45.0% vs. 46.0%). OPD fails to surpass the code teacher (60.6% vs. 61.2%) and barely surpasses the math teacher (+0.4). The claim is technically correct: only ExOPD exceeds both teachers on their respective domain averages. But the margins are small on code, and the comparison rests on domain-averaged numbers. Looking at individual benchmarks: on code, ExOPD underperforms ExPO on HumanEval+ (86.3% vs. 86.7%) and MBPP+ (70.6% vs. 72.0%), while matching on LiveCodeBench (29.0% vs. 29.0%). So ExOPD is not uniformly the best method on code — ExPO edges it out on two of three benchmarks. The claim would be more precise as "ExOPD is the only method that produces a student exceeding the domain-average performance of all teachers," but the "consistently outperforms all domain teachers" framing glosses over per-benchmark code results.

Claim 3: "Reward extrapolation works only when appropriate λ values are chosen and excessive extrapolation (λ = 1.5) is avoided due to implicit reward hacking." Strongly supported. The λ = 1.5 degradation is visible across both math and code (Figures 2, 3), and the paper provides a plausible mechanism (length bias exploitation, as seen in Figure 9a's continued response length growth at λ = 1.5). However, the paper only tested three λ values above 1.0 (1.25 and 1.5). It's unclear whether λ = 1.1 or λ = 1.4 would perform better or worse — the sweet spot is identified as "around 1.25" but not precisely characterized. A finer-grained sweep (e.g., 1.1, 1.2, 1.3, 1.4) would strengthen confidence that 1.25 is genuinely optimal rather than simply the best of two tested options.

Claim 4: "Reward correction in strong-to-weak distillation further improves performance by providing a cleaner implicit reward signal." Supported by Figure 14a, but the evidence is limited. The reward correction experiment uses a different teacher-student pair (Qwen3-4B teachers distilling into Qwen3-1.7B) than the main strong-to-weak experiments (Qwen3-30B teacher distilling into Qwen3-1.7B/4B). The paper explicitly acknowledges this is because they "cannot get the pre-RL variant of Qwen3-30B-A3B-Instruct-2507." This means the reward correction claim is validated only in a setting where the teacher and student base models are from the same family and scale (Qwen3-4B for teacher, Qwen3-1.7B for student), not in the more extreme cross-scale setting where the correction is theoretically most needed. The paper also does not ablate other possible reference model choices (e.g., a different student checkpoint, a randomly initialized model) to confirm that the improvement comes specifically from isolating the teacher's RL improvements rather than from some other property of using a larger reference model.

Experimental design weaknesses:

• No statistical significance reporting. All results are point estimates without confidence intervals, standard errors, or significance tests. Given the relatively small test sets (AIME has 30 problems per year, HMMT likely similar, HumanEval+ has 164 problems), some of the smaller reported gains (+0.3, +0.4) may not be statistically distinguishable from noise. This is particularly concerning for the multi-teacher code results where ExOPD's margin over the code teacher is thin.

• Single model family. All experiments use Qwen3 variants. The paper acknowledges this in the conclusion as future work ("evaluating the effectiveness of ExOPD for on-policy distillation across different model families") but does not test it. The extrapolation mechanism depends on the implicit reward log(π*/π_ref), whose properties may differ across model architectures with different log-probability calibration characteristics. A model with better-calibrated log-probabilities might support higher λ values before reward hacking; a poorly calibrated model might show reward hacking at lower λ.

• λ = 1.25 selected and fixed without validation on a held-out tuning set. The λ sweep in Section 4.1.2 is performed on the single-teacher setting, and the best-performing λ (1.25) is then applied to multi-teacher and strong-to-weak settings without retuning. While this demonstrates some robustness, it's possible that different settings benefit from different λ values. The strong-to-weak setting in particular might benefit from a different λ since the implicit reward characteristics change (noisier signal due to capacity gap). The paper provides no λ sensitivity analysis for strong-to-weak distillation — only the default λ = 1.25 is tested.

• Limited optimization step exploration. The paper uses 50 steps for same-sized distillation and 100 for strong-to-weak, stating that further steps cause overfitting, but provides no learning curves, validation metrics, or step-count ablations. The reader cannot assess whether these step counts are near-optimal or whether longer training with appropriate regularization (early stopping, learning rate decay) would yield further gains.

• No ensemble or multi-seed results. All experiments appear to use single training runs. Without multiple seeds, it's impossible to distinguish genuine method differences from training stochasticity — a concern given the small margins on some benchmarks.

Missing experiments that would strengthen the paper:

• A λ sweep in the strong-to-weak setting, to test whether the optimal λ differs when the implicit reward is noisier.
• An ablation where the reference model in G-OPD is varied systematically — e.g., using a weaker model, a random model, or a model from a different family — to map out when and why the reference model choice matters beyond the specific reward correction case.
• A discount factor ablation comparing the discount-0 gradient (Eq. 21) against the full gradient (Eq. 20) to quantify the approximation cost.
• Token-level rollout correction ablation to isolate its contribution to G-OPD performance.
• Results on a second model family (e.g., LLaMA or DeepSeek) to begin validating the claimed generality.
• A direct ExOPD vs. continued teacher RL comparison with identical FLOP budgets, rather than the current comparison which uses different step counts (50 ExOPD steps vs. 100 additional GRPO steps). The per-step compute differs between methods, making it hard to assess whether ExOPD is genuinely more compute-efficient or just uses a different compute allocation.

6. Limitations and Trade-offs

6.1 The λ Sweet Spot Is Narrow, Uncharacterized, and Setting-Dependent

The assumption or constraint. The paper identifies λ = 1.25 as the empirically optimal reward scaling factor through a sweep in the same-sized single-teacher setting (Section 4.1.2), then fixes this value for all subsequent experiments — multi-teacher (Section 4.1.3) and strong-to-weak (Section 4.2) — without retuning. The sweep itself is coarse: only two values above λ = 1.0 are tested (1.25 and 1.5). The paper acknowledges implicitly that extrapolation is fragile, stating that "excessive reward extrapolation (λ = 1.5) may lead to instability and degrade performance" (Section 4.1.2) and attributing this to the student "aggressively fitting the peak of the log ratio, even if some tokens have excessively large log ratios due to bias." However, the paper never characterizes why λ = 1.25 works and λ = 1.5 fails — whether the degradation is gradual (e.g., λ = 1.3 performs slightly worse than 1.25 but still above 1.0) or abrupt (a cliff between 1.25 and 1.5). Nor does it test whether different distillation settings (multi-teacher, strong-to-weak, different model scales, different domains) have different optimal λ values.

The consequence. A practitioner wanting to apply ExOPD to their own model, dataset, or domain faces an expensive hyperparameter tuning problem with no guidance beyond "try 1.25 first." If the optimal λ is setting-specific — and the paper provides no evidence either way — then the headline results may not transfer. Worse, if the degradation is abrupt, a practitioner who starts at λ = 1.5 might incorrectly conclude that ExOPD hurts performance. The paper's failure to characterize the shape of the λ-performance curve around the optimum leaves users without a tuning strategy. They don't know whether to sweep finely around 1.25 (1.2, 1.3) or more broadly, or whether the optimal λ correlates with measurable quantities (teacher-student capacity gap, dataset size, domain difficulty) that could guide selection.

What evidence exists in the paper. The λ sweep in Figures 2 and 3 shows the full trajectory from 0.0 to 1.5 across all seven evaluation benchmarks, with the peak consistently at 1.25 and clear degradation at 1.5. Figure 9a shows response length continuing to increase at 1.5 even as accuracy plateaus or declines — evidence consistent with implicit reward hacking. However, the sweep contains no values between 1.0 and 1.25 (e.g., 1.1, 1.15, 1.2) and no values between 1.25 and 1.5 (e.g., 1.3, 1.35, 1.4). The paper provides no λ sensitivity analysis for multi-teacher or strong-to-weak settings — only the default λ = 1.25 is tested there. The training dynamics in Figure 12a compare λ = 1.0 vs. λ = 1.25 only, giving no insight into what changes between 1.25 and 1.5.

Mitigation status. Not addressed. The paper treats λ = 1.25 as a fixed constant after the initial sweep, with no discussion of how a practitioner should select λ for a new setting, no characterization of the optimum's sharpness, and no proposed heuristics or automatic tuning methods. The conclusion mentions future work on "validating the generalizability of ExOPD on larger-scale models" and "assessing the robustness of ExOPD in multi-teacher distillation with a broader and more diverse set of domain teachers" (Section 5), but these focus on extending to new settings rather than making the method easier to tune in those settings.

---

6.2 Reward Correction Requires an Additional Large Model and Incurs Unaccounted Inference Cost

The assumption or constraint. The reward correction technique for strong-to-weak distillation (Section 3.2, "Reward correction in strong-to-weak distillation") replaces the default reference model π_base^student (the small student's initial checkpoint) with π_base^teacher (the large teacher's pre-RL variant). The paper explicitly acknowledges two limitations: "this requires access to π_base^teacher and incurs additional computation, since computing log π_base^teacher requires more cost than computing log π_base^student." Furthermore, the main strong-to-weak experiments (Table 3, Section 4.2.2) do NOT use reward correction because the authors "cannot get the pre-RL variant of Qwen3-30B-A3B-Instruct-2507." Instead, the reward correction experiments (Figure 14a, Section 4.2.3) use a proxy setup: Qwen3-4B teachers distilling into Qwen3-1.7B, where the pre-RL variant (Qwen3-4B-Non-Thinking) is available. The paper never quantifies the additional computational cost of using π_base^teacher rather than π_base^student, nor does it include this cost in any efficiency comparison.

The consequence. The practical value of reward correction is conditional on an assumption that many real-world strong-to-weak distillation scenarios will not satisfy. When distilling from a proprietary or API-only large model (e.g., GPT-4, Claude, Gemini), the pre-RL base model is simply unavailable — the provider releases only the post-trained checkpoint. Even when the pre-RL variant is available (e.g., for open-weight models like Qwen or LLaMA), the additional inference cost is non-trivial: every G-OPD training step requires computing log-probabilities under π_base^teacher for every token of every student-generated sequence. If π_base^teacher is the 30B model's base variant and the student is 1.7B, the reference model forward passes dominate the training compute. The paper's headline ExOPD results for strong-to-weak distillation (Table 3) do NOT benefit from reward correction — they use the noisy default reward — so the reported gains are achievable without the extra model, but the claimed further improvement from reward correction is validated only in a smaller-scale proxy setting that may not generalize to the large-scale case where it's most needed.

What evidence exists in the paper. Figure 14a shows consistent improvement from reward correction for Qwen3-4B teachers → Qwen3-1.7B student, validating the theoretical argument. However, this is a 4B → 1.7B transfer (same model family, modest scale gap), not the 30B → 1.7B or 30B → 4B transfer in the main experiments. The paper provides no FLOP comparison, no wall-clock timing, and no cost analysis for the extra reference model inference. Table 3's default ExOPD results (without reward correction) serve as the realistic baseline for scenarios where π_base^teacher is unavailable.

Mitigation status. Partially addressed through transparency — the authors explicitly state the limitation rather than hiding it. However, they do not propose any workaround for the missing-model case, such as approximating π_base^teacher with a smaller model from the same family, using a distilled version, or training a lightweight proxy. The conclusion mentions no specific future work on reducing the cost or relaxing the access requirement for reward correction.

---

6.3 All Experiments Use a Single Model Family; Transfer to Other Architectures Is Unvalidated

The assumption or constraint. All experiments — GRPO teacher training, G-OPD distillation, and evaluation — use models from the Qwen3 family: Qwen3-4B-Non-Thinking as the primary base model, Qwen3-1.7B-Non-Thinking and Qwen3-4B-Non-Thinking as strong-to-weak students, and Qwen3-30B-A3B-Instruct-2507 as the large teacher. The paper acknowledges this limitation in the conclusion: "evaluating the effectiveness of ExOPD for on-policy distillation across different model families" is listed as future work (Section 5, item 3). The implicit reward function log(π*/π_ref) depends on token-level log-probabilities, whose calibration properties vary across model architectures and training procedures. A model family with poorly calibrated log-probabilities (overconfident or underconfident) would produce noisier or systematically biased implicit rewards, potentially changing the optimal λ or making reward extrapolation unsafe at lower λ values.

The consequence. A practitioner using a non-Qwen model (LLaMA, DeepSeek, Mistral, Gemma) cannot assume that λ = 1.25 is the right choice, that ExOPD will outperform standard OPD, or that the interpolation regime will produce the same monotonic accuracy-length tradeoff. The extrapolation mechanism amplifies log(π*/π_ref); if this log-ratio behaves differently in another model family — for instance, if the log-probability shift during RL is systematically larger or smaller, or if length bias is more or less severe — the entire λ sweep could shift. The paper provides no diagnostic for assessing whether a given model family produces well-behaved implicit rewards, and no evidence that the framework transfers.

What evidence exists in the paper. None. All results are on Qwen3 variants. The paper cites prior OPD work that used different model families (Agarwal et al., 2024, used PaLM; Gu et al., 2024, used LLaMA-derived models), but these are citations for standard OPD, not for ExOPD specifically. There are no cross-family experiments, no comparison of log-probability calibration across models, and no analysis of whether the implicit reward's properties (mean, variance, length correlation) are consistent across architectures.

Mitigation status. Acknowledged as future work but not addressed experimentally. The authors state they believe it is "practical to explore" evaluating ExOPD across model families (Section 5), but this is listed alongside other future directions without prioritization or proposed methodology. A minimal validation — testing ExOPD on at least one non-Qwen model pair — would substantially strengthen the generalizability claim.

---

6.4 No Statistical Significance Reporting; Small Test Sets Make Some Reported Gains Indistinguishable from Noise

The assumption or constraint. All results in Tables 1–3 and Figures 2–3 are reported as point estimates (accuracy percentages) without confidence intervals, standard errors, or statistical significance tests. The evaluation benchmarks have relatively small problem counts: AIME has 30 problems per year, HMMT likely has a similar competition format (the paper does not report exact test set sizes, but Math-Verify-based benchmarks of this type typically contain 30–50 problems per contest), HumanEval+ has 164 problems, and MBPP+ has approximately 400 problems. With these sample sizes, a change of ±1 percentage point on a single benchmark corresponds to fewer than 1–2 additional problems solved correctly, which may fall within the range of sampling variability. The paper samples 32 solutions per math problem and 4 per code problem, averaging over multiple samples per problem — this reduces variance from the sampling process but does not eliminate test-set-level uncertainty about whether a model genuinely outperforms another on the underlying problem distribution.

The consequence. Some of the paper's central comparative claims rest on margins that may not be statistically reliable. In the multi-teacher setting (Table 2), ExOPD's improvement over the code teacher is +0.3 points on HumanEval+ (86.3% vs. 86.0%), +0.4 points on MBPP+ (70.6% vs. 70.2%), and +1.7 points on LiveCodeBench (29.0% vs. 27.3%). The first two of these represent approximately 0.5–1.6 additional correct problems on benchmarks of 164 and ~400 problems respectively — differences that could plausibly arise from sampling noise alone, especially given that 4 samples per problem are averaged. The claim that ExOPD is "the only method that yields a unified student that consistently outperforms all domain teachers" (Figure 1 caption) rests partly on these code margins. On math, the margins are larger and more likely meaningful, but on code, the evidence that ExOPD genuinely outperforms the code teacher (rather than matching it within noise) is weak. Similarly, the comparison between ExOPD and ExPO on code (62.0% vs. 62.6% average) — where ExPO wins — involves margins that may not be statistically distinguishable.

What evidence exists in the paper. The paper reports no statistical measures whatsoever. There is no mention of standard deviations across the 32 or 4 samples per problem, no bootstrapped confidence intervals on the benchmark averages, and no significance tests for pairwise comparisons. The evaluation protocol (32 samples per math problem, 4 per code problem) is described in Section 4.1.1, but the paper does not discuss how this sampling configuration affects the reliability of the reported averages. The test set sizes are not explicitly stated — the reader must infer them from knowledge of the benchmarks or prior work.

Mitigation status. Not addressed. The paper's evaluation methodology section focuses on benchmark selection and sampling configuration but does not discuss statistical rigor. This is a common weakness in LLM benchmarking papers, where test sets are treated as fixed populations and all differences are reported as meaningful — but it is a weakness nonetheless, and it affects the interpretation of some of the paper's more marginal claims.

---

6.5 Single-Domain Training Data and Narrow Benchmark Selection; Generalization to Non-Verifiable, Open-Ended, or Broader Tasks Is Unknown

The assumption or constraint. All training and evaluation uses verifiable tasks with binary correctness signals: math reasoning problems with deterministic final answers (verified by Math-Verify) and code generation problems with unit tests. The domain teachers are trained with GRPO using a reward of 1.0 for correct answers / passing tests and 0.0 otherwise (Section 4.1.1). The distillation data is identical to the RL data — the same prompts are used for teacher training and student distillation. This setup relies on the availability of (1) a large dataset of verifiable problems with clean correctness signals, (2) a rule-based verifier or test suite that reliably judges correctness, and (3) a well-defined notion of a "correct" answer that the model can produce through chain-of-thought reasoning or code generation. The training data is domain-specific and relatively narrow: 57K math problems filtered to difficulty ≥ 6, and 25K code problems.

The consequence. The entire G-OPD framework — including reward interpolation, reward extrapolation, and reward correction — is validated only for tasks where the teacher's RL improvements were obtained through verifiable outcome rewards. In open-ended domains (creative writing, dialogue, summarization, instruction following) where reward models are learned rather than rule-based, the implicit reward log(π*/π_ref) would encode the teacher's alignment with a potentially flawed or over-optimizable learned reward model. Extrapolating such an implicit reward further could amplify biases or reward model errors rather than genuine capability improvements. The paper provides no guidance on whether ExOPD would work, would need modification, or would be actively harmful in these settings. Furthermore, the narrow training data distribution means the distilled student's generalization to out-of-distribution problems, different difficulty levels, or different reasoning formats is completely unmeasured — the evaluation benchmarks (AIME, HMMT, HumanEval+, MBPP+, LiveCodeBench) are all competition-style problems of similar format to the training data.

What evidence exists in the paper. None. The paper does not include any non-verifiable tasks, any learned reward model settings, any broader instruction-following benchmarks, or any out-of-distribution evaluation. The benchmarks are carefully chosen to match the training domain (competition math and coding), which is appropriate for a controlled study but means the results provide no signal about broader applicability. The paper's related work section cites RLHF and preference-based reward models (Cai et al., 2024; Dong et al., 2024; Liu et al., 2025a) in the context of RL reward functions (Section 3.1), acknowledging that rewards can be "a parameterized neural reward model trained on the specific preference data for open-domain alignment," but never tests the G-OPD framework with such rewards.

Mitigation status. Not addressed. The paper makes no claims about applicability beyond verifiable reasoning tasks, which is honest but limiting. The conclusion's future work items focus on scaling, multi-teacher diversity, and cross-family transfer — none mention extending to non-verifiable domains or learned reward models. A practitioner working in open-ended text generation would find no evidence that ExOPD applies to their setting.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper's primary contribution is not a new algorithm, but a reconceptualization that unlocks a previously frozen degree of freedom in an existing and widely-used method. The finding that standard on-policy distillation (OPD) is algebraically equivalent to dense KL-constrained reinforcement learning with a fixed 1:1 reward-to-KL ratio (Section 3.2, Eq. 7–11) changes how the field should think about distillation. Prior to this work, OPD was understood through a knowledge-transfer lens: the student imitates the teacher's distribution, but generates its own training data for efficiency. The paper shows this framing is incomplete — OPD is doing policy optimization under a particular implicit reward function, and that optimization has a knob (λ, or equivalently β) that no one knew existed because the distillation framing didn't suggest it should.

The magnitude of this shift is methodological infrastructure rather than paradigm-shattering. It doesn't replace OPD with something new; it reveals that OPD was a special case of a more general family all along, and that breaking the special case yields consistent gains. This pattern — discovering that an effective method is a constrained version of something richer — has precedent in the field. DPO (Rafailov et al., 2023) emerged from showing that preference-based RLHF could be reparameterized as a direct policy optimization problem, unlocking simpler training pipelines. The G-OPD reframing is conceptually analogous but operates in the opposite direction: it takes a simple distillation objective and shows it is RL, then uses that insight to enrich the objective.

The paper resolves a latent tension in the distillation literature that was previously invisible. Standard OPD was simultaneously praised for efficiency (dense token-level signal, on-policy data) and criticized for its implicit performance ceiling — it could at best match the teacher. The RL reframing shows these properties came from the same source: the 1:1 reward-to-KL weighting ensured stability (the student couldn't over-optimize the implicit reward) but also imposed a ceiling (the student could never amplify the teacher's improvement beyond log π*). ExOPD breaks this coupling. The teacher's role shifts from destination to direction vector — the distilled student can extrapolate along log π* − log π_ref and arrive at a policy the teacher never reached. This is genuinely new terrain for distillation methods.

The paper also redirects attention toward the implicit reward as a diagnostic and design object rather than a black-box consequence of the KL-constrained RL derivation. The finding that log(π*/π_base^student) is noisier than log(π*/π_base^teacher) in strong-to-weak distillation (Section 3.2, "Reward correction in strong-to-weak distillation") is a concrete property with practical consequences — it tells practitioners that the reference model choice matters for reward quality, not just for the KL penalty direction. This opens a line of inquiry around implicit reward engineering that was previously unimagined because the implicit reward was a derived quantity, not a tunable component.

Research directions that become more attractive after this work include: (1) understanding the calibration and failure modes of log-probability ratios across model scales and architectures, since ExOPD's success depends on these ratios being meaningful; (2) exploring adaptive or learned λ schedules rather than fixed values, analogous to how KL coefficient annealing is standard in RL; (3) combining ExOPD with other reward-shaping techniques from RL, such as potential-based reward shaping or advantage normalization, which become natural extensions once OPD is recognized as RL. Research directions that become less attractive include: (1) pursuing ever-more-complex distillation objectives without understanding their RL structure, since the paper shows that a single scalar λ — an RL hyperparameter that was sitting in plain sight — produces larger gains than architectural innovations; and (2) treating weight extrapolation (ExPO) as a general solution for teacher-surpassing performance, since the paper demonstrates it is brittle and inconsistent compared to the optimization-based ExOPD approach (Table 2: ExPO multi-teacher math average of 45.0% vs. math teacher's 46.0%, compared to ExOPD's 47.7%).

Follow-Up Research This Work Enables

Characterizing the λ-performance landscape: where is the cliff, and what controls its location? The paper sweeps λ only at coarse intervals (0.0, 0.25, 0.5, 0.75, 1.0, 1.25, 1.5) and finds the peak at 1.25 with degradation at 1.5 — but never maps the region between 1.25 and 1.5, nor tests whether the degradation is gradual or abrupt. A follow-up study sweeping λ finely (e.g., 1.05, 1.1, 1.15, ..., 1.45, 1.5) on the same Qwen3-4B setup from Section 4.1.2 would reveal the shape of the extrapolation cliff. Even more valuable: testing whether the cliff location correlates with measurable quantities — the implicit reward's per-token variance, the KL divergence between teacher and reference, the length of teacher-generated responses, or the teacher's RL training budget. If the optimal λ can be predicted from properties of the teacher and reference models without an expensive sweep, ExOPD becomes a practical tool rather than a tuning headache. The paper's observation that λ = 1.5 produces longer responses and higher entropy (Figure 9a, Figure 12a) suggests diagnostic signals exist — a study correlating these signals with the onset of reward hacking across multiple model pairs and λ values would be immediately useful.

Testing ExOPD on a second model family with calibrated log-probabilities. The paper's single-model-family limitation (Qwen3 throughout) is the most urgent gap to close. A direct replication on LLaMA-3 or DeepSeek-R1 variants would test whether the implicit reward's properties — and therefore the optimal λ — transfer across architectures. The key measurement is not just whether λ = 1.25 works on a different model family, but whether the shape of the λ-performance curve (monotonic interpolation from 0 to 1, peak at ~1.25, degradation by 1.5) is consistent. If it is, ExOPD generalizes as a method without architecture-specific tuning. If the optimal λ shifts substantially — say, to 1.05 or 1.5 — then architecture-specific implicit reward calibration is a first-order concern, and the paper's fixed λ = 1.25 advice is misleading. A well-designed replication would use the same DeepMath / Eurus-RL-Code training pipeline on a different base model of comparable scale, keeping the data and RL procedure constant while varying only the model architecture, to isolate the architecture's effect on implicit reward behavior.

Combining ExOPD with process reward models or outcome reward models for hybrid supervision. The paper's dense token-level implicit reward log(π*(y_t|...)/π_ref(y_t|...)) provides per-token signal, while the GRPO teachers were trained with sparse outcome rewards (1.0 for correct, 0.0 otherwise). A natural extension is to combine both signals: use ExOPD's implicit reward for dense credit assignment, but also incorporate the outcome reward (or a learned process reward model) as an additional term in the G-OPD objective. This would be a straightforward modification of Eq. 11: add a term α · r_outcome(x, y) alongside the implicit reward. The hypothesis is that the implicit reward provides efficient but potentially noisy token-level guidance, while the outcome reward provides a cleaner (if sparse) signal about final answer quality. Their combination might support higher λ values (stronger extrapolation) without hacking, since the outcome reward would penalize degenerate solutions that score highly under the implicit reward but produce wrong answers. The deep connection here is that the implicit reward is already a proxy for the teacher's RL improvement, and the outcome reward is the signal that produced that improvement — using both closes the loop.

Adaptive or learned λ schedules during training. The paper fixes λ as a constant throughout training. But the RL literature has long known that KL coefficient schedules — starting with high β (low λ, strong regularization) and gradually relaxing — can stabilize training and improve final performance. Testing whether a λ schedule (e.g., start at λ = 1.0 for the first half of training, then increase to λ = 1.25 or beyond) outperforms a fixed λ would connect ExOPD to standard RL best practices. More ambitiously, one could train a small meta-controller that adjusts λ per batch based on training diagnostics (reward hacking indicators like response length growth or entropy increase), implementing a form of automated trust-region control. The paper's training dynamics in Figure 12a already show that ExOPD produces longer responses and higher entropy than OPD — these are candidate signals for detecting impending reward hacking. A simple PID controller that reduces λ when response length exceeds a threshold would be a low-effort, high-impact experiment.

Stress-testing ExOPD on tasks without clean verifiability — what happens when the teacher's RL reward is learned rather than rule-based? The paper's teachers were trained with binary outcome rewards from rule-based verifiers (Math-Verify, unit tests). The implicit reward log(π*/π_ref) therefore encodes the teacher's alignment with a near-perfect reward signal. But many RLHF-trained models are optimized against learned reward models that have known biases (verbosity, sycophancy, specific phrasing preferences). If the teacher was trained with a flawed reward model, ExOPD would amplify the teacher's movement toward that flawed model's preferences — potentially making reward hacking worse in the student than in the teacher. This is a critical negative-result experiment: take a teacher trained with a learned reward model known to have a specific bias (e.g., excessive verbosity), apply ExOPD with varying λ, and measure whether the student's bias amplifies proportionally to λ. If it does, ExOPD is unsafe for learned-reward settings without additional mitigation; if the KL penalty provides sufficient regularization, the extrapolation might be self-limiting. Either outcome gives practitioners essential guidance.

Reward correction without the teacher's pre-RL model: can we approximate π_base^teacher from the post-trained teacher alone? The paper's reward correction technique (Section 3.2, 4.2.3) requires access to the teacher's pre-RL base model, which is unavailable for many deployed models. A practical extension would test whether π_base^teacher can be approximated from π* alone — for instance, by exploiting the property that RL training typically modifies the policy most strongly on tokens related to reasoning chains while leaving general language modeling largely unchanged. One could construct a proxy reference model by extracting the base-model-like components from π*: temperature-scaled logits, logit-lens projections onto a smaller base model, or even a separate lightweight model trained to predict "what would the base model have said here?" The experiment would measure whether an approximated π_base^teacher improves over the default π_base^student in strong-to-weak distillation, even if it underperforms the true π_base^teacher. A positive result would make reward correction applicable to API-only or proprietary teachers.

Practical Applications and Downstream Use Cases

Multi-expert consolidation for multi-domain deployment. The most directly actionable result in the paper is the multi-teacher distillation finding (Table 2, bottom section): ExOPD produces a single student that surpasses both domain teachers (47.7% math, 62.0% code) with minimal degradation from single-teacher ExOPD performance (48.0% math, 62.1% code). For organizations that maintain separate RL-fine-tuned experts for different domains (math, code, science, multilingual), this offers a consolidation pipeline: independently train each expert with domain-specific RL, then merge them into a single deployable model via ExOPD with λ = 1.25. The merged model requires no additional inference-time routing logic, no multi-model serving infrastructure, and no tradeoff where one domain's performance is sacrificed for another. The paper's numbers suggest near-lossless merger: the math average drops only 0.3 points from single-teacher ExOPD (48.0% → 47.7%) and the code average drops only 0.1 points (62.1% → 62.0%). For deployment teams currently running separate model instances or complex mixture-of-experts routing, this simplifies serving while improving per-domain performance beyond any individual expert — a rare combination of operational simplicity and capability gain.

Cost-aware capability engineering via reward interpolation. The paper's finding that λ ∈ (0, 1) produces a continuous spectrum of models whose accuracy and response length vary monotonically with λ (Figure 9a, Section 4.1.2) is immediately useful for cost-sensitive deployments. Rather than deploying the full RL-trained teacher (high accuracy, long responses, high inference cost) or the base model (low accuracy, short responses, low cost), an organization can distill a family of students at different λ values — say, λ = 0.25, 0.5, 0.75 — and route queries based on estimated difficulty or user-specified budget constraints. The paper doesn't demonstrate this routing scenario, but the framework makes it straightforward: each λ student is a single model with a fixed cost-capability profile, and inference-time routing is just model selection. The reported monotonicity means there are no pathological λ values where accuracy drops while cost increases — the entire interpolation range is usable. For applications where most queries are easy (and short responses suffice) but occasional hard queries justify longer reasoning, this enables fine-grained cost control without the complexity of adaptive test-time compute strategies.

Strong-to-weak distillation for edge deployment with limited access to large models. The strong-to-weak results (Table 3) matter for scenarios where a powerful but expensive teacher (Qwen3-30B-A3B-Instruct-2507, 59.7% average math accuracy) must be distilled into a deployable small student (Qwen3-1.7B, 8.8% base accuracy). Standard OPD lifts the 1.7B student to 23.1%; ExOPD lifts it further to 25.4% (+2.3 points), and the reward correction technique promises additional gains when the teacher's pre-RL variant is available (Figure 14a). These absolute numbers are modest — the small student still underperforms the large teacher substantially — but the relative improvement matters when every point counts for on-device performance. The practical scenario involves a cloud-based large model that performs the expensive distillation training (teacher forward passes, G-OPD gradient computation) and produces a small student checkpoint that ships to edge devices. The key practical requirement — access to the teacher's log-probabilities during training — is satisfied in this setup since the teacher runs server-side during distillation. The limitation (need for π_base^teacher for reward correction) is also satisfied if the teacher provider releases the base model, or if the organization trained the teacher themselves. ExOPD's advantage over SFT is starkest here: SFT on the 1.7B student achieves only 13.5% — roughly half of ExOPD's 25.4% — demonstrating that on-policy learning with extrapolation extracts substantially more capability from the same large teacher.

Iterative self-improvement loops: ExOPD as a data generation policy. The paper's demonstrated ability to produce a student that surpasses its same-sized teacher suggests a bootstrapping loop: use the ExOPD-surpassing student as the new teacher for the next round of distillation. Each iteration would (1) start with a base model π_base, (2) train a domain teacher π* via GRPO, (3) distill a student π_ExOPD via ExOPD that outperforms π*, (4) treat π_ExOPD as the new base model for the next iteration, with π* from the previous iteration as the pre-RL reference. The paper's Table 1 provides encouraging evidence that ExOPD is not simply equivalent to longer RL training — it finds a qualitatively different improvement (+2.0 average for 50 ExOPD steps vs. +0.9 for 100 additional GRPO steps). If this advantage compounds over iterations — if each round's ExOPD student genuinely encodes capability that the base model didn't have, and if the next round's GRPO training can build on that capability — then iterative ExOPD could produce a self-improvement flywheel. The practical requirement is substantial compute (each iteration requires GRPO teacher training + ExOPD distillation), but for organizations already running RL fine-tuning pipelines, the marginal cost of adding an ExOPD step between rounds is modest. The key risk — catastrophic collapse from iterative distribution shift — would need to be monitored with held-out benchmarks at each iteration.

When to Prefer This Method

The paper articulates specific tradeoffs among OPD variants and competing methods (SFT, ExPO), which can be summarized as decision rules grounded in the reported results:

• Prefer standard OPD (λ = 1.0) when the goal is faithful reproduction of teacher behavior (accuracy, response length, and output style) on the student's own distribution. The paper shows OPD near-losslessly recovers teacher performance in same-sized settings (Table 2: OPD math average 46.5% vs. teacher 46.0%), making it the safe choice when exceeding the teacher is not required or when λ tuning budget is unavailable.

• Prefer ExOPD (λ = 1.25) when (a) the student and teacher share the same base model (the teacher was obtained via RL from the student's initial checkpoint), and (b) surpassing the teacher's domain performance is the explicit goal. The evidence is strongest for same-sized single-teacher distillation (+2.0 average math accuracy over the teacher, Table 2) and multi-teacher consolidation (the only method surpassing all teachers, Table 2 bottom). The paper provides no tuning guidance beyond λ = 1.25 as a fixed default.

• Prefer ExOPD with reward correction when distilling from a larger teacher to a smaller student AND the teacher's pre-RL base model is available. The corrected reward log(π*/π_base^teacher) provides a cleaner signal by isolating the teacher's RL improvements from the base capacity gap (Section 3.2, Figure 14a). The cost is additional inference compute for π_base^teacher log-probabilities during training.

• Prefer SFT (off-policy distillation) only when compute constraints make on-policy training infeasible (SFT requires no student sampling during training and no reference model forward passes) or when the teacher is a black-box API that exposes outputs but not log-probabilities. The paper consistently shows SFT underperforms all on-policy variants: in multi-teacher distillation, SFT achieves 44.3% math average vs. OPD's 46.4% and ExOPD's 47.7% (Table 2); in strong-to-weak, SFT achieves 13.5% vs. ExOPD's 25.4% for the 1.7B student (Table 3).

• Avoid ExPO when consistent teacher-surpassing performance across all domains is required. The paper shows ExPO outperforms ExOPD on some code benchmarks (MBPP+ 72.0% vs. 70.6%) but underperforms substantially on math (45.0% vs. 47.7%) and fails to exceed the math teacher entirely (45.0% vs. 46.0%). ExPO is training-free (cheap to try) but its performance is brittle and domain-dependent in ways the paper does not explain or predict.

• Do not extrapolate beyond λ ≈ 1.25 without validation on the specific setting. The paper finds consistent degradation at λ = 1.5 across all benchmarks (Figures 2, 3), attributed to implicit reward hacking (length bias exploitation). No evidence exists that λ > 1.25 can be made safe without additional mitigation. The interpolation regime (0 < λ < 1) appears safe and monotonic, making it suitable for cost-controlled deployment where the teacher represents an upper accuracy bound.