ArXiv: 2604.13016

🎯 Pitch

A stronger teacher can completely fail at on-policy distillation while a weaker one succeeds, simply because same-family models share indistinguishable thinking patterns from the student's perspective. Successful OPD hinges entirely on progressive alignment of a tiny set of overlapping high-probability tokens—the teacher's performance score itself barely matters. This paper cracks open the black box of distillation dynamics and provides actionable fixes for when the process breaks down.

1. Executive Summary

This paper systematically investigates the training dynamics of on-policy distillation (OPD) in LLMs, analyzing why it succeeds or fails across controlled teacher-student configurations using models from the Qwen3 and DeepSeek families on mathematical reasoning benchmarks. The authors identify two governing conditions — thinking-pattern consistency between student and teacher (operationalized through the overlap ratio of their top-k token distributions) and the presence of genuinely new knowledge beyond what the student encountered during training (operationalized by comparing same-family teachers with and without additional RL post-training) — and validate both through reverse distillation experiments showing that same-family 1.5B and 7B teachers produce indistinguishable training dynamics. At the token level, successful OPD is driven by progressive alignment on high-probability overlap tokens — a small shared set concentrating 97–99% of the probability mass — and restricting supervision to these tokens alone suffices to match full top-k performance, while failing runs exhibit stagnant overlap from the outset. The paper further proposes two practical recipes that recover failing OPD — off-policy cold start via SFT on teacher rollouts and teacher-aligned prompt selection from the teacher's post-training data — and reveals a fundamental trajectory-length ceiling where reward quality degrades systematically with decoding depth, establishing that OPD's dense token-level supervision is effective only when both thinking patterns are compatible and the teacher carries genuinely transferable capabilities.

2. Context and Motivation

The Core Problem: OPD Is Widely Deployed but Poorly Understood

On-policy distillation (OPD) has rapidly become a standard component in the post-training pipelines of state-of-the-art LLMs. Industry-scale systems — including Qwen3 [Yang et al., 2025], MiMo [Xiao et al., 2026], and GLM-5 [Zeng et al., 2026] — all incorporate OPD and report substantial performance gains. Thinking Machines Lab [Lu and Lab, 2025] independently replicated the Qwen3 OPD recipe at a fraction of the RL compute cost, confirming that OPD provides a practically efficient alternative to outcome-reward reinforcement learning. The technique has also been extended to self-distillation settings where a single model serves as its own teacher given privileged information [Hübotter et al., 2026, Shenfeld et al., 2026, Zhao et al., 2026b], enabling continual self-improvement loops.

The fundamental appeal of OPD is straightforward: unlike sparse outcome-level rewards used in RL, OPD provides dense, per-token supervision from the teacher model. At each decoding step, the student receives a signal comparing its own next-token distribution to the teacher's, creating a rich training landscape that should, in principle, guide the student toward the teacher's behavior on states the student actually visits.

However, OPD is fragile. The paper opens with a striking failure mode that motivates the entire investigation:

"We observe a striking failure mode: a stronger teacher can completely fail to improve a student, even when a weaker teacher succeeds from lower initial alignment."

This is not a marginal or hypothetical concern. The authors demonstrate concrete cases where a larger, higher-performing teacher model produces no improvement whatsoever — and in some cases causes regression — while a smaller, weaker teacher from a different training pipeline successfully improves the same student. This counterintuitive behavior suggests that the relationship between teacher quality and distillation success is far more complex than the prevailing assumption that "better teacher → better student."

The significance of understanding OPD's failure modes extends beyond academic curiosity. As major AI labs increasingly rely on OPD in production post-training pipelines, deploying OPD without understanding when and why it fails risks wasted compute, degraded model performance, and missed opportunities to deploy the technique where it would actually help. The paper frames this as an urgent need to move OPD from an empirical "recipe" — where practitioners try different teachers and hope for the best — toward a principled understanding of its governing dynamics.

The Gap in Existing Work: Success Stories Without Failure Analysis

The literature on on-policy distillation has largely focused on demonstrating its potential and extending its scope, rather than systematically analyzing its failure modes.

OPD as a theoretical framework. MiniLLM [Gu et al., 2023] first formalized OPD for LLMs under a reverse KL objective optimized via policy gradient, arguing that reverse KL's mode-seeking behavior prevents the student from spreading probability mass over regions the teacher considers unlikely — a key advantage over forward KL-based distillation. GKD [Agarwal et al., 2024] introduced a unified framework interpolating between on-policy and off-policy data across multiple divergences, demonstrating consistent gains over other KD baselines. Yang et al. [2026b] later formalized OPD theoretically as a special case of dense KL-constrained RL, showing that the teacher's per-token log-ratio constitutes an implicit reward and that scaling this reward beyond its standard weight can push the student past the teacher's performance boundary.

These works established OPD's theoretical foundations and demonstrated its promise, but they share a common limitation: they were designed to show that OPD can work, not to understand when and why it doesn't. The experiments in these papers typically used teacher-student pairs where OPD naturally succeeds, leaving the failure surface unexplored.

The off-policy exposure bias problem. The motivation for OPD itself comes from a well-documented limitation of off-policy distillation: exposure bias [Bengio et al., 2015]. In conventional off-policy distillation, the student is trained on fixed teacher-generated sequences. At inference time, however, the student must generate from its own distribution, meaning it visits states it has never been trained on. This train-inference distribution mismatch causes errors to accumulate over long generations — precisely the regime where strong reasoning models operate.

OPD solves this by having the student generate its own rollouts and computing the teacher's supervision on those student-visited states. This removes the distribution mismatch in principle. But solving one problem potentially introduces others: the teacher is now being evaluated on states it might never naturally visit, raising questions about the reliability of its token-level signal in unfamiliar territory. The paper directly investigates this in Section 6, finding that reward quality degrades systematically with trajectory depth.

The capacity gap and distillability literature — limited to off-policy. A recurring observation in knowledge distillation is that large teacher-student capacity gaps can degrade or even reverse the benefit of distillation. Cho and Hariharan [2019] demonstrated that distillation can hurt student performance when the teacher is substantially more capable. Mirzadeh et al. [2020] proposed using an intermediate-sized teacher assistant to bridge the gap. More recently, Busbridge et al. [2025] provided a quantitative treatment via distillation scaling laws, showing that student loss follows a power law as a function of teacher quality, student size, and data volume, identifying a U-shaped capacity regime where teacher over-capability degrades distillation efficiency.

For LLM reasoning specifically, Li et al. [2025] documented a "learnability gap": training small models on long chain-of-thought traces from strong reasoning teachers consistently underperforms simpler approaches. The suggestion is that the reasoning complexity of teacher outputs must be matched to student capacity.

However, all of these analyses focus on off-policy knowledge distillation. The student is trained on pre-generated teacher outputs, and the analysis concerns the complexity or distribution of those outputs relative to what the student can absorb. OPD introduces a fundamentally different dynamic: the student generates its own trajectories, and the teacher provides token-by-token guidance on those trajectories. The question of "capacity gap" in OPD is not about whether the student can memorize and reproduce the teacher's outputs, but about whether the teacher's local distributional guidance on student-visited states provides a useful training signal. This distinction is critical and unexplored in existing work.

Conflicting Signals in Practice

The paper is motivated by a genuine tension observable in the broader LLM post-training landscape. On one hand, OPD is successful enough to be adopted in major production pipelines. On the other hand, practitioners report configurations where OPD fails entirely — stronger teachers producing worse results, training runs that stall, and performance that regresses to below the student's initial level.

Prior to this work, there was no systematic framework for understanding these failures. A practitioner encountering a failed OPD run would have no principled guidance: should they change the teacher? The student? The data? The training hyperparameters? The field lacked even a vocabulary for diagnosing OPD failures — what signals to monitor, what patterns to look for, what conditions distinguish recoverable from irrecoverable failures.

How This Paper Positions Itself

The paper explicitly positions itself as a systematic investigation of OPD training dynamics, progressing from empirical conditions through token-level mechanism to practical recipe. This is structured as three interlocking questions:

  1. When does OPD succeed or fail? (§3: Phenomenology) — identifying the macroscopic conditions (thinking-pattern consistency, new knowledge) that govern whether OPD produces improvement or stagnation.

  2. Why does OPD work at the token level? (§4: Mechanism) — investigating the token-level signature of successful OPD (progressive alignment on high-probability overlap tokens) and demonstrating that this signature is not merely correlational but causal (optimizing only overlap tokens suffices).

  3. How to rescue failing OPD? (§5: Recipe) — developing practical interventions (off-policy cold start, teacher-aligned prompt selection) that address the identified conditions and recover failing configurations.

The paper's key conceptual contribution is the overlap ratio as a diagnostic that connects all three levels. The overlap ratio — the fraction of tokens appearing in both the student's and teacher's top-k sets at student-visited states — serves simultaneously as:

  • A condition (low initial overlap predicts failure),
  • A mechanism (rising overlap is the dynamical signature of successful training),
  • And a target for intervention (off-policy cold start raises initial overlap; teacher-aligned prompts sharpen mass on overlap tokens).

This unified framework converts OPD from a black-box empirical procedure into a phenomenon with testable predictions and actionable diagnostics. The paper does not claim to solve all OPD problems, but rather to provide the concepts and metrics that enable practitioners and researchers to reason about OPD failures systematically.

Distinguishing from Prior OPD Work

The paper draws a sharp distinction between its focus and that of prior OPD work. Existing studies:

"focus on demonstrating OPD's promise, such as dense rewards and mitigated exposure bias, across varied objectives, tasks, and teacher-student pairs, without systematically analyzing when or why OPD fails."

By contrast, this paper makes failure the primary object of study. The reverse distillation experiments in Section 3.3 — where a model is distilled back toward its own pre-RL checkpoint and regresses exactly — are designed specifically to reveal OPD's mechanism by observing what happens when it goes wrong. Similarly, the comparison between a successful run (JustRL-1.5B → R1-Distill-1.5B) and a failing run (R1-Distill-7B → R1-Distill-1.5B) under matched conditions allows the paper to isolate the dynamical signatures of success and failure without confounding variables.

This "failure-first" approach mirrors how other fields have advanced: understanding when and why a technique breaks down is often more informative than accumulating success stories, because it reveals the boundary conditions and underlying mechanisms that success stories alone obscure.

3. Technical Approach

3.1 Reader Orientation

This paper is primarily an empirical analysis paper with a diagnostic framework — it builds a systematic methodology for monitoring, interpreting, and recovering on-policy distillation (OPD) training runs, rather than proposing a new training algorithm. The core idea is that OPD success or failure is governed by two identifiable conditions (thinking-pattern consistency and the presence of new knowledge) that manifest at the token level as the dynamics of overlap tokens — the small set of high-probability tokens shared between student and teacher distributions at student-visited states — and that these conditions can be diagnosed with lightweight metrics and corrected through targeted data interventions.

3.2 Big-Picture Architecture (Diagram in Words)

The paper's analytical framework has four major components, arranged as a diagnostic pipeline rather than a training system:

  1. OPD Training Engine — the core mechanism that samples student rollouts, queries the teacher for token-level distributions, computes a divergence (typically reverse KL over top-k tokens), and updates the student via gradient descent. This is the "thing being studied" rather than a novel contribution.

  2. Controlled Teacher-Student Configurations — a set of carefully constructed experimental setups where the only variable is the teacher model (or prompt set), allowing isolation of what drives OPD success vs. failure. Configurations include: same-pipeline teachers vs. RL-augmented teachers, larger vs. smaller same-family teachers, thinking vs. non-thinking teachers, and reverse-distillation pairs where the student is distilled back toward its own pre-RL checkpoint.

  3. Dynamic Monitoring Metrics — three token-level statistics computed continuously during OPD training at student-generated decoding positions: (i) the overlap ratio (fraction of tokens in both student and teacher top-k sets), (ii) the overlap-token advantage (distributional agreement within the shared token set), and (iii) the absolute entropy gap (difference in uncertainty profiles). These serve as real-time diagnostics that reveal whether OPD is progressing or stagnating.

  4. Intervention Recipes — two corrective strategies, derived from the phenomenological findings, that alter the data feeding into OPD: (a) off-policy cold start, where the student is first fine-tuned on teacher-generated rollouts (SFT) before OPD begins, raising initial overlap; and (b) teacher-aligned prompt selection, where OPD is run on prompts drawn from the teacher's own post-training data, sharpening the alignment signal on high-probability tokens.

Information flows as follows: a teacher-student pair is configured → OPD training begins → at each step, student rollouts are generated, teacher distributions are queried, and the three dynamic metrics (overlap ratio, overlap-token advantage, entropy gap) are computed → these metrics are monitored to diagnose whether OPD is succeeding (rising overlap, narrowing advantage, shrinking entropy gap) or failing (stagnant metrics) → if failing is diagnosed, one of the two interventions is applied before retraining.

3.3 Roadmap for the Deep Dive

  • First, the formal OPD objective (token-level reverse KL decomposition, the three supervision granularities, and their cost-accuracy tradeoffs), because all subsequent experiments instantiate one of these variants and understanding what is being optimized is prerequisite to interpreting the dynamics.

  • Second, the dynamic monitoring metrics — how they are computed from student and teacher distributions, what each captures mechanistically, and how they connect to the phenomenological conditions — because these metrics are the paper's central diagnostic instrument and appear in every experiment.

  • Third, the controlled experimental configurations that operationalize the paper's core questions — the specific model pairs, datasets, and training recipes used to isolate thinking-pattern consistency, new knowledge, and the effects of scale — because the phenomenology cannot be understood without knowing precisely what is being compared.

  • Fourth, the two corrective recipes (off-policy cold start and teacher-aligned prompt selection), including their data generation pipelines, because these are the actionable outputs of the phenomenological analysis.

  • Fifth, the trajectory-length ceiling analysis (reward degradation with depth, back-to-front entropy propagation, and teacher continuation experiments), because this reveals a fundamental limitation of OPD that bounds its applicability regardless of recipe.

  • Sixth, the overlap sufficiency ablation and support-size experiments, because these establish the causal role of overlap tokens and define the practical regime where OPD's signal is concentrated.

3.4 Detailed, Sentence-Based Technical Breakdown

This is an empirical analysis paper whose core idea is that OPD success is governed by whether the student and teacher (1) share compatible thinking patterns (high initial overlap ratio) and (2) the teacher possesses genuinely new knowledge beyond what the student has already internalized; when these conditions are met, OPD manifests as progressive alignment on a small set of shared high-probability tokens that concentrates 97–99% of the probability mass and drives all measurable improvement; when they are violated, the overlap dynamics stagnate from the outset and OPD fails regardless of the teacher's benchmark performance.

The OPD Objective and Its Three Supervision Granularities

OPD aims to minimize the divergence between the student's output distribution $\pi_\theta$ and the teacher's output distribution $\pi_T$, but computed on trajectories sampled from the student itself. This is the defining characteristic that distinguishes OPD from conventional (off-policy) distillation: the supervision signal is computed on states the student actually visits, not on states the teacher would naturally produce.

Sequence-level formulation. The objective begins at the sequence level as the reverse KL divergence between the student's and teacher's full-sequence distributions:

LOPD(θ)=Ex∼Dx[DKL(πθ(⋅∣x) ∥ πT(⋅∣x))]L_{\text{OPD}}(\theta) = \mathbb{E}_{x \sim \mathcal{D}_x} \left[ D_{\text{KL}} \left( \pi_\theta(\cdot \mid x) \, \| \, \pi_T(\cdot \mid x) \right) \right]

where $\mathcal{D}_x$ is the prompt set, $\pi_\theta(\cdot \mid x)$ is the student's distribution over complete response sequences given prompt $x$, and $\pi_T(\cdot \mid x)$ is the teacher's distribution over the same space.

What it computes: the expected (over prompts) reverse KL divergence from the teacher's sequence distribution to the student's sequence distribution. Reverse KL — as opposed to forward KL $D_{\text{KL}}(\pi_T \| \pi_\theta)$ — penalizes the student for placing probability mass on sequences the teacher considers unlikely (mode-seeking behavior), but does not penalize the student for ignoring modes the teacher covers. This is why OPD uses reverse KL rather than forward KL.

Why this form: forward KL would encourage the student to spread mass over all teacher modes (mean-seeking), which is catastrophic for autoregressive generation because the student has limited capacity relative to the teacher. The student would average over incompatible reasoning paths, producing incoherent outputs. Reverse KL's mode-seeking behavior is a better inductive bias for distillation because it lets the student specialize to a subset of the teacher's modes that it can faithfully reproduce.

Token-level decomposition. Because both the student and teacher generate autoregressively, the sequence-level reverse KL decomposes into a sum of per-token KL divergences over the student's generated prefix:

LOPD(θ)=Ex∼Dx,y^∼πθ(⋅∣x)[∑t=1TDKL(pt∥qt)]L_{\text{OPD}}(\theta) = \mathbb{E}_{x \sim \mathcal{D}_x, \hat{y} \sim \pi_\theta(\cdot \mid x)} \left[ \sum_{t=1}^T D_{\text{KL}}(p_t \| q_t) \right]

where $\hat{y} = (\hat{y}_1, \dots, \hat{y}_T)$ is a response generated by the student, $p_t(v) = \pi_\theta(v \mid x, \hat{y}_{<t})$ is the student's next-token distribution at step $t$ given the student-generated prefix $\hat{y}_{<t}$, and $q_t(v) = \pi_T(v \mid x, \hat{y}_{<t})$ is the teacher's next-token distribution evaluated on the same student-generated prefix.

What it computes: the expected sum of per-step reverse KLs. Each step's term $D_{\text{KL}}(p_t \| q_t)$ measures, for the specific prefix the student generated, how much the student's next-token distribution diverges from the teacher's. The sum over steps aggregates this mismatch across the entire trajectory. Because the expectation is over student-generated trajectories, the supervision is inherently on-policy: the teacher is evaluated on states the student actually visits.

Why this decomposition matters: it reveals that OPD provides dense supervision — a learning signal at every single decoding step, not just at sequence boundaries. This is in contrast to outcome-reward RL, which provides a single scalar reward at the end of the sequence. The dense per-step signal is OPD's claimed advantage, but it also creates a vulnerability: the teacher's token-level distribution must be reliable at every step, including deep into long trajectories where the student-generated prefix may be far from anything the teacher would naturally produce.

Three OPD variants — sampled-token, full-vocabulary, and top-k. The exact token-level KL $D_{\text{KL}}(p_t \| q_t)$ requires summing over the entire vocabulary $\mathcal{V}$ (typically 50K–250K tokens) at every decoding step, which is computationally prohibitive at scale. The paper describes three practical supervision granularities that vary in their cost-accuracy tradeoff:

(1) Sampled-Token OPD. Computes the loss using only the single token the student actually sampled:

ℓtsample=log⁡pt(y^t)−log⁡qt(y^t)\ell_t^{\text{sample}} = \log p_t(\hat{y}_t) - \log q_t(\hat{y}_t)

where $\hat{y}_t \sim p_t$ is the token the student generated at step $t$. The trajectory-level objective is:

LOPDsample(θ)=Ex∼Dx,y^∼πθ(⋅∣x)[∑t=1Tℓtsample]L_{\text{OPD}}^{\text{sample}}(\theta) = \mathbb{E}_{x \sim \mathcal{D}_x, \hat{y} \sim \pi_\theta(\cdot \mid x)} \left[ \sum_{t=1}^T \ell_t^{\text{sample}} \right]

What it computes: for each step, the log-probability difference between student and teacher on the single token the student happened to emit. This is an unbiased single-sample estimator of the full token-level KL — formally, $\mathbb{E}_{\hat{y}_t \sim p_t} [\ell_t^{\text{sample}}] = D_{\text{KL}}(p_t \| q_t)$ — because the expectation of the log-ratio under $p_t$ is exactly the KL divergence.

Why this form: it is the cheapest variant, requiring only one teacher forward pass per generated token (to get $\log q_t(\hat{y}_t)$) rather than $|\mathcal{V}|$ teacher queries. The unbiasedness property means that, over many training steps, the gradient from sampled-token OPD converges to the gradient of the true KL divergence. This is the most common implementation in prior work [Lu and Lab, 2025, Xiao et al., 2026, Yang et al., 2026b] and is the default used throughout this paper's experiments unless otherwise specified.

(2) Full-Vocabulary OPD. Computes the exact KL over all tokens:

LOPDfull(θ)=Ex∼Dx,y^∼πθ(⋅∣x)[∑t=1TDKL(pt∥qt)]L_{\text{OPD}}^{\text{full}}(\theta) = \mathbb{E}_{x \sim \mathcal{D}_x, \hat{y} \sim \pi_\theta(\cdot \mid x)} \left[ \sum_{t=1}^T D_{\text{KL}}(p_t \| q_t) \right]

What it computes: the exact token-level reverse KL at every position, requiring the teacher's log-probabilities for every token in the vocabulary at every decoding step.

Why this form: it provides the densest possible gradient signal per training step, with no sampling noise. However, it costs $O(BTM)$ memory where $B$ is batch size, $T$ is sequence length, and $M = |\mathcal{V}|$ is vocabulary size, making it prohibitive for large models or long sequences.

(3) Top-k OPD. Restricts the KL computation to the $k$ tokens assigned the highest probability under the student's own distribution. Define $S_t = \text{TopK}(p_t, k)$ as the student's top-k tokens at step $t$. The student and teacher distributions are renormalized over this subset:

pˉt(St)(v)=pt(v)1[v∈St]∑u∈Stpt(u),qˉt(St)(v)=qt(v)1[v∈St]∑u∈Stqt(u)\bar{p}_t^{(S_t)}(v) = \frac{p_t(v) \mathbb{1}[v \in S_t]}{\sum_{u \in S_t} p_t(u)}, \quad \bar{q}_t^{(S_t)}(v) = \frac{q_t(v) \mathbb{1}[v \in S_t]}{\sum_{u \in S_t} q_t(u)}

The trajectory-level objective is then:

LOPDtop-k(θ)=Ex∼Dx,y^∼πθ(⋅∣x)[∑t=1TDKL(pˉt(St)∥qˉt(St))]L_{\text{OPD}}^{\text{top-k}}(\theta) = \mathbb{E}_{x \sim \mathcal{D}_x, \hat{y} \sim \pi_\theta(\cdot \mid x)} \left[ \sum_{t=1}^T D_{\text{KL}} \left( \bar{p}_t^{(S_t)} \| \bar{q}_t^{(S_t)} \right) \right]

What it computes: the reverse KL divergence restricted to the student's $k$ most probable next tokens, after renormalizing both distributions to sum to one over just those $k$ tokens. Mass outside $S_t$ is discarded — the objective does not care whether student and teacher agree or disagree on low-probability tokens.

Why this form: it provides an intermediate cost-accuracy tradeoff between sampled-token (cheap but high variance per step) and full-vocabulary (expensive but exact). By restricting to the student's own high-probability region, it captures most of the probability mass where the student and teacher are likely to interact while avoiding the cost of querying the teacher on thousands of near-zero-probability tokens. It also creates a natural connection to the overlap analysis: the top-k tokens become the locus of investigation for whether OPD is succeeding. The paper uses $k = 16$ as the default in all experiments (Table 2), and Section 6.3 systematically varies $k$ to test sensitivity. The default top-k strategy is Student Top-k (tokens selected by the student's probabilities, not the teacher's), which means the overlap set — the intersection of student and teacher top-k — naturally becomes the key object of study.

Key design choice: reverse KL over forward KL. The paper does not experimentally justify this choice (it inherits it from prior work), but the rationale is critical for understanding the mechanism. Under reverse KL:

DKL(p∥q)=∑vp(v)log⁡p(v)q(v)D_{\text{KL}}(p \| q) = \sum_v p(v) \log \frac{p(v)}{q(v)}

the gradient pushes the student to increase $q(v)$ where $p(v)$ is large and $q(v)$ is small, and decrease $p(v)$ where the ratio $p(v)/q(v)$ is unfavorable. Because the sum is weighted by $p(v)$, the student receives the strongest signal on tokens it already assigns high probability to. This creates the "mode-seeking" dynamic that makes overlap tokens the natural locus of learning: the student is incentivized to bring its high-probability tokens into alignment with the teacher's high-probability tokens. If the student's high-probability region is disjoint from the teacher's (low overlap ratio), the gradient signal is weak because the ratio $p(v)/q(v)$ on the student's modes involves very small $q(v)$ values, producing large but potentially noisy advantages that may not point in a coherent direction.

Dynamic Monitoring Metrics

The paper defines three token-level metrics that are computed continuously during OPD training and serve as the primary diagnostic instruments. All three are computed on student-generated trajectories — that is, at each decoding step $t$, the prefix $\hat{y}_{<t}$ is the prefix the student actually generated, and the student and teacher distributions are evaluated at that prefix. The metrics are averaged over steps and over a batch of rollouts.

Overlap Ratio. The overlap ratio quantifies structural alignment between the student's and teacher's high-probability candidate sets:

Moverlap=Et[∣St(p)∩St(q)∣k]M_{\text{overlap}} = \mathbb{E}_t \left[ \frac{|S_t^{(p)} \cap S_t^{(q)}|}{k} \right]

where $S_t^{(p)} = \text{TopK}(p_t, k)$ is the set of $k$ tokens with highest probability under the student at step $t$, and $S_t^{(q)} = \text{TopK}(q_t, k)$ is the analogous set under the teacher. The default $k = 16$.

What it computes: the expected fraction of the student's top-k tokens that also appear in the teacher's top-k set. An overlap ratio of 1.0 means the student and teacher's top-k sets are identical (perfect structural alignment); a ratio of 0.0 means they are completely disjoint (mode mismatch).

Why this form: the overlap ratio measures whether the student is operating in the same token space as the teacher, which is a prerequisite for the teacher's per-token supervision to be useful. The paper emphasizes that overlap tokens carry 97–99% of the total probability mass for both models throughout training (Appendix B.1, Equation 9–10), meaning the overlap ratio captures alignment on the probabilistically dominant tokens, not merely a set-level coincidence among many equally-weighted candidates. Low initial overlap means the student's high-probability region is disjoint from the teacher's, so the teacher's feedback (which is most informative about its own high-probability tokens) provides little signal about the tokens where the student actually concentrates its mass. The metric is monitored throughout training because successful OPD is characterized by progressively rising overlap, not just high final overlap.

Overlap-Token Advantage. Within the overlap set, the paper measures distributional agreement:

Madv=Et[1∣St(p)∩St(q)∣∑v∈St(p)∩St(q)At(v)]M_{\text{adv}} = \mathbb{E}_t \left[ \frac{1}{|S_t^{(p)} \cap S_t^{(q)}|} \sum_{v \in S_t^{(p)} \cap S_t^{(q)}} A_t(v) \right]

where $A_t(v) = \bar{p}_t(v)(\log \bar{q}_t(v) - \log \bar{p}_t(v))$, and $\bar{p}_t, \bar{q}_t$ are the student and teacher distributions renormalized over the overlap set $S_t^{(p)} \cap S_t^{(q)}$.

What it computes: the expected, per-overlap-token contribution to the reverse KL divergence, using renormalized distributions. Each term $A_t(v)$ is the student's renormalized probability on token $v$ multiplied by the log-ratio of teacher-to-student probabilities. A value close to zero means the student and teacher agree on how probability mass is distributed among the shared tokens — the student places mass on teacher-preferred tokens with appropriate confidence. A large negative value means the student is overconfident compared to the teacher on tokens in the overlap set (high $\bar{p}_t(v)$ but lower $\bar{q}_t(v)$).

Why this form: the overlap-token advantage decomposes the KL divergence into its per-token components within the overlap set, isolating whether the student's relative weighting of shared tokens matches the teacher's. This is more fine-grained than the overlap ratio: two runs can have identical overlap ratios but very different overlap-token advantages if, in one run, the student distributes mass among shared tokens differently than the teacher. The renormalization over the overlap set is crucial — it factors out the effect of total mass assigned to non-overlap tokens, isolating the distributional alignment within the shared region. Successful OPD is characterized by the overlap-token advantage approaching zero from below (progressively less negative), indicating that the student is calibrating its confidence within the shared token set.

Entropy Gap. The entropy gap measures divergence in uncertainty profiles:

ΔHt=∣H(qt)−H(pt)∣\Delta H_t = |H(q_t) - H(p_t)|

where $H(p_t) = -\sum_{v} p_t(v) \log p_t(v)$ is the Shannon entropy of the student's next-token distribution at step $t$, and $H(q_t)$ is the teacher's entropy, both computed on the full vocabulary at the student-generated prefix.

What it computes: the absolute difference between student and teacher entropy at each decoding position, averaged over positions. A large gap means the student is substantially more or less uncertain than the teacher at the same state — for instance, the student might be highly confident (low entropy) while the teacher is diffuse (high entropy), indicating the student is overfitting to a specific next token where the teacher sees multiple plausible continuations. Convergence toward zero means the student has matched the teacher's uncertainty profile along its own generated trajectories.

Why this form: the absolute difference is used rather than signed difference because both overconfidence and underconfidence relative to the teacher are problematic. Overconfidence (student entropy lower than teacher entropy) means the student is mode-collapsed — it has lost the teacher's ability to consider alternatives. Underconfidence (student entropy higher) means the student is diffuse where the teacher is focused, indicating the student hasn't absorbed the teacher's knowledge. The paper uses this metric primarily to detect pathological collapse: in the teacher-aligned prompt experiments (Section 5.2), student entropy drops sharply, signaling that the prompt set is overly narrow and suppressing the student's exploration capacity.

Connection to the overlap-token mass. Appendix B.1 defines two complementary quantities:

Moverlap-mass(p)=Et[∑v∈St(p)∩St(q)pt(v)]M_{\text{overlap-mass}}^{(p)} = \mathbb{E}_t \left[ \sum_{v \in S_t^{(p)} \cap S_t^{(q)}} p_t(v) \right]

Moverlap-mass(q)=Et[∑v∈St(p)∩St(q)qt(v)]M_{\text{overlap-mass}}^{(q)} = \mathbb{E}_t \left[ \sum_{v \in S_t^{(p)} \cap S_t^{(q)}} q_t(v) \right]

These measure the total probability mass (not just set membership) that each model assigns to the overlap tokens. The key finding (Figure 18) is that this mass is 97–99% for both student and teacher throughout training, regardless of whether OPD succeeds or fails. This means the overlap ratio's diagnostic power comes from which tokens are in the shared set, not from whether that shared set captures most of the probability mass — it always does. When the overlap ratio is low, the 97–99% of mass is distributed over different tokens for student and teacher. Successful OPD shifts which tokens receive that mass until the same tokens dominate both distributions.

Controlled Experimental Configurations

The paper's phenomenology relies on carefully constructed teacher-student configurations that isolate specific hypotheses. Each configuration controls all variables except the one being tested, enabling causal attribution of OPD success or failure. Unless otherwise specified, all experiments use the default hyperparameters in Table 2 of Appendix A.2: training temperature 1.0, global batch size 64, rollout number 4, LogProb top-K 16 using Student Top-K strategy, top-p 1.0, max prompt length 1024, max response length 7168, learning rate 1e-6, one epoch, KL coefficient 0.0. The training dataset is DAPO-Math-17K [Yu et al., 2025] unless otherwise noted. Evaluation is on AIME 2024 [Li et al., 2024], AIME 2025 [Balunović et al., 2025], and AMC 2023 [Li et al., 2024], with 16 samples per problem at temperature 0.7, top-p 0.95, and a maximum validation response length of 31,744 tokens. The primary metric is avg@16 (average accuracy over 16 samples).

Thinking-pattern consistency configuration (Section 3.1). Student: Qwen3-1.7B-Base. Two teachers compared: Qwen3-4B (Non-thinking) — a base model without specialized reasoning training — and Qwen3-4B-Base-GRPO — the same base model further trained with GRPO [Shao et al., 2024] on DAPO-Math-17K (detailed GRPO hyperparameters in Appendix A.1: learning rate 1e-6, rollout n=8, temperature 1.0, KL regularization disabled, max response length 7168, one epoch on 8 A800 80G GPUs). Both teachers are evaluated on the same benchmarks (Figure 3): Qwen3-4B (Non-thinking) achieves avg@16 of 0.700 on AMC 2023, 0.210 on AIME 2024, and 0.204 on AIME 2025; Qwen3-4B-Base-GRPO achieves 0.599, 0.212, and 0.242 respectively — broadly comparable performance. The critical variable is the thinking pattern: since the student (Qwen3-1.7B-Base) is also a base model, its natural token distribution is expected to align more closely with the GRPO-trained teacher (which also starts from a base checkpoint) than with the non-thinking teacher (which may have a different reasoning style). The hypothesis: the GRPO teacher will produce higher initial overlap ratio and better OPD outcomes, despite not being uniformly stronger on benchmarks.

New knowledge configuration (Section 3.2). Two model families tested in parallel, each contrasting a teacher from the same training pipeline with one that acquired additional capabilities through RL.

  • DeepSeek family. Student: DeepSeek-R1-Distill-Qwen-1.5B. Two teachers: DeepSeek-R1-Distill-Qwen-7B (same-pipeline, same-family, larger scale) vs. Skywork-OR1-Math-7B (obtained by applying RL post-training on top of R1-Distill-7B). Both are 7B models from the same base family, but Skywork-OR1-Math-7B has additional RL training that the student has not seen.

  • Qwen family. Student: Qwen3-1.7B (Non-thinking). Two teachers: Qwen3-4B (Non-thinking, same training pipeline as student but larger) vs. Qwen3-4B-Non-Thinking-RL-Math (obtained by applying RL to Qwen3-4B Non-thinking on a 57K subset of DeepMath [He et al., 2025c]).

The key contrast in both families: same-pipeline teachers have seen essentially the same training data and recipe as the student (just at different scales), so they model the same distribution with different degrees of fit. Post-trained teachers have acquired genuinely new capabilities through RL that the student has not encountered. The hypothesis: even when thinking patterns are broadly aligned (since the post-trained teachers are derived from the same base checkpoints), only the post-trained teachers provide transferable signal because they carry genuinely new knowledge.

Reverse distillation configuration (Section 3.3). A counterintuitive setup designed to reveal the deepest properties of OPD mechanisms. JustRL-DeepSeek-1.5B (JustRL-1.5B) is obtained by applying RL to R1-Distill-1.5B [He et al., 2025a]. The experiment reverses the usual direction: JustRL-1.5B serves as the student, and is distilled toward two teachers: (i) R1-Distill-1.5B, which is JustRL-1.5B's own pre-RL checkpoint — effectively asking the model to unlearn its RL gains; and (ii) R1-Distill-7B, a larger same-family model that scores slightly higher on benchmarks than JustRL-1.5B. This experiment simultaneously tests both conditions: (i) the thinking-pattern effect, because both teachers share the same model family and should have similar token distributions; and (ii) the new-knowledge effect, because R1-Distill-7B, despite being larger, has less knowledge than JustRL-1.5B (it never underwent RL, so it represents an earlier point in the same training trajectory).

Gap recovery rate. The paper quantifies OPD's effectiveness using:

Gap Recovery Rate=Accafter OPD−Accbefore OPDAccteacher−Accbefore OPD\text{Gap Recovery Rate} = \frac{\text{Acc}_{\text{after OPD}} - \text{Acc}_{\text{before OPD}}}{\text{Acc}_{\text{teacher}} - \text{Acc}_{\text{before OPD}}}

This measures what fraction of the performance gap between student and teacher is closed by OPD. It is used in Figure 4 to compare how much of the teacher's advantage is actually transferable: post-trained teachers achieve substantially higher gap recovery rates (58.6% in the Qwen family vs. 15.6% for same-pipeline teachers; 16.9% in the DeepSeek family vs. 5.3%).

Corrective Recipe 1: Off-Policy Cold Start

When initial overlap ratio is low (thinking-pattern mismatch), the paper proposes a two-stage training pipeline: first, supervised fine-tuning (SFT) the student on teacher-generated rollouts to bring the student closer to the teacher's thinking pattern; second, continuing with standard OPD.

Why this addresses the condition. The low initial overlap in failing configurations means the student's high-probability region and the teacher's high-probability region are largely disjoint. The teacher's per-token signal provides gradients, but those gradients push on tokens the student barely considers, resulting in weak effective updates. SFT on teacher-generated data forces the student to mimic the teacher's outputs directly, which by construction moves the student's high-probability region toward the teacher's. Once overlap is raised via SFT, the teacher's OPD signal becomes exploitable because the student is already operating near the teacher's support.

Data generation pipeline (Section 5.1, Appendix C.1). The concrete instantiation uses Qwen3-1.7B-Base as the student and Qwen3-4B (Non-thinking) as the teacher:

  1. Sample 200K math prompts from the math-domain subset of OpenThoughts3-1.2M [Guha et al., 2025].
  2. Generate one offline response for each prompt from the teacher (Qwen3-4B Non-thinking) using temperature 0.7, top-p 0.95, top-k -1 (disabled), max generation length 12,288 tokens, with the prompt template: "{Question} Please reason step by step, and put your final answer within \boxed{}."
  3. Filter out incomplete responses (truncated outputs) and degenerate repetitive responses.
  4. Perform full-parameter SFT of the student (Qwen3-1.7B-Base) on these 200K teacher-generated samples using the LLaMA-Factory framework [Zheng et al., 2024], with the hyperparameters in Table 3: learning rate 1e-5, cosine scheduler with 0.05 warmup ratio, one epoch, sequence length 14,336, per-device batch size 8, BF16 precision. This yields Qwen3-1.7B-SFT.
  5. Continue training with OPD from this SFT initialization, using the remaining prompts from OpenThoughts after deduplicating against the SFT prompt subset (approximately 30K prompts), with the same teacher (Qwen3-4B Non-thinking).

Why SFT on teacher rollouts and not just any data. The SFT phase must use teacher-generated outputs specifically because the goal is to align the student's thinking pattern with the teacher's. Training on a generic math dataset would improve the student's math capabilities but might not shift its token-level distribution toward the specific teacher being used. The SFT phase is not about general capability improvement; it is about distributional alignment between student and teacher to precondition OPD.

Control baseline. Pure OPD starting directly from Qwen3-1.7B-Base with the same teacher and the same 30K OPD prompt set, but without any cold-start SFT phase. This isolates the effect of the SFT preconditioning.

Expected mechanistic effect. The SFT-initialized student should begin OPD with a substantially higher overlap ratio (because SFT forcibly aligned it with the teacher's outputs), a smaller entropy gap (because the student has already matched the teacher's uncertainty profile through imitation), and smoother training dynamics (because the teacher's token-level signal is immediately exploitable rather than requiring the student to first discover the teacher's support region through random exploration).

Corrective Recipe 2: Teacher-Aligned Prompt Selection

Rather than moving the student closer to the teacher (as in cold start), this recipe improves alignment from the data side: by using prompts that the teacher encountered during its own post-training, OPD benefits from the teacher being evaluated on states that are closer to its training distribution.

Why this addresses the condition. The teacher's token-level distribution $\pi_T(\cdot \mid x, \hat{y}_{<t})$ depends on both the prompt $x$ and the student-generated prefix $\hat{y}_{<t}$. When the prompt $x$ is drawn from the teacher's post-training data, the teacher's distribution is sharper and more reliable — the teacher has been optimized specifically to produce correct reasoning on these prompts. When $x$ comes from an out-of-distribution source, the teacher's distribution may be more diffuse or less well-calibrated, providing a weaker or noisier distillation signal. Teacher-aligned prompts strengthen the signal on overlap tokens by making the teacher's distribution more peaked on the correct continuations.

Two granularities tested (Section 5.2).

(a) Prompt template alignment. Teacher: JustRL-1.5B. Student: R1-Distill-1.5B. Prompt set: DAPO-Math-17K (same content in both runs). The only difference is the prompt format:

  • Original DAPO template: "Solve the following math problem step by step. The last line of your response should be of the form Answer: $Answer (without quotes) where $Answer is the answer to the problem. {Question} Remember to put your answer on its own line after 'Answer:'."

  • Teacher-aligned template: "{Question} Please reason step by step, and put your final answer within \boxed{}." — this matches the format used during JustRL-1.5B's post-training.

The hypothesis: even a surface-level format change affects OPD because the teacher's token distribution is sensitive to prompt formatting — the teacher has been trained to produce \boxed{} formatting and may be less well-calibrated when asked to produce Answer: formatting. The student, when generating rollouts under the teacher-aligned template, visits states that the teacher's distribution is better tuned for, producing a stronger token-level signal.

(b) Prompt content alignment. Teacher: Qwen3-4B-Base-GRPO (introduced in Section 3.1). Student: Qwen3-1.7B-Base. Two prompt sets of matched size compared: DAPO-Math-17K (aligned with the teacher's RL training data, since GRPO training used DAPO-Math-17K) vs. a deduplicated subset of DeepMath (in-domain for math but not seen during the teacher's RL training). The DeepMath subset is constructed via two-stage deduplication against DAPO-Math-17K (Appendix C.3): (1) exact-match deduplication on extracted question text, and (2) semantic deduplication using sentence embeddings (all-mpnet-base-v2) with a cosine similarity threshold of 0.6. The teacher-aligned prompts (DAPO-Math-17K) are expected to produce a stronger OPD signal because the teacher's distribution is optimized on these exact prompts.

Tradeoff: entropy collapse risk. The paper identifies an important side effect: using only teacher-aligned prompts causes the student's entropy to drop substantially during training (Figure 10, bottom-right). Low entropy means the student becomes overconfident — it assigns extremely high probability to a few tokens and near-zero to alternatives. While this indicates strong alignment on the overlap tokens, it may also mean the student loses the ability to explore alternative reasoning paths. The paper recommends mixing teacher-aligned prompts with out-of-distribution prompts to preserve policy entropy and maintain exploration capacity. This is not implemented in the paper but stated as a practical guideline.

Trajectory-Length Ceiling Analysis

Section 6.1 investigates how the teacher's reward quality varies with response length, motivated by the observation that OPD's dense per-token supervision assumes the teacher provides reliable signal at every decoding position, including deep into long responses.

Response length sweep setup. R1-Distill-1.5B distilled against JustRL-1.5B for 200 steps under six maximum response lengths: 0.5K, 1K, 3K, 7K, 10K, and 15K tokens. All other hyperparameters held constant.

The sweet-spot phenomenon (Figure 11a). Very short responses (0.5K, 1K) produce the weakest results because too few tokens receive supervision — the student cannot learn efficiently from limited signal. Moderate lengths (3K, 7K) yield the strongest results, providing enough supervised tokens for sample-efficient learning without entering the regime where reward quality degrades. Beyond 7K (10K, 15K), performance plateaus or declines, indicating that the additional supervised tokens at later positions are either uninformative or actively harmful.

Training dynamics at different lengths (Figure 12). The overlap ratio curves reveal why longer responses fail. At 3K and 7K, overlap rises smoothly throughout training. At 10K and 15K, overlap initially rises but then exhibits late-stage collapse — the overlap ratio drops sharply after peaking, accompanied by spikes in student entropy and gradient norm. This suggests that training becomes unstable when too many late-position tokens are included: the teacher's signal at deep positions becomes noisy or unreliable, and this noise propagates back to destabilize learning at earlier positions.

Back-to-front entropy propagation (Figure 13). The paper traces the origin of this instability by analyzing student entropy as a function of output position across training steps in the 15K setting. At early training steps, entropy is uniformly moderate across all positions. As training proceeds, high entropy first emerges at the end of the response (positions near 15K) and progressively propagates backward toward earlier tokens. The paper describes this as a "clear back-to-front pattern." The interpretation: the teacher, when evaluated on long student-generated prefixes, encounters states increasingly far from its training distribution. At late positions, the teacher's conditional distribution becomes diffuse (high entropy), providing weak or noisy supervision. This noisy signal destabilizes the student's policy at late positions, which then feeds back to earlier positions through the autoregressive dependency — if the student's policy at position 14K degrades, the prefixes it generates for position 14K+1 become even worse, accelerating the collapse.

Teacher continuation experiment (Figure 11b). To directly test whether the teacher's capability degrades with prefix depth, the paper designs a controlled probe: sample 2K prompts, generate full student rollouts, select those exceeding 16K tokens, truncate each at multiple prefix positions (1K, 4K, 8K, 16K), and let the teacher continue generation from the resulting prefix. The accuracy gain from teacher continuation (compared to student-only completion) decreases monotonically with prefix depth: +0.37 at 1K prefix, +0.27 at 4K, +0.15 at 8K, and only +0.02 at 16K. This directly demonstrates that the teacher's token-level predictions become progressively less informative about producing correct answers as the student-generated prefix grows longer and diverges further from the teacher's natural trajectories.

What this reveals about OPD's fundamental limitation. The mechanism of degradation is not that the teacher becomes "wrong" in absolute terms — the teacher's conditional distribution remains well-defined. Rather, the student-generated prefix at depth is a state the teacher would never naturally visit, so the teacher's distribution on that state is not calibrated for producing correct continuations. This is a different kind of exposure bias than the one OPD was designed to solve. In off-policy distillation, the student is evaluated on states it never visited during training. In OPD, the teacher is evaluated on states the student visits but the teacher never would. The dense supervision that makes OPD attractive is simultaneously its vulnerability: every additional token of depth introduces more teacher evaluations on out-of-distribution prefixes.

The practical implication. OPD's effectiveness has a trajectory-length sweet spot. For moderate-length reasoning traces (roughly 3K–7K tokens in this experimental setup), the teacher's token-level signal remains reliable and OPD works well. For very long reasoning traces (10K+), the signal degrades and can actively destabilize training. This imposes a ceiling on OPD's applicability to long-horizon reasoning, chain-of-thought, or agentic multi-turn settings where trajectories routinely exceed this sweet spot. The paper does not propose a solution to this ceiling beyond suggesting future work on hybrid approaches that combine dense token-level supervision on short segments with sparse outcome-level rewards for longer horizons.

The Global vs. Local Reward Distinction

Section 6.2 addresses a puzzle: the configuration R1-Distill-7B → R1-Distill-1.5B fails completely (no improvement), yet the teacher (R1-Distill-7B) is a stronger model that one would naively expect to provide better supervision. The paper tests whether this failure is because the teacher's reward signal is globally uninformative (the teacher cannot distinguish good from bad student outputs) or whether the signal is informative but locally unexploitable (the teacher provides useful global assessments that cannot be translated into effective per-token gradients).

Sequence mean reward analysis (Figure 14). For each student rollout $y$, the paper computes the sequence mean reward under sampled-token OPD:

rˉ(y)=1T∑t=1T[log⁡πT(yt∣x,y<t)−log⁡πθ(yt∣x,y<t)]\bar{r}(y) = \frac{1}{T} \sum_{t=1}^T \left[ \log \pi_T(y_t \mid x, y_{<t}) - \log \pi_\theta(y_t \mid x, y_{<t}) \right]

This is the average per-token advantage over the trajectory — how much better the teacher evaluates the student's sampled tokens compared to the student's own evaluation. The distribution of $\bar{r}(y)$ is compared between correct and incorrect student rollouts.

The result. Both teachers — the successful JustRL-1.5B teacher and the failing R1-Distill-7B teacher — produce sequence mean rewards that are globally informative: correct rollouts consistently receive higher mean rewards than incorrect rollouts, with comparable AUROC values (0.73 for JustRL-1.5B, 0.75 for R1-Distill-7B). The failing teacher does not produce a weaker global signal — it is equally correlated with rollout correctness. This means the failure is not because the 7B teacher's reward is inherently noisy or uninformative.

The anisotropy hypothesis. If the global signal is equally good, why does OPD fail? The paper proposes a hypothesis based on the gradient structure. The per-step advantage $\log \pi_T(\hat{y}_t) - \log \pi_\theta(\hat{y}_t)$ is computed independently at each position and then aggregated into a gradient update. For the successful JustRL-1.5B teacher (which shares a compatible thinking pattern with the student), the per-step advantages may be directionally coherent: even though individual token advantages may be modest in magnitude, they point in consistent directions across positions, producing a gradient that effectively steers the student toward the teacher's distribution. For the failing R1-Distill-7B teacher, the per-step advantages may be anisotropic: individually large (explaining why the overlap-token advantage in Figure 6 is more negative — larger in magnitude — for the 7B teacher), but pointing in mutually canceling directions across positions within each sequence. When these heterogeneous signals are aggregated into a single gradient update, they partially cancel, yielding small effective gradients despite large per-token rewards.

Evidence from optimization diagnostics (Appendix B.2, Figure 19). The paper reports several auxiliary metrics supporting this interpretation. The failing run with R1-Distill-7B exhibits: (i) a consistently smaller gradient norm throughout training (suggesting weak effective updates despite large per-token advantages), (ii) persistently large probability differences on the token with the highest absolute advantage (indicating that the student cannot resolve the most extreme local mismatches), and (iii) much smaller reduction in training loss over time (the OPD objective barely decreases from its already-low starting value). By contrast, the successful run with JustRL-1.5B shows sustained gradient magnitude, steady reduction in extreme-token probability mismatches, and substantial loss reduction.

Why this matters for understanding OPD failure. The distinction between global and local signal quality is fundamental. A practitioner might reasonably assume that a stronger teacher that can distinguish correct from incorrect rollouts (high AUROC) will provide better OPD supervision. This analysis shows that assumption is false: global informativeness does not guarantee local exploitability. For OPD to succeed, the teacher must provide per-token advantages that are not just individually correct but collectively coherent — they must point in directions that aggregate into effective gradient updates. This coherence depends on thinking-pattern compatibility: when the teacher's and student's high-probability regions overlap, the advantages on those overlap tokens are directionally aligned; when they are disjoint, the advantages may be individually large but point in incompatible directions.

The paper's caveat. The anisotropy hypothesis is explicitly flagged as unverified: "We have not directly verified this anisotropy hypothesis, and doing so would require analyzing the directional structure of per-token gradients, which we leave to future work." This is a hypothesis, not a proven mechanism. The direct evidence — co-occurrence of high per-token advantage and low gradient norm — is suggestive but not conclusive.

Overlap Sufficiency Ablation

Section 4.2 tests whether the overlap region is not merely correlated with OPD success but is causal — whether optimizing only the overlap tokens suffices to match the performance of standard OPD.

Setup. Using the successful OPD setting from Section 4.1 (JustRL-1.5B → R1-Distill-1.5B with Student Top-k OPD, k=16), three variants are compared that differ only in which subset of the top-k tokens receives the distillation loss:

  • Student Top-k: the standard variant, optimizing on all $k$ tokens in $S_t^{(p)}$ (the student's top-k set).
  • Overlap Top-k: restricting optimization to the intersection $S_t^{(p)} \cap S_t^{(q)}$. Tokens in the student's top-k that are not in the teacher's top-k receive no supervision. This tests whether the overlap tokens alone carry the useful signal.
  • Non-Overlap Top-k: restricting optimization to the symmetric difference $S_t^{(p)} \triangle S_t^{(q)}$ — tokens that are in exactly one of the two top-k sets. This tests whether tokens outside the overlap carry any useful signal.

The result (Figure 7). Overlap Top-k nearly perfectly matches Student Top-k on all three benchmarks across all training steps. Non-Overlap Top-k is substantially weaker and shows unstable training. The overlap-token advantage curves for Student Top-k and Overlap Top-k are "nearly indistinguishable," while Non-Overlap Top-k has much smaller magnitude (weaker effective gradient on overlap tokens). The overlap ratio dynamics are revealing: both Student Top-k and Overlap Top-k steadily raise overlap from about 72% to above 91%. Non-Overlap Top-k causes overlap to initially decrease and then only partially recover.

Why Non-Overlap Top-k is weak. The non-overlap tokens carry very little probability mass — the overlap tokens already concentrate 97–99% of the total mass. Moreover, the teacher provides no useful guidance on non-overlap tokens because, by definition, these are tokens the teacher considers low-probability (they are not in the teacher's top-k). The teacher's signal on non-overlap tokens is essentially "these are unlikely," which provides minimal information for the student to reallocate its probability mass.

The self-reinforcing nature of overlap optimization. The paper identifies a dynamic that explains why Overlap Top-k is sufficient: "once a token enters the shared high-probability region and is favored by the teacher, reverse-KL updates concentrate more mass on it, gradually pushing competing non-overlap tokens out of the student's top-k set." This is because reverse KL's mode-seeking behavior amplifies tokens that the teacher prefers. As training proceeds, tokens that are in the overlap set receive positive reinforcement (the teacher's distribution assigns them higher probability, and the student is penalized for not matching this), causing the student to increase their probability. This pushes other tokens that were previously in the student's top-k below the top-k threshold, increasing the overlap ratio. The overlap set "grows not despite but because of the optimization, creating a virtuous cycle that sustains alignment throughout training."

The causal claim. This ablation establishes that the overlap region is not just where alignment manifests, but the region that drives optimization. The useful gradient signal in OPD is concentrated on the tokens that both student and teacher consider high-probability. All other tokens — whether high-probability for only the student (student-only tokens) or only the teacher (teacher-only tokens) — contribute negligible useful signal. This explains why OPD success depends on initial overlap ratio: if the initial overlap is low, there are few tokens receiving useful gradient signal, and the self-reinforcing dynamic has no fuel to start.

Support Size Analysis

Section 6.3 investigates how many tokens per position are needed for effective OPD by varying $k$ in Top-k OPD and comparing against sampled-token OPD.

Setup. Student: R1-Distill-1.5B. Teacher: JustRL-1.5B. Top-k OPD with $k \in \{1, 4, 16, 64\}$ compared against sampled-token OPD (which uses only the single sampled token per position), all other hyperparameters fixed.

Main result (Figure 15). Sampled-token OPD achieves performance comparable to Top-4, Top-16, and Top-64 on all three benchmarks. The only clearly worse variant is Top-1, which consistently underperforms. Enlarging $k$ beyond 4 brings negligible additional gain while incurring greater computational overhead (the teacher must be queried for $k$ log-probabilities per position rather than one).

Training dynamics (Figure 16). Top-1 exhibits unstable overlap growth with sharp spikes in entropy and gradient norm. Top-4 is substantially more stable but still shows a late-stage dip in overlap. Top-16 and Top-64 remain smooth throughout training, with no collapse. The paper explains the Top-1 failure mode: "Top-1, by contrast, always selects the argmax token, thereby concentrating the reward on a single mode. Small policy changes can flip which token occupies rank 1, creating an unstable reward signal that does not average out over training." The problem is not the number of tokens (one), but the selection rule: picking the argmax token deterministically creates a discontinuous reward landscape where infinitesimal policy changes can produce completely different supervision, preventing stable gradient-based learning.

Why sampled-token OPD works despite using only one token. Sampled-token OPD draws $\hat{y}_t \sim p_t$ — the token is sampled stochastically from the student's own distribution. Although only one token receives supervision at each step, the sampling process provides unbiased coverage of the student's high-probability region across training steps. Over many batches, the aggregated gradient from sampled-token OPD converges to the true KL gradient because $\mathbb{E}_{\hat{y}_t \sim p_t}[\log p_t(\hat{y}_t) - \log q_t(\hat{y}_t)] = D_{\text{KL}}(p_t \| q_t)$. This is in contrast to Top-1, which always trains on $\arg\max_v p_t(v)$ — a deterministic, biased estimator that only sees the most probable token and is blind to the distribution of mass among other high-probability candidates.

Practical guidance. The support size $k$ is not a critical design choice as long as the degenerate Top-1 setting is avoided. Sampled-token OPD, Top-4, Top-16, and Top-64 all produce comparable results, suggesting practitioners can use the cheapest variant (sampled-token) without sacrificing performance, provided they accept the higher per-step variance that averages out over training.

4. Key Insights and Innovations

Innovation 1: OPD Failure as a First-Class Object of Study

Prior work on on-policy distillation — from MiniLLM [Gu et al., 2023] through GKD [Agarwal et al., 2024] to the industrial-scale pipelines in Qwen3 [Yang et al., 2025], MiMo [Xiao et al., 2026], and GLM-5 [Zeng et al., 2026] — has overwhelmingly focused on demonstrating that OPD can work. The research program has been to show gains, extend the technique to new settings (self-distillation, privileged information), and scale it up. The implicit assumption has been that OPD is a generally reliable technique whose effectiveness is limited only by compute budget and engineering quality.

This paper makes a fundamental conceptual pivot: it treats OPD failure as the primary object of study rather than an anomaly to be ignored. The reverse distillation experiments in Section 3.3 are the most vivid instantiation of this shift. Distilling JustRL-1.5B — a model that has acquired genuine reasoning gains through RL — back toward R1-Distill-1.5B, its own pre-RL checkpoint, causes the student to regress exactly to its pre-RL performance, systematically erasing all RL gains. This is not a marginal failure; it is a complete reversal of the intended effect. Even more strikingly, substituting R1-Distill-7B — a larger same-family model that slightly outperforms JustRL-1.5B on benchmarks — produces a training trajectory that is "nearly indistinguishable" from the 1.5B teacher, driving the student to the same regressed level. A stronger teacher produces the same failure as a weaker one, and neither produces any improvement.

This finding directly contradicts the dominant assumption in the distillation literature that teacher quality (measured by benchmark performance) predicts distillation effectiveness. The capacity gap literature [Cho and Hariharan, 2019, Mirzadeh et al., 2020, Busbridge et al., 2025] had already documented that larger teachers can sometimes hurt in off-policy distillation, but those analyses focused on the complexity or length of teacher-generated outputs being mismatched to student capacity. The OPD setting is fundamentally different: the student generates its own rollouts, so the "complexity" of teacher outputs is not the relevant variable. The failure documented here is that the teacher's local token-level guidance on student-visited states can be completely ineffective even when the teacher is manifestly more capable overall. The paper's contribution is to identify this as the central puzzle — not a footnote — and to build an entire diagnostic framework around understanding it.

Why this is fundamental, not incremental. This is a reframing of the research question rather than an improvement to an existing technique. The field has been asking "how can we make OPD work better?" This paper asks "when does OPD work at all, and why does it sometimes fail catastrophically?" The answers to the second question reveal boundary conditions that success stories alone would never uncover — such as the finding that same-family teachers at different scales can be distributionally indistinguishable from the student's perspective, rendering scale alone useless for OPD even when it improves benchmarks.

Innovation 2: The Overlap Ratio as a Unified Diagnostic Concept

The paper's signature conceptual move is the introduction of the overlap ratio — the fraction of tokens in both student and teacher top-k sets at student-visited states — as a diagnostic that connects phenomena across three levels of analysis: conditions (initial overlap predicts success or failure), mechanism (rising overlap is the dynamical signature of effective training), and intervention (off-policy cold start raises initial overlap; teacher-aligned prompts sharpen mass on overlap tokens).

Prior work on OPD dynamics did not have a comparable metric. GKD [Agarwal et al., 2024] used divergence values (KL, reverse KL, JS) to compare methods but did not decompose those divergences into structural components. Yang et al. [2026b] theoretically characterized OPD as dense KL-constrained RL but did not provide operational metrics for monitoring training health. Practitioners monitoring OPD runs had no principled way to diagnose whether a run was progressing or stagnating beyond watching validation accuracy — a lagging indicator that only reveals failure after substantial compute has been spent.

The overlap ratio solves this by providing a leading indicator that reveals whether OPD's self-reinforcing dynamic has engaged. The paper demonstrates that successful OPD is characterized by a steadily rising overlap ratio (from ~72% to ~91% in the successful JustRL-1.5B → R1-Distill-1.5B run, Figure 6), while failing runs exhibit stagnant overlap from the outset. The metric is also causally validated: the overlap sufficiency ablation (Section 4.2, Figure 7) shows that optimizing only the overlap tokens matches full top-k performance, while optimizing non-overlap tokens is substantially weaker. This demonstrates that the overlap ratio is not merely correlated with OPD success — it tracks the region where the actual learning signal is concentrated.

The conceptual power of the overlap ratio lies in its ability to reconcile the paper's two governing conditions into a single observable. Thinking-pattern consistency manifests as high initial overlap ratio. New knowledge manifests as overlap growth during training — the student progressively shifts mass onto teacher-supported tokens, and the teacher's signal on those tokens steers the student toward capabilities it hadn't internalized. When either condition is violated (low initial overlap, or high initial overlap but no new knowledge), the overlap dynamics stagnate and OPD fails, but for mechanistically distinct reasons that the overlap ratio makes visible.

Why this is fundamental, not incremental. The paper is not proposing a new training algorithm or a better loss function. It is providing a diagnostic lens through which any OPD implementation can be monitored and understood. This is akin to how the training loss curve, gradient norm, and weight statistics became standard diagnostic tools in deep learning — not because they improve training directly, but because they make training dynamics interpretable. The overlap ratio serves the same function for OPD, and the paper demonstrates its utility across every experiment. This is a conceptual tool that outlives any specific model pair or dataset.

Innovation 3: Decomposing "Teacher Quality" into Thinking Pattern × New Knowledge

The conventional view in distillation is that teacher quality is a scalar: higher benchmark scores → better teacher → better distillation. This paper demonstrates that this view is not just incomplete but actively misleading for OPD. It decomposes teacher quality into two orthogonal factors that the paper's experiments show are independent and both necessary for successful OPD: thinking-pattern consistency (whether the student and teacher's high-probability token sets overlap at student-visited states) and new knowledge (whether the teacher possesses capabilities the student has not already internalized through training).

The most striking evidence for this decomposition comes from the reverse distillation experiment (Section 3.3, Figure 5). R1-Distill-7B satisfies the "higher benchmark scores" definition of teacher quality — it slightly outperforms JustRL-1.5B on the evaluation benchmarks. But R1-Distill-7B fails completely as a teacher because, despite its scale advantage, it represents an earlier point in the same training trajectory as the student. It has high thinking-pattern consistency (same model family, similar token distributions) but no new knowledge — the student (JustRL-1.5B) actually has more knowledge, having been trained further via RL. Distilling from R1-Distill-7B therefore causes regression: OPD's mode-seeking reverse KL pulls the student back toward the earlier checkpoint's distribution.

The complementary failure mode is demonstrated in the thinking-pattern experiments (Section 3.1, Figure 2). Qwen3-4B (Non-thinking) and Qwen3-4B-Base-GRPO have broadly comparable benchmark performance (Figure 3), but the GRPO teacher substantially outperforms the non-thinking teacher in OPD because its thinking pattern is more compatible with the base-model student. Here, the non-thinking teacher may possess high-quality knowledge, but the knowledge is encoded in token distributions that the student cannot effectively learn from because the student's high-probability region does not overlap with the teacher's. The teacher's signal is like a highly accurate map in a language the student doesn't speak.

The new-knowledge experiments (Section 3.2, Figure 4) demonstrate the third case: same-family teachers where thinking-pattern consistency is high (similar initial overlap ratios of 70–75%) but only the post-trained teachers provide transferable gains. The same-pipeline teachers (R1-Distill-7B, Qwen3-4B Non-thinking) achieve gap recovery rates of only 5.3% and 15.6% respectively, while the RL-augmented teachers (Skywork-OR1-Math-7B, Qwen3-4B-Non-Thinking-RL-Math) recover 16.9% and 58.6%. This establishes that thinking-pattern consistency is necessary but not sufficient: the teacher must also carry genuinely novel capabilities that the student can absorb through token-level alignment.

Why this is fundamental, not incremental. This decomposition resolves the field's conflicting experiences with OPD. A practitioner who tried OPD with a same-family 7B → 1.5B configuration and saw no improvement might conclude "OPD doesn't work for capacity gaps." A practitioner who tried with an RL-augmented small teacher and saw strong gains might conclude "OPD works well." Both are correct for their configuration, but neither understands why. The thinking-pattern × new-knowledge decomposition provides a language for reasoning about which teachers will work and why, converting OPD from trial-and-error into a principled decision. It also explains the puzzle of why larger scale doesn't help: R1-Distill-7B and R1-Distill-1.5B, despite their 4.7× parameter difference, induce "nearly identical local target distributions on student-visited states" — scale alone, within the same training pipeline, does not create the distributional novelty that OPD requires.

Innovation 4: The Global-vs-Local Reward Distinction as an Explanation for OPD Failure

A practitioner observing a failed OPD run would naturally ask: is the teacher's reward signal simply too noisy? Does the teacher fail to distinguish good from bad student outputs? Section 6.2 provides a clear negative answer to this intuition and proposes a more subtle explanation that shifts the focus from signal quality to optimization geometry.

The experiment in Figure 14 shows that both a successful teacher (JustRL-1.5B) and a failing teacher (R1-Distill-7B) produce sequence mean rewards that are globally informative with comparable AUROC values (0.73 vs. 0.75). The failing teacher is equally capable of assigning higher average reward to correct student rollouts and lower average reward to incorrect ones. This eliminates the simplest explanation — that the teacher is providing a bad signal — and forces a more nuanced analysis.

The paper's hypothesis is that the failure lies in local exploitability: per-token advantages from the 7B teacher may be individually large (visible in the more negative overlap-token advantage in Figure 6) but directionally incoherent when aggregated into gradients. The evidence is circumstantial but internally consistent: the failing run exhibits persistently smaller gradient norms (Appendix B.2, Figure 19) despite larger per-token advantages, and the student cannot reduce probability mismatches on the most extreme-advantage tokens. The metaphor is that the 7B teacher provides a detailed but self-contradictory map: every individual instruction is clear, but they collectively point in canceling directions. The 1.5B teacher provides a simpler but coherent map: smaller per-token signals that aggregate into a consistent direction reverse KL can amplify.

This distinction connects directly to the overlap mechanism. When the teacher and student share high overlap, their probability mass is concentrated on the same tokens — the teacher's advantages on those tokens are likely to be directionally aligned because both models agree on which tokens matter, even if they disagree on how much probability each should receive. When overlap is low, the teacher's per-token advantages on student-high-probability tokens (which are teacher-low-probability tokens) may be individually large — the teacher strongly disagrees with the student's choices — but these disagreement signals on disjoint token sets may not point in a consistent direction for gradient updates.

Why this is fundamental, not incremental. This is a negative result with significant implications. It demonstrates that stronger benchmark performance does not translate to better local supervision, and that the mapping from global reward quality to local gradient exploitability is nontrivial and poorly understood. This challenges the implicit assumption in the OPD literature that "dense" automatically means "better" — the density of supervision is only useful if the per-token signals are geometrically coherent. The finding also points to a previously unrecognized limitation of reverse KL as an optimization objective in the OPD setting: reverse KL's mode-seeking behavior amplifies the strongest mode of disagreement, but if the strongest modes of disagreement are mutually contradictory, amplification may produce near-zero net gradient rather than effective learning. This opens a research direction the paper explicitly flags but does not resolve: understanding the directional structure of per-token gradients in OPD and potentially designing objectives that can exploit anisotropic reward landscapes.

Innovation 5: The Trajectory-Length Ceiling as a Fundamental Limitation of OPD

The paper's trajectory-length analysis (Section 6.1) reveals a limitation that is fundamental to the OPD framework itself, not merely an artifact of specific models or datasets. The finding is that OPD has a sweet spot for response length — around 3K–7K tokens in the paper's experimental setup — beyond which additional supervision becomes actively harmful, causing training instability and performance collapse.

The mechanism is a form of teacher-side exposure bias. OPD was designed to solve the student-side exposure bias of off-policy distillation: the student is no longer evaluated on states it never visited during training because it generates its own rollouts. But OPD creates a symmetric problem: the teacher is now evaluated on student-generated prefixes that grow progressively further from the teacher's natural trajectories as sequence length increases. The paper demonstrates this directly through the teacher continuation experiment (Figure 11b): the teacher's advantage over the student in completing a prefix drops from +0.37 at 1K tokens to +0.02 at 16K tokens. The teacher, when starting from a deeply student-generated prefix, is nearly as lost as the student.

The back-to-front entropy propagation pattern (Figure 13) reveals how this causes training collapse. The instability originates at the end of long trajectories where the teacher's signal is weakest, then propagates backward to earlier positions. This is diagnostically important: it means a practitioner monitoring only overall training metrics might see stable performance for many steps before a sudden collapse, unaware that the seeds of failure are already growing at the suffix.

Why this is fundamental, not incremental. This is not a finding about a particular teacher-student pair or dataset — it is a structural limitation that follows from OPD's core design (on-policy student rollouts + teacher evaluation at every position). Any OPD implementation will face some version of this ceiling, though the exact threshold will vary with model scale, domain, and training data. The implication is that OPD is not a free lunch of dense supervision — the denser the supervision, the deeper into student-generated trajectories the teacher must provide reliable signal, and eventually that reliability degrades. For applications requiring long-horizon reasoning (extended chain-of-thought, multi-turn agentic interaction), pure OPD may be fundamentally insufficient, and the paper explicitly suggests hybrid approaches combining dense token-level supervision on short segments with sparse outcome-level rewards for longer horizons as a necessary future direction.

This finding also reframes the value proposition of OPD relative to RL. The appeal of OPD over outcome-reward RL has been that dense, per-token supervision should be more sample-efficient than sparse, outcome-level rewards. But this efficiency advantage has a hard ceiling imposed by trajectory length. For short-to-moderate reasoning traces, OPD's dense signal provides a genuine advantage. For very long traces, the signal degrades to the point where sparse outcome rewards may be more reliable — not because they are inherently better, but because they avoid the teacher-side exposure bias that accumulates with depth. The paper thus identifies a fundamental tradeoff between supervision density and supervision reliability that bounds OPD's applicability.

5. Experimental Analysis

Evaluation Methodology

  • Dataset. DAPO-Math-17K [Yu et al., 2025] serves as the primary training prompt set for OPD across nearly all experiments, with the specific splits and usage described per configuration. Evaluation is conducted on three mathematical reasoning benchmarks: AIME 2024 [Li et al., 2024], AIME 2025 [Balunović et al., 2025], and AMC 2023 [Li et al., 2024]. For the off-policy cold start recipe (Section 5.1), the math-domain subset of OpenThoughts3-1.2M [Guha et al., 2025] provides the SFT prompt source, with 200K prompts sampled for teacher rollout generation and approximately 30K deduplicated remaining prompts used for the subsequent OPD phase. For the prompt content alignment experiment (Section 5.2), a deduplicated subset of DeepMath [He et al., 2025c] is compared against DAPO-Math-17K.

  • Base model(s). The paper spans two model families to test generalizability. From the Qwen family: Qwen3-1.7B-Base serves as the primary student in thinking-pattern and cold-start experiments, with Qwen3-4B (Non-thinking) and Qwen3-4B-Base-GRPO (obtained by applying GRPO to Qwen3-4B-Base on DAPO-Math-17K; see Appendix A.1 for hyperparameters) as teachers. Qwen3-1.7B (Non-thinking) is the student in the Qwen-family new-knowledge experiment, with Qwen3-4B (Non-thinking) and Qwen3-4B-Non-Thinking-RL-Math [Yang et al., 2026b] as teachers. From the DeepSeek family: DeepSeek-R1-Distill-Qwen-1.5B (R1-Distill-1.5B) [Guo et al., 2025] serves as the main student for mechanism analysis (Sections 4 and 6), reverse distillation (Section 3.3), and overlap sufficiency ablation, paired with JustRL-1.5B [He et al., 2025a] (successful teacher) and R1-Distill-7B [Guo et al., 2025] (failing teacher), as well as Skywork-OR1-Math-7B [He et al., 2025b] and R1-Distill-14B for cross-model validation (Appendix B.3). The models span 1.5B, 1.7B, 4B, 7B, and 14B parameter scales, covering both base and reasoning-tuned checkpoints, enabling controlled comparisons across capacity gaps and training pipelines.

  • Metrics. The primary evaluation metric is avg@16 — the average accuracy over 16 sampled solutions per problem, following standard reasoning benchmark evaluation practice. Accuracy is the fraction of problems for which the model's generated answer matches the ground-truth answer (exact match with appropriate formatting). During training, three dynamic monitoring metrics are computed continuously on student-generated trajectories: overlap ratio (Equation 6, the fraction of tokens appearing in both student and teacher top-k sets, averaged over decoding positions), overlap-token advantage (Equation 7, the renormalized per-token contribution to reverse KL within the overlap set, with values approaching zero indicating better distributional agreement), and absolute entropy gap (Equation 8, the absolute difference between student and teacher Shannon entropy at student-visited prefixes). The gap recovery rate is used in Section 3.2 to quantify how much of the teacher-student performance gap is closed by OPD: (Acc_after_OPD − Acc_before_OPD) / (Acc_teacher − Acc_before_OPD). Sequence mean reward (Section 6.2) is computed as the per-step average of log π_T(y_t) − log π_θ(y_t) over a student rollout, and its distribution over correct vs. incorrect rollouts is analyzed via AUROC.

  • Baselines. The primary baseline across all experiments is the student's pre-OPD performance — the avg@16 of the student model before any OPD training begins, evaluated under the same sampling protocol (temperature 0.7, top-p 0.95, max 31,744 tokens). In each controlled comparison, the baseline is a specific OPD configuration rather than an external method: the thinking-pattern experiment (Section 3.1) contrasts two teachers with the same student; the new-knowledge experiment (Section 3.2) contrasts same-family teachers with and without additional RL post-training; the cold-start experiment (Section 5.1) uses pure OPD from Qwen3-1.7B-Base as the control against SFT-initialized OPD; the prompt-content experiment (Section 5.2) contrasts teacher-aligned prompts (DAPO-Math-17K) against in-domain but deduplicated prompts (DeepMath subset). In the overlap sufficiency ablation (Section 4.2), the standard Student Top-k OPD serves as the baseline against Overlap Top-k and Non-Overlap Top-k variants. In the support-size analysis (Section 6.3), sampled-token OPD is the baseline against Top-k OPD with k ∈ {1, 4, 16, 64}. Teacher performance is also reported (dashed lines in figures) as a reference ceiling, representing the avg@16 of the teacher model under the same evaluation protocol.

  • Generation budget / compute accounting. All OPD experiments are run for a fixed number of training steps rather than a fixed FLOPs budget. The default configuration (Table 2, Appendix A.2) specifies: global batch size 64, rollout number 4 (4 student-generated trajectories per prompt), maximum prompt length 1024 tokens, maximum response length 7168 tokens, training for one epoch on the DAPO-Math-17K dataset. The trajectory-length experiments in Section 6.1 sweep the maximum response length across {0.5K, 1K, 3K, 7K, 10K, 15K} tokens. Compute cost is not normalized across different teacher sizes: using a 7B teacher vs. a 1.5B teacher incurs different forward-pass costs per token, but this cost difference is not factored into any efficiency comparison since the paper's focus is on whether OPD works under different configurations, not on FLOPs-matched comparisons. The support-size experiments in Section 6.3 note that larger k incurs greater computational overhead (more teacher log-probability queries per position), but no explicit cost model is provided.

  • Cross-validation / statistical protocol. There is no cross-validation or statistical significance testing reported. All experiments train once per configuration and report a single training curve (accuracy vs. step, overlap ratio vs. step, etc.) — there are no error bars, confidence intervals, or multiple random seeds. The gap recovery rate (Section 3.2) is computed from single-run accuracy numbers. Experimental comparisons rely on visual inspection of training curves and the consensus of three evaluation benchmarks (AIME 2024, AIME 2025, AMC 2023) to establish robustness — claims are supported when all three benchmarks show the same qualitative pattern. The reverse distillation experiment (Section 3.3) runs for 600 steps rather than the default 200 to confirm convergence; the trajectory-length experiments (Section 6.1) run for 200 steps with maximum lengths up to 15K to observe collapse. The paper acknowledges that the anisotropy hypothesis in Section 6.2 is unverified and explicitly leaves verification to future work.

Main Quantitative Results

Phenomenology: Thinking-Pattern Consistency Governs OPD Effectiveness (Section 3.1)

The experiment comparing two teachers — Qwen3-4B (Non-thinking) and Qwen3-4B-Base-GRPO — with the same student (Qwen3-1.7B-Base) on the DAPO-Math-17K dataset demonstrates that thinking-pattern compatibility, rather than raw benchmark performance, determines OPD success. As shown in Figure 2 (left), distillation from the GRPO-trained teacher consistently outperforms distillation from the non-thinking teacher across all 200 training steps, with the final average validation accuracy (avg@16 averaged across AIME 2024, AIME 2025, and AMC 2023) reaching approximately 0.38–0.40 for the GRPO teacher vs. approximately 0.28–0.30 for the non-thinking teacher at step 200. Crucially, both teachers have broadly comparable standalone benchmark performance (Figure 3): the non-thinking teacher achieves avg@16 of 0.700 on AMC 2023, 0.210 on AIME 2024, and 0.204 on AIME 2025; the GRPO teacher achieves 0.599, 0.212, and 0.242 respectively. The GRPO teacher is actually slightly weaker on AMC 2023 yet produces substantially better OPD outcomes.

The overlap ratio dynamics (Figure 2, right) explain this apparent paradox. The GRPO teacher begins with a substantially higher initial overlap ratio (approximately 0.69 at step 0 vs. approximately 0.57 for the non-thinking teacher), confirming that its thinking pattern — having been trained from a base model checkpoint, as is the student — is more aligned with the student's token distribution than the non-thinking teacher's pattern. Both overlap curves converge later in training (reaching approximately 0.90 for the GRPO teacher and approximately 0.88–0.90 for the non-thinking teacher by step 200), but the performance gap established early in training persists throughout. As the paper notes: "Although the two overlap curves converge later in training, the performance gap persists, suggesting that early-stage thinking-pattern mismatch causes a loss of distillation benefit that cannot be recovered later." The benchmark-wise breakdown in Appendix A.3 (Figure 17) confirms that this pattern is consistent across individual benchmarks, with the GRPO teacher advantage more pronounced on AMC 2023 and AIME 2024, and smaller but still present on AIME 2025.

Phenomenology: New Knowledge Beyond the Student's Training Is Necessary (Section 3.2)

Two parallel experiments across the DeepSeek and Qwen families isolate whether a teacher from the same training pipeline — differing only in scale — can provide transferable signal through OPD, or whether genuinely new knowledge acquired through additional post-training is required.

DeepSeek family (Figure 4, left). Student: R1-Distill-1.5B. Same-pipeline teacher: R1-Distill-7B (same training recipe, larger scale). Post-trained teacher: Skywork-OR1-Math-7B (RL applied on top of R1-Distill-7B). At step 200, the post-trained teacher achieves approximately 0.46 avg@16 accuracy, while the same-pipeline teacher reaches only approximately 0.40. The gap recovery rate tells an even starker story: the post-trained teacher recovers 16.9% of the student-teacher performance gap, while the same-pipeline teacher recovers only 5.3%. Both runs begin with comparable initial overlap ratios (Skywork: 71.5%, R1-Distill-7B: 74.7%), indicating that thinking-pattern consistency is not the differentiator here — both teachers share broadly compatible patterns with the student. The improvement must therefore stem from the additional capabilities Skywork-OR1-Math-7B acquired through RL post-training, which the student (having been trained only up to R1-Distill-1.5B) has not internalized.

Qwen family (Figure 4, right). Student: Qwen3-1.7B (Non-thinking). Same-pipeline teacher: Qwen3-4B (Non-thinking). Post-trained teacher: Qwen3-4B-Non-Thinking-RL-Math (RL applied to Qwen3-4B Non-thinking on a 57K DeepMath subset). The post-trained teacher achieves substantially stronger OPD performance, reaching approximately 0.54–0.56 avg@16 at step 200 vs. approximately 0.40 for the same-pipeline teacher. The gap recovery rates are 58.6% and 15.6% respectively — a dramatic difference. Initial overlap ratios are again comparable (Qwen3-RL-Math: 70.3%, Qwen3 Non-thinking: 75.7%), confirming that thinking-pattern compatibility is not the limiting factor. The key interpretation from the paper: "Since the post-trained teachers are derived from the same base checkpoints, their thinking patterns remain broadly aligned... The improvement therefore stems from new capabilities of the teacher acquired through RL."

The unifying pattern. Across both model families, the same-pipeline teachers (which have seen essentially the same training data and recipe as the student, just at larger scale) provide minimal transferable signal through OPD, achieving gap recovery rates of only 5.3% and 15.6%. The post-trained teachers (which have acquired genuinely novel capabilities through additional RL) produce substantially larger improvements with gap recovery rates of 16.9% and 58.6%. This establishes that higher benchmark scores alone do not guarantee OPD effectiveness — the teacher must carry knowledge beyond what the student has already encountered during its own training.

Phenomenology: Reverse Distillation Validates Both Conditions and Reveals OPD's Mechanism (Section 3.3)

The reverse distillation experiment — the most conceptually revealing in the paper — uses JustRL-1.5B [He et al., 2025a] (a checkpoint obtained by applying RL to R1-Distill-1.5B) as the student and distills it toward two teachers: R1-Distill-1.5B (its own pre-RL checkpoint) and R1-Distill-7B (a larger same-family model). The experiment runs for 600 steps and reports accuracy on individual benchmarks rather than averages.

The results (Figure 5) produce three striking findings:

Finding 1: OPD overwrites thinking patterns. Distilling JustRL-1.5B toward R1-Distill-1.5B (its own pre-RL checkpoint) causes the student to regress almost exactly to its pre-RL performance on all three benchmarks. On AIME 2024, accuracy drops from approximately 0.54 (JustRL-1.5B initial performance) to approximately 0.38 (R1-Distill-1.5B teacher performance) by step 200, and remains near that level through step 600. On AIME 2025, the drop is from approximately 0.40 to approximately 0.30. On AMC 2023, from approximately 0.80 to approximately 0.74. The paper's interpretation: "This suggests that OPD actively acquires the teacher's thinking patterns and overwrites the student's own." The RL gains — which represent the student's own acquired capabilities — are systematically erased because OPD's reverse KL minimization pulls the student's distribution toward the teacher's distribution at every student-visited state.

Finding 2: Benchmark performance does not predict OPD outcome. When the teacher is switched to R1-Distill-7B — a model that is both larger (7B vs. 1.5B) and slightly stronger on benchmarks than JustRL-1.5B — the training trajectory is "nearly indistinguishable" from the R1-Distill-1.5B teacher. Despite outscoring JustRL-1.5B, R1-Distill-7B drives the student to the same regressed level (approximately 0.38 on AIME 2024, 0.30 on AIME 2025, 0.74 on AMC 2023 by step 200–600). The paper concludes: "Despite outscoring JustRL-1.5B on benchmarks, R1-Distill-7B drives the student to the same regressed level as the weaker 1.5B teacher." This directly contradicts the assumption that a stronger teacher (in the benchmark sense) provides better OPD supervision.

Finding 3: Same-family models at different scales are distributionally indistinguishable. Since OPD minimizes reverse KL divergence over student-generated trajectories, the convergence of the two teacher runs to the same regressed level implies that "the two teachers induce nearly identical local target distributions on student-visited states, despite their difference in scale." In other words, from the perspective of a student navigating its own generation space, R1-Distill-1.5B and R1-Distill-7B look essentially identical — scale alone, within the same training pipeline, does not create meaningfully different token-level guidance. This explains why the same-pipeline teachers in Section 3.2 failed to provide substantial gains despite being larger.

The paper draws three conclusions from these findings: (i) OPD fundamentally learns thinking patterns, not just benchmark capabilities — consistency in these patterns is necessary; (ii) the teacher's benchmark performance can be completely decoupled from OPD training dynamics, and may even move in the opposite direction (stronger teacher → regression); (iii) higher scores do not imply new knowledge for OPD — if the teacher's knowledge is already encoded in distributions the student has seen during training (albeit at smaller scale), OPD provides no forward signal and may actively regress the student toward an earlier checkpoint.

Mechanism: Successful OPD Is Characterized by Progressive High-Probability Token Alignment (Section 4.1)

The controlled comparison between a successful run (JustRL-1.5B → R1-Distill-1.5B) and a failing run (R1-Distill-7B → R1-Distill-1.5B) — same student, two teachers, identical training recipe — reveals the dynamical signature that distinguishes effective OPD from stagnation.

Performance (Figure 6, top). The successful run with JustRL-1.5B as teacher yields consistent gains across all three benchmarks. At step 200 on AIME 2024, the student reaches approximately 0.44 avg@16, recovering more than 80% of the gap to the teacher (dashed line at approximately 0.47). On AIME 2025, performance reaches approximately 0.31 (teacher: ~0.33). On AMC 2023, approximately 0.77 (teacher: ~0.78). The failing run with R1-Distill-7B as teacher shows essentially no improvement from the student's initial performance — accuracy stays flat near 0.28–0.30 on AIME 2024, 0.225–0.25 on AIME 2025, and 0.63–0.66 on AMC 2023 — despite the 7B teacher being slightly stronger overall.

Overlap ratio dynamics (Figure 6, bottom-left). The successful run exhibits a steadily rising overlap ratio, from approximately 0.72 at step 0 to approximately 0.91 by step 200. This monotonically increasing curve indicates that the student's high-probability token set progressively converges toward the teacher's — the self-reinforcing dynamic described in Section 4.2. The failing run shows near-flat overlap ratio throughout training, starting at approximately 0.74–0.75 and ending at approximately the same level, with only minor fluctuations. The paper interprets this as the decisive diagnostic: "In the successful run, the overlap ratio rises steadily... In the failing run, all three metrics stagnate."

Overlap-token advantage (Figure 6, bottom-middle). The successful run shows the overlap-token advantage progressing from approximately −0.006 at step 0 to near zero (−0.001 to 0.000) by step 200, indicating that the student progressively calibrates its confidence within the shared token set to match the teacher's. The failing run shows a more complex pattern: the overlap-token advantage starts at roughly −0.003 (less negative than the successful run initially), but progressively becomes more negative, reaching approximately −0.005 to −0.004 by step 200. The paper flags this as evidence that the failing teacher produces per-token advantages that are individually large but cannot be resolved: the student continues to assign probability differently from the teacher on the tokens where they nominally agree on what tokens matter.

Entropy gap (Figure 6, bottom-right). The successful run shows the absolute entropy gap narrowing from approximately 0.10 at step 0 to near 0.00 by step 150–200, demonstrating that the student matches the teacher's uncertainty profile along its own trajectories. The failing run shows a persistently larger entropy gap that fluctuates between 0.02 and 0.08 without a clear narrowing trend.

Auxiliary diagnostics (Appendix B.2, Figure 19). The paper reports additional optimization metrics that reinforce this pattern. The successful run shows a large reduction in policy gradient (PG) loss from approximately 1.6 to approximately 0.2, indicating that OPD is effectively reducing the KL divergence. The failing run starts with a much smaller loss (approximately 0.2) and barely changes — not because it's already well-aligned, but because the teacher-induced signal is too weak from the outset. The gradient norm shows an even clearer separation: the successful run maintains gradient norms of 0.4–2.0 throughout training (sustained corrective signal), while the failing run's gradient norms hover near 0.0–0.2 (weak updates). The probability difference on the token with the largest absolute advantage progressively decreases in the successful run (from ~0.12 to ~0.02) but persists at ~0.04–0.08 in the failing run.

Cross-model validation (Appendix B.3, Figure 20). Using R1-Distill-7B as the student and comparing Skywork-OR1-Math-7B (successful) vs. R1-Distill-14B (failing) as teachers, the same pattern replicates. Skywork-OR1-Math-7B produces performance gains (AIME 2024: ~0.56 to ~0.64; AIME 2025: ~0.40 to ~0.48; AMC 2023: ~0.84 to ~0.90), accompanied by rising overlap ratio (0.80 → 0.96), advancing overlap-token advantage toward zero, and a small, stable entropy gap (~0.02–0.04). R1-Distill-14B produces minimal improvement with poor or unstable alignment metrics — the overlap ratio fluctuates around 0.88–0.92 without clear trend, and the entropy gap is larger (0.08–0.16).

Mechanism: Overlap Tokens Alone Suffice to Drive OPD (Section 4.2)

The overlap sufficiency ablation tests whether the overlap region is causally responsible for OPD's training signal. Using the successful JustRL-1.5B → R1-Distill-1.5B configuration with Student Top-k OPD (k=16) as the baseline, two restricted variants are compared: Overlap Top-k (optimizing only on S_t^(p) ∩ S_t^(q)) and Non-Overlap Top-k (optimizing only on S_t^(p) △ S_t^(q)).

Main finding (Figure 7). Overlap Top-k nearly perfectly replicates the performance of Student Top-k across all three benchmarks and all training steps. On AIME 2024, both variants reach approximately 0.44 at step 200; on AIME 2025, approximately 0.31–0.32; on AMC 2023, approximately 0.77. Non-Overlap Top-k consistently underperforms, reaching only approximately 0.38 on AIME 2024, 0.27 on AIME 2025, and 0.72 on AMC 2023 at step 200 — substantially below both Overlap Top-k and Student Top-k, and only marginally above the student's initial performance. The paper states: "optimizing only the overlap region is sufficient to recover nearly the full benefit of standard Student Top-k OPD on all three benchmarks."

Dynamical explanation (Figure 7, bottom). The overlap-token advantage curves for Student Top-k and Overlap Top-k are "nearly indistinguishable" — both approach zero from below with similar trajectories. Non-Overlap Top-k shows much smaller magnitude in overlap-token advantage, confirming that restricting optimization to non-overlap tokens provides a dramatically weaker effective gradient on the overlap tokens that actually matter. The overlap ratio dynamics reveal the self-reinforcing mechanism: Student Top-k and Overlap Top-k both raise the overlap ratio from approximately 0.72 to above 0.91, while Non-Overlap Top-k causes the overlap ratio to initially decrease (from ~0.72 to ~0.66) before partially recovering to ~0.78. The paper explains: "once a token enters the shared high-probability region and is favored by the teacher, reverse-KL updates concentrate more mass on it, gradually pushing competing non-overlap tokens out of the student's top-k set. The overlap region thus grows not despite but because of the optimization."

Why Non-Overlap Top-k fails. The non-overlap tokens carry negligible probability mass (Appendix B.1, Figure 18 shows that the overlap tokens already capture 97–99% of total probability mass for both student and teacher). Moreover, the teacher's signal on non-overlap tokens is inherently weak — by definition, these tokens are not in the teacher's top-k, so the teacher assigns them low probability. Providing the student with supervision that essentially says "the teacher thinks these are unlikely" on tokens the student already considers unlikely provides minimal learning signal relative to the dense, mutually reinforcing signal on tokens where both models concentrate their probability mass.

Recipe: Off-Policy Cold Start Recovers Failing OPD by Raising Initial Overlap (Section 5.1)

Using the failing configuration from Section 3.1 — Qwen3-1.7B-Base as student, Qwen3-4B (Non-thinking) as teacher — the cold-start experiment introduces a two-stage pipeline: first, SFT the student on 200K teacher-generated rollouts (yielding Qwen3-1.7B-SFT), then continue with OPD on the remaining prompts.

Performance (Figure 8). The SFT-initialized student consistently outperforms the pure-OPD baseline across all benchmarks and training steps. At step 200 on AIME 2024, the SFT+OPD run reaches approximately 0.11–0.12 avg@16, compared to approximately 0.05 for pure OPD — more than doubling the final accuracy. On AIME 2025, the gap is approximately 0.065 vs. 0.045; on AMC 2023, approximately 0.38 vs. 0.28. Critically, "the performance gap persists throughout training, indicating that the off-policy cold start improves not only early optimization, but also the final performance ceiling of subsequent OPD."

Overlap dynamics (Figure 8, bottom-left). The SFT-initialized student begins with a substantially higher overlap ratio (approximately 0.68 at step 0 vs. approximately 0.52 for the base-initialized student) and maintains a smooth, stable upward trajectory throughout training, reaching approximately 0.72–0.74 by step 200. The base-initialized student starts much lower and exhibits pronounced instability — the overlap ratio fluctuates between 0.54 and 0.66 before gradually recovering to approximately 0.70 by step 200. The paper's interpretation: "The SFT-initialized student begins with a much higher overlap ratio and maintains a smooth, stable trajectory, whereas the base-initialized student starts lower and exhibits pronounced instability before gradually recovering."

Entropy gap (Figure 8, bottom-right). The SFT-initialized student maintains a substantially smaller entropy gap throughout training (approximately 0.05–0.10) compared to the base-initialized student (approximately 0.15–0.35, with large fluctuations), confirming that off-policy distillation preconditions the student to match the teacher's uncertainty profile before OPD's token-level supervision begins.

Overlap mass analysis (Appendix C.2, Figure 21). Further analysis reveals why the base-initialized student sometimes shows a comparable overlap-token advantage while still underperforming. The SFT-initialized student maintains both student overlap mass and teacher overlap mass at consistently high levels (near 0.95–1.0 for student mass, 0.85–0.95 for teacher mass), indicating that the overlap set covers most of the high-probability regions of both distributions. The base-initialized student exhibits substantially lower and more unstable overlap mass (student mass fluctuating between 0.65 and 0.95). Since overlap-token advantage is averaged only over overlap tokens, it can appear favorable even when the overlap set itself misses substantial high-probability teacher tokens — overlap mass complements this view by revealing whether the shared support covers the most important parts of both distributions.

Recipe: Teacher-Aligned Prompt Selection Improves OPD Signal Quality (Section 5.2)

Two granularities of prompt alignment are tested: template alignment (Does matching the prompt format matter?) and content alignment (Does matching the prompt content matter?).

Prompt template alignment (Figure 9). Student: R1-Distill-1.5B. Teacher: JustRL-1.5B. Two runs on DAPO-Math-17K differ only in the prompt template: the original DAPO template (asking for "Answer: $Answer" format) vs. the teacher-aligned template (asking for "\boxed{}" format, matching the format used during JustRL-1.5B's post-training). The teacher-aligned template yields higher average validation accuracy throughout training, reaching approximately 0.50–0.52 avg@16 at step 200 vs. approximately 0.46 for the original template. The overlap ratio dynamics (Figure 9, right) show that the teacher-aligned run begins with a higher overlap ratio (approximately 0.82 vs. 0.76) and converges to a higher level (approximately 0.92 vs. 0.90). The benchmark-wise breakdown in Appendix C.4 (Figure 22) confirms consistency: the teacher-aligned template yields improvements on all three benchmarks, with larger gains on AIME 2024 (approximately 0.44 vs. 0.38) and AIME 2025 (approximately 0.32 vs. 0.28) and a smaller but positive effect on AMC 2023 (approximately 0.78 vs. 0.75). The paper's interpretation: "even a minor change in prompt template can materially affect OPD by making the student's generated states more compatible with the teacher."

Prompt content alignment (Figure 10). Student: Qwen3-1.7B-Base. Teacher: Qwen3-4B-Base-GRPO. Two prompt sets of matched size: DAPO-Math-17K (teacher-aligned — the GRPO teacher was trained on this data) vs. deduplicated DeepMath (in-domain for math but not the teacher's post-training data). The teacher-aligned prompts yield stronger downstream performance (AIME 2024: ~0.10 vs. ~0.08 at step 200; AIME 2025: ~0.06 vs. ~0.05; AMC 2023: ~0.32 vs. ~0.28). However, the overlap dynamics exhibit a counterintuitive pattern: the teacher-aligned prompts produce a lower overlap ratio throughout training (Figure 10, bottom-left: ~0.68–0.70 for teacher-aligned vs. ~0.70–0.72 for deduplicated) but simultaneously show substantially higher cumulative student probability mass on the overlap tokens (Figure 10, bottom-middle: ~0.85–0.92 vs. ~0.30–0.45). The paper explains: "the student concentrates its mass on fewer but more strongly shared tokens. The effective alignment on high-probability tokens is therefore stronger, even though the overlap set is smaller."

Entropy collapse tradeoff (Figure 10, bottom-right). The teacher-aligned prompts cause a dramatic drop in student entropy — from approximately 7.5 at early training to approximately 2.0 by step 200, compared to 5.0–6.0 for the deduplicated prompts. The paper identifies this as a practical risk: "performing OPD only on prompts seen during teacher post-training may not always be ideal, as it can overly reduce policy entropy." The recommended mitigation (stated but not implemented) is to mix teacher-aligned prompts with out-of-distribution prompts to preserve exploration capacity.

Discussion: Trajectory Length Imposes a Hard Ceiling on OPD (Section 6.1)

The trajectory-length analysis systematically sweeps the maximum response length during OPD training (student: R1-Distill-1.5B, teacher: JustRL-1.5B) across six values: 0.5K, 1K, 3K, 7K, 10K, and 15K tokens, training for 200 steps.

The sweet-spot phenomenon (Figure 11a). On all three benchmarks, 0.5K and 1K responses produce the weakest results (AIME 2024: ~0.15–0.20; AIME 2025: ~0.08–0.10; AMC 2023: ~0.35–0.40 at step 200), because too few tokens receive supervision. Moderate lengths of 3K and 7K yield the strongest results (AIME 2024: ~0.35–0.40; AIME 2025: ~0.20–0.25; AMC 2023: ~0.65–0.70). Beyond 7K, performance plateaus or declines: 10K and 15K achieve similar or lower accuracy than 7K, with 15K showing degradation on AIME 2024 (~0.32 vs. ~0.38 for 7K) and AIME 2025 (~0.18 vs. ~0.22). The paper concludes: "very short responses (0.5K and 1K) provide too few supervised tokens for sample-efficient learning, while moderate lengths (3K and 7K) yield the strongest results. Beyond this range (10K and 15K), performance plateaus or declines."

Training dynamics at different lengths (Figure 12). The overlap ratio curves reveal structural differences in training stability. At 3K and 7K, overlap rises smoothly from approximately 0.72 to approximately 0.90–0.92 over 200 steps. At 10K, overlap initially rises to approximately 0.88 by step 100 but then dips to approximately 0.84 by step 200, with a small recovery. At 15K, the collapse is dramatic: overlap rises to approximately 0.90 by step 140–160, then drops sharply to approximately 0.66 by step 200, accompanied by spikes in student entropy (from ~2.0 to ~12.5) and gradient norm (from ~5.0 to ~12.5). This indicates late-stage training instability: the teacher's signal at deep positions becomes so unreliable that it destabilizes the entire training process.

Back-to-front entropy propagation (Figure 13). Analyzing student entropy as a function of output position in the 15K setting reveals the origin of this collapse. At early training steps (step 180), entropy is uniformly moderate across all positions (2.0–4.0 from 0K to 15K). By step 210–220, elevated entropy (6.0–8.0) first appears at positions 12K–15K, while earlier positions remain stable. By step 240–250, high entropy (8.0–12.0+) has propagated backward to positions 6K–9K. The paper describes this as: "high entropy first appears at the end of the response and progressively propagates toward earlier tokens as training proceeds." Teacher entropy (Appendix D.1, Figure 23) exhibits the same suffix-to-prefix pattern, confirming that the teacher encounters increasingly unfamiliar prefixes at later positions, producing noisier reward that destabilizes the student.

Teacher continuation experiment (Figure 11b). To directly quantify the teacher's degradation with prefix depth, student rollouts exceeding 16K tokens are truncated at four prefix lengths (1K, 4K, 8K, 16K), and the teacher continues generation from each prefix. The accuracy gain from teacher continuation (vs. student-only completion) decreases monotonically: +0.37 at 1K prefix, +0.27 at 4K, +0.15 at 8K, and only +0.02 at 16K. This directly demonstrates that the teacher's token-level predictions become progressively less informative about producing correct answers as the student-generated prefix grows deeper. The paper concludes: "Dense reward is effective on moderately long reasoning traces, but its reliability degrades with depth as the student prefix drifts further from the states familiar to the teacher."

Discussion: Globally Informative Reward Does Not Guarantee Local Exploitability (Section 6.2)

This experiment addresses the puzzle of why the R1-Distill-7B → R1-Distill-1.5B configuration fails completely when R1-Distill-7B is a stronger model than the successful JustRL-1.5B teacher. The key question: is the 7B teacher's reward signal fundamentally uninformative (globally), or is it informative but locally unexploitable?

Sequence mean reward analysis (Figure 14). For both teachers, the sequence mean reward r̄(y) = (1/T) Σ_t [log π_T(y_t) − log π_θ(y_t)] (averaged over student rollouts) is computed separately for correct and incorrect student outputs. For both teachers, correct rollouts consistently receive higher mean reward than incorrect rollouts, with comparable separability: AUROC = 0.73 for JustRL-1.5B, AUROC = 0.75 for R1-Distill-7B. The distributions are clearly separated in both cases, with correct rollouts concentrated at higher reward values (around −0.05 to −0.10) and incorrect rollouts at lower values (−0.15 to −0.25). The paper states: "Correct rollouts consistently receive higher sequence mean reward than incorrect ones, with comparable AUROC values... The failing 7B teacher does not produce a weaker global signal."

The local exploitability hypothesis. If global reward quality is comparable, the failure must lie in how per-token advantages aggregate into effective gradient updates. The paper's hypothesis is that the 7B teacher's per-token advantages, while individually large (visible in the more negative overlap-token advantage in Figure 6, bottom-middle: −0.004 to −0.005 vs. −0.001 for JustRL-1.5B by late training), are directionally incoherent — pointing in mutually canceling directions across positions within a sequence. The evidence is circumstantial but internally consistent: the failing run shows persistently smaller gradient norms (Appendix B.2, Figure 19, middle: 0.0–0.2 vs. 0.4–2.0 for the successful run), larger and unresolved probability differences on the token with the highest absolute advantage (Figure 19, right: ~0.04–0.08 vs. ~0.02–0.04), and minimal loss reduction. The paper's interpretation: "One possible explanation is that the 7B teacher's per-token advantages, while individually large, are anisotropic across positions within each sequence. When these heterogeneous signals are aggregated into a gradient update, they partially cancel, yielding small effective gradients despite large per-token rewards."

Caveat. The paper explicitly notes that this hypothesis is not directly verified: "We have not directly verified this anisotropy hypothesis, and doing so would require analyzing the directional structure of per-token gradients, which we leave to future work." The evidence — co-occurrence of high per-token advantage and low gradient norm — is suggestive but does not constitute proof of the anisotropy mechanism. Alternative explanations (e.g., the 7B teacher's advantages being concentrated on tokens with small student probability, reducing their effective gradient weight) are not ruled out.

Discussion: Sampled-Token OPD Matches Top-k OPD, Top-1 Fails (Section 6.3)

The support-size experiment varies k in Top-k OPD and compares against the cheapest variant: sampled-token OPD.

Main result (Figure 15). Using R1-Distill-1.5B as student and JustRL-1.5B as teacher, sampled-token OPD achieves performance comparable to Top-4, Top-16, and Top-64 on all three benchmarks. On AIME 2024: sampled-token reaches ~0.46 at step 260, Top-4 reaches ~0.46, Top-16 reaches ~0.47, Top-64 reaches ~0.46. On AIME 2025: sampled-token ~0.33, Top-4 ~0.33, Top-16 ~0.33, Top-64 ~0.33. On AMC 2023: sampled-token ~0.79, Top-4 ~0.79, Top-16 ~0.79, Top-64 ~0.79. Top-1 consistently underperforms: ~0.45 on AIME 2024, ~0.31 on AIME 2025, ~0.77 on AMC 2023. The paper concludes: "sampled-token OPD achieves performance comparable to that of the Top-k settings... The only clearly worse configuration is Top-1, which consistently underperforms."

Training dynamics (Figure 16). The overlap ratio and stability metrics reveal why Top-1 fails. Top-1 exhibits unstable overlap growth — the overlap ratio fluctuates between 0.75 and 0.95 with sharp oscillations — accompanied by spikes in student entropy (up to ~3.0) and gradient norm (up to ~24). Top-4 is more stable but still shows a late-stage dip in overlap ratio (from ~0.92 at step 200 to ~0.88 at step 260). Top-16 and Top-64 remain smooth throughout. The explanation: "Top-1, by contrast, always selects the argmax token, thereby concentrating the reward on a single mode. Small policy changes can flip which token occupies rank 1, creating an unstable reward signal that does not average out over training." The failure is not about using too few tokens — sampled-token OPD also uses only one token per position and works well — but about using a biased, mode-concentrated selection rule. Sampled-token OPD draws ŷ_t ~ p_t stochastically, providing unbiased coverage of the high-probability region across training batches; Top-1 deterministically selects argmax, creating a discontinuous reward landscape.

Practical guidance. Enlarging k beyond 4 brings negligible additional gain while increasing computational overhead. The paper implies that sampled-token OPD — the cheapest variant — is sufficient for practical use, provided the degenerate Top-1 setting is avoided.

Ablation Studies and Robustness Checks

Overlap optimization sufficiency (Section 4.2, Figure 7): Restricting OPD supervision to only the intersection of student and teacher top-k tokens (Overlap Top-k) matches the performance of standard Student Top-k OPD across all three benchmarks — AIME 2024 (~0.44 vs. ~0.44 at step 200), AIME 2025 (~0.31 vs. ~0.32), AMC 2023 (~0.77 vs. ~0.77) — while restricting to non-overlap tokens (Non-Overlap Top-k) substantially underperforms (AIME 2024: ~0.38; AIME 2025: ~0.27; AMC 2023: ~0.72). The overlap-token advantage curves for Student Top-k and Overlap Top-k are "nearly indistinguishable." This demonstrates that the overlap region is not merely correlated with OPD success but is causally responsible for the training signal. The non-obvious finding is that the extra tokens in the student-only support contribute negligible useful signal despite being included in standard OPD — practitioners following the standard recipe are computing gradients on tokens that provide no benefit, and those gradients can be safely dropped without performance loss.

Support size sensitivity (Section 6.3, Figures 15–16): Sampled-token OPD matches Top-4, Top-16, and Top-64 performance across all three benchmarks. Top-1 consistently underperforms and exhibits unstable training dynamics (sharp entropy and gradient norm spikes). The non-obvious finding is that support size is not a critical hyperparameter — the cheapest variant (sampled-token OPD, requiring only one teacher query per position) suffices, provided the token selection rule is stochastic (sampling from the student distribution) rather than deterministic (argmax). The failure mode of Top-1 is instructive: it is not a matter of insufficient tokens but of a biased estimator that creates a discontinuous reward landscape.

Prompt template alignment (Section 5.2, Figure 9; Appendix C.4, Figure 22): Simply switching the prompt template from the original DAPO format ("Answer: $Answer") to the teacher-aligned format ("\boxed{}") yields consistent accuracy improvements and higher overlap growth. This holds across all three benchmarks, with larger gains on AIME datasets. The non-obvious implication: prompt formatting is not a superficial detail in OPD — it directly affects whether the student's generated states are compatible with the teacher's token distribution, and mismatched templates can materially degrade OPD outcomes even when the underlying problem content is identical.

Prompt content alignment (Section 5.2, Figure 10): Using prompts from the teacher's post-training data yields stronger downstream performance compared to in-domain but unseen prompts, despite producing a lower overlap ratio. The mechanism is revealed by the overlap mass and entropy metrics: teacher-aligned prompts produce higher concentration of student probability mass on shared tokens (0.85–0.92 vs. 0.30–0.45) but also substantially lower student entropy (2.0 vs. 5.0–6.0 by step 200), indicating stronger alignment at the cost of reduced exploration. The non-obvious finding is that higher overlap ratio does not always mean better alignment — effective alignment can manifest as concentrated mass on fewer but more strongly shared tokens, producing lower set-level overlap but stronger distributional agreement.

Cross-model generalizability of mechanism (Appendix B.3, Figure 20): The progressive alignment signature (rising overlap ratio, advancing overlap-token advantage toward zero, narrowing entropy gap) replicates with a different student (R1-Distill-7B) and different teacher pairs (Skywork-OR1-Math-7B as successful teacher, R1-Distill-14B as failing teacher). Performance gains accompany rising overlap (0.80 → 0.96 for the successful run); stagnation accompanies unstable or poor overlap dynamics. This demonstrates that the mechanism generalizes beyond the specific 1.5B-scale student used in the main experiments.

Overlap mass robustness (Appendix B.1, Figure 18): Across both successful and failing runs, the overlap tokens consistently carry 97–99% of the total probability mass for both student and teacher distributions throughout training. This means the overlap ratio's diagnostic power comes from which tokens are in the shared set — not from whether the shared set captures most of the probability mass, which it always does. Low overlap ratio indicates that even though both models concentrate their mass on a small number of tokens, those tokens are different for student and teacher. Successful OPD shifts which tokens receive that mass until they coincide.

Entropy gap stability complementarity (Section 4.1, Figure 6; Section 5.1, Figure 8): The entropy gap is a more sensitive indicator of mismatch than overlap ratio alone. In the base-initialized cold-start control (Figure 8), the overlap ratio eventually recovers to near-SFT levels (0.70 vs. 0.74 at step 200), but the entropy gap remains substantially larger (0.15–0.35 vs. 0.05–0.10) and performance is correspondingly lower. This indicates that the student can nominally share the teacher's token space (high overlap) without matching the teacher's confidence profile (high entropy gap), and the entropy gap captures this residual misalignment that overlap ratio misses.

Auxiliary optimization diagnostics (Appendix B.2, Figure 19): Three additional metrics — policy gradient loss, gradient norm, and extreme-advantage token probability difference — consistently separate successful from failing runs. Successful OPD shows large loss reduction (1.6 → 0.2), sustained gradient magnitude (0.4–2.0), and progressive resolution of the largest probability mismatches (0.12 → 0.02). Failing OPD shows minimal loss reduction (stays near 0.2), weak gradients (0.0–0.2), and persistent extreme-token discrepancies (0.04–0.08). These are not central metrics but provide optimization-level confirmation of the overlap-based diagnosis.

Deduplication rigor (Appendix C.3): The DeepMath subset used for prompt content comparison is constructed via two-stage deduplication against DAPO-Math-17K: (1) exact-match deduplication on extracted question text, (2) semantic deduplication using sentence embeddings (all-mpnet-base-v2) with a cosine similarity threshold of 0.6. This ensures that the comparison isolates the effect of prompt content alignment with the teacher's training data rather than prompt overlap.

Negative result: ReST^EM not tested. The paper focuses exclusively on standard OPD (SFT + reverse KL minimization). There is no comparison with or ablation of RL-based fine-tuning methods (PPO, GRPO, ReST^EM) as alternatives to OPD for the same teacher-student pairs. This limits claims about OPD's unique advantages or disadvantages relative to RL.

Negative result: No combination of cold start with teacher-aligned prompts. The two corrective recipes are studied independently but never combined. A natural question — whether an SFT cold start plus teacher-aligned prompt selection would yield additive gains — is not addressed.

Negative result: Long-horizon mitigation not tested. Section 6.1 identifies a trajectory-length ceiling but proposes no solution beyond suggesting future hybrid approaches. No experiment tests whether curriculum strategies (progressively increasing max length), mixed-length training, or truncated backpropagation could mitigate the late-position instability.

Critical Assessment

Claim 1: "Thinking-pattern consistency and new knowledge jointly govern OPD effectiveness." The experiments in Section 3 provide strong support for both factors individually, but the joint claim depends on the reader accepting an implicit conjunctive logic rather than an experimentally tested interaction. The thinking-pattern experiment (Section 3.1) shows that low overlap → poor OPD outcome, comparing two teachers with broadly comparable benchmark performance. The new-knowledge experiment (Section 3.2) shows that same-pipeline teachers → poor OPD outcome, comparing two teachers with comparable thinking patterns (similar initial overlap ratios) but different training histories. These are separate experiments demonstrating each condition's necessity in isolation. What the paper does not provide is a fully crossed 2×2 experiment (high vs. low overlap × new vs. no-new knowledge) showing that both conditions must simultaneously hold. The reverse distillation experiment (Section 3.3) comes closest to this by demonstrating a configuration (R1-Distill-7B → JustRL-1.5B) where thinking-pattern consistency is high (same model family) but new knowledge is absent (or negative — the teacher has less knowledge than the student), and OPD fails. This still leaves the complementary cell untested: what happens with low overlap but new knowledge? The paper's framework predicts failure, but no experiment directly tests this. This is a legitimate weakness: the conjunction claim is inferred from separate necessity experiments rather than demonstrated through a comprehensive design.

Claim 2: "Successful OPD is mechanically driven by progressive alignment on high-probability overlap tokens." This claim is strongly supported by the controlled comparison in Section 4.1 (successful vs. failing run with all three dynamic metrics tracked), the overlap sufficiency ablation in Section 4.2 (Overlap Top-k matches Student Top-k), and the cross-model validation in Appendix B.3 (same signature with different model pairs). The causal evidence from the overlap sufficiency ablation is particularly strong: restricting gradients to only overlap tokens recovers full performance, demonstrating that the overlap region is not just where alignment manifests but where the operative gradient signal is concentrated. The self-reinforcing dynamic — overlap rising not despite but because of optimization — is demonstrated by the overlap ratio curves for Student Top-k and Overlap Top-k both rising from ~72% to ~91%, while Non-Overlap Top-k causes overlap to decrease initially. One weakness: the ablation only tests a single successful configuration (JustRL-1.5B → R1-Distill-1.5B). It is possible that in configurations with substantially different overlap dynamics, the overlap tokens would not be sufficient. The cross-model validation in Appendix B.3 mitigates but does not fully address this.

Claim 3: "Off-policy cold start and teacher-aligned prompt selection can recover failing OPD." The cold-start experiment (Section 5.1) convincingly demonstrates that SFT initialization raises initial overlap and improves final performance in a configuration (Qwen3-1.7B-Base + Qwen3-4B Non-thinking) where pure OPD performs poorly. The teacher-aligned prompt experiments (Section 5.2) demonstrate that both template and content alignment improve OPD signal quality. However, three limitations weaken the "recovery" framing. First, the recipes are tested on configurations that are a priori known to be suboptimal, not on configurations that practitioners would independently discover to be failing. The cold-start experiment uses the teacher-student pair from Section 3.1 that the paper already established as poor — it is a demonstration on a known failure, not a blind recovery of an unexpected failure. Second, neither recipe is tested against a configuration where the teacher lacks new knowledge (the other failure mode). SFT cold start raises overlap, but if the teacher has no new knowledge to transfer (e.g., same-pipeline teacher), raising overlap will not help — it may even accelerate regression, as implied by the reverse distillation result. The paper does not test cold start with a same-pipeline teacher to verify this boundary. Third, the entropy collapse observed with teacher-aligned prompts (Figure 10) is identified as a risk but not mitigated experimentally — the recommended mixing strategy is stated but not implemented or evaluated. The recipes are demonstrated to work under specific conditions, but their robustness and failure modes are left unexplored.

Claim 4: "OPD has a trajectory-length ceiling where reward quality degrades with depth." The length sweep (Figure 11a), training dynamics at different lengths (Figure 12), back-to-front entropy propagation (Figure 13), and teacher continuation experiment (Figure 11b) together provide compelling evidence for this claim. The teacher continuation experiment is particularly elegant: it directly measures the teacher's accuracy advantage when completing from student-generated prefixes of varying depth, showing a monotonic decline from +0.37 at 1K to +0.02 at 16K. A legitimate limitation: the experiments use a single teacher-student pair (JustRL-1.5B → R1-Distill-1.5B), and the specific sweet spot (3K–7K tokens) is likely specific to this model pair, dataset, and domain. A larger teacher might maintain reliable signal to greater depths; a smaller student might diverge faster. The paper acknowledges this implicitly by not claiming a universal threshold but rather the existence of a ceiling. A stronger demonstration would sweep across multiple teacher-student pairs or at minimum vary model scale to show how the ceiling shifts. The back-to-front entropy propagation (Figure 13) is shown for only the 15K setting; showing the same visualization for 7K (which doesn't collapse) and 10K (which partially collapses) would strengthen the claim that the collapse mechanism is specifically tied to late-position instability rather than some other training artifact.

Claim 5: "The failing teacher's reward is globally informative but locally unexploitable." The AUROC comparison (Figure 14: 0.73 vs. 0.75) clearly demonstrates that the failing 7B teacher's sequence-level reward is equally correlated with rollout correctness as the successful 1.5B teacher's. This is the strongest and cleanest result in Section 6. However, the local exploitability explanation (anisotropy hypothesis) is explicitly flagged as unverified, and the evidence is circumstantial. The paper does not directly measure gradient directional coherence — it infers incoherence from the co-occurrence of large per-token advantages and small gradient norms. This inference could be wrong: small gradient norms could result from the advantages being concentrated on tokens with very small student probabilities (reducing their effective weight in the gradient), or from the teacher's advantages being large but having high variance that averages out over the batch. The paper acknowledges this: "We have not directly verified this anisotropy hypothesis." This is a hypothesis, not a demonstrated mechanism, and the paper's hedging should be taken seriously.

Structural limitations across all experiments:

  • Single-run reporting with no error bars. Every training curve in the paper (Figures 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 15, 16, 17, 19, 20, 21) shows a single training run per configuration. There are no multiple random seeds, no confidence intervals, no statistical tests. Given that some of the claimed performance differences are modest (e.g., approximately 2–4 percentage points in some comparisons), the absence of any measure of variance makes it impossible to assess whether these differences are statistically reliable or within the noise of training stochasticity. The Benchmark-wise breakdown in Appendix A.3 (Figure 17) and Appendix C.4 (Figure 22) provides some qualitative robustness by showing that trends hold across three benchmarks, but this is a weak substitute for proper replication.

  • Small evaluation set. The MATH-related benchmarks (AIME 2024: 30 problems, AIME 2025: 30 problems, AMC 2023: 25 problems) are competition-level datasets with very few problems. Reporting avg@16 compounds this: each accuracy number is based on at most 30 × 16 = 480 sampled solutions. A single correctly or incorrectly solved problem swings avg@16 by approximately 3.3 percentage points on AIME. The paper reports performance differences of 2–5 percentage points in some comparisons — these are within the resolution of a single problem. The gap recovery rates (Section 3.2: 5.3% vs. 16.9%, 15.6% vs. 58.6%) are based on these same small evaluation sets.

  • Single domain (mathematical reasoning). All experiments are on mathematical reasoning benchmarks using models trained on math data. The paper explicitly acknowledges this limitation (Section 8: "All experiments in this work are conducted on mathematical benchmarks. Whether the same conditions and token-level mechanisms govern OPD in other domains such as code and open-ended settings remains an important open question.") The dynamics — overlap ratio, progressive alignment, trajectory-length ceiling — may be specific to domains where the answer space is highly structured (token sequences leading to a single correct answer). In open-ended generation or creative tasks, the concept of "thinking patterns" and overlap tokens may be less well-defined, and the trajectory-length ceiling may behave differently.

  • No comparison with RL alternatives. The paper studies OPD in isolation, without comparing against outcome-reward RL (PPO, GRPO, ReST^EM) for the same teacher-student pairs. The finding that R1-Distill-7B fails as an OPD teacher does not rule out that outcome-reward RL with R1-Distill-7B as a reward model could be effective — the global reward is informative (AUROC 0.75), and RL might exploit this signal where OPD's local gradients fail. The paper does not address this comparison, making it unclear whether OPD failures represent fundamental limitations of token-level supervision or limitations of OPD's specific optimization approach (reverse KL) that other methods might circumvent.

  • Idealized difficulty estimation absent. The paper does not address how a practitioner would know in advance whether a given teacher-student pair satisfies the thinking-pattern consistency and new-knowledge conditions. The overlap ratio can be measured only after the teacher has been queried on student-generated rollouts, which requires running OPD (or at least a forward pass). There is no method proposed for predicting OPD success from teacher and student properties alone (scale, training data, benchmark scores). The diagnostic framework is therefore retrospective — it explains failures after they occur rather than preventing them.

  • Cold start implementation details may not generalize. The SFT phase uses 200K teacher-generated rollouts from a specific prompt set (OpenThoughts3-1.2M math subset), a specific teacher (Qwen3-4B Non-thinking), with specific generation hyperparameters (temperature 0.7, top-p 0.95, max length 12,288). The effectiveness of cold start likely depends on the quality and coverage of these rollouts — too few rollouts may not raise overlap sufficiently; too many may cause the student to overfit to the teacher's specific outputs. The paper does not ablate the SFT data quantity or quality, making it unclear how robust this recipe is to implementation choices.

Missing experiments that would strengthen the paper:

  1. Crossed 2×2 experiment (overlap × new knowledge): Testing all four combinations — (high overlap, new knowledge), (high overlap, no new knowledge), (low overlap, new knowledge), (low overlap, no new knowledge) — would directly validate the joint necessity claim. The first two are tested; the last two are not.

  2. Cold start with same-pipeline teacher: Testing whether SFT cold start can recover OPD when the teacher has no new knowledge (e.g., R1-Distill-7B → R1-Distill-1.5B with cold start) would establish the boundary condition for the recipe.

  3. Multiple random seeds for key comparisons: Even 3 seeds on the main contrastive experiments (Figures 2, 4, 5, 6) would substantially strengthen confidence in the reported differences.

  4. Unified recipe (cold start + teacher-aligned prompts): Testing the combination would establish whether the two recipes are complementary or redundant.

  5. RL baseline for failing OPD configurations: Comparing OPD against outcome-reward RL (using the same teacher as a reward model) on the R1-Distill-7B → R1-Distill-1.5B configuration would clarify whether the failure is specific to OPD's optimization approach or fundamental to the teacher's token-level signal quality.

  6. Trajectory-length ceiling with different model scales: Sweeping the maximum response length experiment across multiple teacher-student pairs (varying both student and teacher scale) would reveal how the ceiling shifts with model capacity.

  7. Curriculum or mixed-length training for long-horizon mitigation: Testing whether progressively increasing max response length during training, or mixing short and long sequences, could prevent the late-stage entropy collapse observed at 15K.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Remains Unaccounted For

The entire compute-optimal scaling framework (both the phenomenology and the proposed recipes) rests on the ability to diagnose when OPD is failing — specifically, whether the overlap ratio is stagnant, whether new knowledge is present, and whether the teacher's signal is degrading with trajectory depth. The paper provides a powerful retrospective diagnostic toolkit: the overlap ratio, overlap-token advantage, and entropy gap can all be computed continuously during training and clearly separate successful from failing runs. However, the paper provides no method for predicting these conditions before committing to an OPD run. A practitioner cannot know in advance whether a given teacher-student pair satisfies the thinking-pattern consistency or new-knowledge conditions without first running OPD (or at minimum performing teacher forward passes on student-generated rollouts to compute initial overlap), at which point substantial compute has already been invested.

The overlap ratio diagnostic is conceptually elegant but practically reactive: it reveals failure as it occurs rather than preventing it. The paper acknowledges the difficulty estimation problem obliquely — the discussion of local vs. global reward (Section 6.2) notes that the 7B teacher's global reward is informative (AUROC 0.75) yet OPD fails, implying that simple metrics like teacher benchmark performance are insufficient predictors. But the paper does not propose any prospective screening method: no mapping from model properties (scale, training data, benchmark scores) to predicted OPD success, no lightweight pre-check that could be run before committing to a full training run, no threshold on initial overlap that predicts whether training will recover or stagnate. The experiments in Section 3 retrospectively diagnose configurations that were already known to be failing (e.g., Qwen3-4B Non-thinking → Qwen3-1.7B-Base was established as poor in Figure 2; the cold-start experiment in Section 5.1 then "recovers" this known failure). No experiment demonstrates that the diagnostic framework can be applied prospectively to an untested teacher-student pair and correctly predict the outcome.

The practical consequence is that a practitioner evaluating OPD for a new teacher-student pair faces a gamble: invest compute in a full OPD run and hope the conditions are met, or run trial experiments that themselves cost significant compute (computing initial overlap requires generating student rollouts and querying the teacher). The paper does not report what fraction of its tested configurations succeeded vs. failed, making it impossible to estimate the base rate of OPD failure that a practitioner should expect. The 200-step training runs used throughout most experiments (with default batch size 64, rollout number 4, max response length 7168 — from Table 2, Appendix A.2) represent a non-trivial compute investment, especially with 7B+ teachers where each teacher forward pass is expensive.

Mitigation status: The paper does not address this limitation directly. It provides no prospective diagnostic method, no failure prediction model, and no estimate of the computational cost of diagnosing a configuration before full training. This is a gap between the paper's retrospective explanatory power and its prospective practical utility.

All Results Are Limited to a Single Domain (Mathematical Reasoning) and Two Model Families

Every experiment in the paper — every overlap ratio curve, every gap recovery rate, every trajectory-length ceiling observation — is conducted on mathematical reasoning benchmarks (AIME 2024, AIME 2025, AMC 2023) using models from the Qwen3 and DeepSeek families trained or fine-tuned on mathematical data (DAPO-Math-17K, DeepMath, OpenThoughts3 math subset). The paper acknowledges this explicitly in Section 8:

"All experiments in this work are conducted on mathematical benchmarks. Whether the same conditions and token-level mechanisms govern OPD in other domains such as code and open-ended settings remains an important open question."

This is a legitimate and consequential limitation. Mathematical reasoning has structural properties that may make the paper's findings domain-specific rather than universal. In math problems, the answer space is narrow and well-defined — there is typically one correct final answer that can be verified through exact string matching. The "thinking patterns" that the paper operationalizes through overlap ratio may reflect domain-specific reasoning structures (step-by-step derivations, algebraic manipulations, arithmetic calculations) that have clear token-level signatures. In code generation, the reasoning pattern is syntactically constrained and the "correctness" is verifiable through execution, potentially creating even stronger token-level alignment than math. In open-ended generation (dialogue, creative writing, summarization), the concept of a "thinking pattern" may be less crisply defined — there is no single correct continuation, and the teacher's distribution may be inherently more diffuse, producing lower overlap ratios and weaker token-level signal even under ideal conditions. The paper's central metrics (overlap ratio, overlap-token advantage) depend on the existence of concentrated probability mass on a small set of tokens — in math, 97–99% of the mass sits in the top-k tokens (Appendix B.1, Figure 18). In open-ended generation, the probability distribution may be substantially more flat, making overlap ratio a less informative diagnostic.

The confinement to two model families (Qwen3 and DeepSeek) raises questions about whether the reverse distillation finding — that same-family teachers at different scales are distributionally indistinguishable — generalizes across architectures and training pipelines. The DeepSeek models (R1-Distill-1.5B, R1-Distill-7B, R1-Distill-14B) all share the same base architecture (Qwen) and were trained through the same distillation pipeline [Guo et al., 2025], so their distributional similarity at different scales is perhaps unsurprising. Models from different families (e.g., LLaMA vs. Qwen) trained on different data mixtures might exhibit fundamentally different overlap dynamics, and the "new knowledge" condition might manifest differently when the teacher's knowledge comes from architectural or data differences rather than additional RL post-training. The paper's "new knowledge" experiments (Section 3.2) use RL-augmented teachers derived from the same base checkpoints, so "new knowledge" is operationally defined as "additional RL training beyond what the student received." Whether other forms of new knowledge — different pretraining data, different architectural inductive biases, different fine-tuning curricula — behave the same way under OPD is untested.

Mitigation status: The paper explicitly acknowledges this as an open question and does not attempt to address it experimentally. The cross-model validation in Appendix B.3 extends the mechanism finding to a different scale (7B student, 7B and 14B teachers) but remains within the same domain (math) and same model family (DeepSeek). No code generation, no open-ended generation, no cross-family distillation is tested.

The Trajectory-Length Ceiling Is a Fundamental Unresolved Limitation of OPD

Section 6.1 provides compelling evidence that OPD's dense token-level supervision degrades with trajectory depth: the teacher's accuracy advantage over the student drops from +0.37 at 1K prefix to +0.02 at 16K prefix (Figure 11b); training with 15K max response length exhibits catastrophic late-stage collapse where overlap ratio drops from ~0.90 to ~0.66 accompanied by entropy and gradient norm spikes (Figure 12); and the instability originates at the end of trajectories and propagates backward (Figure 13). The paper characterizes this as a "fundamental tradeoff in OPD's token-level supervision": dense reward is effective on moderately long reasoning traces but unreliable at depth.

What the paper does not provide is any solution to this ceiling. The future work section (Section 8) gestures toward "hybrid approaches that combine dense token-level supervision on short segments with sparse outcome-level rewards for longer horizons, as well as curriculum strategies that progressively extend the supervised horizon during training." But no experiment tests any mitigation: no curriculum learning (progressively increasing max response length during training rather than fixing it at 15K), no mixed-length training (sampling both short and long sequences in the same batch), no truncated backpropagation through time (stopping gradients before the unstable suffix), no hybrid OPD+RL baseline on long trajectories. The paper identifies a hard problem and stops there.

This is a practically significant limitation because long-horizon reasoning is precisely where OPD's dense supervision should be most valuable relative to sparse outcome-reward RL. On short reasoning traces (e.g., 1K–3K tokens), outcome-reward RL can be sample-efficient because the credit assignment problem is manageable — there are few enough steps that a sparse terminal reward provides reasonable signal. On very long traces (10K+ tokens), outcome-reward RL suffers from severe credit assignment difficulty, and dense per-token supervision should theoretically shine. But the paper's findings invert this intuition: OPD's dense supervision is most reliable on short-to-moderate traces (3K–7K, the sweet spot) and becomes actively harmful on long traces, precisely where RL needs the most help. This means OPD is most useful in the regime where RL already works reasonably well, and least useful in the regime where RL struggles — a dispiriting complementarity pattern.

Moreover, the trajectory-length ceiling is likely to bite harder as the field pushes toward extended chain-of-thought reasoning (where responses routinely exceed 10K–20K tokens) and agentic multi-turn interactions (where total trajectory length across turns can be orders of magnitude larger). The paper's experiments use models at the 1.5B–7B scale; larger models with more sophisticated reasoning capabilities may produce even longer rollouts, exacerbating the ceiling. The ceiling may also be domain-dependent: in math, reasoning traces have a natural termination (arriving at the answer); in code generation, a correct solution could be arbitrarily long; in agentic settings, trajectories are unbounded. The paper provides no guidance on how the ceiling scales with model capacity, domain, or task structure.

Mitigation status: None experimentally. The paper acknowledges the ceiling and suggests future work directions but provides no mitigation recipe comparable to the cold-start or teacher-aligned prompt strategies for the conditions identified in Sections 3 and 5. This is a gap between the phenomenological analysis (which is thorough) and the practical recipe section (which is incomplete regarding this limitation).

The Anisotropy Hypothesis Is Unverified, Leaving the Deepest Failure Mode Unexplained

Section 6.2 presents one of the paper's most intriguing findings: the R1-Distill-7B teacher's sequence-level reward is globally informative (AUROC 0.75, comparable to the successful JustRL-1.5B teacher's 0.73 — Figure 14), yet OPD fails completely (Figure 6). The paper hypothesizes that the failure is due to "anisotropic" per-token advantages — individually large but directionally incoherent across positions, producing small effective gradients despite large per-token signals. The evidence is circumstantial: the failing run exhibits consistently smaller gradient norms (Appendix B.2, Figure 19, middle panel: 0.0–0.2 vs. 0.4–2.0), larger persistent probability discrepancies on extreme-advantage tokens (Figure 19, right panel: ~0.04–0.08 vs. ~0.02–0.04), and a more negative overlap-token advantage that does not improve (Figure 6, bottom-middle: −0.004 to −0.005 vs. approaching 0.000). But the paper explicitly states:

"We have not directly verified this anisotropy hypothesis, and doing so would require analyzing the directional structure of per-token gradients, which we leave to future work."

This is more than a minor caveat — it is a gap at the center of the paper's explanatory framework. The global-vs-local distinction is the paper's deepest mechanistic claim, the one that most sharply distinguishes OPD failure from naive explanations ("the teacher is bad"). It is also the claim with the weakest direct evidence. The co-occurrence of large per-token advantages and small gradient norms is consistent with the anisotropy hypothesis but does not rule out alternative explanations:

  • Mass-weighting alternative: The 7B teacher's large advantages may be concentrated on tokens where the student assigns very small probability. Because the gradient contribution of each token is weighted by p_t(v) (the student's probability on that token), a large advantage log p_t(v) − log q_t(v) on a token with small p_t(v) contributes little to the overall gradient. The overlap-token advantage is computed with renormalized distributions over the overlap set, which removes this mass-weighting effect — a token can have a large renormalized advantage while contributing negligible gradient if the student assigns it low absolute probability mass.

  • Variance alternative: The 7B teacher's per-token advantages may have high variance across the batch, causing the batch-averaged gradient to have a small norm not because individual advantages are directionally incoherent but because they are noisy and average toward zero. This would be a signal-to-noise problem rather than a geometry problem.

  • Optimization landscape alternative: The 7B teacher's reward landscape may be locally flat around the student's current policy — not because advantages are contradictory but because the landscape curvature is small. This would produce small gradients without requiring directional cancellation.

Distinguishing among these alternatives matters for designing fixes. If the problem is anisotropy (directional cancellation), the solution might involve filtering or reweighting per-token advantages to emphasize coherent subsets. If the problem is mass-weighting (advantages on low-probability tokens), the solution might involve amplifying the gradient on high-advantage tokens regardless of student probability. If the problem is high variance, the solution might involve larger batch sizes or advantage normalization. The paper's failure to diagnose which mechanism is operative means the practical guidance ("avoid large same-family teachers without new knowledge") is empirically grounded but mechanistically opaque — a practitioner knows that R1-Distill-7B fails but not why, and therefore cannot predict whether other large teachers from different families or training pipelines would also fail.

Mitigation status: The paper explicitly flags this as unverified and leaves it to future work. This is intellectually honest but leaves a significant gap in the paper's explanatory framework. A preliminary experiment — computing the cosine similarity between per-position gradient directions within each sequence, or measuring the variance of per-token advantage signs — would have provided suggestive evidence one way or another without requiring full gradient decomposition. Such experiments are absent.

No Statistical Reliability Measures, Single Runs Throughout

Every training curve in the paper represents a single training run per configuration. No experiment is replicated across multiple random seeds, no confidence intervals are reported, no statistical tests are performed. The evaluation benchmarks (AIME 2024: 30 problems, AIME 2025: 30 problems, AMC 2023: 25 problems) are extremely small, and the primary metric (avg@16) compounds this: each reported accuracy number is based on at most 30 × 16 = 480 sampled solutions for AIME, meaning a single problem solved or missed swings avg@16 by approximately 3.3 percentage points.

The paper reports performance differences in the 2–5 percentage point range as meaningful (e.g., the gap between Overlap Top-k and Student Top-k at step 200 on AIME 2024 in Figure 7: ~0.44 vs. ~0.44 — identical; the prompt template effect in Figure 9: ~0.50 vs. ~0.46 at step 200; the support-size comparisons in Figure 15: 0.46–0.47 across variants). These are within the resolution of a single AIME problem. The gap recovery rates reported in Section 3.2 (5.3% vs. 16.9% for DeepSeek family; 15.6% vs. 58.6% for Qwen family) are point estimates from single runs and inherit all the variance of the underlying accuracy measurements. The paper's qualitative conclusions ("successful vs. failing," "consistently outperforms") rely on visual inspection of training curves, not on formal comparisons.

The absence of multiple seeds is particularly problematic for the claims about training dynamics. The overlap ratio curves in Figure 6 show a clear separation: JustRL-1.5B teacher → steadily rising overlap (0.72 → 0.91); R1-Distill-7B teacher → flat overlap (~0.74). Without multiple seeds, it is unclear whether the "flat" curve is genuinely flat or whether it fluctuates around a slowly rising trend whose slope is obscured by noise. The late-stage collapse at 15K max response length (Figure 12: overlap dropping from ~0.90 to ~0.66) is shown for a single run — could this be an artifact of a particular random seed's trajectory, or does it occur reliably? The paper's cross-model validation in Appendix B.3 (Figure 20) provides some qualitative replication (same pattern with different model pairs), but it does not address within-configuration variance.

The small evaluation sets and single-run reporting interact with the paper's narrow domain scope. If the reported differences are within the noise of a 30-problem benchmark, we have weak evidence that the interventions (cold start, teacher-aligned prompts) produce genuine improvements rather than lucky evaluation draws. This is not a fatal flaw — the qualitative patterns are consistent enough across three benchmarks to be credible — but it weakens the precision of the paper's quantitative claims and makes the threshold for "successful OPD" somewhat arbitrary.

Mitigation status: The paper does not address this limitation at all. No mention is made of random seeds, statistical testing, or the resolution of the evaluation benchmarks. The benchmark-wise breakdowns in Appendix A.3 (Figure 17) and Appendix C.4 (Figure 22) provide qualitative robustness by showing that trends hold across individual benchmarks, but this is a weak substitute for within-configuration replication.

The Two Failure Modes Are Diagnosed Independently but Never Tested Jointly, and No Unified Recipe Exists

The paper identifies two distinct failure modes — thinking-pattern mismatch (low initial overlap) and absence of new knowledge (same-pipeline teacher) — and proposes two recipes targeting them — off-policy cold start (raises initial overlap) and teacher-aligned prompt selection (sharpens signal on overlap tokens). However, these recipes are tested on configurations that are selected to isolate exactly one failure mode each. The cold-start experiment (Section 5.1) uses the Qwen3-1.7B-Base + Qwen3-4B (Non-thinking) pair, which Section 3.1 already established has low initial overlap (thinking-pattern mismatch). The prompt-alignment experiments (Section 5.2) use JustRL-1.5B → R1-Distill-1.5B (template alignment, already a successful configuration) and Qwen3-1.7B-Base + Qwen3-4B-Base-GRPO (content alignment, a configuration with reasonable initial overlap of ~70% — Figure 4 right). No experiment tests either recipe on a configuration where both failure modes are present, nor on a configuration where the failure mode is unknown.

This matters for practical deployment because a practitioner encountering a failed OPD run will typically not know which of the two conditions is violated without conducting the retrospective diagnostics the paper itself introduced. The overlap ratio can be measured during training to diagnose thinking-pattern mismatch, but the new-knowledge condition has no simple real-time diagnostic — it requires comparing the teacher to the student's training history, which may not be fully known if the student and teacher are from different organizations or training pipelines. A practitioner might apply cold start to a configuration that is failing due to lack of new knowledge (e.g., same-pipeline 7B → 1.5B). The paper's framework predicts this would not help — SFT on same-pipeline teacher rollouts would move the student closer to a teacher whose knowledge is already encoded in the student's training distribution, potentially even causing regression (as implied by the reverse distillation experiment in Section 3.3 where SFT on the pre-RL checkpoint erased RL gains). But this prediction is not tested.

The paper also never combines cold start with teacher-aligned prompt selection. Do the two recipes yield additive gains? Do they interact? The cold-start recipe addresses initial overlap; the prompt-alignment recipe sharpens signal quality. In principle, a two-stage pipeline (SFT cold start → OPD with teacher-aligned prompts) could address both conditions simultaneously. But this combination is not tested, leaving practitioners to guess whether deploying both recipes is beneficial or redundant. The entropy collapse tradeoff with teacher-aligned prompts (Figure 10, bottom-right) adds a complication: mixing in out-of-distribution prompts is recommended but not implemented or evaluated. A combined recipe would need to balance SFT data quantity, teacher-aligned prompt ratio, and OOD prompt ratio — a multi-dimensional design space the paper does not explore.

Mitigation status: None. The two recipes are presented as independent fixes for independent problems, with no cross-testing or integration. The paper does not provide guidance on how to choose which recipe to apply to a given failing configuration, nor on whether applying the "wrong" recipe (e.g., cold start for a new-knowledge failure) could be harmful. This leaves a gap between the paper's diagnostic clarity (it can explain why a run failed retrospectively) and its prescriptive utility (it cannot reliably tell practitioners what to do about a failure prospectively without first characterizing the failure mode, which requires running OPD).

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the research conversation around on-policy distillation from demonstration to diagnosis. Prior work — from MiniLLM [Gu et al., 2023] and GKD [Agarwal et al., 2024] to the industrial pipelines in Qwen3, MiMo, and GLM-5 — established OPD as a technique that can work, but treated failures as anomalies to be worked around rather than phenomena to be understood. The field's shared mental model was that OPD is generally reliable, with effectiveness limited primarily by compute budget and engineering quality. A stronger teacher was assumed to be a better teacher.

This paper systematically disproves that assumption and provides the conceptual vocabulary to replace it. Three findings are particularly landscape-shifting:

The two-condition decomposition (thinking-pattern consistency × new knowledge) replaces the scalar notion of teacher quality. The reverse distillation experiment (Section 3.3) — where a model distilled toward its own pre-RL checkpoint regresses exactly, and where a larger same-family teacher produces an indistinguishable trajectory — demonstrates that benchmark performance is not merely an unreliable predictor of OPD success; it can be orthogonal to it. This means the field can no longer evaluate OPD configurations by comparing teacher benchmark scores. Instead, practitioners must assess whether the teacher's token-level distribution overlaps with the student's at student-visited states (thinking-pattern consistency) and whether the teacher encodes genuinely novel capabilities beyond what the student internalized during training (new knowledge). This is a conceptual upgrade: it decomposes "teacher quality" into two factors that are independently necessary and experimentally separable.

The overlap ratio becomes a portable diagnostic that spans conditions, mechanisms, and recipes. Unlike benchmark accuracy, which is a lagging indicator, the overlap ratio reveals whether OPD's self-reinforcing dynamic has engaged while training is in progress. The paper demonstrates its diagnostic power across every experiment: low initial overlap predicts failure (Section 3.1), rising overlap is the dynamical signature of success (Section 4.1), and overlap dynamics explain why SFT cold start works (Section 5.1, Figure 8: SFT initialization raises initial overlap from ~0.52 to ~0.68) and why non-overlap top-k fails (Section 4.2, Figure 7: overlap initially decreases when training only on non-overlap tokens). Practitioners now have a leading indicator that can be monitored continuously — akin to how training loss, gradient norm, and weight statistics became standard tools in deep learning, not because they improve training directly, but because they make training dynamics interpretable and failures diagnosable before compute is wasted.

The global-vs-local reward distinction (Section 6.2) exposes a previously invisible failure mechanism. The finding that the failing R1-Distill-7B teacher produces globally informative reward (AUROC 0.75) that is locally unexploitable (small gradient norms despite large per-token advantages, Figure 19) reveals that OPD failure is not about the teacher providing a bad signal, but about the signal being geometrically incoherent. This distinction has no analog in the off-policy distillation literature, where the teacher's signal quality is typically assessed holistically (e.g., by comparing student performance against teacher-generated targets). It changes how researchers should think about improvement: making OPD more robust is not primarily about finding better teachers (in the benchmark sense), but about understanding and improving the directional coherence of per-token gradients. This redirects research attention from teacher selection toward optimization geometry — a shift that is both more fundamental and more actionable.

The trajectory-length ceiling (Section 6.1) reframes the value proposition of OPD relative to RL. The appeal of OPD over outcome-reward RL has been that dense per-token supervision should be more sample-efficient, especially on long reasoning traces where sparse outcome rewards create severe credit assignment problems. The paper's finding inverts this intuition: OPD's dense signal is most reliable at moderate lengths (3K–7K tokens, the sweet spot in Figure 11a) and becomes actively harmful at long lengths (15K tokens, where training collapses — Figure 12). But 3K–7K token traces are precisely where outcome-reward RL also works reasonably well, because the credit assignment horizon is manageable. On 10K+ traces where RL struggles most, OPD's signal degrades to the point of destabilizing training. This means OPD and RL are complementary in a dispiriting way: OPD is most effective where RL already works, and least effective where RL most needs help. The practical implication is that pure OPD cannot be the solution for long-horizon reasoning, and hybrid approaches that combine dense token-level supervision on short segments with sparse outcome rewards for depth are not merely an incremental improvement — they are a necessary direction.

Reconciliation of conflicting practitioner experiences. The paper's framework explains why different practitioners report different OPD outcomes: they are implicitly testing different difficulty regimes defined by thinking-pattern compatibility and knowledge novelty. A team distilling from an RL-augmented teacher within the same model family (high overlap, new knowledge) sees strong gains. A team distilling from a larger same-family teacher without additional post-training (high overlap, no new knowledge) sees stagnation or regression. A team distilling from a cross-family teacher with no SFT preconditioning (low overlap, possibly new knowledge) sees failure. Before this work, these experiences appeared contradictory. After this work, they are predictable from the two-condition framework. This is directly analogous to how the paper's reference example (the compute-optimal test-time scaling paper) reconciled conflicting findings about self-correction by showing that effectiveness depends on problem difficulty — a structural pattern that individual studies, testing on different implicit difficulty distributions, could not see.

What this work does not change. The paper does not propose a new training algorithm, a new loss function, or a new model architecture. It is not a paradigm shift in the sense of replacing OPD with a fundamentally different technique. It is a diagnostic reframing of an existing technique: it provides the concepts, metrics, and experimental methodology to reason about OPD systematically rather than empirically. This is a different kind of contribution — it changes how researchers think about and investigate OPD rather than what algorithm they run. The downstream impact will depend on whether the diagnostic framework (overlap ratio, overlap-token advantage, entropy gap) is adopted as a standard monitoring toolkit in OPD implementations, and whether the two-condition decomposition guides teacher selection and data curation in practice.

Research directions that become more attractive. The paper makes verifier quality and optimization geometry central concerns, suggesting that work on improving OPD should focus on: (1) understanding the directional structure of per-token gradients and developing objectives that can exploit anisotropic reward landscapes, (2) designing teachers specifically for OPD rather than for benchmark performance — teachers optimized for token-level distributional alignment rather than final-answer accuracy, (3) hybrid OPD+RL approaches that combine dense short-range supervision with sparse long-range rewards, and (4) prospective screening methods that can predict OPD success from teacher and student properties without requiring a full training run.

Research directions that become less attractive. The paper's findings diminish the case for: (1) naively scaling teacher size within the same training pipeline and expecting OPD gains — the reverse distillation experiment shows that same-family 1.5B and 7B teachers are distributionally indistinguishable from the student's perspective, (2) more complex OPD variants (e.g., full-vocabulary over top-k) without addressing the fundamental conditions — the support-size analysis (Section 6.3) shows that sampled-token OPD already works well when conditions are met, and no amount of increased supervision granularity can compensate for low initial overlap or absent new knowledge, and (3) treating OPD as a universal post-training step to be applied uniformly to all teacher-student pairs — the failure modes are systematic and predictable, and applying OPD to configurations that violate the two conditions wastes compute and can actively degrade the student.

Follow-Up Research This Work Enables

Prospective OPD success prediction from model properties alone. The paper's diagnostic framework is retrospective: overlap ratio reveals failure during training, but cannot prevent it before compute is invested. A natural follow-up is to develop a lightweight predictor of OPD success from properties measurable before any OPD training begins. The paper's own experiments suggest candidate features: the initial overlap ratio computed from a small number of student-generated rollouts (say, 100–500 prompts rather than 2048), the teacher's entropy on student-generated prefixes, and the student-teacher benchmark performance gap. A strong study would collect these features across 30–50 teacher-student pairs spanning different model families (Qwen, LLaMA, DeepSeek), scales (1B–14B), training pipelines (base, SFT, RL-tuned), and domains (math, code, general reasoning), then train a classifier to predict whether OPD will produce meaningful improvement (e.g., gap recovery rate above some threshold) based solely on pre-training measurements. The key advance would be a practical screening tool that tells practitioners "this teacher-student pair is unlikely to benefit from OPD" before they commit to a full training run. The paper's existing data — the gap recovery rates in Figure 4 (5.3% vs. 16.9%, 15.6% vs. 58.6%) and the initial overlap ratios that predict them — provide a proof-of-concept that such prediction is feasible.

Direct verification of the anisotropy hypothesis through gradient directional analysis. Section 6.2 proposes that OPD failure with R1-Distill-7B as teacher results from per-token advantages that are directionally incoherent — individually large but mutually canceling when aggregated into gradient updates. The paper explicitly leaves this unverified. A strong follow-up would directly measure this: for each position within a student-generated sequence, compute the per-token gradient contribution ∇_θ [p_t(v) (log p_t(v) − log q_t(v))] for each token v in the overlap set, then measure the average pairwise cosine similarity between gradient contributions at different positions within the same sequence. If the anisotropy hypothesis is correct, the 7B teacher should show substantially lower pairwise cosine similarity than the 1.5B teacher, indicating that advantages point in mutually canceling directions. A variant would measure the norm of the sequence-aggregated gradient vs. the sum of per-position gradient norms — if cancellation is occurring, the aggregated norm should be much smaller than the sum of per-position norms. If the anisotropy hypothesis is disconfirmed — if the 7B teacher's per-position gradients are as directionally coherent as the 1.5B teacher's, but simply smaller in magnitude — the explanation for failure would shift toward the mass-weighting or variance alternatives the paper mentions. Either outcome advances understanding: verification of anisotropy opens the door to gradient filtering or reweighting methods; disconfirmation forces a search for the true mechanism.

Cross-domain replication of the overlap mechanism in code generation and open-ended tasks. The paper is confined to mathematical reasoning. A critical test of generality is whether the overlap-ratio diagnostic and the two-condition framework transfer to structurally different domains. Code generation (HumanEval, MBPP, LiveCodeBench) is the natural first extension because it shares math's property of having verifiable correctness (execution-based evaluation) but differs in token distribution: code tokens follow syntactic constraints that may produce even higher overlap ratios between models, or may reveal different overlap dynamics (e.g., overlap on keywords vs. variable names). Open-ended generation (dialogue, creative writing, summarization evaluated by LLM-as-judge) is the harder test: the teacher's distribution is inherently more diffuse, so the 97–99% overlap mass finding (Appendix B.1, Figure 18) may not hold, and the concept of "thinking patterns" may need to be operationalized differently. A strong study would replicate the controlled comparison from Section 4.1 in both domains — a successful OPD run vs. a failing run with the same student — and measure whether the progressive overlap alignment signature appears in code but not in open-ended generation, or whether it generalizes across all three domains. Negative results (e.g., overlap ratio does not rise in successful code OPD) would establish boundary conditions on the paper's mechanism and motivate domain-specific diagnostics.

The cold-start × teacher-aligned prompt interaction surface. The paper presents off-policy cold start (Section 5.1) and teacher-aligned prompt selection (Section 5.2) as independent recipes targeting different failure conditions — thinking-pattern mismatch and weak signal quality, respectively. But they are never combined, and neither is tested against a new-knowledge failure mode. A comprehensive follow-up would systematically cross these recipes: (1) cold start alone vs. teacher-aligned prompts alone vs. both combined, tested on a configuration with both failure modes present (e.g., a cross-family teacher-student pair where initial overlap is low and the teacher is from the same training pipeline — or better, a cross-family teacher from a different training pipeline with genuinely new knowledge but mismatched thinking patterns). (2) Cold start tested on a same-pipeline teacher where new knowledge is absent (e.g., R1-Distill-7B → R1-Distill-1.5B with SFT initialization on R1-Distill-7B rollouts): the paper's framework predicts this would not help, or would cause faster regression, because SFT would pull the student toward a distribution it has already internalized. Confirming this negative prediction would establish the boundary condition for cold start and clarify that it only helps when the teacher carries genuinely new knowledge. (3) The entropy collapse tradeoff with teacher-aligned prompts (Figure 10, bottom-right) tested with different mixing ratios of in-distribution to out-of-distribution prompts (0%, 25%, 50%, 75%, 100% teacher-aligned). The paper states that mixing is recommended but provides no empirical guidance on the ratio — a sweep would produce a directly actionable recommendation.

Curriculum and hybrid strategies for the trajectory-length ceiling. Section 6.1 identifies a hard ceiling: reward quality degrades with depth, and training with 15K max response length collapses. The paper proposes but does not test two mitigations: curriculum strategies that progressively extend the supervised horizon, and hybrid approaches combining dense token-level supervision on short segments with sparse outcome-level rewards for longer horizons. A strong follow-up would test both: (1) A curriculum experiment that starts OPD training with max response length 3K for 50 steps, then increases to 7K for 50 steps, then 10K, then 15K, compared against the fixed-15K baseline (which collapses — Figure 12) and a fixed-7K baseline (which succeeds). The hypothesis is that progressively extending the horizon allows the student's policy to stabilize at each length before the teacher's signal at deeper positions degrades, preventing the back-to-front entropy propagation (Figure 13). (2) A hybrid experiment that applies OPD (dense token-level supervision) to positions 0 to 7K, and applies an outcome-reward RL objective (sparse terminal reward) to the full 15K trajectory, compared against pure OPD-15K (collapses), pure RL-15K (baseline), and pure OPD-7K (ceiling). The hypothesis is that OPD provides efficient learning on the reliable early segment while RL handles credit assignment on the problematic suffix. The key measurement would be whether the trajectory-length sweet spot shifts rightward (e.g., from 7K to 10K or 15K) under curriculum or hybrid strategies — indicating that the ceiling is not fundamental to OPD but can be engineered around.

Self-distillation dynamics under the two-condition framework. The paper's Section 8 notes that self-distillation — where a single model serves as its own teacher given privileged information — is an increasingly important setting where thinking-pattern consistency is guaranteed (same model) but knowledge novelty arises from privileged access rather than a separate teacher. The paper does not study self-distillation experimentally. A natural extension would apply the overlap-ratio diagnostic to a self-distillation setup: a strong student model generates rollouts, receives privileged information (e.g., ground-truth solutions, execution feedback, or retrieval-augmented context), and serves as its own teacher by producing token-level distributions conditioned on that privileged information. The key questions: Does the overlap ratio rise during self-distillation even though the student and teacher are the same model (conditioned differently)? Does the trajectory-length ceiling manifest similarly, or does the shared model architecture mitigate the teacher-side exposure bias? Does the new-knowledge condition manifest as the gap between the model's privileged-information distribution and its standard distribution, and does this gap predict self-distillation gains? Extending the diagnostic framework to self-distillation would test whether the paper's findings are specific to cross-model OPD or reflect deeper properties of KL-based distribution matching that apply even within a single model.

Practical Applications and Downstream Use Cases

Teacher selection in industrial OPD pipelines becomes principled rather than trial-and-error. Organizations deploying OPD in production post-training (e.g., Qwen3, MiMo, GLM-5 scale systems) currently select teachers based on benchmark performance — a natural heuristic that this paper shows is actively misleading. A practitioner reading this paper would instead: (1) before committing to OPD, compute the initial overlap ratio between candidate teachers and the student by generating a small number of student rollouts (e.g., 200–500 prompts) and querying teacher top-k distributions at each position. Teachers with initial overlap below ~65–70% (based on the Qwen3-4B Non-thinking → Qwen3-1.7B-Base failure in Figure 2, where initial overlap was ~0.57 and performance was poor) are flagged for off-policy cold start before OPD. (2) Teachers from the same training pipeline as the student (same pretraining data, same SFT recipe, just larger scale) are deprioritized regardless of their benchmark scores — the gap recovery rates of 5.3% (DeepSeek family) and 15.6% (Qwen family) in Figure 4 indicate that same-pipeline scale provides minimal transferable signal. (3) RL-augmented teachers derived from the same base checkpoint are prioritized — the 16.9% and 58.6% gap recovery rates from post-trained teachers (Figure 4) demonstrate that additional RL training creates genuinely transferable capabilities. This selection protocol, informed by the paper's two-condition framework, would prevent the most common OPD failure mode (deploying a large same-family teacher and expecting gains) and direct compute toward configurations with a high probability of success.

Real-time training monitoring with overlap ratio as an early-stopping signal. In current practice, OPD training runs are monitored primarily through downstream validation accuracy — a lagging indicator that may only reveal failure after hundreds of steps and substantial compute expenditure. The overlap ratio provides a leading indicator that can trigger intervention much earlier. A practitioner monitoring an OPD run would track the overlap ratio curve: if it remains flat or fluctuates without a clear upward trend for the first 30–50 steps (as in the failing R1-Distill-7B run, Figure 6 bottom-left, where overlap stagnated at ~0.74), the run can be terminated early, saving the remaining compute. If the overlap ratio shows late-stage instability (as in the 15K max-length run, Figure 12, where overlap dropped from ~0.90 to ~0.66 after step 140), the practitioner can revert to an earlier checkpoint before the collapse propagates. The paper's finding that overlap-token advantage and entropy gap provide complementary signals (Appendix B.2, Figure 19, and Section 5.1, Figure 8 bottom-right) means a dashboard of these three metrics can disambiguate failure modes: stagnant overlap + stable entropy gap → thinking-pattern mismatch; rising overlap + large persistent entropy gap → the student is sharing the teacher's token space but not matching the teacher's confidence profile; rising overlap + narrowing entropy gap + improving advantage → healthy training. This dashboard is implementable immediately in any OPD training loop that already queries teacher top-k distributions — the additional instrumentation cost is negligible relative to the training compute.

SFT cold start as a standard preconditioning step for cross-family OPD. The paper's cold-start experiment (Section 5.1) demonstrates that SFT on teacher-generated rollouts before OPD raises initial overlap ratio (from ~0.52 to ~0.68, Figure 8) and more than doubles final accuracy on AIME 2024 (~0.11 vs. ~0.05 at step 200) for a configuration where pure OPD performs poorly. This recipe is directly actionable for practitioners attempting cross-family distillation — e.g., distilling from a Qwen teacher into a LLaMA student, or from an RL-tuned teacher into a base student from a different family. The procedure has clear steps: (1) generate 100K–200K rollouts from the teacher on a domain-relevant prompt set, (2) perform full-parameter SFT of the student on these rollouts, (3) continue with standard OPD on a separate (deduplicated) prompt set. The paper's hyperparameters (Table 3: learning rate 1e-5, cosine schedule, 1 epoch, sequence length 14,336) and data pipeline (teacher rollout template matching the teacher's training format, filtering incomplete and degenerate responses) provide a concrete starting point. The primary uncertainty — which the paper does not resolve — is the required SFT data quantity: 200K rollouts were used, but no ablation on data volume is provided. A practitioner would want to sweep SFT data quantities (50K, 100K, 200K) to find the minimum that raises overlap sufficiently, since teacher rollout generation at scale is expensive. The practical insight is that cold start transforms OPD from a technique that fails on mismatched teacher-student pairs into one that can be deployed across model families, substantially expanding the set of viable configurations.

Prompt engineering for OPD is not superficial — template alignment matters systematically. The paper's prompt template alignment experiment (Section 5.2, Figure 9) demonstrates that simply switching the prompt format from "Answer: $Answer" to "\boxed{}" (matching the teacher's post-training format) yields consistent accuracy improvements (~0.50 vs. ~0.46 avg@16 at step 200, Figure 9) and higher overlap growth (initial overlap ~0.82 vs. ~0.76, converging to ~0.92 vs. ~0.90). For practitioners deploying OPD, this means that prompt template selection should be treated as a hyperparameter to be optimized, not an implementation detail: before running OPD, test a small number of candidate templates (the teacher's training template, the student's training template, a neutral template) by measuring the initial overlap ratio on a small set of student rollouts, and select the template that maximizes overlap. The prompt content alignment experiment (Figure 10) adds the nuance that using the teacher's exact post-training prompts sharpens alignment on overlap tokens but risks entropy collapse — the practical recipe is to use teacher-aligned prompts for a fraction of the training data (e.g., 50–70%) with the remainder drawn from a broader in-domain set to preserve exploration. This is a low-cost intervention (changing prompt strings costs no FLOPs and minimal engineering effort) with measurable benefit, making it one of the most immediately actionable findings in the paper. The caveat is that the optimal mixing ratio and the entropy collapse threshold are likely domain-specific and require per-deployment tuning.

When to Prefer This Method

The paper does not position OPD against named alternative post-training methods with an explicit tradeoff matrix. It studies OPD's internal failure modes and recipes, but does not compare OPD against outcome-reward RL, SFT, or other distillation approaches under matched conditions. Consequently, a formulaic "Prefer OPD when ... Prefer RL when ..." decision rule would be imposed on the paper rather than extracted from it.

The closest the paper comes to a comparative claim is the trajectory-length ceiling discussion (Section 6.1), which suggests that OPD is most effective on moderate-length reasoning traces (roughly 3K–7K tokens in the paper's experimental setup) and degrades on very long traces (10K+). The implied comparison is with outcome-reward RL, which the paper mentions as suffering from credit assignment difficulty on long traces — but the paper does not experimentally compare OPD and RL at different trajectory lengths, so this remains an inference rather than a demonstrated tradeoff. Similarly, the paper's two-condition framework implies that OPD should be preferred when the teacher carries new knowledge beyond the student's training (Section 3.2) and the thinking patterns are compatible (Section 3.1), but these are conditions on when OPD itself works, not conditions that favor OPD over specific alternatives.

The paper's contribution is a diagnostic framework for OPD, not a comparative evaluation of post-training methods. Including a "when to prefer" matrix would fabricate tradeoffs the paper does not establish.