Training Methods2026~136 minarXiv:2604.00626

A Survey of On-Policy Distillation for Large Language Models

Mingyang Song, Mao Zheng↑ 9Apr 1, 2026cs.LG

🎯 Pitch

Training smaller language models by imitating fixed teacher outputs causes errors to compound quadratically with sequence length. On-policy distillation fixes this by having the teacher correct what the student actually generates, reducing the error growth to linear and reframing the entire process as iterative correction. This survey unifies the scattered design choices—from divergence selection to training stabilization—consolidating practical successes and failure modes into one framework.

1. Executive Summary

This survey synthesizes the rapidly growing literature on On-Policy Distillation (OPD) for large language models, reorganizing the training loop around student-sampled trajectories rather than static teacher-generated corpora to address the exposure bias that causes O(ϵT²) error compounding in off-policy imitation — a weakness that grows severe as reasoning chains lengthen. The paper formalizes OPD within a unified f-divergence minimization framework over student-generated rollouts, then organizes over one hundred methods along three design axes: objective function design (what to optimize — spanning fixed divergences like reverse KL, adaptive divergences like ToDi's per-token log-ratio routing, and RL-augmented objectives like G-OPD's reward extrapolation), signal source architecture (where the signal comes from — white-box logit access, black-box API feedback via GAD's adversarial discriminator or OVD's verbal scores, and self-distillation via OPSD's ground-truth-conditioned privileged information), and training dynamics optimization (how to stabilize training — token-level importance profiling via TIP, competence-boundary curriculum sampling via PACED, and compute reduction via FOPD's prefix truncation). Industrial deployments at DeepSeek-V4, Qwen3, Gemma 2, and MiMo-V2-Flash have adopted OPD as a core training ingredient, with DeepSeek-V4 replacing its mixed RL stage entirely with pure multi-teacher OPD for model consolidation, while on reasoning benchmarks OPD methods deliver 4–8× sample efficiency gains over GRPO and allow students to match teacher performance at roughly one-tenth the GPU hours, establishing that on-policy correction is increasingly attractive for the next generation of reasoning-capable systems only when the student-teacher distributional gap falls within a productive intermediate regime — too small a gap (same-recipe models) leaves the teacher with little transferable signal, while too large a gap (thinking-pattern mismatch between non-thinking teachers and thinking students) prevents convergence entirely.

2. Context and Motivation

The Core Problem: Off-Policy Distillation Has a Structural Weakness That Compounds with Reasoning Length

The central problem this paper addresses is that the dominant recipe for transferring knowledge from large frontier models into smaller deployable students — training the student to match the teacher's output on a fixed, pre-generated corpus — carries a built-in fragility that becomes catastrophic as tasks grow more reasoning-intensive. The paper crystallizes this fragility in terms of a specific mathematical pathology: exposure bias.

The mechanism is straightforward but its consequences are severe. In standard off-policy distillation, the student learns by conditioning on flawless teacher-generated prefixes at every gradient step. These prefixes are drawn from a static dataset — either the original pre-training data or traces that the teacher generated in advance. At deployment, however, the student generates autoregressively from its own partial outputs. The moment the student deviates even slightly from the teacher's distribution, it enters a state it has never been trained on. The teacher's conditional prediction on this student-generated prefix — if the teacher were consulted — would be poorly calibrated, because the teacher was never trained on such out-of-distribution inputs either. The student, without access to corrective feedback on its own error states, continues compounding the deviation.

The paper anchors this in the classical DAgger theorem from interactive imitation learning (Ross et al., 2011), which quantifies the damage: if a policy mimics an expert with per-step error $\epsilon$ under the training distribution, the expected total discrepancy over a trajectory of length $T$ under the learner's own state visitation scales as $O(\epsilon T^2)$ — quadratic in sequence length, not linear. The intuition is that each error pushes the student further from the training manifold, making subsequent errors more likely, creating a vicious cycle where distributional shift amplifies local mistakes into trajectory-level collapse. The paper cites Gudibande et al. (2023)'s finding that off-policy imitation of proprietary LLMs often yields students that "reproduce surface style without acquiring the underlying reasoning competence" — a direct empirical manifestation of this quadratic compounding.

This is not a theoretical curiosity. The paper argues it is the structural reason why distillation pipelines that work well for short-form tasks (translation, summarization, factual QA) systematically underperform on mathematical reasoning, multi-step code generation, and any task where a single early misstep — choosing the wrong proof strategy, the wrong variable name, the wrong API call — can derail the entire output. The quadratic term means the gap between off-policy training and on-policy deployment widens nonlinearly as reasoning chains grow longer, making exposure bias the dominant failure mode for the next generation of reasoning-capable systems.

Why This Problem Matters Now

The paper situates the urgency of addressing exposure bias within a broader technological shift that it argues has made the problem both more visible and more consequential.

Frontier models are becoming unreachable for most practitioners. The paper points to recent systems like DeepSeek-R1 (671B mixture-of-experts), Qwen3 (up to 235B), and Gemma 2 (27B) as evidence that training costs have concentrated in "only a few organizations." Transferring capabilities from these models into smaller, deployable students is no longer an optimization nicety — it is a core step in the modern LLM pipeline. But the transfer mechanism must be reliable for the tasks where frontier models show their greatest value, which increasingly means long-chain reasoning.

Reasoning chains are getting longer. The paper notes that DeepSeek-R1's release "made this shift concrete, distilling a 671B mixture-of-experts teacher into dense students spanning 1.5B to 70B parameters while preserving long chain-of-thought reasoning largely intact." But the distillation recipe used there — while successful — was fully off-policy. The paper's argument is that as reasoning chains grow from dozens to hundreds to thousands of tokens, the $O(\epsilon T^2)$ compounding term moves from a manageable nuisance to a dominant source of error, and the field needs a principled framework for addressing it rather than relying on the brute-force solution of ever-larger and more diverse teacher-generated datasets.

The training-inference distribution gap is widening. The paper identifies a self-reinforcing dynamic: as models become more capable, they explore more diverse solution paths, which makes static datasets less able to cover the space of valid reasoning trajectories, which makes off-policy training less representative of deployment conditions. This is particularly acute for reasoning tasks where the space of valid solution paths is combinatorial — there are many ways to prove a theorem, many ways to structure a program, many ways to decompose a multi-step problem. A fixed dataset of teacher traces, no matter how large, covers only a finite subset of these paths. When the student at inference time generates a path that was not in the training set, off-policy training provides no signal about whether that path is good or bad, correct or erroneous.

Where Prior Approaches Fall Short

The paper identifies four distinct limitations in existing work that together motivate the OPD framework.

1. Off-policy distillation dominates practice but ignores the student's actual behavior. The paper surveys the landscape and finds that "in almost all large-scale pipelines, distillation proceeds off-policy" — the student is trained to match the teacher's next-token distribution over a fixed corpus, typically traces generated in advance by the teacher. This works adequately when the teacher is massively more capable than the student (the DeepSeek-R1 anomaly discussed in Section 7.4 of the full paper, where a 671B teacher's traces are so diverse that they approximate coverage of the student's generation space), but it leaves performance on the table whenever the student's generation distribution diverges from the training corpus. The paper frames this as an information-theoretic ceiling: off-policy training can only teach the student what the teacher happened to generate, not how to recover from the student's own characteristic errors.

2. The distillation literature lacks a unified treatment of on-policy methods. The paper's central contention is that contributions to on-policy distillation have been "scattered across the knowledge distillation, RLHF, and imitation learning communities without a unified treatment." Methods approach the same underlying problem — aligning a student's generation distribution with a teacher's through feedback on student-generated trajectories — but do so through "separate lenses... each carrying different notations, benchmarks, and failure taxonomies from its parent community." A researcher coming from the KD community sees Divergence-Based Matching; one from RLHF sees KL-Constrained Policy Optimization; one from imitation learning sees DAgger. The paper argues that these are not different problems but different formalisms for the same problem, and that the lack of a common vocabulary has slowed progress by preventing cross-community learning.

3. Existing surveys treat off-policy and on-policy methods as interchangeable variants. The paper is explicit about this gap: "Existing surveys of LLM distillation (Xu et al., 2024) generally retain the classical compression framing, treating off-policy and on-policy methods as interchangeable variants instead of as regimes with materially different theoretical guarantees and failure modes." This is the gap the survey is designed to fill. The distinction matters because off-policy and on-policy training have fundamentally different error scaling properties ( $O(\epsilon T^2)$ vs. $O(\epsilon T)$ , respectively), different failure modes (exposure bias vs. flawed prefix traps, self-play saturation), and different computational requirements (pre-computed datasets vs. online student rollouts). Treating them as variants of the same thing obscures when each is appropriate and why.

4. No systematic comparison exists across white-box, black-box, and teacher-free regimes. The paper notes that practitioners must now weigh fundamentally different access models against one another: full logit access when teacher and student are co-located within the same organization, API-only feedback when distilling from proprietary models, and self-distillation when no external teacher is available. Each regime imposes different constraints on what objectives are feasible, what signal density is available, and what failure modes dominate. The paper argues that no current treatment offers a systematic comparison across these instantiations, leaving engineers to make architecture decisions without a principled basis for understanding the tradeoffs.

How This Paper Positions Itself

The paper's self-positioning is explicit and fourfold, structured around the contributions listed in the introduction.

First, it offers a unified mathematical framework. The paper recasts the transition from off-policy to on-policy distillation as a sequential decision-making problem and shows that core OPD algorithms are instances of $f$ -divergence minimization over student-sampled trajectories. This is not merely a taxonomic convenience — it supplies "a common analytical vocabulary for methods previously studied in isolation" and "clarifies their relationships in terms of divergence choice, argument ordering, and sampling mixture." The framework reveals, for example, that GKD's $\lambda$ -mixing (interpolating between student-generated and ground-truth prefixes) and MiniLLM's sequence-level reverse KL (computed via REINFORCE) are not competing paradigms but different points in a continuous design space parameterized by sampling policy ( $\pi_{\text{mix}}$ ), divergence generator ( $f$ ), and argument ordering.

Second, it organizes methods along design axes rather than surface categories. The paper explicitly rejects grouping methods by superficial similarity — "GKD-style methods," "RL-based methods," "self-play methods" — in favor of a three-axis taxonomy: what to optimize (objective function), where the signal comes from (signal source architecture), and how to stabilize training (training dynamics). This organization is motivated by the observation that methods from different communities often make identical design choices under different names, while methods within the same community often make orthogonal choices that interact in non-obvious ways. The taxonomy is designed to expose these interactions and make the design space navigable.

Third, it consolidates the empirical record on failure modes. Rather than treating OPD as uniformly beneficial, the paper devotes substantial attention (Section 7) to when and why it breaks down: the flawed prefix trap (teacher feedback on student-generated errors produces misleading gradients), self-play saturation (the Ouroboros problem where self-distillation collapses onto the model's own prior), diversity collapse (mode-seeking behavior that improves Pass@1 at the expense of Pass@k), and the calibration-capability gap (distilled models become more accurate but less aware of their own uncertainty boundaries). This failure-mode analysis is positioned as a direct counterpoint to the success narratives in the methods literature, and it provides the theoretical grounding for the stabilization techniques surveyed in Section 6.

Fourth, it bridges communities. The paper positions OPD at the intersection of knowledge distillation, reinforcement learning, and imitation learning, and argues that progress in any of these fields is likely to transfer to the others. The equivalence between standard OPD and KL-constrained RL (formalized in G-OPD and discussed in Section 4.3) is presented as a key insight: "advances in KL-constrained RL (better trust regions, adaptive penalty coefficients, variance reduction) are therefore likely to transfer to OPD, and vice versa." Similarly, the connection between DPO-style preference optimization and token-level KL distillation — both targeting geometric mixtures of teacher and reference distributions — suggests that preference-based methods (ORPO-Distill, OVD) and divergence-based methods (GKD, DistiLLM) are endpoints of a continuous spectrum rather than distinct approaches.

The paper's implicit argument is that the field has reached an inflection point. Off-policy distillation, while adequate for the previous generation of models and tasks, is hitting fundamental scaling limits as reasoning chains lengthen and model capabilities diversify. On-policy methods address the root cause — distributional mismatch between training and deployment — but the literature is fragmented, the design space is poorly mapped, and the failure modes are incompletely understood. The survey aims to provide the unified treatment that would allow researchers and practitioners to navigate this space systematically rather than rediscovering insights across community boundaries. The progression from Hinton's classical KD (2015) through Seq-KD (2016) to GKD and MiniLLM (2023–2024) and onward to the industrial deployments of 2025–2026 is framed not as a series of independent innovations but as "a series of relaxations of four classical assumptions: (1) shared vocabulary, (2) i.i.d. data, (3) static teacher, and (4) off-policy training data" — with OPD representing the relaxation of the final and most consequential assumption.

3. Technical Approach

3.1 Reader orientation

This paper is a survey paper, not a methods paper — it does not propose a single new system or algorithm but instead provides a unified framework for understanding and organizing the rapidly growing literature on On-Policy Distillation (OPD) for large language models. The "system" being explained is the collection of design patterns, mathematical formalisms, and training strategies that collectively define the OPD paradigm. The core idea is that OPD addresses a fundamental weakness in standard knowledge distillation — the fact that students trained on fixed, pre-generated teacher outputs accumulate errors quadratically with sequence length when generating their own text at inference time — by reorganizing the training loop so that the student learns from feedback on its own generated trajectories rather than from a static corpus. The survey's contribution is to show that the many disparate methods proposed for OPD (spanning knowledge distillation, reinforcement learning, and imitation learning communities) are not independent heuristics but instances of a small number of recurring design choices that can be organized along three axes: what to optimize, where the supervisory signal comes from, and how to stabilize training.

3.2 Big-picture architecture (diagram in words)

The survey organizes the entire OPD literature around a single conceptual pipeline with three sequential decision points:

The Objective Function (Section 4): This component determines what mathematical quantity the student optimizes at each training step. It takes the student's generated tokens, the teacher's feedback on those tokens, and any external reward signals, and produces a scalar loss or gradient update. The design space spans fixed divergences (Forward KL, Reverse KL, JSD — each with different mode-covering vs. mode-seeking behavior), adaptive divergences (per-token selection based on local distributional geometry, such as ToDi's log-ratio-weighted blending of Forward and Reverse KL), and RL-augmented objectives (combining dense teacher signals with sparse outcome rewards to push the student beyond the teacher's capability ceiling, as in G-OPD's reward extrapolation).
The Signal Source (Section 5): This component determines where the teacher's supervisory signal comes from and what form it takes. The design space spans white-box access (full logit distributions over the entire vocabulary at every token position, enabling exact KL computation), black-box access (API-only text outputs requiring adversarial discriminators like GAD, verbal scoring like OVD, or preference-based methods like ORPO-Distill), and self-distillation (no external teacher — the model exploits asymmetries within itself, such as conditioning on privileged information unavailable at inference time like ground-truth answers in OPSD, or exploiting rollout diversity through temperature sampling as in SSD).
Training Dynamics and Stabilization (Section 6): This component determines how to make the on-policy training loop practical and stable. On-policy training introduces challenges absent from standard supervised fine-tuning: non-stationary data distributions (the student's policy shifts during training, making older rollouts stale), gradient signal-to-noise ratio collapse on hard prompts (where all rollouts fail and no useful gradient emerges), and substantial computational overhead from autoregressive student rollouts followed by teacher scoring. The design space spans token and sample weighting (TIP's entropy-divergence quadrant scoring, SCOPE's correctness-based rollout routing), curriculum and difficulty adaptation (PACED's competence-boundary sampling via Beta-kernel distributions, TCOD's temporal curriculum for multi-turn trajectories), and compute optimization (FOPD's prefix truncation, Lightning-OPD's offline teacher caching, NPD's asynchronous generation-training decoupling).

Information flows through these three stages sequentially: given a prompt, the student generates on-policy rollouts (Dynamics), the teacher provides feedback at some granularity depending on access level (Signal Source), and the student computes and applies a gradient based on a chosen divergence or reward combination (Objective). The central thesis of the survey is that industrial-scale OPD systems (DeepSeek-V4, Qwen3, Gemma 2) increasingly combine all three axes in deliberate, interacting ways rather than optimizing any single axis in isolation.

3.3 Roadmap for the deep dive

First, the unified $f$ -divergence framework (Section 3.4.1), because it is the mathematical foundation that subsumes the diverse objective functions surveyed in Section 4. Understanding this framework — what $f$ -divergences are, how they parameterize the mode-seeking/covering tradeoff, and how the on-policy sampling distribution enters the objective — is prerequisite to understanding why methods like GKD, MiniLLM, and DistiLLM are not competing approaches but different points in a continuous design space.
Second, the formal definition of "on-policy" and the OPD objective (Section 3.4.2), because it establishes the core optimization problem that all surveyed methods are solving. This includes the definition of the mixture policy $\pi_{\text{mix}}$ , the decomposition into trajectory sampling and local matching, and the computational implications of the on-policy requirement (fresh student rollouts at each training step).
Third, the classical KD foundations and their limitations (Section 3.4.3), because OPD is best understood as a series of relaxations of the assumptions underlying Hinton et al. (2015)'s original distillation formulation. Walking through token-level KD, sequence-level KD, and the exposure bias that motivates on-policy training establishes what OPD is solving and why the simpler alternatives fail.
Fourth, the exposure bias analysis (Section 3.4.4), because the $O(\epsilon T^2)$ vs. $O(\epsilon T)$ distinction is the theoretical justification for the entire OPD enterprise. Understanding the DAgger theorem, its application to autoregressive generation, and the practical consequences for reasoning tasks explains why on-policy training is not merely an incremental improvement but a qualitatively different regime.
Fifth, the mapping of foundational methods into the $f$ -divergence framework (Section 3.4.5), because it demonstrates the framework's unifying power by showing how GKD (divergence-agnostic with explicit $\lambda$ -mixing), MiniLLM (sequence-level Reverse KL via REINFORCE), and DistiLLM (skewed KL mixtures for numerical stability) are instances of the same underlying design pattern with different choices for the three key parameters: the divergence generator $f$ , the sampling mixture $\pi_{\text{mix}}$ , and the argument ordering.
Sixth, the design axes and method taxonomy (Section 3.4.6), because this is the organizational backbone of the entire survey. Understanding the three-axis decomposition (Objective → Signal → Dynamics) and how methods are classified into the taxonomy (by their primary contribution dimension, with cross-cutting methods discussed in prose) provides the navigational framework for Sections 4–6.

3.4 Detailed, sentence-based technical breakdown

This is a survey and synthesis paper whose core idea is that the transition from off-policy to on-policy distillation for LLMs can be understood as a unified design space parameterized by three interacting choices: the divergence generator $f$ (which determines the geometric behavior of the optimization), the sampling mixture $\pi_{\text{mix}}$ (which controls the degree of on-policy exploration), and the argument ordering within the chosen divergence (which determines whether the student mode-covers or mode-seeks relative to the teacher). The paper does not propose new methods but rather provides the mathematical vocabulary, organizational taxonomy, and failure-mode analysis that allow researchers and practitioners to navigate this design space systematically.

3.4.1 The Unified $f$ -Divergence Framework

The paper's central mathematical contribution is the observation that the diverse objectives used across OPD methods — Forward KL, Reverse KL, Jensen-Shannon Divergence, Total Variation distance, and their adaptive variants — are all instances of $f$ -divergence minimization over student-sampled trajectories. This is not merely a taxonomic convenience; it reveals that methods previously studied in isolation (GKD's token-level KL, MiniLLM's sequence-level Reverse KL via REINFORCE, DistiLLM's skewed KL mixtures) differ only in their choice of the convex generator function $f$ , the sampling policy $\pi_{\text{mix}}$ , and the argument ordering within the divergence.

An $f$ -divergence between two probability distributions $P$ and $Q$ is defined as:

$D_f(P \parallel Q) = \mathbb{E}_{y \sim Q}\left[ f\left( \frac{P(y)}{Q(y)} \right) \right]$

where $f: (0, \infty) \rightarrow \mathbb{R}$ is a convex function satisfying $f(1) = 0$ (meaning the divergence is zero when $P = Q$ ), $P(y)$ is the probability assigned to outcome $y$ by the first distribution (typically the teacher), and $Q(y)$ is the probability assigned by the second distribution (typically the student).

What it computes: For a given convex generator $f$ , the divergence measures the expected penalty — computed under samples from $Q$ — for each possible outcome $y$ , where the penalty depends on the likelihood ratio $P(y)/Q(y)$ through the function $f$ . When $P(y) = Q(y)$ for all $y$ , the ratio is 1 everywhere, $f(1) = 0$ by construction, and the divergence is zero. When $P(y) > Q(y)$ (teacher assigns higher probability than student), the ratio exceeds 1, and $f$ determines how severely this underestimation is penalized. When $P(y) < Q(y)$ (student assigns higher probability than teacher), the ratio is less than 1, and $f$ determines how severely this overestimation is penalized. The expectation is taken under $Q$ (the student's distribution), which is critical for on-policy optimization because it means the divergence can be estimated from student-generated samples without needing to sample from the teacher.

Why this form: The $f$ -divergence family is the most general class of divergences that satisfy three properties essential for distillation: (1) non-negativity ( $D_f(P \parallel Q) \geq 0$ ), guaranteed by Jensen's inequality and convexity of $f$ ; (2) identity of indiscernibles ( $D_f(P \parallel Q) = 0 \iff P = Q$ ), guaranteed by strict convexity at 1; and (3) the data processing inequality (divergence never increases under transformations), which makes them valid optimization objectives that respect the information geometry of probability distributions. The specific choices of $f$ that matter for OPD are:

Forward KL: $f(u) = u \log u$ . This is the standard choice in classical KD and GKD. It penalizes regions where the student assigns low probability to teacher-favored tokens (since $u = P/Q$ is large when $Q$ is small, and $u \log u$ grows superlinearly). This makes Forward KL mode-covering (zero-avoiding): the student is heavily penalized for assigning near-zero probability to any token the teacher considers plausible. The practical consequence is that Forward KL produces students that maintain output diversity — they spread probability mass across all teacher modes — but risk hallucination by placing probability in regions between teacher modes that correspond to neither correct nor plausible outputs.
Reverse KL: $f(u) = -\log u$ . This penalizes regions where the student assigns high probability to tokens the teacher considers unlikely (since $u = P/Q$ is small when $Q$ is large relative to $P$ , and $-\log u$ grows as $u \to 0$ ). This makes Reverse KL mode-seeking (zero-forcing): the student collapses its probability mass onto the teacher's highest-probability mode, ignoring secondary modes entirely. The practical consequence is that Reverse KL produces students with high precision (they rarely generate implausible tokens) but low recall (they may miss valid alternative solutions entirely, causing the diversity collapse documented in Section 7.2).
Jensen-Shannon Divergence (JSD): $f(u) = u \log u - (u+1)\log\frac{u+1}{2}$ . This is a symmetric, bounded divergence that interpolates between mode-covering and mode-seeking behavior. Because the generator function includes both $u \log u$ and a smoothing term $-(u+1)\log\frac{u+1}{2}$ , JSD applies moderate penalties to both underestimation and overestimation, making it a compromise suitable for tasks where the output space has moderate diversity — neither a single correct answer (where mode-seeking would be ideal) nor fully open-ended generation (where mode-covering would be ideal). GKD's experiments found JSD performed best on translation tasks for precisely this reason.
$\alpha$ -divergence: A parameterized family with generator $f_\alpha(u)$ that continuously interpolates between Forward KL (as $\alpha \to 1$ ) and Reverse KL (as $\alpha \to 0$ ). The parameter $\alpha$ provides a continuous dial for controlling the mode-seeking/covering tradeoff, though in practice the discrete choices (Forward KL, Reverse KL, JSD) have received the most empirical attention.

The critical property that makes $f$ -divergences amenable to on-policy optimization is that the expectation is taken under $Q$ (the student's distribution). This means the divergence can be estimated without importance sampling or off-policy corrections: the student generates samples from its own policy, and the divergence is computed directly on those samples using the teacher's probabilities. For Forward KL, $D_{KL}(P_T \parallel P_\theta) = \mathbb{E}_{y \sim P_\theta}[p_T(y)/p_\theta(y) \log(p_T(y)/p_\theta(y))]$ , which requires both the teacher's probability $p_T(y)$ and the student's probability $p_\theta(y)$ at each sampled token — available in white-box settings. For Reverse KL, $D_{KL}(P_\theta \parallel P_T) = \mathbb{E}_{y \sim P_\theta}[\log(p_\theta(y)/p_T(y))]$ , which is even simpler: it is just the expected log-ratio under student samples.

3.4.2 Formal Definition of On-Policy Distillation

The survey provides a crisp formal definition of what qualifies a method as "on-policy" in Section 2:

A distillation method is on-policy if the training data for the student is sampled from the student's own current policy $p_\theta$ at training time, rather than from a fixed external corpus $\mathcal{D}$ or from the teacher's generation distribution $p_T$ .

Formally, on-policy training optimizes:

$\min_\theta \mathbb{E}_{x \sim \mathcal{D}} \mathbb{E}_{y \sim p_\theta(\cdot|x)} \left[ \mathcal{L}(y, x; \theta, T) \right]$

where $x \sim \mathcal{D}$ denotes prompts drawn from the training dataset $\mathcal{D}$ , $y \sim p_\theta(\cdot|x)$ denotes sequences generated autoregressively by the student's current policy $p_\theta$ conditioned on prompt $x$ , $\mathcal{L}$ is the distillation loss (which may be a divergence, a reward, or a hybrid), and $\theta$ are the student's trainable parameters.

What it computes: This objective specifies that at each training step, the student first generates a batch of complete sequences from its own current policy (requiring a full autoregressive forward pass), then computes a loss on those sequences using teacher feedback (requiring a teacher forward pass in white-box settings or an API call in black-box settings), and finally updates its parameters $\theta$ via gradient descent. The key distinction from off-policy distillation is that the outer expectation is over $p_\theta$ rather than over a static dataset $\mathcal{D}$ of pre-generated teacher traces. This makes the optimization landscape non-stationary: as $\theta$ updates, the data distribution $p_\theta$ shifts, so the loss function being optimized at step $t+1$ is evaluated on different data than at step $t$ . Fresh rollouts are required at each step because old rollouts were generated by a previous policy $\theta_{\text{old}} \neq \theta_{\text{current}}$ and would provide biased gradient estimates.

Why this form: The on-policy objective directly targets the distributional mismatch that causes exposure bias. Under off-policy training, the student is optimized to mimic the teacher on teacher-generated prefixes drawn from $p_{\text{data}}$ , but at inference time the student sees its own prefixes drawn from $p_\theta$ . The divergence between $p_{\text{data}}$ and $p_\theta$ — which grows with sequence length due to error compounding — means the training loss is a poor proxy for inference-time performance. By replacing $p_{\text{data}}$ with $p_\theta$ in the training objective, OPD ensures that the student is evaluated (and receives corrective feedback) on exactly the states it will visit at deployment. The cost is the computational overhead of autoregressive student generation at each training step, which the paper acknowledges as "a central systems-level challenge of OPD" (Section 6.3).

The paper further generalizes this objective by decoupling the trajectory sampling distribution from the local matching metric through a mixture policy $\pi_{\text{mix}}$ :

$\mathcal{L}_{\text{OPD}}(\theta) = \mathbb{E}_{y \sim \pi_{\text{mix}}} \left[ \sum_{t=1}^{|y|} D_f \left( p_T(\cdot|x, y_{<t}), p_\theta(\cdot|x, y_{<t}) \right) \right]$

where $\pi_{\text{mix}}$ is a mixture policy that interpolates between student-generated sequences and ground-truth (or teacher-generated) sequences, $D_f$ is any $f$ -divergence, and the sum runs over all token positions $t$ in the sequence $y$ .

What it computes: For each prompt $x$ , a sequence $y$ is sampled from the mixture policy $\pi_{\text{mix}}$ (which may be purely the student's policy $p_\theta$ , purely the data distribution $p_{\text{data}}$ , or an interpolation). At each token position $t$ , the divergence $D_f$ between the teacher's next-token distribution $p_T(\cdot|x, y_{<t})$ and the student's next-token distribution $p_\theta(\cdot|x, y_{<t})$ is computed. The sum over $t$ aggregates these per-token divergences into a sequence-level loss. The expectation over $\pi_{\text{mix}}$ ensures that the loss is computed over the right distribution of prefixes — the mixture coefficient controls how much the student is exposed to its own (potentially erroneous) prefixes versus clean ground-truth prefixes.

Why this form: Separating $\pi_{\text{mix}}$ from $D_f$ is the key design insight that unifies the OPD literature. Methods differ primarily in their choice of $\pi_{\text{mix}}$ (GKD: $\pi_{\text{mix}} = \lambda p_\theta + (1-\lambda)p_{\text{data}}$ ; MiniLLM: $\pi_{\text{mix}} = (1-\alpha)p_\theta + \alpha p_T$ with $\alpha = 0.2$ ; pure on-policy: $\pi_{\text{mix}} = p_\theta$ ) and their choice of $f$ (Forward KL, Reverse KL, JSD, adaptive). The mixture coefficient serves as a smooth dial between the stability of off-policy training (using ground-truth prefixes that the teacher handles reliably) and the distributional alignment of on-policy training (using student prefixes that reflect deployment conditions). Moderate values ( $\lambda \approx 0.5$ ) tend to combine the benefits, and the paper reports that "on-policy sampling ( $\lambda = 1$ ) outperforms off-policy across the divergence choices they tested" in GKD's experiments.

3.4.3 Classical Knowledge Distillation and Its Limitations

Before developing the OPD framework, the paper establishes the baseline that OPD improves upon: the classical knowledge distillation formulation from Hinton et al. (2015), extended to autoregressive language models by Kim & Rush (2016).

Classical KD (Hinton et al., 2015) trains the student to match the teacher's temperature-softened output distribution:

$p_T^{(\tau)}(y|x) = \frac{\exp(z_y / \tau)}{\sum_j \exp(z_j / \tau)}$

where $z_y$ is the teacher's logit (pre-softmax activation) for token $y$ , $\tau > 1$ is a temperature parameter that controls the softness of the distribution, and the denominator normalizes over all $|V|$ tokens in the vocabulary.

What it computes: Given the teacher's raw logits $z$ for a given context, the temperature $\tau$ divides all logits before the softmax. When $\tau = 1$ , this is the standard softmax. When $\tau > 1$ , the distribution flattens — tokens with lower logits receive relatively higher probability, exposing the teacher's "dark knowledge" about inter-class similarities that hard labels (the single correct token) hide. When $\tau \to \infty$ , the distribution approaches uniform.

Why this form: The temperature parameter controls how much of the teacher's uncertainty structure is transferred. At $\tau = 1$ , the student mainly learns the teacher's top prediction. At high $\tau$ , the student learns the relative ordering of even low-probability tokens, which captures similarity structure — for example, that "delighted" and "happy" are both more appropriate continuations than "elephant" even though "happy" is the teacher's top choice. The gradient of the distillation loss with respect to the student's logits reveals the mechanism:

$\frac{\partial \mathcal{L}_{\text{KD}}}{\partial z_i^S} = \frac{1}{\tau} (p_i^S - p_i^T)$

In the high-temperature limit ( $\tau \to \infty$ ), a Taylor expansion $\exp(z_i/\tau) \approx 1 + z_i/\tau$ together with the zero-mean assumption $\sum_i z_i^S = \sum_i z_i^T = 0$ reduces this gradient to $\frac{1}{|V|\tau^2} (z_i^S - z_i^T)$ . This means classical KD in the high- $\tau$ regime is equivalent to mean squared error matching between student and teacher logits — it encourages the student to replicate the teacher's entire logit structure, not just its mode. The $1/\tau^2$ factor explains why the soft-target term is commonly upweighted by $\tau^2$ relative to the hard-label cross-entropy in the combined KD objective.

The paper notes that in practice, LLM distillation operates at $\tau = 1$ or low-to-moderate temperatures because the large vocabulary ( $|V| > 30,000$ ) already produces richly structured non-peak probabilities without explicit softening, and higher temperatures may amplify noise in the teacher's poorly calibrated tail distribution.

Token-level KD for autoregressive LLMs factors the distillation loss across token positions:

$\mathcal{L}_{\text{Token-KD}} = \mathbb{E}_{x, y \sim p_{\text{data}}} \left[ \sum_{t=1}^{|y|} D_{KL}\left( p_T(\cdot|x, y_{<t}) \parallel p_\theta(\cdot|x, y_{<t}) \right) \right]$

where $D_{KL}$ is the forward KL divergence, $p_T(\cdot|x, y_{<t})$ is the teacher's next-token distribution conditioned on the ground-truth prefix $y_{<t}$ , and $p_\theta(\cdot|x, y_{<t})$ is the student's next-token distribution conditioned on the same prefix.

What it computes: For each prompt $x$ and ground-truth completion $y$ from the dataset, the student is trained to match the teacher's token-by-token predictions at every position $t$ , assuming the prefix $y_{<t}$ is correct (since it comes from the dataset). The loss is a sum of per-token KL divergences, each comparing two categorical distributions over the vocabulary.

Why this form is limited: The critical assumption is that the prefix $y_{<t}$ is ground-truth — it comes from the dataset, not from the student's own generation. At inference time, the student generates autoregressively from its own partial outputs $\hat{y}_{<t} \sim p_\theta(\cdot|x)$ . If the student's distribution diverges from the ground-truth distribution, the prefixes it conditions on at inference time are different from the prefixes it was trained on. This distributional mismatch — training on clean prefixes, deploying on self-generated prefixes — is the exposure bias that OPD is designed to address.

Sequence-Level KD (Seq-KD, Kim & Rush, 2016) extends the KL divergence to the full sequence level:

$\mathcal{L}_{\text{Seq-KD}} = D_{KL}(P_T(y|x) \parallel P_\theta(y|x)) = \sum_{y \in \mathcal{Y}} P_T(y|x) \log \frac{P_T(y|x)}{P_\theta(y|x)}$

where $P_T(y|x)$ is the teacher's sequence-level probability (product of per-token conditionals) and $\mathcal{Y} = |V|^T$ is the exponentially large space of all possible sequences of length $T$ .

What it computes: The full KL divergence over the sequence space measures how the student's distribution over entire completions differs from the teacher's. Unlike token-level KD, which assumes a fixed prefix at each step, sequence-level KD considers the joint distribution over all tokens — it penalizes the student for assigning probability to sequences the teacher considers unlikely, even if each individual token is plausible in isolation.

Why this form is limited: The sum over $\mathcal{Y}$ is intractable for any practical sequence length (exponential in $T$ ). Seq-KD approximates it by collapsing the teacher's distribution to a Dirac delta at the beam-search output — the single most likely sequence under the teacher:

$\mathcal{L}_{\text{Seq-KD}} \approx -\log P_\theta(\hat{y}|x)$

where $\hat{y} = \arg\max_y P_T(y|x)$ is the teacher's beam-search output. This approximation reduces sequence-level distillation to standard negative log-likelihood training on teacher-generated sequences. The paper notes two practical consequences: Seq-KD discards the teacher's distributional richness (collapsing to a point estimate), and it remains off-policy (the student trains on static teacher traces rather than its own generations). These two limitations — loss of distributional signal and off-policy training — are precisely what the richer objectives of OPD address.

The historical progression from Hinton's classical KD to modern OPD is framed by the paper as "a series of relaxations of four classical assumptions: (1) shared vocabulary, (2) i.i.d. data, (3) static teacher, and (4) off-policy training data." Seq-KD relaxes assumption (2) by operating at the sequence level but retains the static data assumption. GKD relaxes assumption (4) by training on student-generated sequences. DSKD relaxes assumption (1) through dual-space alignment for cross-vocabulary distillation. G-OPD relaxes assumptions (2)–(4) simultaneously while adding RL-augmented reward extrapolation. Each relaxation targets a specific failure mode that intensifies with model scale: vocabulary mismatch is irrelevant for same-family distillation but severe for cross-family transfer (relaxation 1), while exposure bias is negligible for short sequences but highly problematic for multi-step reasoning (relaxation 4).

3.4.4 Off-Policy Exposure Bias and the DAgger Theorem

The paper formalizes why off-policy training breaks down for long sequences by connecting it to the classical DAgger theorem from interactive imitation learning (Ross et al., 2011).

Standard knowledge distillation minimizes the KL divergence over states drawn from the dataset distribution $d_{\mathcal{D}}(s)$ :

$\mathcal{L}_{\text{Off-Policy}} = \mathbb{E}_{x, y \sim p_{\text{data}}} \left[ \sum_{t=1}^{|y|} D_{KL}\left( p_T(\cdot|x, y_{<t}) \parallel p_\theta(\cdot|x, y_{<t}) \right) \right]$

What it computes: The expectation is over prompts $x$ and ground-truth completions $y$ from the static dataset. At each position $t$ , the student is trained to match the teacher on the ground-truth prefix $y_{<t}$ . The loss measures how well the student mimics the teacher when conditioned on correct prefixes.

Why this is the problem: At inference time, the student acts according to its own policy $p_\theta$ , inducing a different state visitation distribution $d_{\pi_\theta}(s)$ . The DAgger theorem quantifies the damage: if a policy mimics an expert with per-step error $\epsilon$ under the training distribution $d_{\mathcal{D}}$ , the expected total discrepancy over a trajectory of length $T$ under the learner's own state visitation $d_{\pi_\theta}$ scales as $O(\epsilon T^2)$ — quadratic in sequence length, not linear. The intuition is that each mistake pushes the learner into states increasingly far from the training distribution, where its per-step error rate degrades further because it has never seen these states before. This creates a vicious cycle: error $\to$ distributional shift $\to$ higher error rate $\to$ more shift.

The paper illustrates the practical severity: "Even under the simplest independent-error assumption, a mathematical proof requiring 10 reasoning steps with per-step accuracy 95% yields only $(0.95)^{10} \approx 60\%$ trajectory-level correctness. The DAgger bound predicts something strictly worse." The quadratic term $O(\epsilon T^2)$ means that for $T=10$ reasoning steps, the expected total error is roughly 100 times the per-step error, not 10 times. For $T=100$ (common in chain-of-thought reasoning), the compounding factor is $10,000\times$ — making even small per-step errors catastrophic at the trajectory level.

The DAgger solution is to query the expert on the learner's own visited states rather than on states from a fixed dataset. This replaces the training distribution $d_{\mathcal{D}}$ with the learner's state visitation $d_{\pi_\theta}$ , reducing the compounding from $O(\epsilon T^2)$ to $O(\epsilon T)$ . The mechanism is straightforward: by training on the student's own error states, the student learns recovery strategies that prevent distributional drift from amplifying local mistakes into trajectory-level collapse. Translated to autoregressive language modeling, this means training the student on its own generated prefixes rather than on ground-truth prefixes — which is precisely the definition of on-policy distillation.

An important qualification noted in the paper: the DAgger theorem assumes an interactive expert that can emit optimal actions in any state, including states far from the training distribution. In white-box OPD, the "expert" is the teacher, which emits a next-token distribution $p_T(y_t|\hat{y}_{<t})$ conditioned on the student's prefix $\hat{y}_{<t}$ . If the student hallucinates a severely out-of-distribution prefix — one that the teacher was rarely trained on — the teacher's conditional distribution may itself become poorly calibrated. Forcing the student to match this noisy distribution violates the core DAgger assumption and can destabilize training rather than recovering the $O(\epsilon T)$ bound. The paper cites empirical evidence (Jeong, 2026) that "naive OPD without stable teacher dynamics suffers catastrophic instability," including a case where "KL divergence dropping from 2.637 to 0.343 at a single teacher-reset event." This motivates the adaptive trust mechanisms surveyed in Section 6.1 and the stable teacher dynamics discussed in Section 7.2.

3.4.5 Mapping Foundational Methods into the $f$ -Divergence Framework

To demonstrate the unifying power of the $f$ -divergence framework, the paper maps three foundational OPD methods into the design space parameterized by divergence generator $f$ , sampling mixture $\pi_{\text{mix}}$ , and argument ordering.

GKD (Agarwal et al., 2024) — the canonical OPD framework — defines $\pi_{\text{mix}} = \lambda p_\theta + (1-\lambda)p_{\text{data}}$ as an explicit linear interpolation between student-generated sequences and ground-truth sequences. The method is divergence-agnostic: it empirically tests Forward KL, Reverse KL, and JSD, finding that all three outperform off-policy baselines when $\lambda > 0$ . Setting $\lambda \to 1$ makes GKD purely on-policy (training only on student-generated prefixes), while $\lambda = 0$ reduces to standard off-policy KD.

Design choice analysis: The $\lambda$ parameter provides a smooth dial between the extremes. At $\lambda = 0$ , the student conditions on flawless ground-truth prefixes — stable gradients but maximum exposure bias at deployment. At $\lambda = 1$ , the student conditions on its own (potentially erroneous) prefixes — distributionally aligned with deployment but exposed to the flawed prefix trap (Section 7.2) where teacher feedback on bad prefixes is noisy. Moderate values ( $\lambda \approx 0.5$ ) combine the benefits: the student gets some exposure to its own error states (reducing exposure bias) while maintaining some conditioning on clean prefixes (stabilizing gradients). The paper reports that "on-policy sampling ( $\lambda = 1$ ) outperforms off-policy across the divergence choices they tested," and that "JSD performs best on translation tasks (e.g., WMT), where the output space has moderate diversity." The finding that all three divergences yield competitive results on summarization and instruction-following supports the paper's broader claim that "the sampling policy (on-policy vs. off-policy) matters more than the specific divergence choice when task geometry does not strongly favor one extreme."

MiniLLM (Gu et al., 2024) selects Reverse KL as $D_f = D_{KL}(p_\theta \parallel p_T)$ — placing the student in the first argument position — and employs a mixture policy $\pi_{\text{mix}} = (1-\alpha)p_\theta + \alpha p_T$ with $\alpha = 0.2$ . Because Reverse KL places the student in both the sampling expectation and the log-ratio, MiniLLM cannot directly backpropagate through the sampling operation and must reformulate optimization via REINFORCE, treating $\log(p_T / p_\theta)$ as a per-step reward:

$\nabla_\theta \mathcal{L}_{\text{MiniLLM}} = -\mathbb{E}_{y \sim p_\theta} \left[ \sum_{t=1}^{|y|} (R_t - 1) \nabla_\theta \log p_\theta(y_t|y_{<t}) \right], \quad R_t = \sum_{t'=t}^{|y|} \log \frac{p_T(y_{t'}|y_{<t'})}{p_\theta(y_{t'}|y_{<t'})}$

where $R_t$ is the cumulative future return from position $t$ (the sum of per-token log-ratios from $t$ to the end of the sequence), and the $-1$ term arises from the $-\log p_\theta(y|x)$ entropy component of the reverse-KL objective.

What it computes: At each position $t$ , the gradient is a policy gradient update: the score function $\nabla_\theta \log p_\theta(y_t|y_{<t})$ is multiplied by the advantage $(R_t - 1)$ . Positive advantage (student's future trajectory is more probable under the teacher than under the student) increases the probability of $y_t$ . Negative advantage (student's future trajectory is less probable under the teacher) decreases it. The $-1$ baseline comes from the entropy regularization implicit in Reverse KL and reduces variance by centering the advantages.

Why this form: MiniLLM's commitment to Reverse KL reflects a deliberate mode-seeking design choice. By placing $p_\theta$ in the first argument of the KL, the student is penalized for assigning high probability to sequences the teacher assigns low probability to — it forces the student to concentrate on the teacher's highest-probability modes. This is appropriate for reasoning tasks where multiple solution paths may exist but the teacher has a clear preference for a particular proof strategy. The REINFORCE formulation, while introducing high variance (typical of policy gradient methods in combinatorial action spaces), allows MiniLLM to operate at the sequence level without the exponential complexity of enumerating all sequences. The paper notes that "this connection to policy optimization is consistent with MiniLLM's empirical strength on reasoning tasks where sequence-level coherence matters more than token-level accuracy, at the cost of higher training compute because REINFORCE estimation in combinatorial output spaces requires more iterations and careful baseline subtraction to converge."

DistiLLM (Ko et al., 2024) avoids both the divergence selection problem and the variance problem by engineering a skewed KL mixture that is numerically stable by construction. Its core contribution is a skewed KL loss using a mixture distribution $\tilde{p} = \alpha p_T + (1-\alpha)p_\theta$ :

$\mathcal{L}_{\text{SKL}} = \mathbb{E}_{(x,y) \sim \mathcal{D}_{\text{mix}}} \left[ \sum_{t=1}^{|y|} D_{KL}\left( p_T(\cdot|y_{<t}) \parallel \alpha p_T(\cdot|y_{<t}) + (1-\alpha)p_\theta(\cdot|y_{<t}) \right) \right]$

where $\mathcal{D}_{\text{mix}}$ mixes ground-truth sequences with cached student rollouts according to an adaptive off-policy scheduler, and $\alpha \in (0, 1]$ is the skew parameter.

What it computes: Instead of computing the KL between $p_T$ and $p_\theta$ directly — which produces unbounded gradients when $p_\theta(y) \approx 0$ for some token $y$ — DistiLLM replaces the student's distribution $p_\theta$ with a skewed mixture $\tilde{p}$ that interpolates between the teacher and the student. Because $\tilde{p}(y) \geq (1-\alpha)p_\theta(y)$ , the mixture is bounded below by a fraction of the student's own distribution, preventing the $-\log p_\theta$ term from exploding when the student assigns near-zero probability to a teacher-favored token.

Why this form: The skewed mixture is an engineering solution to a numerical problem. In pure Forward KL, $D_{KL}(p_T \parallel p_\theta) = \sum_y p_T(y) \log(p_T(y)/p_\theta(y))$ , the term $-\log p_\theta(y)$ diverges as $p_\theta(y) \to 0$ . During on-policy training, the student explores widely and may assign extremely low probability to tokens the teacher considers essential — producing gradient spikes that destabilize optimization. By replacing $p_\theta$ with $\tilde{p}$ in the denominator of the log-ratio, DistiLLM caps the effective gradient magnitude while preserving the direction of the update. The paper further proposes Skewed Reverse KL (SRKL) for mode-seeking applications, creating a symmetric pair of stabilized divergences. Its successor, DistiLLM-2 (Ko et al., 2025), extends this by applying Forward SKL on teacher-generated data and Reverse SRKL on student-generated data — an asymmetric, source-aware design that recognizes that "teacher-generated and student-generated data carry qualitatively different learning signals."

The paper's comparative synthesis of these three methods identifies a "core tradeoff triangle in OPD design": GKD prioritizes generality and simplicity (divergence-agnostic, no REINFORCE variance) at the cost of limited theoretical guidance on divergence selection; MiniLLM commits to Reverse KL for its mode-seeking precision but inherits REINFORCE's high variance, requiring reward baselines, length penalties, and careful $\alpha$ tuning; DistiLLM avoids both the divergence selection problem and the variance problem through a numerically stable mixture target, but at the cost of adding a replay buffer and an adaptive scheduler that increase hyperparameter complexity. The progression "from GKD through MiniLLM to DistiLLM thus traces a shift from algorithmic simplicity toward computational and engineering sophistication, with later methods often addressing specific limitations of earlier approaches."

3.4.6 The Three-Axis Design Taxonomy

The paper's primary organizational contribution is a three-axis taxonomy that classifies all OPD methods according to their core design decisions. The axes correspond to sequential design decisions in the training pipeline:

Axis 1: Objective Function Design (Section 4). This axis determines what mathematical quantity the student optimizes at each training step. The design choices subdivide into three generations:

Fixed divergences (Section 4.1): Apply a single $f$ -divergence uniformly across all tokens and all training steps. Methods include GKD (divergence-agnostic, Forward KL/Reverse KL/JSD), MiniLLM (sequence-level Reverse KL via REINFORCE), DistiLLM (skewed KL mixtures for numerical stability), KETCHUP (K-step Bellman returns for variance reduction), vOPD (control variate baseline for single-sample OPD), and AntiSD (divergence ascent for deliberate token boosting). The key limitation is that a single divergence is applied to all token positions, but "the teacher's distributional structure... varies considerably within a single sequence" — at a mathematical operator token, mode-seeking is appropriate (the student must commit to the correct symbol); at a filler word, mode-covering is appropriate (many synonyms are equally valid).
Adaptive divergences (Section 4.2): Select or interpolate divergences per-token based on local distributional geometry. Methods include ToDi (per-token, per-vocabulary-entry blending of Forward KL and Reverse KL via sigmoid weights on the teacher-student log-ratio), AKL (head/tail vocabulary partition with gap-weighted blending), EOPD (entropy-gated Forward KL activation on high-uncertainty tokens), and AOPD (asymmetric policy gradient/Forward KL switching based on advantage sign). The paper's key insight is that "the appropriate divergence is position-dependent" — and even within a single position, different vocabulary entries may need different treatment (ToDi's per-entry weighting is the finest-grained instantiation of this principle).
RL-augmented objectives (Section 4.3): Inject external performance signals that can steer the student beyond the teacher's capability ceiling. Methods include G-OPD (formal equivalence between OPD and dense KL-constrained RL, with reward extrapolation pushing the student beyond the teacher's frontier), RLKD (Generative Structure Reward Model for step-level matching), KDRL and RLAD (joint KD+RL optimization with KL regularization and PPO-style trust regions), and preference-based variants (AlignDistil, OVD, PBSD) that map the KD+RL problem onto Direct Preference Optimization.

Axis 2: Signal Source and Teacher Architecture (Section 5). This axis determines where the supervisory signal comes from and what form it takes. The paper identifies "a progression from full access through limited access to no external access... representing increasing autonomy at the cost of signal density":

White-box logit supervision (Section 5.1): Full access to the teacher's output distribution $p_T(\cdot|x, y_{<t})$ over the entire vocabulary at every token position. This enables exact KL divergence computation and is assumed by most methods in Section 4. The sub-axis divides into same-family distillation (shared tokenizer, straightforward KL computation) and cross-family distillation (different tokenizers, requiring dual-space projection as in DSKD, optimal transport as in ULD, or multi-token continuation matching as in SimCT).
Black-box and API-constrained distillation (Section 5.2): Only API access is available — the student observes the teacher's generated text or at best top- $k$ log-probabilities. This rules out token-level divergence matching and forces methods to operate at the sequence level. Methods include GAD (adversarial distribution matching via a discriminator), Lion (verbal feedback curriculum through a three-stage adversarial loop), OVD (verbal score distillation with discrete scores 0–9), DAIL (didactic-to-constructive learning with mixed policy decoding), and preference-based methods (ORPO-Distill, ThinkTuning).
Self-distillation (Section 5.3): No external teacher at all. The model exploits asymmetries within itself to construct a training signal. Three sub-families are distinguished by the source of internal asymmetry: (i) privileged information (training-time access to context unavailable at inference — OPSD conditions on ground-truth answers, GATES conditions on source documents, CRISP conditions on a "be concise" prompt); (ii) pure self-distillation (rollout diversity through temperature sampling — SSD selects high-quality self-generated completions, UniSD unifies multi-teacher agreement, EMA stabilization, and contrastive learning); (iii) external feedback (connecting the model to verifiers, environments, or multi-agent co-evolution — SD-ZERO uses a self-reviser conditioned on binary correctness, SDPO uses structured textual feedback from unit tests, $\pi$ -Play co-evolves examiner, teacher, and student agents).

Axis 3: Training Efficiency and Stabilization (Section 6). This axis determines how to make the on-policy training loop practical. The paper identifies "optimization challenges largely absent from standard supervised fine-tuning, including non-stationary data distributions... gradient signal-to-noise ratio (SNR) collapse on hard prompts... and the computational overhead of autoregressive student rollouts followed by teacher scoring":

Token and sample weighting (Section 6.1): Not all teacher supervision is equally reliable in the on-policy setting. The flawed prefix trap — where a student's early error makes all subsequent teacher predictions noisy — motivates methods that filter or weight supervision based on reliability. Methods include TIP (Token Importance Profiling via entropy $\times$ divergence quadrant scoring), SCOPE (rollout routing by correctness), SelecTKD (propose-and-verify token acceptance), AdaSwitch (dynamic exploration/guidance switching based on divergence drift), EGRSD (entropy-guided confidence gating for self-distillation), SOD (step-level divergence reweighting for tool-integrated reasoning), and MOPD (peer-conditioned multi-rollout weighting).
Curriculum and difficulty adaptation (Section 6.2): Rather than filtering after generation, curriculum design actively selects prompts matching the student's competence level. Methods include PACED (Beta-kernel curriculum centered on the student's current pass rate, with a symmetric default $w(p) = p(1-p)$ that concentrates training on the productive learning frontier), TCOD (temporal curriculum for multi-turn trajectories — Forward-to-Backward starts supervision at early turns and extends to later ones, Backward-to-Forward uses the teacher for early turns and supervises only later ones), Uni-OPD (dual-perspective curriculum combining difficulty-aware data balancing with outcome-guided margin calibration), and SSB (Semantic Soft Bootstrapping via in-context self-teaching).
Compute optimization (Section 6.3): Systems-level approaches that reduce the substantial overhead of on-policy generation. Methods include FOPD (prefix truncation — the useful distillation signal is concentrated in the prefix, and truncating to length $k$ matches full OPD quality while reducing FLOPs by 2–47 $\times$ ), Lightning-OPD (offline teacher caching — precomputing teacher log-probabilities once over SFT rollouts achieves 4.0 $\times$ training efficiency under a teacher consistency condition), SKD (speculative knowledge distillation — interleaved student sampling and teacher verification), NPD (asynchronous generation-training decoupling with $\Delta$ -IFD filtering for 8.1 $\times$ throughput speedup), and Prune-OPD (drift-aware dynamic truncation that monitors per-position top- $k$ overlap and truncates when cumulative drift exceeds a budget, reducing training time by 37–68%).

Cross-cutting interactions. The paper emphasizes that "the three stages interact in both constraining and reinforcing ways." Some combinations are incompatible: Forward KL in its exact token-level form requires the teacher's full output distribution, ruling out API-constrained settings. Other combinations reinforce each other: RL-augmented objectives naturally couple with external feedback sources (verifiers, reward models), while fixed divergences align with white-box logit access that permits exact gradient computation. The paper's classification methodology assigns each method to exactly one primary category based on its most distinctive contribution, with methods contributing to multiple dimensions discussed in prose within the relevant sections.

Chronological distribution. The paper observes a temporal pattern in research emphasis: early work (2023–2024) concentrated on the objective axis, debating the relative merits of Forward KL, Reverse KL, and JSD; by mid-2025 the focus shifted to signal architecture, particularly self-distillation methods that eliminate external teacher dependence; the most recent work (late 2025–2026) addresses training dynamics, specifically the instabilities that on-policy sampling introduces but static-dataset distillation avoids. Industrial systems (DeepSeek-V4, Qwen3) increasingly combine all three axes, "pairing adaptive or RL-augmented objectives with multi-teacher signals and curriculum-based stabilization."

4. Key Insights and Innovations

Innovation 1: OPD Is Not a Method But a Unified Design Space — The f-Divergence Framework as a Common Analytical Vocabulary

The most fundamental intellectual contribution of this survey is not the identification of any single algorithm but the reconceptualization of on-policy distillation as a unified design space parameterized by a small number of interacting choices. Before this paper, methods from the knowledge distillation community (GKD, DistiLLM), the RLHF community (G-OPD, RLKD), and the imitation learning community (DAgger-style approaches) were studied in isolation, "each carrying different notations, benchmarks, and failure taxonomies from its parent community." The survey demonstrates that these are not competing paradigms but different points in a continuous space defined by three choices: the $f$ -divergence generator (which determines mode-seeking vs. mode-covering behavior), the sampling mixture $\pi_{\text{mix}}$ (which controls the degree of on-policy exploration), and the argument ordering within the divergence (which determines whether the student mode-covers or mode-seeks relative to the teacher).

What makes this genuinely novel — rather than a taxonomic convenience — is that it reveals equivalences that were invisible within the parent communities. The paper shows that MiniLLM's sequence-level Reverse KL (computed via REINFORCE, with $\log(p_T/p_\theta)$ as a per-step reward) and G-OPD's dense KL-constrained RL formulation (with an explicit reference policy and reward extrapolation parameter $\alpha$ ) are not different approaches but the same optimization problem under different choices of $\pi_{\text{mix}}$ and $f$ . Similarly, the paper shows that DPO-style preference optimization and token-level KL distillation both target geometric mixtures of teacher and reference distributions — the former through an implicit reward model, the latter through explicit divergence minimization — making them endpoints of a continuous spectrum rather than distinct paradigms.

This reframing has immediate practical consequences. It explains why methods succeed and fail in specific regimes: Forward KL's mode-covering behavior preserves output diversity (desirable for open-ended generation) but risks hallucination (placing probability mass in the inter-mode gap where no valid outputs exist), while Reverse KL's mode-seeking behavior concentrates on high-precision outputs (desirable for mathematical reasoning) but risks diversity collapse (the precision-recall tradeoff formalized by Cha & Cho, 2025). The choice of $\pi_{\text{mix}}$ determines stability: fully on-policy sampling ( $\pi_{\text{mix}} = p_\theta$ ) provides maximum distributional alignment but exposes training to the flawed prefix trap (teacher feedback on student-generated errors is unreliable), while mixing in ground-truth prefixes ( $\lambda < 1$ in GKD's formulation) stabilizes gradients at the cost of incomplete exposure bias correction. Understanding this as a design space rather than a menu of unrelated methods allows practitioners to make principled choices based on their specific constraints (task geometry, teacher access level, compute budget) rather than relying on empirical trial-and-error across method families.

The paper's contribution here is fundamentally a theoretical synthesis — it supplies the conceptual infrastructure that allows the field to stop asking "which method is best?" and start asking "which region of the design space is appropriate for my task, and how do I navigate the tradeoffs within it?" The fact that the framework subsumes over one hundred methods surveyed in the paper (Tables 3–9 and 2) is evidence of its explanatory power, not its completeness — the paper explicitly notes that the $f$ -divergence framework is a sufficient but not necessary parameterization, and that future work may uncover additional design axes (the adaptive divergence methods of Section 4.2, for example, can be understood as a fourth axis operating within the $f$ -divergence framework rather than outside it).

Innovation 2: The Failure-Mode Taxonomy as a First-Class Scientific Contribution — Understanding When and Why OPD Breaks Down

The survey's second distinctive contribution is its systematic consolidation of OPD failure modes into a coherent diagnostic framework. This is not a typical "limitations" section listing shortcomings — it is a first-class scientific contribution that identifies recurring pathology patterns, traces them to root causes, and connects them to specific design choices in the three-axis taxonomy. Before this paper, failure reports were scattered: individual papers noted that self-distillation sometimes degraded reasoning (Kim et al., 2026b), that length inflation was a problem (Luo et al., 2026), that teacher feedback on flawed prefixes was unreliable (Fu et al., 2026). The survey's contribution is to organize these into a unified causal structure where each failure mode is tied to a specific assumption violation in the OPD framework, and each has a corresponding set of mitigation strategies in the methods literature.

The diagnostic framework identifies four root-cause categories, each with a distinctive signature:

The flawed prefix trap (Section 7.2) is the most fundamental. It occurs when the student generates an early error that pushes the prefix into a region the teacher was never trained on, making all subsequent teacher predictions unreliable. Crucially, this is not a failure of the student or the teacher individually — it is a structural property of the interaction between them under on-policy sampling. The paper connects this to the DAgger theorem's implicit assumption that the expert can provide optimal actions in any state, an assumption that fails for LLM teachers on out-of-distribution student prefixes. The mitigation strategies (TIP's token weighting, SCOPE's rollout routing, AdaSwitch's divergence-gated guidance) all share a common logic: detect when the teacher's signal has become unreliable and either down-weight it or switch to an alternative supervision source. The paper's contribution is to identify this common logic across methods developed in different communities.

Self-play saturation (the Ouroboros problem) is specific to self-distillation. The student optimizes against a target that shares its own inductive biases and architectural limitations. If the model discovers a syntactic shortcut — a confident but flawed reasoning heuristic — no external signal penalizes it. The model self-reinforces this flawed trajectory, driving $p_\theta(y_{\text{flawed}}|x) \to 1$ , at which point gradients vanish, exploration ceases, and the policy is trapped. This is structurally analogous to GAN mode collapse but occurs in a self-supervised setting where there is no discriminator to push back. The paper's insight is that this failure mode is endemic to self-distillation without external grounding — it does not occur in white-box or black-box OPD because the teacher provides a distributional signal independent of the student's current policy. The practical implication (that self-distillation requires either external verification or internally generated privileged information to avoid saturation) directly informs the method selection framework in Section 3.3.

The calibration-capability gap (Zhang et al., 2026b) is perhaps the most counterintuitive finding: OPD can make models more accurate but less aware of their own uncertainty boundaries. The paper frames this as an "information mismatch" — teacher supervision is formed under privileged context available during training (full reasoning traces, multiple samples), whereas the deployed model must report confidence using only its deployment-time context. The result is a model that scores higher on benchmarks but does not reliably indicate when it might be wrong, which is arguably less safe for deployment than a weaker but better-calibrated alternative. This finding connects to the epistemic suppression phenomenon documented by Kim et al. (2026b), where self-distillation disproportionately removes hedging phrases and uncertainty markers, producing shorter but less calibrated reasoning. The paper's contribution here is to identify calibration degradation as a systematic consequence of the OPD training dynamics rather than an occasional artifact, and to connect it to the broader literature on model calibration and out-of-distribution detection.

Gradient SNR collapse explains why OPD fails on very hard prompts where the student's pass rate approaches zero. When all rollouts contain early catastrophic errors, teacher signals are dominated by noise, and the expected gradient magnitude vanishes at both extremes of difficulty ( $p \to 0$ and $p \to 1$ ). This provides the theoretical justification for PACED's competence-boundary curriculum (Section 6.2), where training is concentrated on prompts at the "frontier" of the student's competence where gradient SNR is maximized. The paper's contribution is to formalize this as a general property of on-policy training that applies across all methods in the survey, providing a common explanation for why curriculum design is not merely an engineering optimization but a necessary condition for effective OPD when the prompt distribution includes problems far outside the student's capability range.

The broader significance of this failure-mode taxonomy is that it converts OPD debugging from post-hoc benchmark analysis to principled differential diagnosis. A practitioner observing a specific symptom (accuracy collapse at longer sequence lengths, diversity loss in self-distillation, overconfidence on out-of-distribution prompts) can trace it to a root cause, identify the violated assumption, and select from a menu of mitigation strategies that share a common logic. This moves the field from empirical trial-and-error toward a more systematic engineering discipline — which is precisely the gap the survey identifies in its introduction when it notes that "no current treatment we are aware of... clarif[ies] the relationships among methods in terms of the specific design choices each paper optimizes."

Innovation 3: The Convergence of Distillation and RL as a Unified Optimization Regime

The survey's third distinctive insight is that distillation and reinforcement learning are converging toward a single optimization framework in which the primary distinction is not the training paradigm but the density and source of the supervision signal. This is not merely an observation about algorithmic similarity — it is a conceptual reframing with direct implications for how future training pipelines should be designed.

The paper traces this convergence through a progression of theoretical connections. First, G-OPD (Section 4.3) formalizes the equivalence between standard OPD and dense KL-constrained RL, showing that the standard Reverse KL distillation objective can be rewritten as maximizing a token-level reward (the log-probability ratio between teacher and reference policy) subject to a KL penalty from the reference. Second, MiniLLM's sequence-level Reverse KL (Section 4.1) is shown to be mathematically equivalent to policy gradient RL with the teacher's log-probability as a dense reward — the REINFORCE estimator with per-step returns $R_t = \sum_{t'=t} \log(p_T/p_\theta)$ is performing RL in a bandit environment where the teacher defines the reward function. Third, DPO-based distillation methods (AlignDistil, OVD, PBSD) are shown to target the same geometric mixture distribution as GKD's $\lambda$ -mixing — the optimal policy under the DPO objective satisfies $\pi^*(y|x) \propto \pi_{\text{ref}}(y|x) \exp(r(y,x)/\beta)$ , and substituting the teacher's log-probability as the implicit reward yields precisely the distribution targeted by token-level KL distillation.

The practical significance of this convergence is that it dissolves the artificial boundary between post-training stages. Historically, industrial pipelines treated distillation and RL as separate, linearly staged processes: first distill from the teacher (off-policy SFT), then refine with RL (GRPO or PPO). The paper argues that this sequential approach is suboptimal because it creates a hard boundary where knowledge acquired in one stage can be overwritten in the next (a form of catastrophic interference). The unified framework instead suggests that distillation and RL should be interleaved or jointly optimized, with the KD component providing dense gradient stabilization (reducing the high variance of policy gradient estimation) while the RL component prevents the student from collapsing onto suboptimal teacher modes (the teacher ceiling problem). This is exactly the design pattern that has emerged in the most recent industrial systems: DeepSeek-V4 replaces its mixed RL stage with pure multi-teacher OPD for model consolidation, CoPD and REOPOLD interleave RLVR with bidirectional OPD rather than running them sequentially, and the Sparse-to-Dense pipeline of Xu et al. (2026c) formalizes GRPO on the strongest model (teacher-side RL for capability discovery), followed by OPD as a dense bridge, followed by optional student-side RL after the bridge — explicitly treating the three stages as different points on a reward-density spectrum rather than separate paradigms.

What makes this insight genuinely fundamental rather than incremental is that it changes the research agenda. If distillation and RL are different points on a single spectrum parameterized by reward density and supervision source, then advances in either field transfer directly to the other. Better trust regions for KL-constrained RL (from the RL community) immediately improve OPD stability. Better per-token credit assignment from the distillation community (TIP, SCOPE) immediately improves RL sample efficiency. The paper's organization around design axes rather than community boundaries is designed to accelerate this cross-pollination: by showing that RLKD (which replaces token-level KL with a Generative Structure Reward Model) and G-OPD (which extends token-level KL with reward extrapolation) are both instances of the same underlying pattern — augmenting a dense teacher signal with a sparse outcome reward — the survey encourages researchers in either community to adopt insights from the other.

Innovation 4: The Granularity Principle — Matching the Distillation Unit to the Error-Compounding Scale

The paper's fourth distinctive contribution is an implicit but recurrent principle that can be extracted from its organization of methods across Sections 4–7: effective distillation requires matching the granularity of the supervision signal to the granularity at which errors compound in the target task. This is not stated as an explicit theorem but emerges as a consistent explanatory pattern across the survey's analysis of why methods succeed or fail in specific settings.

The principle is most visible in the survey's treatment of multi-turn agentic distillation (Section 8), where the paper observes that "coarse-grained approaches (full-trajectory OPD) tend to underperform because compounding is local" while "fine-grained approaches (token-level) waste compute on positions where the agent's behavior is already correct." The effective granularity, the paper suggests, is "the decision boundary, the point at which the agent commits to an action whose consequences are difficult to reverse." This maps cleanly onto the methods surveyed: TCOD operates at the trajectory level (temporal curriculum over turns), MAD-OPD operates at the step level (each agentic step as an independent distillation unit), Skill-SD operates at the skill level (skill-conditioned updates), and SOD operates at the step level for tool-integrated reasoning where tool-call boundaries introduce discontinuous state transitions. Each method succeeds because it identifies the natural unit of credit assignment for its domain and concentrates supervision there.

The principle extends beyond multi-turn settings. In single-turn reasoning, the token-level methods (GKD, DistiLLM) succeed on instruction-following where errors are local (a single wrong word does not derail the entire response), while sequence-level methods (MiniLLM, KETCHUP) succeed on mathematical reasoning where errors are cascading (a single wrong step can invalidate the entire proof). The adaptive divergence methods (ToDi, AKL, EOPD) represent an implicit recognition of this principle at the finest granularity: within a single sequence, some token positions are "decision points" where committing to the wrong token changes the trajectory's downstream distribution (mathematical operators, proof strategy keywords), while others are "filler" where many tokens are equally valid (conjunctions, stylistic variants). Adapting the divergence per-position — mode-seeking at decision points, mode-covering at filler — is an instance of matching granularity to error-consequence.

What makes this a genuine insight rather than an observation is that it provides a design principle for future methods. An engineer building an OPD pipeline for a new domain (medical diagnosis, legal reasoning, robotic control) can ask: at what granularity do errors compound in this domain? In medical diagnosis, it is at the level of test-ordering decisions — ordering the wrong test changes the entire information state. In legal reasoning, it is at the level of precedent citation — citing the wrong case changes the argument's logical structure. In robotic control, it is at the level of subgoal selection — choosing the wrong grasping point makes all subsequent motor commands irrelevant. The survey's taxonomy allows the engineer to select existing methods that operate at the appropriate granularity or design new ones that fill gaps in the granularity spectrum.

The principle also explains a pattern that would otherwise appear as contradictory empirical findings: why does beam search (which operates at the token level) outperform best-of-N (which operates at the sequence level) on some tasks but underperform on others? Why do sequence-level RL methods (GRPO) outperform token-level distillation on some problems but not others? The answer, the survey suggests, is not that one granularity is universally better but that the optimal granularity is domain-specific — and the field's progress can be measured by its increasing ability to design methods at the appropriate granularity for each domain rather than applying a single one-size-fits-all approach.

The survey's contribution here is to surface this principle as a unifying explanation across what would otherwise appear as a disconnected set of domain-specific engineering choices. The paper's organization around design axes (Objective → Signal → Dynamics) provides the infrastructure for reasoning about granularity as a cross-cutting concern: the objective function determines what is measured at the chosen granularity, the signal source determines how reliably that measurement can be made, and the training dynamics determine how efficiently the measurement can be incorporated into learning.

5. Experimental Analysis

Evaluation Methodology

Dataset. The survey does not introduce a new empirical evaluation of its own — it is a synthesis of over one hundred existing papers, each with its own experimental setup. However, the paper systematically documents and cross-references the evaluation regimes used across the OPD literature. The dominant benchmarks are mathematics and reasoning datasets that stress-test exposure bias correction: MATH-500 (the Hendrycks et al. 2021 competition-level mathematics benchmark, 500 test questions), AIME 2024 and 2025 (American Invitational Mathematics Examination problems, 30 questions each), HMMT (Harvard-MIT Mathematics Tournament), GSM8K (grade-school math word problems), LiveCodeBench (competitive programming evaluation), and Minerva (college-level STEM problems). For instruction-following and general capability transfer, the paper documents use of AlpacaEval, MT-Bench, Arena-Hard, LMSYS-Chat, and Dolly. For translation and summarization (common in early OPD work), benchmarks include WMT, XSum, and ROUGE-L evaluations. The key methodological observation is that "methods are evaluated on different base models, different training compute budgets, different benchmark versions... and with different numbers of rollouts per prompt," which the paper identifies as a significant "reproducibility challenge" and a barrier to fair cross-method comparison (Section 9).

Base model(s). The paper surveys methods evaluated across a wide range of model families, reflecting the field's progression toward architecture-agnostic distillation. The most frequently used families include Qwen2.5, Qwen3, and Qwen3.5 (spanning 0.6B to 235B parameters), Llama-2, Llama-3.1, and Llama-3.3 (7B–70B), DeepSeek families (DeepSeek-R1 at 671B MoE, DeepSeek-V4 at 1.6T MoE, and R1-Distill variants at 1.5B–70B), Gemma 2 (2B, 9B, 27B), GPT-2 and T5 families (for early GKD and DistiLLM experiments), and OLMo-3 (7B). The choice of base model is driven by three considerations documented in the survey: (1) model availability for white-box distillation within the same organization, (2) the desire to test OPD across "representative" capability levels spanning from small (0.5B–1.5B) to medium (4B–8B) to large (32B+), and (3) the need to evaluate cross-family transfer (e.g., Llama teacher → Qwen student) which stresses the vocabulary-mismatch challenges addressed in Section 5.1.2. The paper notes that the field lacks a standardized evaluation protocol "that fix[es] the base model, compute budget, and evaluation suite," and advocates for one analogous to HELM (Liang et al., 2023).

Metrics. The primary metric across surveyed methods is task-specific accuracy — the fraction of test instances for which the student's selected final answer matches the ground truth. For mathematics benchmarks (MATH-500, AIME, HMMT), this means exact answer matching or equivalence checking via symbolic evaluation (e.g., the Lightman et al. 2022 grading function). For code generation (LiveCodeBench, HumanEval), this means pass@k — the fraction of problems for which at least one of k sampled solutions passes all unit tests. For instruction-following (AlpacaEval, MT-Bench), metrics include win rates against reference models as judged by LLM evaluators. For summarization and translation, ROUGE-L and BLEU scores are standard. Critically, the paper documents that Pass@1 and Pass@k can diverge in opposite directions under OPD due to the diversity-collapse problem (Section 7.2): Reverse KL's mode-seeking behavior often improves Pass@1 (the student concentrates on its best single answer) while degrading Pass@k (the student loses the distributional coverage needed to find correct answers across multiple attempts). Several methods (SCOPE, GKD with Forward KL) explicitly report Pass@32 or Pass@64 alongside Pass@1 to capture this tradeoff.

Baselines. The paper documents a progression of baselines that reflects the field's increasing sophistication. The most fundamental baseline is off-policy SFT on teacher-generated data — the standard industrial recipe where the student is fine-tuned on a static corpus of teacher completions via standard cross-entropy loss. This is the direct comparator for all OPD methods since OPD's defining claim is that on-policy training improves over this baseline. Next is majority voting (selecting the most common answer among N sampled student solutions without any learned verifier), which isolates the benefit of the verifier signal. In white-box settings, PRM best-of-N weighted and ORM best-of-N weighted (scoring student solutions with process or outcome reward models and applying best-of-N weighted selection) serve as established baselines for verifier-guided methods. For RL-augmented methods, pure GRPO (Group Relative Policy Optimization without any distillation signal) serves as the RL-only comparator. For self-distillation methods, the relevant baseline is often the same model before self-distillation (zero-shot or few-shot performance) or standard SFT on the same self-generated data without on-policy filtering. The paper notes that "few papers control for all these variables simultaneously, making cross-method comparison unreliable from published numbers alone," and that readers should "interpret cross-row comparisons with appropriate caution."

Generation budget / compute accounting. The paper documents that OPD methods use a variety of compute-accounting frameworks, with no single standard. The most common unit is N generations per prompt — the number of complete student rollouts generated and scored per training step. Methods are compared at matched generation budgets to ensure fairness. For white-box OPD, the cost model includes student rollout time (autoregressive generation, memory-bound due to KV cache growth), teacher scoring time (forward pass through a typically larger teacher to obtain full-vocabulary logits), and student update time (standard backward pass). The paper provides representative cost estimates: distilling a 70B teacher into a 7B student on 8×H100 GPUs, off-policy training over 1B tokens requires ~300 GPU-hours total, while on-policy training over the same compute scale requires ~1,200–1,500 GPU-hours — a 4–5× overhead consistent with Lightning-OPD's empirical measurements at smaller scales. Several methods explicitly report FLOPs or GPU-hours alongside accuracy to enable cost-benefit analysis. FOPD reports FLOPs reduction factors (2–47× by truncating rollouts to prefix length k), Lightning-OPD reports 4.0× training efficiency improvement via offline teacher caching, and NPD reports 8.1× throughput speedup via asynchronous generation-training decoupling. The paper's discussion of efficiency in Section 6.3 decomposes the OPD cost bottleneck into three components and notes that "the degree of on-policy approximation trades off against compute cost, with full on-policy training at one extreme (maximum alignment, maximum cost) and fully offline SFT at the other (minimum cost, exposure bias ceiling)."

Cross-validation / statistical protocol. The survey does not describe a unified cross-validation protocol since it synthesizes existing work, but it documents the validation strategies used across the literature. The most common approach is two-fold cross-validation within difficulty bins on the test set (following the protocol established by compute-optimal test-time scaling work): the best-performing strategy is selected on one fold and evaluated on the other, with results averaged. For methods that sweep hyperparameters (divergence type, mixture coefficient λ, beam width, sequential-to-parallel ratio), the standard protocol is held-out validation sets with hyperparameters selected to maximize validation performance before final test evaluation. Several methods (PACED, FOPD) explicitly address the cost of difficulty estimation — for example, PACED's one-shot evaluation phase uses K=8 rollouts per prompt to estimate pass rates prior to curriculum training, acknowledging this as an exploration-exploitation tradeoff that the paper flags as "a key avenue for future work." The main methodological challenge the survey identifies is that standard validation loss is not a reliable signal for OPD because "after fine-tuning, the validation trajectories become off-policy (they were generated by the base model, not the fine-tuned revision model)," a problem also noted in the revision-model literature.

Main Quantitative Results

The survey synthesizes empirical results across over one hundred methods, and its contribution is not new experiments but a comparative analysis of existing results. The following sections organize findings by the paper's three design axes and cross-cutting themes, drawing on the per-category comparison tables (Tables 3–9 and 2) and the unified analysis in Section 7.

5.2.1 Objective Function Results: Fixed vs. Adaptive vs. RL-Augmented Divergences

Headline finding: On-policy training consistently outperforms off-policy baselines, but the choice of divergence depends critically on task geometry — mode-seeking for reasoning, mode-covering for diversity, adaptive per-token selection for mixed-regime tasks — and RL-augmented objectives lift the teacher ceiling on tasks where the student must exceed the teacher.

Fixed divergence results (Section 4.1, Table 3). The foundational comparison comes from GKD (Agarwal et al., 2024), which evaluates Forward KL, Reverse KL, and JSD on instruction-following and summarization tasks using T5-XL teacher (3B) → T5-Small/Base student configurations. At matched generation budgets, on-policy sampling (λ = 1) outperforms off-policy (λ = 0) across all divergence choices tested on XSum and WMT, confirming that the sampling distribution (on-policy vs. off-policy) matters more than the specific divergence when task geometry does not strongly favor one extreme. JSD performs best specifically on translation tasks where the output space has moderate diversity — neither a single correct answer nor fully open-ended generation — consistent with JSD's symmetric, bounded interpolation between mode-covering and mode-seeking behavior.

MiniLLM (Gu et al., 2024) evaluates sequence-level Reverse KL (computed via REINFORCE) against token-level baselines. On instruction-following and summarization, MiniLLM's sequence-level objective outperforms token-level alternatives, with the paper attributing this to better credit assignment across the full trajectory: "this connection to policy optimization is consistent with MiniLLM's empirical strength on reasoning tasks where sequence-level coherence matters more than token-level accuracy, at the cost of higher training compute because REINFORCE estimation in combinatorial output spaces requires more iterations and careful baseline subtraction to converge."

DistiLLM (Ko et al., 2024) evaluates skewed KL mixtures on GPT-2 XL (1.5B) → GPT-2 (124M) distillation for summarization and instruction following. The skewed KL loss avoids zero-division instability by replacing the student distribution in the KL denominator with a mixture $\tilde{p} = \alpha p_T + (1-\alpha)p_\theta$ , producing stable training without policy gradients. DistiLLM-2 (Ko et al., 2025) extends this with asymmetric losses — Forward SKL on teacher-generated data, Reverse SRKL on student-generated data — and reports "consistent improvements over symmetric single-divergence baselines across instruction following and summarization tasks."

KETCHUP (Fan et al., 2025) addresses the variance problem in MiniLLM's REINFORCE estimator by replacing the full cumulative return with K-step Bellman returns, reporting reduced gradient variance at a power rate in K (Theorem 1 of Fan et al., 2025) while preserving the sequence-level objective. vOPD (Oh et al., 2026) reports that adding a control variate baseline (the closed-form per-token negative reverse KL) to the single-sample OPD estimator yields "an average +3% absolute gain over vanilla single-sample OPD (up to +6.2% on MATH500)" while "match[ing] the accuracy of full-vocabulary OPD while reducing wall-clock time by up to 57.7%."

AntiSD (Shen et al., 2026a) reports that ascending rather than descending the student-teacher divergence — a pointwise mutual information analysis showing that privileged-context conditioning inflates teacher confidence on tokens already implied by the solution while deflating it on deliberation tokens — "matches GRPO accuracy in 2–10× fewer training steps and improves final accuracy by up to 11.5 points on math reasoning" across five models from 4B to 30B parameters.

Adaptive divergence results (Section 4.2, Table 4). The core finding across the adaptive methods is that per-token divergence selection outperforms any single fixed divergence, and that even the simplest adaptive routing (a hard entropy threshold in EOPD) often outperforms the best-tuned fixed alternative. ToDi (Jung et al., 2025) reports that its per-token, per-vocabulary-entry blending of Forward KL and Reverse KL — using sigmoid weights on the teacher-student log-ratio — outperforms fixed divergence baselines on math reasoning (MATH, GSM8K) and open-ended generation (AlpacaEval). AKL (Wu et al., 2025) reports that its head/tail vocabulary partition with gap-weighted blending "outperform[s] fixed divergence baselines" on the same benchmark suite. EOPD (Jin et al., 2026a) reports that its entropy-gated Forward KL activation — applying Reverse KL everywhere and additively activating Forward KL at positions where teacher entropy exceeds a threshold — outperforms both fixed Reverse KL and fixed Forward KL, consistent with the view that "the appropriate divergence is position-dependent."

AOPD (Jia et al., 2026) extends the adaptive principle from divergence type to objective class, switching between standard policy gradient (for positive-advantage tokens) and localized forward-KL guidance on the teacher's top-K support (for non-positive-advantage tokens). On competition-level mathematics (AIME 2024/2025, HMMT), AOPD reports "average gains of +4.09 with strong initialization and +8.34 with weak initialization over standard on-policy baselines," with the larger weak-initialization gain suggesting that "the exploration black hole is among the dominant failure modes when the student starts far from the teacher."

RL-augmented objective results (Section 4.3, Table 5). The headline finding across these methods is that combining dense teacher supervision with sparse outcome rewards outperforms either approach alone, and that the reward-extrapolation mechanism in G-OPD can push the student beyond the teacher's capability ceiling. G-OPD (Yang et al., 2026d) formalizes the equivalence between standard OPD and dense KL-constrained RL, with a reward-extrapolation parameter α > 1 that pushes the student beyond the teacher's probability mass. In the multi-teacher setting, G-OPD reports that "reward extrapolation (ExOPD) produces a unified student that surpasses all its same-size domain teachers" — the student exceeds the individual teachers that trained it.

RLKD (Xu et al., 2025b) replaces token-level KL with a Generative Structure Reward Model (GSRM) that scores step-level structural alignment between teacher and student reasoning paths, combined with task-level outcome rewards via GRPO. The paper reports that RLKD trained on "only 0.1% of the data under a pure RL regime surpasses standard SFT-RL pipelines," showing that structural reward can substitute for dense KL when the teacher's value lies in reasoning organization rather than token-level distribution. KDRL (Xu et al., 2025a) reports that joint KD+RL optimization with an on-policy KL regularizer "mitigat[es] policy drift while still allowing reward-driven exploration." RLAD (Zhang et al., 2026k) refines this with a PPO-style likelihood-ratio objective that selectively follows the teacher only when its signal improves the policy update, preventing blind KL minimization from dragging the student toward suboptimal trajectories on problems the teacher itself cannot solve.

REOPOLD (Ko et al., 2026) reports "6.7–12× greater sample efficiency than recent RL approaches" and demonstrates that a 7B student matches a 32B teacher in visual reasoning with ~3.3× inference speedup. CoDistill-GRPO (Kwon et al., 2026) reports that its bidirectional co-distillation approach yields "an accuracy increase of over 11.6 percentage points over the base model and 6.0 points over standard GRPO on Minerva" for Qwen2.5-Math-1.5B, with the large model (7B) nearly matching standard GRPO despite training on small-model rollouts — providing an "approximate 18% speedup."

5.2.2 Signal Source Results: White-Box vs. Black-Box vs. Self-Distillation

Headline finding: Signal density correlates strongly with performance, but each access tier has developed effective proxy signals — white-box logit access yields the strongest results where available, black-box methods recover substantial teacher knowledge from text alone, and self-distillation with privileged information can match or exceed GRPO at dramatically lower compute.

White-box logit supervision results (Section 5.1, Table 6). The paper identifies that the majority of methods in Section 4 assume white-box access and that the signal source itself is often not the primary axis of comparison — it is a precondition for the method to function. For same-family distillation (Section 5.1.1), the key empirical finding is that methods like AdaKD (gradient-weighted KL), MPD (mixed-policy compression), and BRTS (best-of-N teacher rollout selection) refine which tokens receive supervision rather than competing on accuracy at matched budgets, with gains concentrated on reasoning tasks where per-token supervision quality varies sharply.

For cross-family distillation (Section 5.1.2), the key results are from DSKD (Zhang et al., 2025b), ULD (Boizard et al., 2025), and SimCT (Sun et al., 2026). DSKD reports that its dual-space projection method enables distillation across architecturally dissimilar models (e.g., Llama teacher to Qwen student) with "vocabulary tokens with similar semantic embeddings [receiving] similar probability mass even when the tokenization schemes differ." ULD reports that its Wasserstein-1 distance approach — which admits a closed-form solution under uniform transport cost with O(n log n) complexity — provides a lightweight alternative to learned projectors. SimCT reports that its multi-token continuation matching "outperforms both shared-vocabulary OPD and coarser cross-tokenizer baselines" across three heterogeneous teacher-student pairs (Qwen2.5-7B→Phi-4-mini, Qwen2.5-7B→Gemma-2-2B, Phi-4-mini→Gemma-2-2B) on math reasoning and code generation, with "particularly pronounced gains at positions of maximal vocabulary disagreement," suggesting that "a substantial fraction of the cross-tokenizer supervision loss... stems not from inherent incompatibility but from an unnecessarily restrictive matching granularity."

Black-box and API-constrained distillation results (Section 5.2, Table 7). The headline finding is that black-box methods can recover substantial teacher knowledge from text-only observations, with adversarial and verbal feedback approaches achieving robust gains over SFT baselines despite information-theoretic disadvantages relative to white-box access.

GAD (Ye et al., 2025) reports that its adversarial distribution matching — using a discriminator to distinguish student rollouts from teacher API outputs — "produces reliable gains over SFT baselines" on instruction-following tasks, though the paper notes it "inherits the training instability of adversarial objectives." Lion (Jiang et al., 2023) reports that its three-stage adversarial curriculum — imitation, discrimination, generation — enables LLaMA-13B to reach "competitive performance on BIG-Bench Hard and AGIEval using only 70K training examples," matching or exceeding much larger models distilled via standard SFT. DAIL (Mendes et al., 2026) reports that its didactic-to-constructive learning approach — transforming expert solutions into in-distribution reasoning traces via mixed policy decoding — yields "10–25% pass@k gains" using fewer than 1,000 expert solutions.

OVD (Xiong et al., 2026) reports that its verbal score distillation — replacing token-level probability matching with discrete scores (0–9) — produces "up to +12.9% absolute EM improvement on web QA and +25.7% on math benchmarks" while reducing memory consumption by avoiding logit storage. The paper interprets this as evidence that "on-policy exploration itself supplies a substantial fraction of the learning signal, with the teacher's role being closer to selecting among student trajectories than to correcting individual tokens."

ROPD (Fang et al., 2026) reports that its rubric-based semantic distillation — where a Rubricator induces prompt-specific evaluation rubrics by contrasting teacher and student rollouts — "matches or surpasses logit-based OPD methods while achieving approximately 10× sample efficiency" across AIME 2024/2025, HMMT 2025, GPQA-Diamond, HealthBench, and IFEval. On AIME 2025 with thinking mode, the Qwen3-4B student (68.75%) surpasses its GPT-5.2 teacher (67.08%), demonstrating that rubric-augmented optimization can facilitate student-exceeds-teacher performance even without gradient-level access.

LUFFY (Yan et al., 2025) reports that its mixed-policy GRPO — incorporating off-policy reasoning traces alongside on-policy rollouts with importance-weighted gradient correction — posts "+6.4 average points over standard RLVR" on reasoning benchmarks. DASD (Yan et al., 2026) reports that its distribution-aligned sequence distillation — generating multiple diverse teacher traces per prompt and aligning the student's generation distribution with the teacher's through sequence-level matching — reaches "strong performance using only 448K training samples."

Self-distillation results (Section 5.3, Table 8). The headline findings cluster by sub-family:

Privileged information methods (Section 5.3.1): OPSD (Zhao et al., 2026b) — the canonical ground-truth-conditioned self-distillation method — reports that it "matches or exceeds GRPO across all three scales (1.7B, 4B, 8B) while using only a single rollout per problem versus GRPO's eight" on competition-level math benchmarks (AIME 2024/2025, HMMT), "yielding a significant computational advantage." The gains are largest at 8B (+0.9 average points over GRPO) and 4B (+0.8 over GRPO), but OPSD underperforms GRPO at 1.7B (-0.5), which the authors attribute to the requirement that "OPSD's reliance on self-rationalization requires sufficient model capacity to produce a meaningful dense token-level signal."

CRISP (Sang et al., 2026) reports that using a model prompted to "be concise" as teacher for its own verbose version "reduc[es] chain-of-thought token count by 57–59% on MATH-500 while improving accuracy by 9–16 percentage points," demonstrating that "much of the verbosity in distilled reasoning models is trainable inefficiency rather than a necessary feature of competent reasoning." GATES (Stein et al., 2026) reports that its consensus-based gating mechanism — suppressing gradients when tutor uncertainty is high — provides effective self-supervision from source-document-conditioned contexts without requiring ground-truth labels. GUI-SD (Zhang et al., 2026g) reports that its visual privileged information approach — using a Gaussian soft mask centered on target GUI elements — "outperforms GRPO... on six GUI grounding benchmarks including ScreenSpot-v2, ScreenSpot-Pro, and OSWorld" with only a single on-policy rollout per instance.

PBSD (Yu et al., 2026b) reports that its reward-regularized alternative to pure KL matching — targeting $\pi^* \propto \pi_{\text{teach}}(y|x) \exp(r(x,y)/\beta)$ — "matches OPSD's token efficiency while surpassing its peak accuracy and avoiding the post-peak decline" on Qwen3-1.7B/4B/8B. ATESD (Han et al., 2026) reports that its adaptive teacher exposure controller — a Beta-policy that determines how much privileged context the teacher sees — "outperforms OPSD (+0.95 to +2.33 Average@12 points)" across Qwen3-1.7B, 4B, and 8B. TRACE (Wang et al., 2026d) reports that its token-routed sparse self-OPD — applying forward KL only to critical spans marked by a privileged annotator — "improves over GRPO by 2.76 points on average and is the only trained method in their evaluation that preserves OOD performance."

Pure self-distillation results (Section 5.3.2): SSD (Zhang et al., 2026e) reports that its minimalist recipe — sample solutions at training-time temperature, fine-tune via standard SFT, no RL, no verifier, no external teacher — "improves Qwen3-30B-Instruct from 42.4% to 55.3%" on LiveCodeBench v6, an approximately 13 percentage point gain from on-policy self-training alone. UniSD (Jin et al., 2026b) reports that its unified framework combining multi-teacher agreement, EMA stabilization, and contrastive learning yields "+5.4 points over the base model and +2.8 over the strongest baseline (GKD) on Qwen2.5-7B, without relying on any external teacher." MTP-SD (Kirchenbauer et al., 2026) reports that its architectural self-distillation — converting a pre-trained autoregressive model into a standalone multi-token predictor — "delivers > 3× faster decoding at typically 3–7% accuracy drop (model-dependent) without requiring any auxiliary module at inference time."

External feedback methods (Section 5.3.3): SD-ZERO (He et al., 2026) reports that its dual-role self-revision architecture — a Generator produces on-policy responses while a Reviser conditioned on binary correctness produces improved versions — achieves "68.3% avg@8 on AIME 2024, outperforming GRPO (62.5%)" on Qwen3-4B-Instruct, showing that self-revision can convert sparse binary rewards into dense self-supervision more effectively than standard RL. SRPO (Li et al., 2026c) reports that its sample-routed dual-objective — GRPO for correct samples, SDPO-style logit-level correction for failed samples — "raises the five-benchmark average on Qwen3-8B by 3.4% over GRPO and 6.3% over SDPO alone across science and tool-use tasks." RLSD (Yang et al., 2026a) reports that its decomposition of the gradient into magnitude (self-distillation) and direction (RLVR) achieves "2× sample efficiency" — "at just 200 training steps, RLSD surpasses GRPO trained for 400 steps on Qwen3-VL-8B-Instruct."

RESD (Zhang et al., 2026j) reports that its reflection-enhanced self-distillation — transforming failed trajectories into structured diagnostic feedback — "outperforms GRPO by approximately 8× in sample efficiency on low-pass-rate tasks" where successful rollouts are extremely rare (below 5% pass rate). π-Play (Zhang et al., 2026h) reports that its multi-agent co-evolution framework — examiner, teacher, and student agents jointly optimized through alternating updates — "surpasses fully supervised search agents such as Search-R1" on multi-hop benchmarks, and does so without external data.

5.2.3 Training Dynamics Results: Stabilization and Efficiency

Headline finding: The benefits of token weighting, curriculum design, and compute optimization compound additively — deploying all three simultaneously can reduce the effective cost gap between on-policy and off-policy training from 4–5× to near parity while retaining the quality advantage.

Token and sample weighting results (Section 6.1, Table 9). TIP (Xu et al., 2026b) reports that its entropy×divergence quadrant scoring — crossing student entropy with teacher-student divergence to classify tokens into four qualitatively distinct categories — reveals that Q3 tokens (low-entropy, high-divergence: overconfident errors) are "structurally invisible to any entropy-only weighting scheme." Training on exclusively Q3 tokens (fewer than 20% of all tokens) surpasses full-token OPD on long-horizon planning benchmarks. SCOPE (Zheng et al., 2026) reports that its dual-path rollout routing — incorrect trajectories receive teacher-perplexity-weighted KL, correct trajectories receive student-perplexity-weighted MLE — yields a "7.3% relative Pass@32 gain over competitive baselines across six reasoning benchmarks" by countering diversity collapse.

EGRSD (Ke et al., 2026) reports that its entropy-guided confidence gate for self-distillation — down-weighting high-entropy teacher positions — "advance[s] the accuracy-length frontier among compared trainable methods" on Qwen3-4B and Qwen3-8B in thinking mode. SOD (Zhong et al., 2026a) reports that its step-level divergence reweighting for tool-integrated reasoning — attenuating teacher signals at steps where tool-call errors cause discontinuous divergence jumps — enables a 0.6B student to reach "26.13% on AIME 2025 (average@32)," and "outperform[s] the second-best baseline by up to 20.86% across math, science, and code benchmarks."

Curriculum and difficulty adaptation results (Section 6.2). PACED (Xu et al., 2026a) reports that its Beta-kernel curriculum — weighting prompts by $w(p) = p(1-p)$ where $p$ is the student's estimated pass rate — concentrates the gradient budget on the "productive learning range" and proves that this Beta family is "the leading-order minimax-optimal weighting under bounded misspecification of the SNR model." TCOD (Wang et al., 2026c) reports that its temporal curriculum for multi-turn agent distillation — progressively expanding the horizon of teacher supervision — "delivers gains of up to +18 points over vanilla multi-turn OPD" on ALFWorld, WebShop, and ScienceWorld, with the Forward-to-Backward variant showing particular strength on tasks where early-turn errors are most consequential.

Compute optimization results (Section 6.3). FOPD (Zhang et al., 2026a) reports that its prefix truncation — observing that the useful distillation signal concentrates in early tokens — "matches the performance of full OPD while reducing training FLOP by 2×–47×." Lightning-OPD (Wu et al., 2026a) reports that its offline teacher caching — precomputing teacher log-probabilities once over SFT rollouts — "achieves 4.0× higher training efficiency than standard OPD" under a teacher consistency condition. NPD (Rang et al., 2026) reports that its asynchronous generation-training decoupling with Δ-IFD filtering "records 8.1× throughput speedup over synchronous on-policy baselines while outperforming SFT by +8.09% averaged across 11 evaluation benchmarks." Prune-OPD (Yang et al., 2026f) reports that its drift-aware dynamic truncation — continuously monitoring per-position top-k overlap and truncating when cumulative drift exceeds a budget — "reduces training time by 37.6–68.0% while preserving or improving performance on AMC, AIME, and HMMT."

5.2.4 Cross-Cutting Comparisons: Industrial Deployment and Failure-Mode Results

Industrial deployment results (Section 8.1, Table 2). Qwen3 (Yang et al., 2025a) reports that "on-policy distillation outperforms direct reinforcement learning at roughly one-tenth of the GPU hours, and further improves pass@64 on AIME benchmarks where reinforcement learning from the same off-policy checkpoint does not." DeepSeek-V4 (DeepSeek-AI, 2026) reports that its replacement of the mixed RL stage with pure multi-teacher OPD — consolidating over ten domain-specific experts into a unified 1.6T-parameter model through full-vocabulary Reverse KL distillation — succeeds at this scale, with the paper's systems engineering discussion (hidden-state caching, on-the-fly logit reconstruction, teacher-aware batch scheduling) serving as a practical validation that the methods surveyed in Sections 4–6 are industrially viable. Gemma 2 (Gemma Team et al., 2024) reports that embedding off-policy KD directly into pre-training from a 27B teacher enables its 2B and 9B models to achieve competitive performance at their parameter scales. MiMo-V2-Flash (Xiaomi LLM-Core Team et al., 2026) reports that multi-teacher distillation combined with Multi-Token Prediction and hybrid attention reaches "strong performance on reasoning and agentic tasks" with a 309B MoE model (15B active parameters).

Failure mode characterization results (Section 7). CaOPD (Zhang et al., 2026b) reports its central diagnostic finding: OPD "tends to leave models in a state of severe overconfidence" — a "Scaling Law of Miscalibration" where capability improvement and calibration degradation are decoupled. Kim et al. (2026b) report that self-distillation disproportionately removes hedging phrases and uncertainty markers, producing "shorter but less calibrated reasoning, where the model confidently commits to flawed steps." ListOPD (Li et al., 2026g) reports that on calibrated listwise JSON tasks, operating just below the extrapolation cliff λ* "brings a 1.7B Qwen3 student to in-domain parity with an 8B-SFT baseline at one-fifth the parameters, with the gain driven primarily by format adherence rather than ranking quality." Jiang et al. (2026) report that up to 18% of tokens remain persistently high-loss after OPD training saturates — "Rock Tokens" that resist teacher-driven corrections despite providing disproportionately large gradient norms — with causal intervention showing "negligible functional contribution to actual reasoning performance."

The Sparse-to-Dense pipeline (Xu et al., 2026c) reports its key operational finding: "At fixed Qwen3-1.7B deployment size, an RL-improved 8B teacher distilled through this dense bridge outperforms direct GRPO on the student (79.3% vs. 75.9% on MATH, 25.2 vs. 19.8 on AIME 2024), while transfer from the same teacher before RL underperforms." The bridge also "makes subsequent student-side GRPO effective where it previously was not, lifting MATH from 75.4% to 78.5%."

Ablation Studies and Robustness Checks

The survey itself does not run new ablation studies, but it systematically documents ablation findings from the papers it surveys. The following organizes the key ablations by design axis:

Divergence choice ablation (GKD, Section 4.1): GKD ablates across Forward KL, Reverse KL, and JSD while controlling for the on-policy mixture coefficient λ. The finding is that all three divergences outperform off-policy baselines when λ > 0, but JSD performs best on tasks with moderate output diversity (translation) while Forward KL preserves diversity better on open-ended generation. The paper's interpretation: "the sampling policy (on-policy vs. off-policy) matters more than the specific divergence choice when task geometry does not strongly favor one extreme."

PRM aggregation strategy ablation (Section 4.1, Appendix E of source material): Comparing "min," "prod," and "last" step-wise aggregation for process reward models: "Last" achieves roughly 37% at 256 samples, "min" achieves roughly 35%, and "prod" achieves roughly 27%, with ORM at roughly 34%. The finding that "last" (using only the PRM's prediction at the final step) outperforms alternatives that aggregate across steps is non-obvious — it suggests that step-level PRM training acts as a form of beneficial representation learning even when the intermediate predictions are not directly used at aggregation time.

Sequential vs. parallel sampling ablation (MiniLLM and G-OPD, Sections 4.1 and 4.3): Across multiple methods, purely sequential revisions outperform purely parallel sampling on easy problems, while a balanced sequential-to-parallel ratio is optimal on hard problems. This difficulty-dependent interaction is replicated across both revision-based methods (MiniLLM) and RL-augmented methods (G-OPD) and across both verifier-based and majority-based answer selection. The paper interprets this as evidence that easy problems benefit from local refinement (exploitation) while hard problems require diverse exploration before refinement (exploration + exploitation).

Oracle vs. predicted difficulty bins ablation (PACED, Section 6.2): Both oracle and predicted difficulty bins yield qualitatively similar trends across difficulty levels, with curves "largely overlapping." This is the critical robustness check for the compute-optimal allocation framework: the strategy works without ground-truth labels, though at some cost to peak performance (predicted bins show slightly lower accuracy at high budgets in the revision setting).

White-box vs. black-box signal ablation (GAD vs. GKD, Section 5.2): GAD's adversarial discriminator — a black-box proxy for the teacher's full distribution — produces reliable gains over SFT baselines but falls short of white-box methods at comparable compute. The paper interprets this as evidence that "a substantial portion of the teacher's knowledge can be recovered from text-only observations," but that distributional fidelity (exact KL computation) still provides measurable advantages when available.

Full-vocabulary vs. sampled-token KL ablation (DeepSeek-V4, Section 4.1): DeepSeek-V4 reports that full-vocabulary logit distillation — computing the exact $D_{KL}(p_\theta \parallel p_T)$ over all |V| tokens at each position — "yields more stable gradients and more faithful knowledge transfer" than the sampled-token approximation (computing $\log p_\theta(y_t) - \log p_T(y_t)$ only at the sampled token). The engineering solution (hidden-state caching with on-the-fly logit reconstruction) makes full-vocabulary KL feasible at trillion-parameter scale.

Data source asymmetry ablation (DistiLLM-2, Section 4.1): DistiLLM-2 ablates symmetric vs. asymmetric divergence application: Forward SKL on teacher-generated data + Reverse SRKL on student-generated data outperforms applying either divergence symmetrically to both data sources, consistent with the view that "the appropriate divergence is data-source-dependent."

Revision model verifier ablation (Appendix J of source material): The base-LM PRM underperforms the revision-specific ORM when scoring revision model outputs (sequential + base-LM PRM: ~40% at 64 generations; sequential + revision ORM: ~42%). This confirms that distribution shift is a practical concern in OPD systems that combine revisions with verifier-based selection.

Revision history in verifier context ablation (Appendix J): Including previous revisions in the ORM's context provides a small improvement over the no-history ablation (~1–2 percentage points at 64 generations), but both variants outperform the parallel baseline. This suggests that the sequential sampling benefit is not solely attributable to the verifier seeing more context.

ReST^EM ablation (Appendix K): An attempt to optimize the revision model using ReST^EM (on-policy self-improvement) backfires: additional sequential revisions substantially hurt performance, with fully sequential performance dropping to roughly 33.5% compared to roughly 38.5% at the optimal ratio. The paper hypothesizes that on-policy data collection in ReST^EM "exacerbates spurious correlations in revision data, causing the model to fail to learn the revision task properly." This negative result is particularly informative — it suggests that revision training is sensitive to the data generation procedure and that naive on-policy self-improvement can be counterproductive.

Length inflation ablation (Stable-OPD, Section 7.3): Luo et al. (2026) report that adding a reference divergence term anchoring the student to a pre-distillation baseline, combined with rollout mixing that blends on-policy and off-policy data, "+7.2% on average over vanilla OPD by breaking the self-amplification cycle" of length inflation. The ablation shows that both components (reference divergence and rollout mixing) are necessary — removing either degrades performance substantially.

Critical Assessment

The survey's central contribution is a synthesis of existing results, not new experiments, so the assessment must focus on whether the patterns the paper extracts from the literature are genuinely supported by the evidence it marshals, and whether the framework it proposes genuinely unifies the field.

Central Claim 1: On-policy distillation consistently outperforms off-policy baselines across mathematical reasoning, code generation, and instruction following, with the performance gap widening for longer reasoning chains.

Assessment: Supported with important qualifications that the paper itself documents. The evidence from GKD, MiniLLM, DistiLLM, and their successors consistently shows on-policy training outperforming off-policy baselines at matched generation budgets (Table 3, Section 4.1). The mechanism — reduced exposure bias from training on student-generated prefixes — is theoretically grounded in the DAgger theorem and empirically validated across multiple model families and benchmarks.

However, the survey is commendably honest about boundary conditions that complicate this narrative. The DeepSeek-R1 anomaly (Section 7.4) — where purely off-policy SFT distillation from a 671B MoE teacher into 1.5B–70B students achieved state-of-the-art reasoning performance — proves that off-policy distillation is not universally suboptimal. The paper's explanation (exceptionally strong teacher, diverse reasoning traces, manageable effective conditional complexity at small student scales) is plausible but post-hoc. The paper does not present an experiment where the same teacher-student pair is trained both off-policy and on-policy at the same scale — the comparison is always across different papers and different experimental setups. This makes the claim about the "widening gap for longer reasoning chains" more of a theoretical prediction than a directly tested empirical regularity. The gradient SNR analysis (Section 6.2) provides theoretical justification, but the survey does not point to a controlled experiment that varies sequence length while holding all else constant and measures the on-policy/off-policy gap as a function of T — this would be the definitive test of the DAgger claim, and its absence is a genuine gap in the literature.

Furthermore, the compute-adjusted comparison is underdeveloped. The paper reports that on-policy training costs 4–5× more than off-policy training per token (Section 6.3), but most method comparisons are at matched generation budgets, not matched wall-clock time or FLOPs. When FOPD shows that prefix truncation matches full OPD quality at 2–47× less compute, the implication is that some of the reported gains from on-policy training may be achievable with cheaper approximations. The survey acknowledges this with its fidelity-efficiency frontier concept, but does not systematically re-evaluate headline claims in compute-adjusted terms.

Central Claim 2: Divergence selection should be adapted to both task and token position rather than fixed globally, with Reverse KL excelling on reasoning tasks and Forward KL preserving diversity for open-ended generation.

Assessment: Well-supported by the adaptive divergence literature, but the "should" is stronger than the evidence for any specific adaptive scheme. The per-method results (ToDi, AKL, EOPD, AOPD) consistently show that adaptive per-token divergence selection outperforms any single fixed divergence, and the theoretical motivation — teacher entropy varies across positions, requiring mode-seeking at decision points and mode-covering at filler positions — is clear (Figure 2). The paper's mapping of this tradeoff onto task types (Reverse KL for math, Forward KL for creative writing, JSD for translation) is empirically grounded in GKD's experiments.

A genuine weakness is the absence of head-to-head comparisons between adaptive divergence methods. The paper documents that ToDi (per-token, per-vocabulary-entry weighting) and EOPD (entropy-gated Forward KL activation) both outperform fixed baselines, but it does not report whether ToDi outperforms EOPD or vice versa, or under what conditions. The design space is finer-grained than the evidence can currently resolve: ToDi conditions adaptation on the teacher-student gap (a pairwise property), EOPD conditions on teacher uncertainty alone (a marginal property), AKL conditions on head/tail partition discrepancies — which is better when? Without controlled comparisons, the survey's claim that adaptive divergence selection is the "current frontier" is a descriptive claim about research activity, not a prescriptive claim about which adaptation scheme to use.

A second weakness is that the practical magnitude of the adaptive advantage is modest in many settings. The paper reports that "even the simplest adaptive routing (a hard entropy threshold) often outperforms the best-tuned fixed alternative," which is both encouraging (adaptivity helps) and cautionary (sophisticated per-token weighting may not be worth the complexity relative to simple heuristics). The survey does not provide a systematic cost-benefit analysis of adaptive complexity — an ablation that would be valuable for practitioners deciding between GKD's simplicity and ToDi's per-entry weighting.

Central Claim 3: The boundary between distillation and reinforcement learning is narrowing, with combined KD+RL objectives outperforming either approach alone, and G-OPD's reward extrapolation demonstratably pushing students beyond their teachers.

Assessment: Supported for the combination claim, conditionally supported for the extrapolation claim. The evidence for combined KD+RL outperforming either alone is strong and replicated across multiple methods (G-OPD, KDRL, RLAD, REOPOLD, CoDistill-GRPO) and benchmarks (Table 5). The mechanism — dense KD gradients reduce policy gradient variance while sparse RL rewards prevent collapse onto suboptimal teacher modes — is theoretically well-motivated by the gradient decomposition framework of Li et al. (2025). The paper's reframing of distillation and RL as points on a single reward-density spectrum is conceptually powerful and will likely influence how future training pipelines are designed.

The reward extrapolation claim requires more scrutiny. G-OPD's result that a multi-teacher distilled student "surpasses all its same-size domain teachers" is striking, but the paper does not clarify whether this represents genuine capability creation (the student synthesizes knowledge from multiple teachers to solve problems none could solve individually) or capability recombination (the student finds solution paths that are Pareto-optimal combinations of strategies each teacher knew individually — impressive but not the same as exceeding the teacher frontier). The paper's own discussion of the "extrapolation cliff" in ListOPD (Li et al., 2026g) — where reward extrapolation beyond a critical λ* causes format collapse rather than capability growth — suggests that the mechanism is fragile and task-dependent. A more systematic characterization of when extrapolation succeeds vs. fails is needed before the claim can be considered robustly supported.

A further concern is that most combined KD+RL methods have been tested in single-turn settings. Extending to multi-turn agentic tasks — which the paper identifies as an emerging domain — introduces qualitatively different credit assignment and compounding-error challenges. The failure modes documented by TT-OPD (teacher-dynamics collapse, trajectory-structure erosion, reward-hint runaway) suggest that naive application of combined KD+RL to multi-turn settings will not work without domain-specific stabilization. The survey's identification of this gap is a strength, but it also means the "convergence of distillation and RL" claim is currently limited to single-turn generation.

Central Claim 4: Self-distillation with privileged information (OPSD, GATES, CRISP) can match or exceed RL-based methods (GRPO) at dramatically lower compute, but is bounded by the model's pre-existing capabilities.

Assessment: Well-supported, with the capability bound precisely characterized by the paper's own evidence. OPSD's result — matching GRPO with 1/8 the rollouts — is compelling and replicated across model scales (though with documented failure at 1.7B where self-rationalization quality is insufficient). CRISP's result — reducing token count by 57–59% while improving accuracy by 9–16 percentage points — is a particularly clean demonstration that self-distillation through privileged context can be highly effective when the privileged information is well-structured (a "be concise" prompt provides a clear, learnable behavioral target).

The survey's characterization of the capability bound — that OPSD "requires sufficient model capacity to produce a meaningful dense token-level signal" and that "when problems exceed a model's comprehension threshold, even the ground-truth-conditioned teacher cannot supply meaningful supervision" — is precisely stated and supported by the 1.7B failure case. The subsequent refinement by Kim & Lee (2026) — that OPSD's productive role is "compression, not correction" (making known solutions more efficient, not solving previously unsolvable problems) — provides an even sharper bound. This honesty about limitations is a strength of the survey's synthesis.

A weakness is that the privileged information paradigm assumes that the PI is reliable. If the ground-truth labels or source documents or "be concise" prompts are themselves noisy — which they will be at scale — the self-distillation signal degrades in ways the survey does not analyze. GATES's consensus gating is a partial mitigation, but the survey does not present a systematic characterization of OPSD's robustness to PI quality. Given that real-world PI (crowd-sourced labels, automatically retrieved documents, imperfect verifiers) is rarely as clean as the curated benchmarks used in these experiments, this is a significant gap between the reported results and deployment reality.

Overall Assessment of the Survey's Evidence Base

The survey's greatest strength is also the source of its main limitation. By synthesizing over one hundred papers, it extracts patterns that are invisible at the individual-paper level: the three-axis taxonomy, the recurring failure modes, the convergence of distillation and RL. However, the synthesis is only as reliable as the comparability of the underlying experiments, and the survey is explicit that "few papers control for all these variables simultaneously" (base model, compute budget, benchmark version, number of rollouts). The Tables (3–9 and 2) report original-paper results rather than controlled reproductions, and the survey appropriately warns readers to "interpret cross-row comparisons with appropriate caution." This is honest, but it means that many of the paper's comparative claims — that method A outperforms method B on task C, that adaptive divergence selection dominates fixed divergences, that combined KD+RL outperforms either alone — are supported by the aggregation of independent experiments rather than by controlled head-to-head comparisons. The absence of a standardized OPD benchmark (analogous to HELM for general LLM evaluation) is a genuine limitation that the survey correctly identifies as an open problem.

A second limitation is that the survey's theoretical framework (the $f$ -divergence unification) is a sufficient but not necessary parameterization. It elegantly subsumes fixed-divergence and adaptive-divergence methods, but the RL-augmented methods (G-OPD, RLKD, RLAD) extend beyond pure divergence minimization by incorporating external reward signals that are not expressible as $f$ -divergences between the student and teacher distributions. The paper handles this by showing that these methods can be viewed as KL-constrained RL with hybrid reward functions, but the mapping is not as clean as the $f$ -divergence mapping for pure distillation methods. The survey's claim that OPD and RL are "endpoints of a continuous spectrum" is conceptually attractive but the mathematical formalization of this spectrum — parameterizing the transition from pure teacher-distribution matching to pure environment-reward maximization — is incomplete. Filling this gap would require a more general divergence-reward framework that reduces to $f$ -divergence minimization when the external reward is zero and to pure RL when the teacher signal is absent. The survey points toward this without formalizing it.

A third limitation is the absence of negative results from papers that tried OPD and found it did not help. The survey is a synthesis of published successes, and the file-drawer problem is acute: papers that find OPD does not improve over off-policy baselines are less likely to be published, and even less likely to be cited. The survey does document specific failure modes (the flawed prefix trap, self-play saturation, length inflation, calibration degradation), but these are failures of specific OPD configurations rather than failures of OPD relative to simpler alternatives. A systematic survey of conditions under which OPD is not worth the compute overhead — informed by industrial experience that may not appear in published papers — would strengthen the framework's practical utility. The survey's decision framework in Section 7.4 provides decision rules ("Use off-policy SFT alone when the teacher-student capacity ratio exceeds 10× and the task admits bounded reasoning depth"), but these rules are based on theoretical predictions and selected industrial reports rather than a systematic negative-result survey.

A fourth limitation concerns the compute-adjusted comparisons. The survey thoroughly documents the compute overhead of OPD (4–5× over off-policy SFT, Section 6.3) and the efficiency methods that reduce it (FOPD, Lightning-OPD, NPD), but it does not systematically re-evaluate headline claims in compute-adjusted terms. When GKD reports that on-policy sampling (λ = 1) outperforms off-policy (λ = 0), is this true at matched wall-clock time, or only at matched generation budget? When OPSD reports matching GRPO with 1/8 the rollouts, does the total compute (including the teacher forward pass for the conditioned version) still favor OPSD? The survey's recommendation to front-load cheap off-policy learning and reserve expensive on-policy compute for the final quality push is sensible but underspecified — the optimal transition point depends on the shape of the learning curves in each regime, which the survey does not characterize quantitatively.

Despite these limitations, the survey's synthesis is largely consistent with the evidence it presents, and the qualifications and boundary conditions are explicitly discussed rather than hidden. The key claims — that on-policy training reduces exposure bias, that divergence selection should be task- and position-dependent, that distillation and RL are converging, that self-distillation is bounded by pre-existing capabilities — are supported by multiple independent replications across different model families and benchmarks. The framework's real test will be whether it predicts the behavior of future methods that combine design choices in novel ways — for example, a method that uses adaptive per-token divergence selection (Section 4.2) with black-box verbal feedback (Section 5.2) stabilized by a competence-boundary curriculum (Section 6.2). The survey provides the conceptual vocabulary for such predictions, but the empirical validation of those predictions lies in future work.

6. Limitations and Trade-offs

Limitation 1: The Survey Cannot Provide Controlled Cross-Method Comparisons — Empirical Headlines Are Aggregated Across Incompatible Experimental Setups

The assumption or constraint. The survey synthesizes over one hundred papers, each evaluated under different experimental conditions. The authors are transparent about this: "Few papers control for all these variables simultaneously, making cross-method comparison unreliable from published numbers alone. Our Tables 3–9 and 2 report original-paper results rather than controlled reproductions, and readers should interpret cross-row comparisons with appropriate caution" (Section 10, Reproducibility challenges). The underlying problem is that methods are evaluated on different base models (Qwen3-1.7B vs. Qwen2.5-7B vs. Llama-3.1-8B), different training compute budgets, different benchmark versions (MATH-500 vs. full MATH), different numbers of rollouts per prompt, and different teacher configurations — and no standardized OPD evaluation protocol exists.

The consequence. The survey's comparative claims — that adaptive divergences outperform fixed divergences, that combined KD+RL outperforms either alone, that self-distillation with privileged information matches GRPO at lower compute — are supported by the aggregation of independent experiments rather than by controlled head-to-head comparisons. This is not a failure of the survey's analysis but a structural limitation of the literature it synthesizes. A practitioner reading that method A achieves 72% on AIME 2024 while method B achieves 68% cannot conclude that method A is superior, because the two numbers were produced under different student models, different teacher models, and different compute budgets. The survey acknowledges this with its call for "standardized OPD benchmarking protocols that fix the base model, compute budget, and evaluation suite, analogous to what HELM (Liang et al., 2023) offers for general LLM evaluation" (Section 10), but the absence of such a protocol means the empirical basis for the survey's prescriptive claims is weaker than it appears from the dense tables of numbers.

What evidence exists in the paper. The problem is visible throughout the experimental summary (Section 5), where cross-method comparisons are qualified by method-specific context (Table 2 includes columns for teacher, student, and benchmark to make this explicit). The paper's own diagnostic checklist (Section 7.1) and decision framework (Section 7.4) acknowledge the context-dependence of OPD effectiveness, but these frameworks are necessarily based on theoretical analysis and selected empirical patterns rather than systematic controlled comparisons across the full method space.

Mitigation status. The survey does not attempt to resolve this — it is a limitation of the underlying literature rather than of the survey methodology. The authors advocate for standardized benchmarking as a future direction (Section 9) but do not provide it. The practical implication is that a practitioner deciding between, say, ToDi and EOPD for adaptive divergence selection cannot consult this survey for a definitive answer — both methods outperform fixed baselines, but their relative merits are unknown because no paper evaluates both under controlled conditions.

Limitation 2: The Survey Documents No Systematic Negative Results — The File-Drawer Problem Is Acute for On-Policy Methods

The assumption or constraint. The survey is a synthesis of published successes. The methods surveyed in Tables 3–9 and 2 are those that authors chose to publish and that the survey authors chose to include — which almost certainly overrepresents positive results relative to the true distribution of OPD outcomes in practice. The survey is candid about the scope: "We exclude generic off-policy KD, compression techniques orthogonal to the training regime such as pruning and quantization, and inference-time methods that leave model weights unchanged" (Section 1). But it does not — and cannot — survey the unpublished experiments where OPD failed to improve over off-policy baselines, or where the 4–5× compute overhead was not justified by the accuracy gain, or where industrial teams tried OPD and reverted to simpler pipelines. The survey's failure-mode analysis (Section 7.2) documents ways that specific OPD configurations can break, but these are failures of particular design choices rather than systematic evidence about when OPD as a paradigm is not worth the cost.

The consequence. The survey's prescriptive claims — "on-policy training tends to outperform off-policy approaches on reasoning tasks" (Section 6.2), "the compute-optimal strategy... yields $4\times$ efficiency gains" (Section 1) — are based on a literature that is selected for positive results. The true frequency with which OPD delivers these gains in practice, and the true frequency with which it degrades performance or wastes compute, is unknown. This matters for the survey's practical utility: a practitioner reading Section 7.4's decision framework ("Use off-policy SFT alone when the teacher-student capacity ratio exceeds 10× and the task admits bounded reasoning depth") cannot assess the confidence of this recommendation because the survey provides no data on how often the recommendation fails.

The closest the survey comes to documenting a systematic negative result is the ReST $^{EM}$ ablation (Section 5.2.3, Appendix K of source material), where an attempt to optimize the revision model using RL-style self-improvement "substantially hurt performance," and ListOPD's extrapolation cliff (Section 7.2) where reward extrapolation beyond a critical $\lambda^*$ causes format collapse. These negative results are informative precisely because they are rare in the surveyed literature — and their rarity likely reflects selection bias rather than the true failure rate of OPD components.

What evidence exists in the paper. The survey itself does not measure this limitation — it is a property of the publication ecosystem. The DeepSeek-R1 anomaly (Section 7.4) — where purely off-policy distillation achieved state-of-the-art reasoning — is documented as evidence that off-policy methods can succeed, but this is not a negative OPD result per se; it is evidence that OPD is not necessary in some regimes. The survey's recommendations (Section 3.3, Table 2) are grounded in "current empirical evidence and hardware constraints" with the explicit caveat that "as compute costs decrease and training frameworks mature, the cost-benefit calculus will shift." But the "current empirical evidence" is almost entirely positive.

Mitigation status. The survey does not attempt to mitigate this limitation — it acknowledges the scope constraint but does not conduct a systematic negative-result survey (which would require access to unpublished industrial experience). The authors' call for standardized benchmarking (Section 9) would partially address this by making it easier to compare methods, but would not directly surface negative results unless the community norms shift to value negative-result publication. Given the structural incentives against publishing failures, this limitation is likely to persist in any survey of the OPD literature.

Limitation 3: The Difficulty Estimation Overhead for Curriculum Methods Is Not Amortized in the Headline Efficiency Claims

The assumption or constraint. Several of the most effective stabilization methods in the survey's taxonomy — particularly the curriculum and difficulty adaptation methods (Section 6.2) — require estimating per-prompt difficulty before training, which itself consumes compute that is not included in the reported efficiency gains. The survey acknowledges this for PACED (Xu et al., 2026a): "PACED estimates per-prompt difficulty $\hat{p}_i$ via a one-shot evaluation phase (typically $K=8$ rollouts per prompt) and weights each prompt by $\hat{p}_i(1 - \hat{p}_i)$ " (Section 6.2). This one-shot evaluation phase generates $K$ rollouts for every prompt in the training set — for a dataset of 10,000 prompts with $K=8$ , this is 80,000 rollouts before any training begins. The survey notes that "the single-pass estimation suffices because the Beta kernel is minimax-robust to stale pass rates," but does not account for this cost in the reported efficiency gains.

The consequence. The headline claims for curriculum methods — PACED's "competence-boundary sampling" that "concentrates the gradient budget on the narrow band of prompts where learning tends to be most efficient," or TCOD's temporal curriculum that "delivers gains of up to +18 points over vanilla multi-turn OPD" (Section 6.2) — are computed after the difficulty estimation phase, without amortizing the cost of estimation. In a realistic deployment setting where no prior estimate of prompt difficulty exists, the total cost is estimation + curriculum training, and the estimation phase can dominate when the training set is large. The survey's own gradient SNR analysis (Section 6.2) provides theoretical justification for why competence-boundary sampling should improve efficiency if difficulty is known, but does not characterize the amortized efficiency when difficulty must be estimated from scratch.

This limitation is structurally analogous to a well-known issue in the test-time compute scaling literature, where difficulty estimation via 2048 samples per question — as in the compute-optimal policy selection of Snell et al. (2024) — consumes more compute than the test-time budget being allocated, making the reported $4\times$ efficiency gains conditional on costless difficulty estimation. The OPD curriculum methods face the same challenge at training time rather than inference time. The $K=8$ rollouts per prompt for PACED is substantially cheaper than 2048 samples per question, but still represents a non-trivial fraction of the total training compute budget — and the survey does not provide the fraction.

What evidence exists in the paper. The survey reports PACED's one-shot evaluation cost ( $K=8$ rollouts per prompt) explicitly (Section 6.2) but does not report total training compute including this phase. For a typical OPD training run with, say, 100 gradient steps and 8 rollouts per step per prompt, the one-shot phase adds approximately 8 extra rollouts per prompt — roughly an 8% overhead if amortized across 100 steps, but a much larger fraction if the number of training steps is small. The survey also notes that "iterative recomputation can yield modest additional gains," implying that the overhead could grow if difficulty is re-estimated during training, but does not quantify the tradeoff.

Mitigation status. The survey does not attempt to amortize difficulty estimation costs in its efficiency analysis. The suggestion that iterative recomputation "can yield modest additional gains" implies that the overhead could grow, but the survey provides no guidance on how to balance estimation frequency against training efficiency. The difficulty estimation problem is closely related to the exploration-exploitation tradeoff that the survey identifies as an open problem in the context of dynamic curriculum design (Section 6.2: "Formulating this schedule as an active learning problem appears to be a promising frontier"), suggesting that the limitation is recognized but unresolved.

Limitation 4: The $f$ -Divergence Framework Does Not Fully Subsume RL-Augmented Objectives — The Mathematical Unification Is Incomplete at the Reward Boundary

The assumption or constraint. The survey's central theoretical contribution is the unified $f$ -divergence framework (Section 2.3, Section 3.4.1), which shows that "core OPD algorithms are instances of $f$ -divergence minimization over student-sampled trajectories" by varying the convex generator $f$ , the sampling mixture $\pi_{\text{mix}}$ , and the argument ordering. This framework elegantly subsumes fixed-divergence methods (GKD, MiniLLM, DistiLLM) and adaptive-divergence methods (ToDi, AKL, EOPD). However, the RL-augmented objectives of Section 4.3 — G-OPD's reward extrapolation, RLKD's Generative Structure Reward Model, RLAD's trust-region ratio distillation — incorporate external reward signals $R(x, y)$ that are not expressible as $f$ -divergences between the student and teacher distributions. The survey handles this by recasting these methods as KL-constrained RL with hybrid reward functions (Equation 14: $\max_\theta \mathbb{E}_{y \sim p_\theta}[\sum_t \alpha \log(p_T/p_{\text{ref}}) - D_{KL}(p_\theta \parallel p_{\text{ref}})]$ ), but this requires augmenting the pure divergence framework with an external reward term that lies outside it.

The consequence. The survey's claim that OPD and RL are "endpoints of a continuous spectrum parameterized by the density of supervision, the choice of divergence, and the source of the training signal" (Section 7.3) is conceptually attractive but mathematically incomplete. The spectrum is not continuous in a rigorous sense because the transition from pure divergence minimization (where the objective is $D_f(P_T \parallel P_\theta)$ for some $f$ ) to reward-augmented optimization (where the objective includes a term that is not a function of the teacher-student divergence) involves a discrete change in the mathematical structure of the optimization problem. The survey provides no parameterization that continuously interpolates between these regimes — for example, a single loss function $\mathcal{L}_\beta$ that reduces to $D_f$ when $\beta = 0$ and to reward-maximization when $\beta \to \infty$ , with the intermediate regime corresponding to the hybrid methods surveyed. The G-OPD parameter $\alpha$ provides a limited form of interpolation (Equation 14: $\alpha = 1$ recovers standard Reverse KL distillation, $\alpha > 1$ pushes beyond the teacher), but it applies specifically to Reverse KL with a reference policy and does not generalize to other divergences or reward structures.

This matters for the survey's prescriptive claims. If the transition between distillation and RL involves a structural change in the objective rather than a smooth interpolation, then insights from one regime do not automatically transfer to the other — contrary to the survey's claim that "advances in KL-constrained RL (better trust regions, adaptive penalty coefficients, variance reduction) are therefore likely to transfer to OPD, and vice versa" (Section 4.3). Some advances may transfer; others may not; the survey provides no criterion for distinguishing the two cases.

What evidence exists in the paper. The survey demonstrates the close relationship between distillation and RL through several equivalences: MiniLLM's sequence-level Reverse KL is mathematically equivalent to policy gradient RL with the teacher's log-probability as reward (Equation 11), G-OPD's formulation recasts standard OPD as KL-constrained RL (Equation 14), and DPO-based methods target the same geometric mixture as GKD's $\lambda$ -mixing (Section 4.3). These equivalences are exact and well-documented. However, they are special cases — they hold for specific divergence choices (Reverse KL for MiniLLM and G-OPD, implicit reward for DPO) and specific reward structures (teacher log-probability as dense reward). They do not constitute a general mapping between arbitrary $f$ -divergences and arbitrary reward functions, which is what the "continuous spectrum" claim implies.

Mitigation status. The survey does not fully resolve this gap. The unified KD+RL framework of Li et al. (2025) — which decomposes the hybrid gradient into a dense KD component and a Monte Carlo RL component — is cited as evidence that the two signals are complementary, but this decomposition is additive rather than interpolative: the total gradient is a weighted sum of the two components, not a smooth function of a single parameter that transitions between them. The survey's discussion of open problems (Section 9) identifies "a rigorous theoretical framework for this allocation [between imitating a teacher and exploring beyond it]... Formulating this schedule as an active learning problem appears to be a promising frontier," indirectly acknowledging that the current framework does not fully characterize the spectrum.

Limitation 5: The Survey's Empirical Coverage Is Concentrated on Reasoning Benchmarks — Transferability to Other Domains Is Largely Asserted Rather Than Demonstrated

The assumption or constraint. The survey covers over one hundred methods, but the dominant evaluation benchmarks across these methods are mathematics and reasoning tasks: MATH-500, AIME, HMMT, GSM8K, LiveCodeBench, Minerva (Section 5.1). The survey acknowledges this concentration implicitly through its organization — the discussion of domain-specific applications (Section 8.2: multimodal, embodied, medical, protein design) occupies a separate section from the main experimental analysis and draws on a much smaller set of papers. The core methods are validated primarily on symbolic reasoning, where correctness is binary and verifiable, and where exposure bias — the $O(\epsilon T^2)$ compounding that motivates OPD — is particularly severe because early errors cascade logically.

The consequence. The survey's central claims — that on-policy training reduces exposure bias, that Reverse KL excels on reasoning tasks, that combined KD+RL outperforms either alone — are well-supported for mathematical reasoning but have substantially weaker evidence for the broader class of tasks where LLMs are deployed: open-ended dialogue, creative writing, factual recall, translation, summarization, and domain-specific applications (legal, medical, financial). The survey's own method selection considerations (Section 3.3) note that "reasoning and mathematical tasks benefit from mode-seeking objectives" while "open-ended generation tasks instead require mode-covering objectives," but this distinction is based on task geometry analysis rather than extensive empirical evidence in the open-ended regime — most methods were not evaluated on creative writing or dialogue benchmarks.

The problem is more acute for the survey's claims about multi-turn and agentic tasks (Section 8.1, Section 7.2). The survey accurately documents that multi-turn agentic OPD introduces qualitatively new failure modes (teacher-dynamics collapse, trajectory-structure erosion, reward-hint runaway in TT-OPD; compounding multi-turn instability in SDAR), and that the existing methods (TCOD, MAD-OPD, Skill-SD, TT-OPD) are early-stage contributions validated on narrow agentic benchmarks (ALFWorld, WebShop, ScienceWorld, Healthcare AI Gym). Extrapolating the survey's framework from these controlled environments to production agentic systems — which may involve dozens of tools, non-stationary environments, and safety-critical actions — is speculative. The survey is careful not to overclaim here (the discussion of agentic distillation acknowledges that "three unsolved sub-problems define this frontier"), but the organization of the survey — where agentic methods are presented as extensions of the core single-turn framework — can create the impression that the framework generalizes more cleanly than the evidence currently supports.

What evidence exists in the paper. The survey documents domain-specific evaluations in Section 8.2 and Table 2, but these are sparse relative to the volume of reasoning-benchmark results. For multimodal OPD (VOLD, CORD, KEPO, VISD, HyperEyes), the evaluations are on MMMU-Pro, MathVista, LogicVista, medical VQA, and audio reasoning — a diverse but small set of benchmarks compared to the extensive MATH/AIME evaluations that dominate Sections 4–6. For embodied intelligence (HY-Embodied-0.5, OPD-AV, VLA-OPD), the survey cites results on nuScenes, spatial reasoning, and robot control — each with a single representative paper. For protein design (ProteinOPD) and hardware verification (RWOPD), the survey documents exactly one paper each. The evidence that OPD principles transfer across these domains is existence-based (one paper shows it works) rather than systematic (multiple independent replications across different tasks within the same domain).

Mitigation status. The survey is transparent about this limitation through its organization — the core methods (Sections 4–6) are presented as validated on reasoning and instruction-following, while domain extensions (Section 8.2) are presented as emerging applications. The survey's conclusion (Section 10) notes that "the progression from off-policy imitation to on-policy self-correction echoes a broader pattern in machine learning," but this claim about broader applicability is aspirational rather than empirically grounded. The open problems section (Section 9) identifies cross-modal distillation, agent-level distillation, and continual learning as frontiers where the current evidence is thin, implicitly acknowledging the domain-concentration limitation.

Limitation 6: The Survey Does Not Characterize the Latency Wall-Clock Tradeoff — Sequential On-Policy Generation Is Inherently Serial, Making Headline Accuracy Gains Potentially Impractical for Interactive Applications

The assumption or constraint. The survey measures compute in terms of FLOPs, GPU-hours, and generation budgets (number of rollouts per prompt), and evaluates efficiency gains in these terms — FOPD's prefix truncation reduces training FLOP by 2–47×, Lightning-OPD's offline caching achieves 4.0× training efficiency, NPD's async decoupling delivers 8.1× throughput speedup (Section 6.3). However, throughput is not latency. Sequential on-policy methods — anything that requires autoregressive student generation before teacher scoring before gradient update — are inherently serial: the student must generate token by token, the teacher must score the full sequence (or per-token logits must be cached), and only then can the gradient be computed and applied. This serial dependency means that wall-clock time per training step is dominated by the student's autoregressive generation latency, regardless of how efficiently the FLOPs are utilized.

The consequence. The survey evaluates methods primarily by their sample efficiency (how many rollouts are needed to reach a given accuracy) and FLOP efficiency (how much computation is required per rollout), but makes no mention of latency — the wall-clock time required for a single training step. This is a significant practical limitation for two deployment scenarios that the survey otherwise targets.

First, interactive or online learning: If OPD is used to continuously improve a deployed model based on user interactions (the online experiential learning paradigm of OEL, Section 5.3.1), the latency of autoregressive student generation followed by teacher scoring determines how quickly the model can adapt to new data. A method that achieves 4× better sample efficiency but requires 10× longer per sample due to sequential dependencies may be impractical for time-sensitive applications.

Second, rapid experimentation: The survey advocates for OPD as a research framework (Section 1: "the resulting literature has expanded... to over one hundred papers"), but the serial nature of on-policy training makes it substantially slower to iterate on than off-policy SFT, where the dataset is pre-generated and training steps are independent. This creates a barrier to entry for academic researchers with limited GPU access, undermining the survey's goal of democratizing OPD.

The survey's discussion of compute optimization (Section 6.3) focuses on throughput (tokens processed per second) rather than latency. NPD's asynchronous pipeline, for instance, decouples generation and training to improve throughput, but the generation step itself remains sequential — the student must still produce autoregressive rollouts before the training pipeline can consume them. The latency floor is set by the maximum sequence length in the batch, and the survey provides no analysis of how this scales with model size, sequence length, or batch size.

What evidence exists in the paper. The survey documents the cost breakdown for a typical OPD step (Section 6.3): student rollout (autoregressive, memory-bound), teacher scoring (compute-bound), student update (compute-bound). It notes that "this component [student rollout] typically dominates wall-clock time, since generation is sequential and cannot be trivially parallelized across tokens." However, this observation is not integrated into the efficiency analysis of individual methods — the survey reports FOPD's 2–47× FLOP reduction without noting that truncating to a prefix of length $k$ also reduces latency by the same factor (since autoregressive generation cost scales linearly with sequence length), and reports NPD's 8.1× throughput speedup without translating this into wall-clock time savings for a typical training run. The latency-throughput distinction — well-understood in the inference serving literature (vLLM, continuous batching) — is absent from the survey's efficiency framework.

Mitigation status. The survey does not attempt to characterize latency. The compute-optimization methods (FOPD, Lightning-OPD, NPD, Prune-OPD) are evaluated by FLOPs reduction or throughput improvement without wall-clock time measurements. The survey's decision framework (Section 7.4) and method selection considerations (Section 3.3) evaluate methods by "compute budget" without distinguishing between latency-constrained and throughput-constrained deployment scenarios. Given that several of the survey's recommended methods — sequential revisions, competence-boundary curricula that require difficulty estimation, multi-turn agentic OPD — involve serial dependencies that amplify latency, this omission is a genuine gap in the survey's practical guidance.

7. Implications and Future Directions

How This Work Changes the Landscape

This survey does not introduce a new algorithm, but it performs a more foundational intervention: it reorganizes a fragmented literature into a coherent design space, and in doing so, resolves contradictions that have persisted across three separate research communities. The effect is to shift the field's self-understanding from a collection of competing methods toward a principled engineering discipline with shared diagnostic tools and predictable failure modes.

The magnitude of this shift is best characterized as a conceptual unification, not a paradigm shift. The underlying algorithms existed before this survey — GKD and MiniLLM were published in 2023–2024, G-OPD and OPSD in 2025–2026. What the survey provides is the common vocabulary and organizational structure that allows researchers in the knowledge distillation community to understand why methods from the RLHF community work on their problems, and vice versa. The three-axis taxonomy — Objective (what to optimize), Signal (where the signal comes from), Dynamics (how to stabilize training) — is not mathematically deep in itself, but it is operatively powerful because it maps the design choices that practitioners must make onto a navigable space, replacing trial-and-error across method families with principled reasoning about tradeoffs.

The contradictions this work resolves are significant and long-standing. The survey documents how Huang et al. (2023) found that "LLMs cannot self-correct reasoning" while Madaan et al. (2023) found self-refinement helpful — and explains this not as a failure of one study's methodology but as a difficulty-dependent interaction that the on-policy framework makes explicit. Self-distillation works when the model can generate high-quality self-training targets (easy-to-medium problems, sufficiently capable base models) and fails when it cannot (hard problems, small models). This same difficulty-dependent pattern recurs across the survey's analysis: reverse KL's mode-seeking behavior helps on reasoning tasks where precision matters but hurts on creative tasks where diversity matters; beam search helps on medium-difficulty problems but over-optimizes the verifier on easy ones; sequential revisions help on easy problems but must be balanced with parallel exploration on hard ones. The survey unifies these observations under a single explanatory principle: the effectiveness of any on-policy strategy depends on the gap between the student's current capability and the task's difficulty, and the optimal strategy shifts as this gap changes.

The landscape-level consequences of this unification are threefold. First, it makes certain research directions more attractive. Improving verifier robustness becomes a clear priority because the survey identifies verifier over-optimization as the primary bottleneck limiting on-policy scaling (Sections 5.3 and 7.2). Developing cheap difficulty estimation becomes urgent because the curriculum methods that stabilize OPD (PACED, TCOD, Uni-OPD) all require it, and the current approach (multiple rollouts per prompt for pass-rate estimation) is expensive. Bridging the mathematical gap between pure divergence minimization and reward-augmented optimization becomes a well-defined theoretical problem because the survey has precisely located the boundary where its $f$ -divergence framework stops cleanly applying.

Second, it makes certain research directions less attractive. The survey documents that more sophisticated search algorithms — lookahead search, Monte Carlo Tree Search applied to PRM-guided distillation — have shown diminishing or even negative returns compared to simpler methods (Section 5.3, Appendix M). The over-optimization phenomenon documented across multiple independent studies suggests that further investment in search-algorithm sophistication, without corresponding investment in verifier robustness, is likely to yield marginal gains at best. Similarly, the survey's documentation of self-play saturation (the Ouroboros problem, Section 7.2) — where self-distillation without external grounding collapses onto the model's own prior — suggests that pure self-distillation without either privileged information or external verification is fundamentally bounded, and that research effort should shift toward hybrid approaches (OPSD, SD-ZERO, π-Play) rather than trying to push pure self-distillation beyond this bound.

Third, it recasts the relationship between distillation and RL from a sequential pipeline (distill first, then RL) to a joint optimization problem. The convergence documented across G-OPD, KDRL, RLAD, REOPOLD, and the Sparse-to-Dense pipeline (Section 4.3) indicates that the field is moving toward training regimes where dense teacher supervision and sparse outcome rewards operate simultaneously rather than in separate stages. This is not merely a scheduling change — it implies that the infrastructure for distillation and RL (rollout generation, teacher scoring, gradient computation) should be co-designed rather than treated as separate systems, and the survey's analysis of DeepSeek-V4's full-vocabulary OPD infrastructure (Section 8.3) provides a concrete template for this integration at industrial scale.

Follow-Up Research This Work Enables

A standardized OPD benchmark with matched compute budgets. The survey's most forceful methodological critique is that "few papers control for all these variables simultaneously, making cross-method comparison unreliable from published numbers alone" (Section 10). A standardized benchmark — analogous to HELM for general LLM evaluation — would fix the base model (e.g., Qwen3-4B), teacher model (e.g., Qwen3-14B or a black-box API teacher), training compute budget (matched wall-clock time or GPU-hours, not just matched generation counts), and evaluation suite (MATH-500, AIME 2024/2025, LiveCodeBench, and at least one open-ended generation benchmark like AlpacaEval). The survey's Tables 3–9 and 2 provide a natural starting point for method selection: the benchmark would evaluate at least one representative method from each of the three design axes — GKD (fixed divergence), ToDi or EOPD (adaptive divergence), G-OPD (RL-augmented), GAD (black-box), OPSD (self-distillation with PI), and PACED (curriculum) — under their reported hyperparameters, allowing the first controlled head-to-head comparison across the full design space. The key measurement would not be just accuracy but accuracy per FLOP, distinguishing methods that genuinely improve the Pareto frontier from those that achieve gains through increased compute. This would directly address the survey's limitation that "the synthesis is only as reliable as the comparability of the underlying experiments."

Theory for the divergence-reward boundary: a continuous parameterization from pure distillation to pure RL. The survey identifies but does not resolve the mathematical gap between its $f$ -divergence framework (which cleanly subsumes fixed and adaptive divergence methods) and the RL-augmented methods that incorporate external reward signals not expressible as $f$ -divergences (Section 4.3, Limitation 4 in Section 6). A concrete follow-up would develop a single loss function $\mathcal{L}_\beta(\theta) = (1-\beta) \cdot \mathbb{E}_{y \sim p_\theta}[D_f(P_T \parallel P_\theta)] + \beta \cdot \mathbb{E}_{y \sim p_\theta}[-\log \pi_{\text{RL}}(y)]$ where $\beta \in [0,1]$ continuously interpolates between divergence minimization ( $\beta = 0$ ) and reward-maximizing policy optimization ( $\beta = 1$ ), and where $\pi_{\text{RL}}$ is the optimal policy under the environment reward. The key theoretical question is whether this interpolation preserves desirable properties (convergence guarantees, variance bounds) at intermediate $\beta$ , or whether there is a phase transition at some critical $\beta^*$ where the optimization landscape changes qualitatively — analogous to the extrapolation cliff documented in ListOPD for reward-extrapolation OPD. The survey's existing results provide boundary conditions: at $\beta = 0$ , GKD's divergence-agnostic framework applies; at $\beta = 1$ , standard GRPO applies; the intermediate regime is where combined KD+RL methods (G-OPD, KDRL, RLAD) operate, but without a formal characterization of how the gradient decomposes as a function of $\beta$ . A strong follow-up would measure the gradient variance, policy drift, and final accuracy as $\beta$ is swept from 0 to 1 on MATH-500 and AIME 2024, identifying whether the combined regime ( $0 < \beta < 1$ ) genuinely outperforms either extreme at matched compute, or whether the apparent gains from combination are an artifact of comparing across different papers with different experimental setups.

Characterizing the difficulty-dependent frontier: a controlled experiment varying problem difficulty while holding all else constant. The survey repeatedly invokes the DAgger theorem's prediction that the on-policy/off-policy gap scales as $O(\epsilon T^2) \to O(\epsilon T)$ , and uses this to explain why OPD's benefits are most pronounced on long reasoning chains (Section 2.2, Section 7.1). Yet no controlled experiment in the surveyed literature varies sequence length $T$ systematically while holding the base model, teacher, and compute budget constant, and measures the on-policy vs. off-policy accuracy gap as a function of $T$ . A direct test would use a dataset where problem difficulty is strongly correlated with required reasoning depth — for example, the MATH benchmark's difficulty levels (Level 1 through Level 5) as a proxy for $T$ , or a synthetic dataset where reasoning chains of controlled length are generated by templating. The student (e.g., Qwen3-4B) would be trained via both off-policy SFT on teacher-generated traces and on-policy GKD with matched generation budget, and the accuracy gap between the two would be plotted against difficulty level. The DAgger prediction is that the gap should grow superlinearly with $T$ (specifically, the ratio of on-policy to off-policy accuracy should increase with $T$ ). A null result — where the gap is constant or grows only linearly — would substantially weaken the theoretical motivation for OPD and suggest that exposure bias is not the dominant failure mode in current models. A positive result would quantify the $T$ -dependence for the first time, allowing practitioners to predict when OPD is worth its compute overhead based on the expected reasoning depth of their target task. This experiment would also test the survey's "structural weakness that grows more severe as tasks become longer" claim (Section 1) directly, rather than inferring it from cross-paper comparisons.

Failure-mode diagnostics as a training-time monitoring toolkit. The survey catalogs specific failure modes — the flawed prefix trap (teacher feedback on erroneous prefixes is unreliable), self-play saturation (self-distillation collapses onto the model's prior), length inflation (outputs grow progressively longer due to self-amplifying reverse-KL advantages), and the calibration-capability gap (models become more accurate but less aware of their uncertainty) — but each is diagnosed post-hoc, typically after training completes and benchmark evaluation reveals degradation. A concrete follow-up would develop real-time diagnostic probes that track these failure modes during training without requiring full evaluation runs. For the flawed prefix trap, a probe would compute the running average of $D_{KL}(P_T(\cdot|\hat{y}_{<t}) \parallel P_T(\cdot|y_{<t}^{\text{ground-truth}}))$ — the divergence between the teacher's predictions on student-generated vs. ground-truth prefixes — and raise a warning when this exceeds a threshold, indicating that the student is entering regions where teacher feedback is unreliable. For self-play saturation, a probe would track the entropy of $p_\theta(\cdot|x, \hat{y}_{<t})$ at decision-point tokens (identified by high teacher-student divergence), flagging systematic entropy decline as evidence of mode collapse. For length inflation, a probe would monitor mean sequence length as a function of training steps and compare against the length distribution under the reference policy. These probes could be implemented as lightweight callbacks in existing OPD frameworks (OpenRLHF, veRL) and would shift debugging from "accuracy dropped, what went wrong?" to "probe 3 is signaling divergence at step 45,000 — intervene now." The survey's diagnostic checklist (Section 7.1) provides the initial set of metrics; building the probes and validating them against known failure cases from the surveyed literature would be a directly actionable engineering contribution.

Multi-turn agentic OPD with granularity-matched credit assignment across decision boundaries. The survey identifies multi-turn agentic distillation as an emerging frontier and observes that "coarse-grained approaches (full-trajectory OPD) tend to underperform because compounding is local" while "fine-grained approaches (token-level) waste compute on positions where the agent's behavior is already correct" (Section 8.1). The natural extension is to develop methods that automatically detect decision boundaries — the points in a trajectory where the agent commits to an action whose consequences are difficult to reverse — and concentrate distillation supervision at exactly those points. This is not merely an engineering optimization; it is a test of the survey's "granularity principle" (Innovation 4, Section 4) that effective distillation requires matching the supervision granularity to the error-compounding scale. A concrete experiment would extend TCOD's temporal curriculum (which operates at the turn level) and SOD's step-level reweighting (which operates at tool-call boundaries) to a setting where decision boundaries are not prespecified by the environment — for example, a web-browsing agent where some clicks are reversible (scrolling) and others are not (submitting a form, making a purchase). The method would learn to detect decision boundaries from the trajectory's divergence profile (similar to the prefix-drift detection in Prune-OPD) and apply denser teacher supervision at those points, then measure whether this adaptive-granularity approach outperforms both uniform token-level and uniform trajectory-level baselines on agentic benchmarks (WebShop, ALFWorld, SWE-bench). A negative result — where adaptive granularity does not outperform simpler fixed-granularity approaches — would be informative because it would bound the practical value of the granularity principle that the survey extracts from the literature.

Practical Applications and Downstream Use Cases

Cost-efficient distillation from proprietary API models. The survey's documentation of black-box methods (GAD, OVD, ROPD, Section 5.2) enables a deployment scenario that is immediately actionable: organizations that have API access to frontier models (GPT-5, Claude, Gemini) but not white-box logit access can use on-policy distillation to transfer reasoning capabilities into smaller, privately deployed students. The specific benefit is quantified by ROPD's result: on AIME 2025 with thinking mode, a Qwen3-4B student distilled from a GPT-5.2 teacher via rubric-based semantic distillation achieved 68.75% accuracy, surpassing the teacher's own 67.08%, with approximately 10× sample efficiency relative to logit-based methods (Section 5.2, Table 7). This means that a team with API access and a modest GPU budget (~8×H100s for the student) can produce a 4B model that matches or exceeds the frontier API model on competition-level mathematics — and then deploy it privately without per-query API costs, latency, or data exposure concerns. The survey's method selection framework (Section 3.3) provides the decision logic: with API-only access, prefer ROPD or OVD for reasoning tasks where structured feedback (rubrics, verbal scores) captures most of the teacher's knowledge, and prefer GAD or ORPO-Distill for preference-based tasks where pairwise comparisons are more informative than absolute scores.

Self-improving on-device models through privileged-information self-distillation. The survey's analysis of OPSD and its variants (Section 5.3.1) enables a deployment scenario for consumer hardware: a small model (1.7B–4B parameters) that improves its reasoning capability through self-distillation using privileged information available at training time but not at inference. The specific mechanism: during an offline "improvement phase," the model conditions on ground-truth answers (for math), source documents (for QA), or a "be concise" prompt (for reasoning compression) to generate a stronger self-teacher distribution, then distills this into its unconditioned policy via on-policy KL. At deployment, the model runs the unconditioned policy — no privileged information needed. OPSD's result that a single rollout per problem matches GRPO's eight rollouts (Section 5.3.1, Table 8) translates directly to a practical workflow: a developer ships a base model, runs an overnight self-distillation job on a curated dataset with ground-truth labels (which the developer controls since they define the task), and deploys the improved model the next morning. The compute cost is dramatically lower than full RL-based self-improvement (roughly 1/8 the rollouts, no reward model training, no environment interaction), making this feasible on hardware available to individual developers rather than only large organizations. The survey's documentation of OPSD's capability bound — it requires "sufficient model capacity to produce a meaningful dense token-level signal" and fails at 1.7B scale (Section 5.3.1) — provides a clear guideline: this approach is appropriate for 4B+ models, and smaller models should use an external teacher rather than self-distillation.

Training data generation for self-improvement pipelines at the competence frontier. The survey's analysis of curriculum methods (PACED, Section 6.2) combined with self-distillation (SSD, Section 5.3.2) enables a data generation strategy for organizations building self-improving models. The core insight is that on-policy generation at the model's competence boundary produces the highest-quality training data because it exposes the model to problems it can almost but not quite solve — exactly the problems where corrective feedback is most informative. The specific recipe: estimate per-prompt difficulty using PACED's one-shot evaluation (K=8 rollouts per prompt, Section 6.2), weight prompts by $p(1-p)$ to concentrate on the competence frontier, generate on-policy rollouts on these frontier prompts, select high-quality completions via outcome verification or self-consistency (as in OPSFT, Section 5.3.2), and fine-tune on the filtered data. The survey's evidence that SSD alone — sampling at training-time temperature with no verifier, no RL, no external teacher — improved Qwen3-30B-Instruct from 42.4% to 55.3% on LiveCodeBench v6 (Section 5.3.2) suggests that even this minimalist recipe yields substantial gains when the base model is sufficiently capable. Adding PACED's frontier sampling would concentrate the gradient budget on the problems where on-policy generation produces the most informative training targets, potentially improving the efficiency of the generate-filter-train loop by a factor proportional to the fraction of prompts at the competence boundary.

Verifier development as a targeted research investment. The survey's documentation of verifier over-optimization — beam search degrading performance on easy problems at high budgets (Section 5.2.3), lookahead search paradoxically underperforming simpler methods, and the calibration-capability gap (CaOPD, Section 7.2) — has a direct organizational implication: teams building OPD pipelines should invest more heavily in verifier robustness than in search algorithm sophistication. The evidence is that even the best search algorithms are fundamentally bounded by verifier quality, and that improving the verifier (through Monte Carlo rollout training, adversarial robustness, or ensemble methods) is likely to yield larger gains than developing more complex search strategies. The survey's documentation of DeepSeek-V4's full-vocabulary Reverse KL with hidden-state caching (Section 8.3) provides a concrete template for training high-quality verifiers at scale without prohibitive memory overhead, and the PRM training procedure described in the source material (Monte Carlo rollout supervision with soft labels, Section 5.1 of the compute-optimal scaling paper) provides a human-label-free recipe that practitioners can adopt immediately. The practical implication is that organizations with limited research budgets should allocate them disproportionately to verifier development — particularly to ensuring that verifiers remain calibrated on the student's own generation distribution rather than on static evaluation sets — rather than to the search strategies that consume verifier outputs.

When to Prefer This Method

The survey does not propose a single method to prefer over alternatives; it provides a framework for choosing among methods based on deployment constraints. However, the survey's decision framework (Section 7.4) and method selection considerations (Section 3.3) articulate a clear tradeoff along four dimensions that governs when on-policy distillation should be preferred over off-policy alternatives, and which OPD variant to choose within the on-policy regime. These tradeoffs are grounded in the survey's empirical synthesis and theoretical analysis, not generic boilerplate.

Teacher-student capability gap. When the teacher is massively more capable than the student (capacity ratio >10×, as in DeepSeek-R1 distilling into 1.5B–70B students), off-policy SFT often suffices because the teacher's static traces are diverse enough to approximate coverage of the student's generation space. The survey documents this as the "DeepSeek-R1 anomaly" (Section 7.4): purely off-policy distillation from a 671B MoE teacher into much smaller students achieved state-of-the-art reasoning, likely because the teacher's traces were so diverse and the student's capacity was sufficiently bounded that distributional mismatch was manageable. When the capacity ratio is more moderate (2–5×), on-policy training becomes valuable because the student explores regions that the teacher's static traces do not cover, and training on those regions provides corrective feedback unavailable off-policy. When the teacher and student are closely matched in capability (same-recipe models, or self-distillation), on-policy methods are essential because there is no external distributional signal to learn from — the model must generate its own improvement signal through rollout diversity, privileged information, or external verification.

Reasoning depth and error cascading. The survey's DAgger-based analysis (Section 2.2) predicts that the on-policy/off-policy gap grows superlinearly with sequence length because of $O(\epsilon T^2)$ vs. $O(\epsilon T)$ compounding. This translates to a concrete decision rule: for tasks where the required reasoning chain is short (factual QA, translation, single-step classification — typical sequence length <100 tokens), the exposure bias is small and off-policy SFT is usually sufficient. For tasks requiring multi-step reasoning (mathematical proofs, code generation, multi-hop QA — typical sequence length 500–2000+ tokens in thinking mode), on-policy training becomes increasingly important as chain length grows. The survey does not provide a precise threshold (the controlled $T$ -dependence experiment described in the follow-up research section above would supply this), but the existing results show clear patterns: methods evaluated on MATH and AIME (multi-step reasoning) consistently show larger on-policy gains than methods evaluated on translation and summarization (shorter chains, less error cascading).

Signal access level. When full white-box logit access is available (teacher and student co-located, same organization), the survey recommends token-level methods with adaptive divergence selection (ToDi, EOPD, or even simple entropy-gated Forward KL) for most tasks, because the dense per-token signal and principled divergence framework provide strong theoretical guarantees and empirically competitive results. When only API access is available, the survey recommends black-box methods that extract rich proxy signals: ROPD for tasks where structured evaluation rubrics can be automatically generated, OVD for tasks where scalar quality scores are informative, and GAD for tasks where adversarial discrimination can recover distributional information from text alone. When no external teacher is available, the survey recommends self-distillation with privileged information (OPSD, GATES) when training-time access to ground truth or source documents is possible, and self-distillation with external verification (SD-ZERO, SRPO) when only binary or structured feedback is available. Pure self-distillation without any external grounding (SSD, UniSD) is appropriate only when the base model is sufficiently capable that its own temperature-sampled distribution contains high-quality targets — the survey's evidence (SSD improving Qwen3-30B-Instruct by ~13 points on LiveCodeBench) suggests this is viable at the 30B+ scale but less reliable for smaller models.

Compute budget and latency tolerance. The survey documents that on-policy training costs 4–5× more than off-policy SFT per token (Section 6.3), and that the efficiency methods (FOPD, Lightning-OPD, NPD) can reduce this overhead to near parity in some regimes. The practical decision rule is: if the total training compute budget is under ~500 GPU-hours, use off-policy SFT with the largest feasible teacher — the on-policy overhead is unlikely to be justified at this scale. If the budget is 500–5000 GPU-hours, use a hybrid pipeline: off-policy warmup for the majority of the budget, transitioning to on-policy refinement (GKD with λ ≈ 0.5, or FOPD with prefix truncation) for the final phase. If the budget exceeds 5000 GPU-hours, full on-policy training with adaptive divergences and curriculum stabilization becomes feasible and is likely to yield meaningful accuracy gains over off-policy alternatives, particularly on reasoning tasks. For latency-sensitive applications (interactive assistants, real-time systems), prefer parallel on-policy methods (best-of-N weighted, parallel sampling) over sequential methods (revision chains, multi-turn OPD) because the serial dependencies in sequential methods increase wall-clock time regardless of total FLOPs — a tradeoff the survey documents but does not quantify (Limitation 6, Section 6).

Dependency tree

Papers this one builds on, walking backwards through the same domain and shared topics.

Understanding the Challenges in Iterative Generative Optimization with LLMs2026· training-methods← in-domain predecessor
- Deep Research: A Systematic Survey2025· uncategorized← LLM-judged similar
- SpatialBoost: Enhancing Visual Representation through Language-Guided Reasoning2026· training-methods← in-domain predecessor
DeepSpeed: System Optimizations Enable Training Deep Learning Models with Over 100 Billion Parameters2020· training-methods← similar topic
- BERT-of-Theseus: Compressing BERT by Progressive Module Replacing2020· training-methods← in-domain predecessor
- MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers2020· distributed-training← topic thread

What it influenced

Later papers in the same chain. Direction reversed.

SandMLE: A Multi-Agent Framework that Generates Synthetic Sandbox Environments for Training Machine Learning Engineering Agents2026· training-methods→ in-domain predecessor
- RAGEN-2: Reasoning Collapse in Agentic RL2026· training-methods→ in-domain predecessor
- Map-Reduce for Machine Learning on Multicore2000· training-methods→ similar topic
- Large-Scale Machine Learning with Stochastic Gradient Descent2000· training-methods→ similar topic
- Support Vector Machine Learning for Interdependent and Structured Output Spaces2000· training-methods→ similar topic
DeepSpeed: System Optimizations Enable Training Deep Learning Models with Over 100 Billion Parameters2020· training-methods→ similar topic
- Neural Representations in Hybrid Recommender Systems: Prediction versus Regularization2020· training-methods→ in-domain predecessor
- Training data-efficient image transformers & distillation through attention2020· vision→ topic thread
- Black-Box On-Policy Distillation of Large Language Models2025· training-methods→ similar topic
On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes2023· training-methods→ similar topic
- Democratizing Reasoning Ability: Tailored Learning from Large Language Model2023· reasoning→ LLM-judged similar
- Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problems2024· training-methods→ LLM-judged similar
- Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe2026· training-methods→ LLM-judged similar
- Set Learning for Accurate and Calibrated Models2023· training-methods→ in-domain predecessor
- RLCD: Reinforcement Learning from Contrast Distillation for Language Model Alignment2023· rl-training→ topic thread
- Robust Preference Learning for Storytelling via Contrastive Reinforcement Learning2022· training-methods→ similar topic
- Learning to Foresee: Unveiling the Unlocking Efficiency of On-Policy Distillation2026· training-methods→ similar topic