Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillation

ArXiv: 2604.13010

🎯 Pitch

On-policy distillation can be done completely offline—and actually works—as long as you use the same teacher for both SFT and distillation. Violating this "teacher consistency" was the hidden reason previous offline attempts failed. Lightning OPD exploits this insight to match standard OPD's accuracy while slashing training cost by 4×, even scaling to MoE models on a single node.

---

1. Executive Summary

This paper proposes Lightning OPD, an offline on-policy distillation framework that eliminates the need for a live teacher server during post-training by precomputing teacher log-probabilities once over SFT rollouts and reusing them throughout training. The central insight is the identification of teacher consistency as a previously overlooked condition for effective on-policy distillation — requiring that the same teacher model be used for both the SFT stage (generating training trajectories) and the OPD stage (providing the reference distribution) — and the paper proves that violating this condition introduces an irreducible gradient bias that degrades both offline and online OPD. Evaluated on math reasoning (AIME 2024/2025, HMMT 2025) and code generation (LiveCodeBench v5/v6) using Qwen3-4B-Base and Qwen3-8B-Base students with Qwen3-8B and Qwen3-32B teachers, Lightning OPD matches standard OPD performance while delivering 4.0× higher training efficiency at the 8B scale (reaching 69.9% on AIME 2024 in 30 GPU hours versus 120 GPU hours for standard OPD), and further scales to a 30B MoE architecture on a single 8×H100 node where standard OPD runs out of memory. The paper establishes that on-policy distillation's benefits can be preserved without a live teacher server, but only when the SFT and OPD teachers are identical — a design principle shown to be necessary for both paradigms to converge to the same optimum.

2. Context and Motivation

The Core Problem: Standard OPD Requires a Costly Live Teacher Server

The fundamental problem this paper tackles is the infrastructure bottleneck in on-policy distillation (OPD). OPD is an effective post-training paradigm for LLMs: a student model generates responses, and a stronger teacher model scores every token of those responses to provide dense, per-token supervision. This dense signal — richer than the sparse reward in standard RLVR (Reinforcement Learning from Verifiable Rewards) — gives OPD greater training stability, lower cost relative to RL, and competitive or superior reasoning performance across math, code, and planning tasks.

But this benefit comes at a steep operational price. In standard OPD, the teacher must score the student's current rollouts at every gradient step. Since the student's policy changes at each step, the rollouts are fresh (on-policy), and the teacher must be queried repeatedly throughout training. This requires a dedicated multi-GPU teacher server running in parallel with the training job for the entire duration of OPD training. Concretely, at the 8B scale with a Qwen3-32B teacher, the standard OPD pipeline costs 120 GPU hours — not just because training is compute-intensive, but because the teacher serving infrastructure runs continuously, fragmenting GPU resources and making large-scale experiments difficult to reproduce, particularly for academic researchers without access to extensive serving infrastructure.

The paper phrases this as a natural question in Section 1:

"A natural question is whether the benefits of on-policy supervision can be preserved while eliminating the need for a live teacher server."

The intuition behind this question is that the student's distribution during OPD training might not drift far from its SFT initialization. The paper cites two pieces of supporting evidence from prior work: Yue et al. (2025) found that reasoning trajectories in RL models are "largely a reweighted subset of those present in the SFT model," and Shenfeld et al. (2025) showed that "on-policy updates are inherently biased toward solutions that minimize KL divergence from the reference policy." If the student stays close to the SFT reference, then perhaps the teacher's log-probabilities could be computed once over SFT-model rollouts before training begins, and those precomputed values could be reused throughout OPD — eliminating the need for a live teacher server entirely.

This is an important practical question. If this offline approximation works reliably, OPD becomes accessible to anyone with a single training node. If it doesn't, the community must either accept the infrastructure burden or abandon OPD for alternatives like RLVR. The paper positions itself as investigating this exact tension.

Why This Problem Matters

The importance of solving the OPD infrastructure bottleneck can be understood along several dimensions.

Democratization of LLM post-training. The paper emphasizes this explicitly in the abstract and introduction. OPD has been validated across numerous studies (Agarwal et al., 2024; Lu & Thinking Machines Lab, 2025; Yang et al., 2026a; Xiao et al., 2026; Yang et al., 2026b; Zeng et al., 2026) as an effective post-training approach, but its practical adoption is limited by the serving infrastructure requirement. Reducing OPD to a single standard training job — with teacher log-probabilities precomputed once offline — would make high-quality on-policy distillation accessible to academic labs and smaller organizations that cannot afford to maintain a dedicated teacher server. The paper's MoE result (Table 3) makes this point concretely: on a single 8×H100 node, standard OPD runs out of memory because co-hosting both a 30B student and a 30B teacher exceeds available GPU memory, while Lightning OPD trains the same model successfully.

Practical deployment decisions. For organizations building post-training pipelines, the choice between OPD and RLVR is partly an infrastructure question. RLVR only requires a reward signal (often verifiable from the output alone, e.g., unit test pass/fail), so it can run without a separate model server. OPD provides richer supervision and often converges faster, but the live teacher server makes it harder to deploy at scale. If the infrastructure cost of OPD can be eliminated, the tradeoff shifts substantially in OPD's favor for many applications. This paper's 4.0× training efficiency improvement at the 8B scale (30 vs. 120 GPU hours; Table 2) provides a concrete number for that shift.

Theoretical significance: understanding what makes on-policy methods work. Beyond the practical motivation, the paper's investigation reveals something surprising: the obstacle to offline OPD is not primarily the distribution shift from fixing the rollout distribution (the "offline approximation"), but rather a more fundamental condition — teacher consistency — that was being violated in existing OPD practice even in the online setting. This means the paper's contribution is not merely an engineering optimization but a conceptual clarification of what conditions are necessary for OPD to be effective in the first place.

Where Prior Approaches Fall Short

The paper identifies several limitations in the prior landscape.

Standard OPD is expensive and infrastructure-heavy. As described above, Agarwal et al. (2024) introduced OPD, and Lu & Thinking Machines Lab (2025) expanded and popularized it, but both require a live teacher server throughout training. The paper cites this as the primary practical limitation motivating the work. Various OPD variants have been proposed — reward extrapolation (Yang et al., 2026a), self-distillation (Zhao et al., 2026; Shenfeld et al., 2026a; Hübotter et al., 2026), entropy-aware divergence (Wang et al., 2026), reinforcement-aware distillation (Xu et al., 2026), controllable reasoning (Liang et al., 2026), and black-box or privileged settings (Ye et al., 2025; Penaloza et al., 2026) — but none of these eliminate the live teacher server from the core OPD loop. They modify the objective or the teacher form, but not the fundamental online setup.

Existing post-training pipelines often violate teacher consistency without realizing it. This is perhaps the paper's most important prior-work critique. The paper observes that OPD practitioners have inherited conventions from RLVR, where the SFT stage and RL stage are largely independent: the SFT dataset is curated using whichever teacher produces the highest-quality demonstrations, regardless of which model is used during RL. The paper gives a concrete example in Section 1:

"Thinking Machines Lab [9] trains a Qwen3-8B-Base model on OpenThoughts-3 [7], whose trajectories are generated by QwQ-32B, while using Qwen3-32B as the OPD teacher, resulting in a mismatch that our analysis predicts to be detrimental."

In RLVR, this mismatch might be acceptable because the reward signal (e.g., unit test pass/fail) is independent of the teacher. But in OPD, the teacher's token-level distribution defines both the SFT training target and the OPD advantage signal. Using different teachers introduces an inconsistency that the paper proves (Theorems 3.8–3.9) introduces an irreducible gradient bias — a bias that degrades both online and offline OPD, and that no amount of training or data can remove. Prior OPD work did not identify or analyze this condition, and the paper argues it is a necessary design constraint for OPD to be effective in general.

Naive offline precomputation fails. The paper explicitly states in Section 1:

"In practice, however, naively applying this offline precomputation fails to reliably match the performance of standard OPD."

The paper does not merely claim that offline OPD can be made to work; it acknowledges that a naive approach — precompute teacher log-probabilities on SFT rollouts without considering teacher consistency — does not work reliably. The investigation then traces this failure not primarily to the offline approximation itself, but to the teacher consistency violation that was being inherited from online OPD practice as well. This means the naive offline attempt failed for two compounding reasons: (1) the teacher mismatch introduces gradient bias, and (2) offline OPD is more sensitive to this bias than online OPD because the fixed rollout distribution amplifies the effect (a point shown empirically in Table 4).

Offline RL methods are not directly applicable. The paper includes an extended discussion in Appendix D comparing Lightning OPD to offline RL, and clarifies why offline RL techniques don't solve the problem. Offline RL addresses "OOD action overestimation arising from sparse reward signals" through conservatism mechanisms like value pessimism or policy constraints. In OPD, the teacher provides dense per-token log-probability supervision everywhere, so there is no sparse reward signal to cause high-variance estimation, and no OOD region where the value function is unreliable. The challenge is not distributional coverage but teacher inconsistency — a "structural property of the gradient field itself, not an estimation artifact" — and standard offline RL conservatism mechanisms are neither necessary nor applicable.

Offline (off-policy) knowledge distillation is a fundamentally different paradigm. The paper distinguishes Lightning OPD from standard offline KD (Kim & Rush, 2016; Gu et al., 2024) in Appendix D. In offline KD, the student trains on teacher-generated sequences, so it only receives supervision on trajectories the teacher would produce — never on its own mistakes. Lightning OPD, by contrast, collects rollouts from the student's own policy $\pi_{\text{ref}}$ and evaluates the teacher's log-probabilities on these student-generated sequences. This preserves the core advantage of on-policy methods: the teacher provides corrective signals on exactly the distribution the student will encounter during inference. The distinction is preserved even though teacher log-probabilities are precomputed, because those log-probabilities are evaluated on student-generated rollouts, not teacher-generated ones.

Rang et al. (2025) independently explored a similar direction but without the theoretical framework. The paper acknowledges in Appendix D that Rang et al. (2025) also precompute teacher signals over student-generated responses and train without a live teacher server. However, the paper identifies a fundamental difference: Rang et al. treat SFT and distillation as independent sequential stages — SFT on curated data, distillation with a separate teacher, no constraint linking the two. They formulate distillation as supervised learning with a composite cross-entropy/KL loss, and provide no analysis of when or why the offline approximation is reliable. Lightning OPD identifies teacher consistency as the necessary condition and formulates distillation as policy gradient optimization with per-token advantages ($\log \pi_T - \log \pi_\theta$), which enables the theoretical analysis (gradient discrepancy bounds, shared fixed points, implicit regularization) that is absent in Rang et al.

How This Paper Positions Itself

The paper's positioning is best understood through three nested claims, each building on the previous.

Claim 1: Offline OPD fails naively, and the root cause is teacher inconsistency — not just the offline approximation. The paper does not position itself as simply proposing a better offline algorithm. It first investigates why naive offline OPD fails, and the answer turns out to expose a condition that was being violated in standard (online) OPD practice as well. Teacher consistency — the requirement that the SFT teacher and OPD teacher be the same model — is presented not as a trick to make offline OPD work, but as a general design principle for OPD that was overlooked in prior work, with consequences for both offline and online settings. The proof (Theorems 3.8–3.9) shows that teacher mismatch introduces a gradient bias $\sigma_\Delta$ that degrades the convergence point of both paradigms, and the empirical ablation (Table 4) confirms this across both standard OPD and Lightning OPD.

Claim 2: Under teacher consistency, offline OPD provably shares the same optimum as online OPD. This is the paper's central theoretical contribution. Theorem 3.6 shows that when the teacher is representable within the student's capacity class, both online and offline OPD share the same global optimum ($\pi_\theta$ that minimizes $\text{KL}(\pi_\theta \parallel \pi_T)$), and the gradient is exactly zero at that point for both methods. Theorem 3.5 bounds the gradient discrepancy between the two methods by $G \cdot \sigma_A \cdot \sqrt{\chi^2(\pi_\theta \parallel \pi_{\text{ref}})}$, which is zero at initialization (when $\pi_\theta = \pi_{\text{ref}}$) and grows only with policy drift, which Theorem 3.7 shows is naturally constrained by an implicit covariance-based regularization effect. These results together mean that Lightning OPD is not just a heuristic approximation — it is a principled offline counterpart to standard OPD that converges to the same solution when the key condition (teacher consistency) is satisfied.

Claim 3: The efficiency gains are substantial and the method scales to settings where standard OPD is infeasible. The paper backs this with three empirical findings:

• At the 4B scale, 3.6× speedup (20 vs. 72 GPU hours; Table 2).
• At the 8B scale, 4.0× speedup (30 vs. 120 GPU hours; Table 2), with matched or marginally better performance across all five benchmarks (Table 1).
• At the 30B MoE scale, standard OPD runs out of memory on a single 8×H100 node, while Lightning OPD trains successfully, reaching 71.0% on AIME 2024 (Table 3).

The paper positions these results as evidence that "much of the practical benefit of on-policy distillation can be retained without the conventional deployment burden of maintaining a live teacher" (Section 5) — but only if teacher consistency is respected as a design constraint.

Positioning relative to the literature. The paper is not proposing a new distillation objective or a new RL algorithm. It is proposing a deployment framework (offline OPD with teacher consistency) and a design principle (teacher consistency as a general requirement for OPD). The theoretical analysis connects these two: the deployment framework works because the design principle ensures the offline approximation remains faithful to the online objective. The paper's contribution is the identification and formalization of this connection, which was missing in both the OPD literature and the nascent offline-OPD work (Rang et al., 2025).

The practical significance of the theoretical results. One might ask: if the student stays close to $\pi_{\text{ref}}$ during training (as Figure 3a shows — importance weights remain near 0.94), why do we need the formal theory? The paper's answer is twofold. First, the theory explains why the student stays close to $\pi_{\text{ref}}$ (the implicit regularization of Theorem 3.7) — it is not an accident of hyperparameter tuning but a structural property of the offline objective. Second, and more importantly, the theory explains why teacher consistency matters: when $\sigma_\Delta > 0$, the gradient bias $G\sigma_\Delta$ persists at initialization (Theorem 3.9) and corrupts the update direction throughout training, regardless of how small the policy drift is. This means that even if the offline approximation is perfectly accurate (policy drift near zero), teacher inconsistency alone can cause OPD to fail. The theory identifies teacher consistency as a necessary condition, not just a helpful trick, which has direct implications for how OPD pipelines should be designed in practice.

3. Technical Approach

3.1 Reader Orientation

This is primarily a system design and theoretical analysis paper whose core idea is that on-policy distillation can be reformulated as an entirely offline procedure — eliminating the need for a live teacher server — provided a single, previously unrecognized condition called teacher consistency is enforced. The "system" being built is not a new model architecture or a new learning objective, but rather a two-stage deployment framework where teacher log-probabilities are precomputed once before OPD training begins, and the student trains on this fixed dataset using the same per-token advantage signal as standard OPD. The problem it solves is the infrastructure bottleneck of standard OPD: the requirement that a dedicated multi-GPU teacher server run continuously throughout training to score the student's evolving rollouts. The solution's "shape" is a principled offline approximation to an online objective, where the approximation's fidelity is guaranteed not by ad-hoc heuristics but by a formal condition — the SFT and OPD stages must use the identical teacher model — that, when satisfied, ensures the offline and online procedures share the same optimal solution and have bounded gradient discrepancy throughout training.

3.2 Big-Picture Architecture (Diagram in Words)

The Lightning OPD system has five major components, organized into two sequential stages:

1. Base Model ($\pi_{\text{base}}$) — the pretrained language model (e.g., Qwen3-4B-Base or Qwen3-8B-Base) that will be fine-tuned into a reasoning model. It serves as the starting point for both stages.

2. Teacher Model ($\pi_T$) — a fixed, stronger model (e.g., Qwen3-8B teaching Qwen3-4B, or Qwen3-32B teaching Qwen3-8B) that provides supervision in both stages. Critically, the same teacher must be used in Stage 1 and Stage 2 — this is the teacher consistency requirement.

3. Stage 1: Supervised Fine-Tuning (SFT) — the teacher generates high-quality trajectories on a prompt dataset $\mathcal{Q}_{\text{SFT}}$, forming $\mathcal{D}_{\text{SFT}} = \{(q_i, x_i) \mid x_i \sim \pi_T(\cdot \mid q_i)\}$. The base model is fine-tuned on these trajectories via standard maximum likelihood estimation to produce $\pi_{\text{ref}}$, the reference policy. This stage requires the teacher to be served once for trajectory generation, but not during training.

4. Stage 2, Phase 1: Offline Preprocessing — the reference policy $\pi_{\text{ref}}$ samples rollouts on a (potentially different) prompt dataset $\mathcal{Q}_{\text{OPD}}$. The teacher is queried once to compute and store per-token log-probabilities $\log \pi_T(a_t \mid s_t)$ for every token in every sampled response. This forms the offline dataset $\mathcal{D}_{\text{OPD}} = \{(q_j, x_j, \{\log \pi_T(a_t^j \mid s_t^j)\}_{t=1}^{T_j})\}$. After this phase, the teacher server is permanently shut down.

5. Stage 2, Phase 2: Offline OPD Training — the student $\pi_\theta$ is initialized from $\pi_{\text{ref}}$ and trained on $\mathcal{D}_{\text{OPD}}$ using the standard OPD update: at each step, the per-token advantage is computed as $A_t(\theta) = \log \pi_T(a_t \mid s_t) - \log \pi_\theta(a_t \mid s_t)$, where the teacher term is read directly from the precomputed dataset and only the student term is computed online. The student is updated via advantage-weighted policy gradient. No teacher server runs during this phase.

Information flows as follows: prompts → teacher generates SFT trajectories → base model fine-tuned to $\pi_{\text{ref}}$ → $\pi_{\text{ref}}$ samples OPD rollouts → teacher scores rollouts once and is discarded → student trains on fixed $\mathcal{D}_{\text{OPD}}$ using precomputed teacher scores. The critical constraint threading through the entire pipeline is that the teacher model used to generate $\mathcal{D}_{\text{SFT}}$ (producing $\pi_{\text{ref}}$) and the teacher model used to score $\mathcal{D}_{\text{OPD}}$ (providing the reference log-probabilities) must be the identical model instance.

3.3 Roadmap for the Deep Dive

• First, the formal definitions of the online and offline OPD objectives (Equations 3–4), because understanding what Lightning OPD optimizes — and how it differs from standard OPD — requires establishing these two optimization problems and their relationship.
• Second, the two-stage procedure in full operational detail (SFT then offline OPD), including exactly where teacher log-probabilities are precomputed, what data structures are stored, and how training proceeds without a live teacher — because this is the practical deployment recipe the paper contributes.
• Third, the per-token advantage function and the importance-sampling decomposition, because these are the mathematical tools that connect the online and offline objectives and are used throughout the theoretical analysis.
• Fourth, the theoretical results (Theorems 3.5–3.9), in order: gradient discrepancy bound, shared fixed point, gradient decomposition / implicit regularization, and the two teacher consistency theorems — because these form the paper's formal justification that Lightning OPD is not a heuristic but a principled approximation to standard OPD whose fidelity is governed by a single, verifiable condition.
• Fifth, the assumptions underpinning the theory, because understanding their restrictiveness and when they hold in practice is essential for assessing the scope of the theoretical guarantees.

This order builds from concrete operational definition → mathematical connection → theoretical guarantees → scope conditions, which mirrors how the paper justifies Lightning OPD: first defining what it does, then proving why it works, then specifying when the proofs apply.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a system design and theoretical analysis paper whose core idea is that on-policy distillation can be performed entirely offline — with teacher log-probabilities precomputed once before training begins — provided the SFT and OPD stages use the identical teacher model, a condition the paper calls teacher consistency. The contribution is not a new objective function but rather (1) the identification that teacher consistency is the necessary and sufficient condition for offline OPD to match online OPD, (2) the theoretical proof that under this condition the two methods share the same optimum with bounded gradient discrepancy, and (3) a practical deployment recipe that eliminates the live teacher server entirely.

---

3.4.1 The Online and Offline OPD Objectives

The paper begins by formalizing the two optimization problems at the heart of the comparison.

Preliminaries: policy, teacher, and advantage. Let $\pi_T$ denote a fixed teacher model — a stronger LLM whose capabilities we want to transfer to a smaller student. Let $\pi_\theta$ denote a trainable student model parameterized by $\theta$. Given a prompt $q$ drawn from a distribution $p(q)$, a response $x = (a_1, a_2, \ldots, a_T)$ is a sequence of tokens generated autoregressively:

$\pi_\theta(x \mid q) = \prod_{t=1}^{T} \pi_\theta(a_t \mid s_t)$

where $s_t = (q, a_1, \ldots, a_{t-1})$ is the state (context) at step $t$, consisting of the prompt and all previously generated tokens. The term $\pi_\theta(a_t \mid s_t)$ is the scalar probability the student assigns to token $a_t$ given context $s_t$.

Let $\pi_{\text{ref}}$ denote the reference policy — the student model after supervised fine-tuning but before any OPD training. This is the SFT-initialized student.

The per-token OPD advantage is defined as:

$A_t(\theta) = \log \pi_T(a_t \mid s_t) - \log \pi_\theta(a_t \mid s_t)$

where $\log \pi_T(a_t \mid s_t)$ is the log-probability the teacher assigns to the same token $a_t$ given the same context $s_t$, and $\log \pi_\theta(a_t \mid s_t)$ is the log-probability the current student assigns.

What it computes: the difference in log-probability between teacher and student for a specific token $a_t$ at position $t$. When $A_t(\theta) > 0$, the teacher is more confident than the student in that token — the student should increase its probability for $a_t$. When $A_t(\theta) < 0$, the teacher is less confident — the student should decrease its probability. The advantage is therefore a signed, token-level signal of how the student's distribution should shift to align with the teacher's.

Why this form: log-probability difference is the natural per-token signal for distribution matching because it measures relative confidence on a logarithmic scale, which connects directly to KL-divergence minimization. An alternative like raw probability difference ($\pi_T - \pi_\theta$) would be dominated by high-probability tokens and insensitive to differences in low-probability regions, which is where the teacher's knowledge is often most valuable.

The online OPD objective. Standard OPD (Agarwal et al., 2024; Lu & Thinking Machines Lab, 2025) optimizes:

$\mathbf{J}_{\text{on}}(\theta) = \mathbb{E}_{q \sim p, \, x \sim \pi_\theta}\left[\sum_{t=1}^{T} A_t(\theta)\right]$

where the expectation is over prompts drawn from $p(q)$ and responses $x$ sampled from the current student policy $\pi_\theta$, not from a fixed dataset. The sum aggregates per-token advantages over the entire response.

What it computes: the expected sum of per-token advantages when the student generates its own responses. At each training step, the student samples fresh rollouts from its current policy, the teacher scores every token of every rollout, and the student is updated to increase the probability of tokens the teacher assigns higher probability to (and decrease the probability of tokens the teacher assigns lower probability to). The objective is maximized when $\pi_\theta = \pi_T$ on the support of $\pi_\theta$, meaning the student perfectly mimics the teacher's distribution.

Why this form: the expectation over $x \sim \pi_\theta$ makes this on-policy — the student learns from its own mistakes on its own distribution, which is the core advantage of OPD over offline knowledge distillation (where the student only sees teacher-generated sequences). This on-policy sampling means the teacher provides corrective signals on exactly the distribution the student will encounter during inference, leading to stronger post-training gains. The sum over $t$ provides dense per-token supervision, which is richer than the sparse outcome-level reward in RLVR and contributes to OPD's greater training stability.

The Lightning OPD (offline) objective. Lightning OPD replaces the on-policy sampling distribution with the fixed reference policy:

$\mathbf{J}_{\text{off}}(\theta) = \mathbb{E}_{q \sim p, \, x \sim \pi_{\text{ref}}}\left[\sum_{t=1}^{T} A_t(\theta)\right]$

where the only change is that responses are sampled from $\pi_{\text{ref}}$ instead of $\pi_\theta$.

What it computes: the same expected sum of per-token advantages, but evaluated on responses sampled once from the SFT-initialized student, before any OPD training begins. The teacher's log-probabilities are computed on these fixed responses and stored; during training, the student reads precomputed $\log \pi_T(a_t \mid s_t)$ from the dataset and computes only $\log \pi_\theta(a_t \mid s_t)$ online.

Why this form: the advantage function $A_t(\theta) = \log \pi_T - \log \pi_\theta$ is identical to the online case — the teacher reference and student term are the same mathematical objects. The only difference is the distribution over which the expectation is taken. This means Lightning OPD is not changing what is being optimized (the per-token divergence from the teacher) but rather changing which responses the optimization is evaluated on. The paper's central theoretical claim is that, under teacher consistency, this distributional shift does not change the optimum.

Critical design choice: stop-gradient on the advantage. Following standard OPD practice, the advantage $A_t(\theta)$ is treated as a fixed scalar with respect to gradient computation — i.e., a stop-gradient is applied so that $\nabla A_t(\theta)$ does not contribute to parameter updates. The actual gradient used for parameter updates is:

$\nabla \mathbf{J}_{\text{off}}(\theta) = \mathbb{E}_{x \sim \pi_{\text{ref}}}\left[\sum_{t=1}^{T} A_t(\theta) \cdot \nabla \log \pi_\theta(a_t \mid s_t)\right]$

where the gradient operator is applied only to the $\log \pi_\theta$ term inside the advantage, not to $A_t(\theta)$ itself. This is an advantage-weighted policy gradient: it scales the policy gradient $\nabla \log \pi_\theta$ by the scalar advantage $A_t$, pushing the student's probability mass toward tokens the teacher favors and away from those it disfavors. The paper formalizes this in Appendix A.1 (Proposition A.1), defining the per-trajectory gradient as $f(x; \theta) = \sum_t A_t(\theta) \cdot \nabla \log \pi_\theta(a_t \mid s_t)$ and noting that $\nabla \mathbf{J}_{\text{on}}$ and $\nabla \mathbf{J}_{\text{off}}$ denote these stop-gradient versions throughout. This is not a contribution of the paper but rather a faithful reproduction of how OPD is implemented in practice (the paper explicitly cites this as "the standard update rule used in all OPD implementations [8, 9, 22]").

---

3.4.2 The Two-Stage Lightning OPD Pipeline

The Lightning OPD procedure is specified in Algorithm 1 of the paper and consists of two sequential stages, with the second stage itself divided into two phases.

Stage 1: Supervised Fine-Tuning (SFT). The goal is to produce a reference policy $\pi_{\text{ref}}$ that is both (a) a capable reasoning model in its own right and (b) the distribution from which OPD rollouts will later be sampled.

Inputs: a base model $\pi_{\text{base}}$ (e.g., Qwen3-4B-Base), a teacher model $\pi_T$ (e.g., Qwen3-8B), and a prompt dataset $\mathcal{Q}_{\text{SFT}}$ (e.g., the prompts from OpenThoughts-3).

Procedure:

1. For each prompt $q_i \in \mathcal{Q}_{\text{SFT}}$, sample a response $x_i \sim \pi_T(\cdot \mid q_i)$ from the teacher. This produces the SFT dataset $\mathcal{D}_{\text{SFT}} = \{(q_i, x_i) \mid q_i \in \mathcal{Q}_{\text{SFT}}, x_i \sim \pi_T(\cdot \mid q_i)\}$. Each data point is a prompt–response pair where the response is the teacher's own generation.
2. Fine-tune $\pi_{\text{base}}$ on $\mathcal{D}_{\text{SFT}}$ via standard maximum likelihood estimation (next-token prediction):

$\pi_{\text{ref}} = \arg\max_\theta \mathbb{E}_{(q, x) \sim \mathcal{D}_{\text{SFT}}}\left[\sum_{t=1}^{T} \log \pi_\theta(a_t \mid s_t)\right]$

where the objective is to maximize the log-probability the student assigns to each token in the teacher-generated response. This is standard SFT: the student learns to mimic the teacher's output distribution on the SFT prompts.

Hyperparameters (Table 5): 3000 training steps, global batch size 256 (4B scale) or 128 (8B scale), maximum sequence length 16,384 tokens, learning rate $8 \times 10^{-5}$, cosine learning rate schedule with 0.1 warmup ratio, sequence packing enabled, DeepSpeed ZeRO-0 (4B) or ZeRO-1 (8B).

Why this procedure: the SFT stage serves two purposes in the Lightning OPD framework. First, it produces a strong initialization for the student — $\pi_{\text{ref}}$ already has non-trivial reasoning capabilities before any OPD training begins. Second, and more subtly, it establishes $\pi_{\text{ref}}$ as a distribution that is anchored to the teacher's preferences, because the SFT data was generated by $\pi_T$ itself. This anchoring is what makes the offline OPD stage well-behaved: rollouts sampled from $\pi_{\text{ref}}$ are similar to what the teacher would produce, so the precomputed teacher log-probabilities are computed on a distribution that is already somewhat aligned with the teacher. If the SFT data came from a different teacher (or from humans, or from another model), the reference policy $\pi_{\text{ref}}$ would be anchored to a different distribution, and the offline OPD stage would suffer from teacher inconsistency — a point formalized in Theorems 3.8–3.9 and validated empirically in Table 4.

The teacher consistency requirement. The paper explicitly states:

"While standard OPD may use any high-quality SFT dataset regardless of its source, Lightning OPD requires that $\mathcal{D}_{\text{SFT}}$ be generated by the same teacher $\pi_T$ used in the OPD stage."

This is the single constraint that Lightning OPD imposes beyond standard OPD practice. It is not a constraint on the model architecture or the training algorithm, but on the data generation pipeline: the SFT trajectories and the OPD reference log-probabilities must both come from the identical teacher model.

Stage 2, Phase 1: Offline Preprocessing (Rollout Collection and Teacher Scoring). The goal is to build a fixed dataset $\mathcal{D}_{\text{OPD}}$ that contains everything needed for OPD training — student-generated responses and teacher log-probabilities for every token — so that no teacher server is needed during the actual training phase.

Inputs: the reference policy $\pi_{\text{ref}}$ (from Stage 1), the teacher model $\pi_T$ (the same one used in Stage 1), and a prompt dataset $\mathcal{Q}_{\text{OPD}}$ (e.g., DAPO-Math-17k for math, a 30K subset of EpiCoder-func-380k for code).

Procedure:

1. For each prompt $q_j \in \mathcal{Q}_{\text{OPD}}$, sample a response $x_j \sim \pi_{\text{ref}}(\cdot \mid q_j)$ from the reference policy. The sampling uses temperature 0.8 and top-$p$ 1.0 (Table 6). Only a single response is sampled per prompt.
2. For each response $x_j = (a_1^j, \ldots, a_{T_j}^j)$, query the teacher $\pi_T$ to compute the log-probability $\log \pi_T(a_t^j \mid s_t^j)$ for every token $t = 1, \ldots, T_j$.
3. Store the triple $(q_j, x_j, \{\log \pi_T(a_t^j \mid s_t^j)\}_{t=1}^{T_j})$ in the offline dataset $\mathcal{D}_{\text{OPD}}$.

The key operational detail: the teacher is queried exactly once per response and then discarded. There is no teacher server running during the subsequent training phase. The per-token log-probabilities are stored as precomputed scalars in the dataset.

Cost of this phase (Table 2, lower panel): at the 8B scale, rollout collection costs 10 GPU hours and teacher log-probability precomputation costs 4 GPU hours — both are one-time offline operations requiring no specialized infrastructure. The actual OPD training phase costs 16 GPU hours. The total is 30 GPU hours, compared to 120 GPU hours for standard OPD where the teacher server runs continuously.

Why sample from $\pi_{\text{ref}}$ rather than from $\pi_T$: this is the critical distinction between Lightning OPD and offline knowledge distillation. If responses were sampled from the teacher (as in standard KD), the student would only receive supervision on trajectories the teacher is confident about — never on its own mistakes or on the distribution it will actually encounter at inference. By sampling from $\pi_{\text{ref}}$ (the student's own SFT-initialized policy), Lightning OPD preserves the on-policy characteristic of OPD: the teacher evaluates the student's own outputs, providing corrective signals on exactly the distribution the student will generate from. As the paper states in Appendix D:

"Lightning OPD instead collects rollouts from the student's own policy $\pi_{\text{ref}}$ and evaluates the teacher's per-token log-probabilities on these student-generated sequences. This means the teacher provides corrective signals on exactly the distribution the student will encounter during inference."

The precomputation step merely fixes the timing of teacher queries (all upfront, instead of during training), not the distribution on which teacher scores are evaluated (student rollouts, not teacher rollouts).

Stage 2, Phase 2: Offline OPD Training. The goal is to train the student to match the teacher's distribution using the precomputed dataset, with no teacher server required.

Inputs: the offline dataset $\mathcal{D}_{\text{OPD}}$, the reference policy $\pi_{\text{ref}}$ (used to initialize the student).

Procedure:

1. Initialize the student $\pi_\theta \leftarrow \pi_{\text{ref}}$.
2. For each training step:
   • Sample a mini-batch from $\mathcal{D}_{\text{OPD}}$.
   • For each token $a_t$ in each response in the batch, read the precomputed teacher log-probability $\log \pi_T(a_t \mid s_t)$ from the dataset.
   • Compute the student's current log-probability $\log \pi_\theta(a_t \mid s_t)$ by running a forward pass of $\pi_\theta$.
   • Compute the per-token advantage $A_t(\theta) = \log \pi_T(a_t \mid s_t) - \log \pi_\theta(a_t \mid s_t)$ and clip it to the range $[-10, 10]$ (per the hyperparameters in Table 6).
   • Compute the gradient $\nabla \mathbf{J}_{\text{off}}(\theta) = \sum_t A_t(\theta) \cdot \nabla \log \pi_\theta(a_t \mid s_t)$ (with stop-gradient on $A_t$).
   • Update $\theta \leftarrow \theta + \eta \cdot \nabla \mathbf{J}_{\text{off}}(\theta)$.

Hyperparameters (Table 6): 150 training steps (sufficient for convergence per Figure 3b), global batch size 256, maximum response length 4,096 tokens (shorter than evaluation length for efficiency but sufficient for optimal performance), learning rate $2 \times 10^{-6}$, constant learning rate schedule (no decay), weight decay 0.1, Adam optimizer with $\beta_1 = 0.9$ and $\beta_2 = 0.98$, rollout temperature 0.8, rollout top-$p$ 1.0, advantage clip range $[-10, 10]$, tensor parallel size 2 (4B) or 4 (8B).

Why 150 steps: Figure 3b shows that the AIME 2024 score converges rapidly, with the student capturing nearly all of its performance gain within the first 50 steps and remaining stable thereafter. Training for 150 steps provides a margin beyond convergence without overfitting.

Why constant learning rate: a constant learning rate without decay is used because the training duration is short (only 150 steps) and the advantage signal naturally diminishes as the student approaches the teacher's distribution, providing an implicit form of learning rate decay. A decaying schedule would unnecessarily complicate the hyperparameter configuration.

Why maximum response length 4,096 tokens for training but up to 40,960 for evaluation: the paper states that "training with 4,096 tokens already achieves optimal performance while offering substantially better training efficiency; increasing the rollout length beyond this threshold does not improve results" (Appendix B). This is an empirical finding: the per-token advantage signals in the early tokens are sufficient to drive the student toward the teacher's distribution, and the extra tokens beyond 4,096 do not provide additional useful supervision.

Cross-domain training strategy (Appendix B). For code generation, the OPD stage is initialized from the math-trained OPD checkpoint rather than from the SFT model directly. The paper states this "consistently outperforms initializing code OPD from the SFT model, consistent with prior findings that math reasoning training provides a stronger initialization for code training." This is not a core contribution but a practical training recipe choice.

---

3.4.3 The Importance-Sampling Decomposition (Proposition A.1)

The theoretical analysis hinges on a single identity that connects the online and offline objectives. The paper formalizes this as Proposition A.1 in Appendix A.1.

Definition: sequence-level importance weight. Define:

$w(x; \theta) = \frac{\pi_\theta(x \mid q)}{\pi_{\text{ref}}(x \mid q)}$

where $\pi_\theta(x \mid q)$ is the probability the current student assigns to the full response $x$ given prompt $q$, and $\pi_{\text{ref}}(x \mid q)$ is the probability the reference policy assigns to the same response. This is the importance sampling ratio — it measures how much more (or less) likely the current student is to generate response $x$ compared to the reference policy.

What it computes: a per-response scalar weight. When $w(x; \theta) = 1$, the student and reference policies assign identical probability to $x$. When $w(x; \theta) > 1$, the student over-weights $x$ relative to the reference; when $w(x; \theta) < 1$, the student under-weights $x$. The vector of these weights across all possible responses encodes the entire distributional shift from $\pi_{\text{ref}}$ to $\pi_\theta$.

Why this decomposition exists: it relies on Assumption 3.2 (support coverage), which requires that any response the current student can generate must also be generatable by the reference policy — formally, $\text{supp}(\pi_\theta(\cdot \mid q)) \subseteq \text{supp}(\pi_{\text{ref}}(\cdot \mid q))$. This holds naturally when $\pi_\theta$ is initialized from $\pi_{\text{ref}}$ and trained with finite learning rate, because the student's distribution is a continuous perturbation of the reference distribution and does not suddenly assign non-zero probability to sequences that had zero probability under $\pi_{\text{ref}}$.

The identity. Under Assumption 3.2:

$\nabla \mathbf{J}_{\text{on}}(\theta) = \mathbb{E}_{x \sim \pi_{\text{ref}}}\left[ w(x; \theta) \cdot \sum_{t=1}^{T} A_t(\theta) \cdot \nabla \log \pi_\theta(a_t \mid s_t) \right]$

and the offline gradient is the special case with $w \equiv 1$:

$\nabla \mathbf{J}_{\text{off}}(\theta) = \mathbb{E}_{x \sim \pi_{\text{ref}}}\left[ \sum_{t=1}^{T} A_t(\theta) \cdot \nabla \log \pi_\theta(a_t \mid s_t) \right]$

What it says operationally: the online gradient is an importance-weighted version of the offline gradient. Every response in the offline dataset is weighted by $w(x; \theta)$ — responses that the current student over-produces relative to the reference get weighted up, and those the student under-produces get weighted down. This reweighting adjusts the offline expectation to match what the online expectation would be with rollouts from the current student.

Why this is the central identity for the entire theoretical analysis: it decomposes the difference between online and offline OPD into a single quantity — the deviation of the importance weights from 1. When $w(x; \theta) = 1$ for all $x$ (i.e., $\pi_\theta = \pi_{\text{ref}}$), the online and offline gradients are identical. As $w$ deviates from 1 (i.e., as the student drifts from the reference), the gradients diverge. Every theorem in Section 3.3 bounds this divergence or characterizes its effect. The identity also makes clear why Lightning OPD is not an arbitrary offline approximation: the offline gradient is the online gradient with a specific known reweighting factor omitted, and the reweighting factor is controlled by how far the student has moved from $\pi_{\text{ref}}$.

---

3.4.4 Theoretical Analysis: Gradient Discrepancy, Shared Optimum, and Implicit Regularization

The paper provides five theoretical results (Theorems 3.5–3.9) that collectively establish Lightning OPD as a principled offline approximation to standard OPD. The first three assume teacher consistency holds; the last two analyze what happens when it is violated. All proofs are in Appendix A, but we walk through each theorem's meaning, assumptions, and operational significance.

---

3.4.5 Theorem 3.5: Gradient Discrepancy Bound

Statement:

$\|\nabla \mathbf{J}_{\text{on}}(\theta) - \nabla \mathbf{J}_{\text{off}}(\theta)\|_2 \leq G \cdot \sigma_A \cdot \sqrt{\chi^2(\pi_\theta \parallel \pi_{\text{ref}})}$

where $G$ is the bound on the score function norm, $\sigma_A$ is the bound on the squared absolute advantage, and $\chi^2(\pi_\theta \parallel \pi_{\text{ref}}) = \mathbb{E}_{x \sim \pi_{\text{ref}}}[w(x; \theta)^2] - 1$ is the $\chi^2$-divergence between the current student and the reference policy.

Symbol definitions:

• $\|\cdot\|_2$ is the Euclidean ($L_2$) norm of the gradient vector — a single non-negative scalar measuring the magnitude of the gradient difference.
• $G$ comes from Assumption 3.3: $\|\nabla \log \pi_\theta(a_t \mid s_t)\|_2 \leq G$ for all $\theta$ and all $t$. This bounds how large the policy gradient can be for any single token. In practice, $G$ is controlled by gradient clipping and the softmax temperature.
• $\sigma_A$ comes from Assumption 3.1: $\mathbb{E}_{x \sim \pi_{\text{ref}}}[(\sum_{t=1}^T |A_t(\theta)|)^2] \leq \sigma_A^2$. This bounds the expected squared sum of absolute per-token advantages. When advantages are clipped to $[-\tau, \tau]$ (in practice, $[-10, 10]$ per Table 6), $\sigma_A \leq T \tau$ is a conservative bound.
• $\chi^2(\pi_\theta \parallel \pi_{\text{ref}})$ is the chi-squared divergence: $\mathbb{E}_{\pi_{\text{ref}}}[(w - 1)^2]$. It is zero when $\pi_\theta = \pi_{\text{ref}}$ (because $w \equiv 1$) and grows as the two distributions diverge.

What it computes: an upper bound on how different the online and offline gradient vectors can be, as a function of how far the student has drifted from the reference policy. The bound is a product of three factors: how sensitive the policy gradient is to parameter changes ($G$), how large the advantages are on average ($\sigma_A$), and how far the student's distribution has moved from the reference ($\sqrt{\chi^2}$).

Key property: zero gap at initialization. At $\pi_\theta = \pi_{\text{ref}}$, the importance weights are identically 1, so $\chi^2 = 0$ and the bound is zero — the online and offline gradients are exactly equal. This means that at the start of OPD training, Lightning OPD takes the identical first gradient step as standard OPD. The divergence only grows as training progresses and the student moves away from $\pi_{\text{ref}}$.

Why this form rather than a KL-based bound: the paper includes a remark (Appendix A.2) explaining that a naive Pinsker-inequality-based bound using KL-divergence would be $\|\nabla \mathbf{J}_{\text{on}} - \nabla \mathbf{J}_{\text{off}}\|_2 \leq M T G \sqrt{2 \text{KL}(\pi_\theta \parallel \pi_{\text{ref}})}$, where $M = \sup_t |A_t(\theta)|$. This bound is vacuous in practice because $M$ diverges whenever $\pi_\theta(a_t \mid s_t) \to 0$ (the student assigns near-zero probability to a token the teacher favors, making the advantage arbitrarily large in magnitude). The $\chi^2$-based bound avoids this by using the $L_2$ constant $\sigma_A$ (which remains finite under advantage clipping) and replacing KL with $\chi^2$. Although $\chi^2 \geq \text{KL}$, advantage clipping and the implicit regularization of Lightning OPD keep $\chi^2$ small throughout training, making the bound practically meaningful.

Operational significance: this theorem guarantees that Lightning OPD's gradient updates are close to standard OPD's as long as the student doesn't drift too far from $\pi_{\text{ref}}$. The empirical evidence (Figure 3a) shows that importance weights stay near 0.94 with standard deviation below 0.1 throughout training, confirming that $\chi^2$ remains small and the gradient discrepancy bound remains tight in practice. This means Lightning OPD is not an uncontrolled offline approximation — its updates are provably close to the online updates that would be computed with a live teacher.

---

3.4.6 Theorem 3.6: Shared Fixed Point (When Teacher Is Representable)

Statement: The online objective satisfies $\mathbf{J}_{\text{on}}(\theta) = -\text{KL}(\pi_\theta \parallel \pi_T) \leq 0$, with global maximum at $\theta^* \in \arg\min_{\theta \in \Theta} \text{KL}(\pi_\theta \parallel \pi_T)$. When $\pi_T \in \Pi_\Theta$ (the teacher is representable within the student's model class), $A_t(\theta^*) = 0$ almost surely, and $\theta^*$ is a shared zero of both the online and offline OPD updates.

What it computes: the online OPD objective is exactly the negative KL-divergence from the student to the teacher, summed over the response distribution. Maximizing $\mathbf{J}_{\text{on}}$ is therefore equivalent to minimizing $\text{KL}(\pi_\theta \parallel \pi_T)$ — making the student's distribution as close as possible to the teacher's.

Why this identity holds: $\mathbf{J}_{\text{on}}(\theta) = \mathbb{E}_{x \sim \pi_\theta}\left[\sum_t (\log \pi_T(a_t \mid s_t) - \log \pi_\theta(a_t \mid s_t))\right] = \mathbb{E}_{x \sim \pi_\theta}\left[\log \frac{\pi_T(x \mid q)}{\pi_\theta(x \mid q)}\right] = -\text{KL}(\pi_\theta \parallel \pi_T)$

The subtlety is that the autoregressive factorization allows the sum of per-token log-ratio differences to be rewritten as the log-ratio of the full sequence probabilities. This connects the per-token advantage formulation back to the standard distribution-matching interpretation of knowledge distillation.

Key property: shared fixed point under representability. When the teacher can be exactly represented within the student's parameter space — i.e., there exists some $\theta^*$ such that $\pi_{\theta^*} = \pi_T$ — then at that $\theta^*$, the advantage $A_t(\theta^*) = \log \pi_T - \log \pi_{\theta^*} = 0$ for every token. Plugging this into both gradient expressions:

$\nabla \mathbf{J}_{\text{on}}(\theta^*) = \mathbb{E}_{\pi_{\theta^*}}\left[\sum_t 0 \cdot \nabla \log \pi_{\theta^*}(a_t \mid s_t)\right] = 0$ $\nabla \mathbf{J}_{\text{off}}(\theta^*) = \mathbb{E}_{\pi_{\text{ref}}}\left[\sum_t 0 \cdot \nabla \log \pi_{\theta^*}(a_t \mid s_t)\right] = 0$

Both gradients vanish at the same point — neither method would update away from $\theta^*$. This means that, when the teacher is representable, Lightning OPD and standard OPD converge to exactly the same solution.

When the teacher is not representable (capacity-limited case): the paper acknowledges that this fixed-point argument does not directly apply when $\pi_T \notin \Pi_\Theta$ — the teacher is strictly more capable than any student in the model class can represent. In this case, both methods converge to the best approximation within the student's capacity: $\min_{\theta \in \Theta} \text{KL}(\pi_\theta \parallel \pi_T)$. The irreducible approximation error $\varepsilon_{\text{approx}} = \min_{\theta} \text{KL}(\pi_\theta \parallel \pi_T)$ lower-bounds both methods, and Theorem 3.5 ensures that the per-step gradient discrepancy between the two remains small. The paper provides an informal error decomposition (Appendix A.3):

$\text{KL}(\pi_{\theta_{\text{final}}} \parallel \pi_T) \approx \underbrace{\varepsilon_{\text{approx}}}_{\text{irreducible}} + \underbrace{\varepsilon_{\text{opt}}}_{\text{optimisation}} + \underbrace{O(G \sigma_A \sqrt{\chi^2})}_{\text{offline-online gap}}$

The third term (offline-online gap) is what Lightning OPD pays for eliminating the live teacher. The decomposition shows that this term is additive to the irreducible approximation error and the optimization error, not multiplicative — meaning that switching from online to offline rollouts does not fundamentally change the asymptotic solution quality; it only adds a small controlled gap that is bounded by how far the student drifts.

Operational significance: this theorem is the paper's strongest theoretical guarantee. It states that Lightning OPD and standard OPD optimize the same underlying objective (minimizing KL to the teacher) and share the same ideal solution — they are not different algorithms converging to different points. The offline approximation only affects the path to the solution (via the gradient discrepancy bound of Theorem 3.5), not the solution itself.

---

3.4.7 Theorem 3.7: Gradient Decomposition and Implicit Regularization

Statement:

$\nabla \mathbf{J}_{\text{off}}(\theta) = \nabla \mathbf{J}_{\text{on}}(\theta) - \text{Cov}_{\pi_{\text{ref}}}[w(x; \theta), f(x; \theta)]$

where $f(x; \theta) = \sum_t A_t(\theta) \cdot \nabla \log \pi_\theta(a_t \mid s_t)$ is the per-trajectory gradient and $\text{Cov}_{\pi_{\text{ref}}}[w, f]$ is the covariance between the importance weight and the gradient vector under the reference distribution.

What it computes: the offline gradient can be expressed as the online gradient minus a covariance correction term. The correction is the expected outer product $(w - \mathbb{E}[w])(f - \mathbb{E}[f])^\top$ under $\pi_{\text{ref}}$. Since $\mathbb{E}_{\pi_{\text{ref}}}[w] = 1$ (because $w$ is an importance weight with expectation 1), this simplifies to $\mathbb{E}[(w - 1)f] - \mathbb{E}[w - 1]\mathbb{E}[f] = \mathbb{E}[(w - 1)f]$, which is exactly the difference between the importance-weighted and unweighted expectations.

Why this is a regularization effect: at initialization ($\pi_\theta = \pi_{\text{ref}}$), $w \equiv 1$ for all $x$, so the covariance is exactly the zero vector and $\nabla \mathbf{J}_{\text{off}} = \nabla \mathbf{J}_{\text{on}}$. As the student drifts from $\pi_{\text{ref}}$:

• $w(x; \theta)$ becomes large on responses that $\pi_\theta$ over-weights relative to $\pi_{\text{ref}}$,
• On those same responses, $f(x; \theta)$ (the gradient vector) also tends to have large magnitude because the advantages $A_t$ are larger when the student diverges more from the teacher,
• The covariance $\text{Cov}[w, f]$ therefore grows in norm and, when projected onto the drift direction $\theta - \theta_{\text{ref}}$, opposes further movement away from $\pi_{\text{ref}}$.

This produces what the paper calls an implicit trust-region effect: the offline update naturally penalizes large deviations from $\pi_{\text{ref}}$ without any explicit KL penalty term in the objective. The gradient decomposition reveals that the covariance subtraction acts as a restoring force — it subtracts a component aligned with the drift direction from the online gradient, keeping the student near the reference distribution.

Why this matters for training stability: standard OPD often requires an explicit KL penalty (e.g., $\beta \cdot \text{KL}(\pi_\theta \parallel \pi_{\text{ref}})$) to prevent the student from collapsing to a degenerate distribution or overfitting to the teacher's scoring. Lightning OPD achieves this stabilization automatically through the structure of the offline objective. The empirical evidence in Figure 3a validates this: importance weights remain near 0.94 with standard deviation below 0.1 throughout training, confirming that the student does not drift far from $\pi_{\text{ref}}$ despite no explicit KL penalty being applied.

Operational significance: this theorem explains why Lightning OPD works well in practice without needing to tune a KL penalty coefficient. The fixed rollout distribution provides built-in stabilization. It also explains why naive offline OPD (without teacher consistency) can fail: if the SFT teacher differs from the OPD teacher, the implicit regularization anchors the student to the wrong reference distribution, and the restoring force pulls the student away from the OPD teacher rather than keeping it close to a meaningful reference.

---

3.4.8 Theorem 3.8: Teacher Consistency and the Offline-Online Gap

This theorem extends Theorem 3.5 to the case where the SFT-stage teacher ($\pi_T^{\text{SFT}}$) and the OPD-stage teacher ($\pi_T^{\text{OPD}}$) are different models.

Statement:

$\|\nabla \mathbf{J}_{\text{on}}(\theta) - \nabla \mathbf{J}_{\text{off}}(\theta)\|_2 \leq G \cdot (\sigma_A + \sigma_\Delta) \cdot \sqrt{\chi^2(\pi_\theta \parallel \pi_{\text{ref}})}$

where $\sigma_\Delta$ is the bound on the teacher mismatch from Assumption 3.4. Additionally, the mismatched offline gradient carries a persistent bias: $\|\nabla \mathbf{J}_{\text{off}}(\theta) - \nabla \mathbf{J}_{\text{off}}^{\delta=0}(\theta)\|_2 \leq G \sigma_\Delta$, where $\nabla \mathbf{J}_{\text{off}}^{\delta=0}$ is the consistent offline gradient, and this bias is independent of $\chi^2$ — it persists even at initialization.

Symbol definitions:

• $\sigma_\Delta$ comes from Assumption 3.4: define $\Delta_t = \log \pi_T^{\text{SFT}}(a_t \mid s_t) - \log \pi_T^{\text{OPD}}(a_t \mid s_t)$ as the per-token difference between the two teachers' log-probabilities. Assumption 3.4 states $\mathbb{E}_{x \sim \pi_{\text{ref}}}[(\sum_t |\Delta_t|)^2] \leq \sigma_\Delta^2$. When teacher consistency holds ($\pi_T^{\text{SFT}} = \pi_T^{\text{OPD}}$), $\Delta_t = 0$ everywhere and $\sigma_\Delta = 0$.
• $\nabla \mathbf{J}_{\text{off}}^{\delta=0}$ is the offline gradient that would be computed if the same teacher were used for both SFT and OPD (i.e., the consistent case).

What this theorem reveals:

1. The offline-online gap is larger under teacher mismatch. The gradient discrepancy bound inflates from $G \sigma_A \sqrt{\chi^2}$ (Theorem 3.5) to $G (\sigma_A + \sigma_\Delta) \sqrt{\chi^2}$. The mismatch adds an extra term proportional to how different the two teachers are ($\sigma_\Delta$).

2. The offline gradient has a persistent bias even at initialization. When $\pi_\theta = \pi_{\text{ref}}$, $\chi^2 = 0$, so the offline-online gap is zero as in Theorem 3.5 — but now the offline gradient itself is biased relative to the consistent offline gradient by $G \sigma_\Delta$. This bias is independent of policy drift and persists even at step 0, meaning the very first Lightning OPD update under teacher mismatch is corrupted — it points in a direction that would not be taken under teacher consistency.

3. Why naive offline OPD fails. The offline gradient under mismatch decomposes into the consistent offline gradient plus a bias term from the mismatch: $\nabla \mathbf{J}_{\text{off}} = \nabla \mathbf{J}_{\text{off}}^{\delta=0} + \mathbb{E}_{\pi_{\text{ref}}}[f_\Delta]$, where $f_\Delta(x; \theta) = -\sum_t \Delta_t \cdot \nabla \log \pi_\theta(a_t \mid s_t)$. The bias term $\mathbb{E}_{\pi_{\text{ref}}}[f_\Delta]$ is non-zero whenever $\sigma_\Delta > 0$, and it pushes the student toward a distribution that reconciles two conflicting teacher signals — one from SFT (embedded in $\pi_{\text{ref}}$) and one from OPD (embedded in the advantage function). No amount of training can eliminate this bias because it is a structural property of the gradient field, not an estimation artifact.

Operational significance: this theorem explains the empirical finding that "naively applying this offline precomputation fails to reliably match the performance of standard OPD" (Section 1). The failure is not due to the offline approximation being inaccurate (the gradient discrepancy bound at $\chi^2 = 0$ is zero regardless of mismatch), but due to the offline gradient itself being biased toward the wrong reference distribution. The fix is not to improve the offline approximation (e.g., better importance sampling) but to enforce teacher consistency so that $\sigma_\Delta = 0$ and the bias vanishes.

---

3.4.9 Theorem 3.9: Teacher Consistency and Standard OPD

Statement: Let $\nabla \mathbf{J}_{\text{on}}^{\delta=0}(\theta)$ denote the standard OPD gradient under a consistent teacher ($\pi_T^{\text{SFT}} = \pi_T^{\text{OPD}}$). At initialization $\theta = \theta_{\text{ref}}$:

$\|\nabla \mathbf{J}_{\text{on}}(\theta_{\text{ref}}) - \nabla \mathbf{J}_{\text{on}}^{\delta=0}(\theta_{\text{ref}})\|_2 \leq G \cdot \sigma_\Delta$

What it reveals: teacher mismatch degrades standard (online) OPD as well, not just Lightning OPD. At initialization, the mismatched online gradient differs from the consistent online gradient by up to $G \sigma_\Delta$ — the exact same bias magnitude we saw for the offline gradient in Theorem 3.8. When $\sigma_\Delta > 0$, the standard OPD gradient at step 0 is pointing in a direction corrupted by the teacher mismatch, and the training dynamics are biased away from the consistent fixed point.

Why this matters for OPD practice beyond Lightning OPD: this theorem establishes teacher consistency as a general design principle for OPD, not specific to the offline setting. The paper's motivating example — Thinking Machines Lab using QwQ-32B for SFT data generation and Qwen3-32B for OPD teacher — is predicted to degrade standard OPD performance, not just Lightning OPD. The empirical validation in Table 4 confirms this: at the 8B scale, standard OPD drops from 68.5% (consistent: Qwen3-32B both stages) to 65.0% (inconsistent: QwQ-32B SFT, Qwen3-32B OPD), a 3.5-point degradation.

Why previous OPD work didn't notice this: the paper argues that prior OPD practice inherited conventions from RLVR, where the SFT and RL stages are decoupled — the reward signal is independent of the teacher used for SFT. In OPD, the teacher's token-level distribution defines both the SFT training target and the OPD advantage signal, coupling the stages in a way that RLVR practitioners may not have recognized. The teacher consistency principle is a co-design constraint that was "largely ignored in prior OPD practice" (Section 2).

Why the bias cannot be eliminated by more training: the bias $G \sigma_\Delta$ is a property of the gradient field itself — it comes from the fact that the advantage function $A_t(\theta) = \log \pi_T^{\text{OPD}} - \log \pi_\theta$ is evaluated relative to the OPD teacher, but the reference policy $\pi_{\text{ref}}$ was trained on trajectories from the SFT teacher. These two teacher signals are structurally inconsistent, and the gradient at every step reflects this inconsistency. Importance sampling corrections cannot fix it because the issue is not that the offline expectation is a poor approximation of the online expectation — it's that the online expectation itself is evaluating the wrong thing (advantages relative to the OPD teacher on a distribution anchored to the SFT teacher).

Operational significance: teacher consistency is not just a trick to make Lightning OPD work; it is a necessary condition for OPD to converge to the intended solution in both online and offline settings. This recontextualizes the paper's contribution: Lightning OPD is not merely an efficient deployment framework but the natural consequence of enforcing a design principle that was already required for OPD correctness.

---

3.4.10 The Assumptions: When Do the Theoretical Guarantees Hold?

All proofs in Appendix A rely on four assumptions, stated in Section 3.3. Understanding their scope is essential for assessing when the theory applies.

Assumption 3.1 (Bounded Absolute Advantage). There exists $\sigma_A < \infty$ such that for all $\theta \in \Theta$:

$\mathbb{E}_{x \sim \pi_{\text{ref}}}\left[\left(\sum_{t=1}^T |A_t(\theta)|\right)^2\right] \leq \sigma_A^2$

What it means: the expected squared sum of absolute per-token advantages, taken over responses from the reference policy, is bounded by a finite constant. This is an $L_2$ condition on how large the per-token log-ratio differences can be in aggregate.

When it holds: advantage clipping to $[-10, 10]$ (the practical setting in Table 6) provides a trivial bound: $|A_t(\theta)| \leq 10$ for all tokens, so $(\sum_t |A_t|)^2 \leq (10T)^2 = 100 T^2$, giving $\sigma_A \leq 10 T$ where $T$ is the maximum response length. For $T = 4096$ (the training response length), this is $\sigma_A \leq 40,960$, which is finite and satisfies the assumption. Without clipping, advantages could theoretically diverge when the student assigns near-zero probability to tokens the teacher favors, making $\sigma_A$ potentially unbounded. Advantage clipping is therefore not just a practical stabilization trick — it is what makes the theoretical analysis well-defined.

Assumption 3.2 (Support Coverage). For all $\theta$ encountered during optimization and all prompts $q$:

$\text{supp}(\pi_\theta(\cdot \mid q)) \subseteq \text{supp}(\pi_{\text{ref}}(\cdot \mid q))$

What it means: any response the current student can generate must also be generatable by the reference policy. The student's support is a subset of the reference's support.

When it holds: this is the standard coverage assumption in importance sampling and off-policy evaluation. It holds naturally when $\pi_\theta$ is initialized from $\pi_{\text{ref}}$ and trained with finite step sizes, because the student distribution is a smooth perturbation of the reference — it may increase or decrease probabilities of existing tokens, but it does not suddenly invent new tokens that had zero probability under $\pi_{\text{ref}}$. In practice, LLM softmax outputs assign non-zero probability to all tokens in the vocabulary (due to the softmax temperature), so the support is actually the entire vocabulary for all policies, and the assumption holds trivially. The finite-vocabulary setting (noted in Appendix A.1) makes this especially well-behaved.

Assumption 3.3 (Bounded Score Function). There exists $G < \infty$ such that $\|\nabla \log \pi_\theta(a_t \mid s_t)\|_2 \leq G$ for all $\theta \in \Theta$ and all $t$.

What it means: the gradient of the log-probability with respect to the model parameters has bounded Euclidean norm for every token. This is a smoothness condition on the policy parameterization.

When it holds: this is standard in policy gradient analyses and is satisfied by neural network architectures with Lipschitz-continuous activations and bounded parameter norms. In practice, gradient clipping (commonly used in LLM training) ensures this bound holds. The assumption prevents pathological cases where a small change in parameters causes an arbitrarily large change in log-probability for some token, which would make gradient-based optimization unstable.

Assumption 3.4 (Bounded Teacher Mismatch). Define $\Delta_t = \log \pi_T^{\text{SFT}}(a_t \mid s_t) - \log \pi_T^{\text{OPD}}(a_t \mid s_t)$. There exists $\sigma_\Delta < \infty$ such that:

$\mathbb{E}_{x \sim \pi_{\text{ref}}}\left[\left(\sum_{t=1}^T |\Delta_t|\right)^2\right] \leq \sigma_\Delta^2$

What it means: the per-token log-probability difference between the two teachers, summed and squared in expectation under the reference distribution, is bounded by a finite constant. When teacher consistency holds ($\pi_T^{\text{SFT}} = \pi_T^{\text{OPD}}$), $\Delta_t = 0$ everywhere and $\sigma_\Delta = 0$ — this case is the baseline for Theorems 3.5–3.7.

When it holds: for any pair of finite-precision models, the per-token log-probability difference is bounded (log-probabilities are finite because softmax outputs are never exactly 0 or 1 in practice), so the sum over a finite number of tokens is finite, and the expectation under any distribution is finite. The assumption is always satisfied for real models; the nontrivial question is the magnitude of $\sigma_\Delta$. When the two teachers are similar (e.g., Qwen3-32B vs. Qwen3-32B-Instruct), $\sigma_\Delta$ is small. When they are different (e.g., Qwen3-32B vs. QwQ-32B, which are different model families with different training procedures), $\sigma_\Delta$ is larger. The paper's empirical ablation (Table 4) confirms that QwQ-32B as SFT teacher with Qwen3-32B as OPD teacher (or vice versa) causes substantial performance degradation, consistent with larger $\sigma_\Delta$.

When the assumptions hold in the paper's experiments:

• Assumptions 3.1–3.3 are standard and satisfied by the practical training setup (advantage clipping, softmax over finite vocabulary, gradient clipping).
• Assumption 3.4 with $\sigma_\Delta = 0$ holds in the main Lightning OPD experiments (Tables 1–3) because the same teacher is used for SFT and OPD by construction.
• Assumption 3.4 with $\sigma_\Delta > 0$ is tested in the ablation (Table 4) where teachers are intentionally mismatched, confirming the theoretical prediction that performance degrades.

A note on what the assumptions do NOT require: the theory does not require that the student actually converge to the teacher (the capacity-limited case is explicitly acknowledged in Theorem 3.6's discussion), does not require that the offline dataset be large or diverse (only that expectations under $\pi_{\text{ref}}$ are well-defined), and does not require that the teacher be a specific architecture or scale — only that it be the identical model in both stages. This makes the theory relatively permissive in practice, with the single non-trivial requirement being teacher consistency.

---

3.4.11 Design Choices and Their Justifications

Why precompute log-probabilities rather than probabilities, logits, or distilled features? Log-probabilities are the natural form because the OPD advantage is defined as $A_t = \log \pi_T - \log \pi_\theta$ — a difference of log-probabilities. Storing raw logits would require storing a vector of size $|\mathcal{V}|$ (vocabulary size, typically 50K–250K tokens) per token, which is memory-prohibitive. Storing probabilities would lose numerical precision in the tails (where $\pi_T(a_t \mid s_t)$ can be very small, and $\log \pi_T$ would underflow). Storing the scalar log-probability for only the token that actually appears in the response is exponentially more compact and numerically stable.

Why sample rollouts from $\pi_{\text{ref}}$ rather than from $\pi_T$? As discussed in Section 3.4.2, sampling from $\pi_{\text{ref}}$ preserves the on-policy characteristic: the teacher evaluates student-generated responses, providing corrective signals on the student's own distribution. This is what distinguishes Lightning OPD from offline knowledge distillation and is the core reason the offline approximation remains faithful to the online objective.

Why a single response per prompt in $\mathcal{D}_{\text{OPD}}$? The paper states that for each prompt, "we sample a single response from $\pi_{\text{ref}}$" (Section 4.1). This is a practical choice balancing dataset size and diversity. Multiple responses per prompt would provide more coverage of the reference distribution but would linearly increase the teacher scoring cost in the preprocessing phase. The paper does not ablate this choice, but the strong empirical results suggest a single response per prompt is sufficient when the prompt dataset is large (17K for math, 30K for code).

Why 150 training steps? Figure 3b shows the AIME 2024 score converges within the first 50 steps and remains stable to 150. Training beyond this point would not improve performance and risks overfitting to the fixed dataset. The short training duration is a feature, not a limitation — it contributes to the training efficiency gains.

Why constant learning rate with no warmup or decay in the OPD stage? The OPD stage is short (150 steps), and the advantage signal naturally decays as the student approaches the teacher (since $A_t \to 0$ when $\pi_\theta \approx \pi_T$). This provides an implicit form of learning rate annealing. A constant learning rate is simpler and avoids the need to tune a schedule, which is consistent with the paper's goal of making OPD more accessible. The SFT stage uses cosine decay because it is substantially longer (3000 steps) and benefits from standard LR scheduling practices.

Why code OPD is initialized from the math-trained checkpoint (Appendix B)? The paper states this "consistently outperforms initializing code OPD from the SFT model" and cites prior work (Nemotron-Cascade) showing math reasoning provides a stronger initialization for code. This is a transfer learning observation, not a core methodological contribution, but it is a practical recipe choice that contributes to the strong code results.

Why not use KL-divergence as the objective instead of advantage-weighted policy gradient? The paper's formulation follows standard OPD practice, which uses policy gradient with stop-gradient advantages rather than directly minimizing the reverse KL via gradient descent on $\text{KL}(\pi_\theta \parallel \pi_T)$. The advantage-weighted policy gradient is what enables the theoretical analysis (connection to importance sampling, covariance decomposition, implicit regularization). A direct KL minimization would produce different gradient dynamics and might not exhibit the same implicit trust-region properties. The paper does not claim the policy-gradient formulation is superior to direct KL minimization — it inherits this choice from the OPD literature and shows that under this formulation, teacher consistency is the key condition.

Why teacher consistency specifically requires the same model, not just a similar model? The theoretical analysis requires $\sigma_\Delta = 0$ for the bias terms to vanish. A similar but not identical model would have $\sigma_\Delta > 0$, introducing the bias characterized in Theorems 3.8–3.9. The magnitude of the degradation depends on how different the models are — similar models from the same family might have small $\sigma_\Delta$ and suffer minimal degradation, while models from different families (Qwen3 vs. QwQ) have larger $\sigma_\Delta$ and show more substantial degradation (Table 4: up to 6.8 points for Lightning OPD at 8B scale). The paper does not explore the continuous relationship between model similarity and performance degradation, but the theoretical framework predicts a monotonic relationship: larger $\sigma_\Delta$ → larger gradient bias → worse convergence.

4. Key Insights and Innovations

Innovation 1: Teacher Consistency as a Necessary Design Principle for On-Policy Distillation

The paper's most conceptually distinctive contribution is not a new algorithm but a diagnostic finding: that a specific condition — teacher consistency, the requirement that the identical teacher model be used for both the SFT and OPD stages — is a previously unrecognized prerequisite for on-policy distillation to work correctly, and that its violation explains why naive offline OPD fails and why standard OPD underperforms in common deployment practice. This is a conceptual shift rather than a metric improvement: the paper reframes OPD from a two-stage pipeline where SFT and distillation are independent choices into a coupled system with a hard coupling constraint.

The prior default assumption was that SFT and OPD teachers are interchangeable. The field inherited this assumption from RLVR post-training pipelines, where the SFT stage and the RL stage are decoupled: the SFT dataset is curated using whichever model produces the highest-quality demonstrations, and the RL reward signal (e.g., unit test pass/fail) is independent of that choice. OPD practitioners imported this same convention without recognizing that OPD creates a fundamentally different coupling. The paper's motivating example in Section 1 makes this concrete: Thinking Machines Lab trained Qwen3-8B-Base on OpenThoughts-3 trajectories generated by QwQ-32B while using Qwen3-32B as the OPD teacher — a teacher-mismatch that the paper's theory (Theorems 3.8–3.9) predicts to be detrimental. Prior OPD work — from Agarwal et al. (2024) through the variants in Section 2 — did not identify or analyze this constraint.

Why this is a reframing rather than a heuristic trick. The paper proves that teacher mismatch introduces an irreducible gradient bias of magnitude $G\sigma_\Delta$ that persists at initialization (Theorem 3.9) and corrupts the offline gradient independently of policy drift (Theorem 3.8). The bias is a structural property of the gradient field — it arises because the advantage function $A_t = \log \pi_T^{\text{OPD}} - \log \pi_\theta$ evaluates the student relative to the OPD teacher, but the reference policy $\pi_{\text{ref}}$ was trained on trajectories from a different SFT teacher, creating a fundamental inconsistency in the optimization landscape. No amount of training, data, or importance-sampling correction can remove it. This is fundamentally different from the standard offline-RL challenge (OOD value estimation), which offline RL methods address through conservatism — the OPD challenge is not distributional coverage but a teacher-signal consistency that offline RL methods are not designed to handle (a point the paper develops in Appendix D).

The empirical force of the claim is clearest in Table 4. At the 8B scale, standard OPD drops from 68.5% (Qwen3-32B for both SFT and OPD) to 65.0% (QwQ-32B SFT, Qwen3-32B OPD) — a 3.5-point degradation for online OPD, confirming Theorem 3.9. Lightning OPD drops even more sharply under mismatch: from 69.9% to 62.1% with the same cross, a 7.8-point degradation. This asymmetry is theoretically motivated: Theorem 3.8 shows that teacher mismatch adds a persistent bias term to the offline gradient that is independent of $\chi^2$, and the fixed rollout distribution of Lightning OPD compounds the damage because $\pi_{\text{ref}}$ is anchored to the "wrong" teacher in two ways (both as the rollout source and as the reference for implicit regularization). The result is more than a performance delta — it is evidence that teacher consistency is a necessary condition for OPD correctness, not a helpful optimization, and that violating it is more costly for offline OPD than online, which previous attempts at offline OPD (like Rang et al., 2025) did not anticipate because they lacked the theoretical analysis.

The significance beyond this paper. Teacher consistency implies that OPD practitioners cannot freely curate SFT data from multiple teacher sources (a common practice in RLVR pipelines). When adopting a new OPD teacher, the SFT dataset must be regenerated — a nontrivial cost that the paper acknowledges as a limitation (Appendix E). But the theoretical contribution is that this cost is not optional; it is required for correctness. This changes the design conversation around OPD from "which teachers work best independently in each stage?" to "which teacher provides the best coupled SFT+OPD pipeline?", a more constrained but more principled optimization problem.

---

Innovation 2: A Principled Offline Reduction of an Online Objective — with Formal Guarantees

The paper's second distinctive contribution is the theoretical framework establishing that offline OPD is not a heuristic approximation but a principled reduction of the online objective, with provable guarantees on optimum equivalence, gradient discrepancy bounds, and implicit regularization — all conditioned on the single, verifiable requirement of teacher consistency. This is significant because it converts a practical efficiency question ("can we precompute teacher scores?") into a formal statement: Lightning OPD and standard OPD optimize the same underlying objective (minimizing KL to the teacher, Theorem 3.6) and converge to the same solution, with the offline approximation's fidelity controlled entirely by how far the student drifts from $\pi_{\text{ref}}$ (Theorem 3.5).

Prior work on offline distillation lacked this formal scaffolding. Rang et al. (2025) independently explored precomputing teacher signals over student-generated responses, but they formulated distillation as supervised learning with a composite loss and provided no analysis of when or why the offline approximation is reliable. The broader offline KD literature (Kim & Rush, 2016; Gu et al., 2024) trains on teacher-generated sequences, which is a fundamentally different distribution from the student's own outputs. Lightning OPD's contribution is not just "precompute teacher scores" — it is the identification that the offline approximation is faithful if and only if the SFT teacher and OPD teacher are the same model, and the construction of the theoretical machinery (importance-sampling decomposition, $\chi^2$-based discrepancy bounds, covariance decomposition) to prove this claim.

The shared fixed-point theorem (Theorem 3.6) is the strongest formal guarantee in the paper. It shows that the online OPD objective is exactly the negative KL-divergence: $\mathbf{J}_{\text{on}}(\theta) = -\text{KL}(\pi_\theta \parallel \pi_T)$. When the teacher is representable within the student's model class, both online and offline OPD have zero gradient at the same optimal parameters $\theta^*$ — a shared global optimum. This means Lightning OPD and standard OPD are not different algorithms converging to different points; they are different gradient dynamics converging to the same solution. The offline approximation only affects the path, not the destination. When the teacher is not representable (the capacity-limited case), both methods are bounded by the same irreducible approximation error, and the gradient discrepancy bound (Theorem 3.5) controls how different the paths can be. The paper's informal error decomposition in Appendix A.3 makes this explicit: the offline-online gap is an additive $O(G\sigma_A\sqrt{\chi^2})$ term on top of the irreducible capacity-limited error, not a multiplicative degradation.

The gradient discrepancy bound (Theorem 3.5) is tighter than a naive KL-based alternative. The paper explicitly shows in Appendix A.2 that a Pinsker-based bound would be $\|\nabla\mathbf{J}_{\text{on}} - \nabla\mathbf{J}_{\text{off}}\|_2 \leq M T G \sqrt{2\text{KL}(\pi_\theta \parallel \pi_{\text{ref}})}$, where $M = \sup_t |A_t(\theta)|$ can diverge when $\pi_\theta(a_t \mid s_t) \to 0$. The $\chi^2$-based bound avoids this divergence by using the $L_2$ constant $\sigma_A$ (finite under advantage clipping) and the $\chi^2$-divergence (which remains small when policy drift is small, as Figure 3a empirically validates). This is a nontrivial theoretical refinement: it means the bound is not vacuous in practice, and the empirical evidence (importance weights near 0.94, standard deviation below 0.1) provides direct validation that $\chi^2$ stays small and the gradient discrepancy remains tight throughout training.

The implicit regularization insight (Theorem 3.7) reframes an apparent weakness as a strength. One might expect that fixing the rollout distribution to $\pi_{\text{ref}}$ would cause the student to overfit to the static dataset. The gradient decomposition shows the opposite: $\nabla\mathbf{J}_{\text{off}} = \nabla\mathbf{J}_{\text{on}} - \text{Cov}[w, f]$, and the covariance term acts as a restoring force that opposes further drift from $\pi_{\text{ref}}$. This is not a bug introduced by the offline approximation — it is a structural property that provides built-in trust-region stabilization without any explicit KL penalty term. Standard OPD often requires tuning a KL penalty coefficient to prevent policy collapse; Lightning OPD gets this stabilization automatically from the covariance subtraction. Figure 3a provides empirical confirmation: importance weights plateau near 0.94 with bounded standard deviation, neither collapsing to zero nor exploding — exactly the dynamics predicted by the implicit regularization analysis.

Connecting theory to the 4.0× efficiency claim. The practical force of this theoretical framework is that it transforms the 4.0× training speedup (Table 2) from an attractive empirical observation into a theoretically justified efficiency gain. One could have observed the speedup without the theory, but the theory explains why the speedup doesn't come at the cost of solution quality — the offline approximation is provably faithful under teacher consistency. This matters for adoption: practitioners can deploy Lightning OPD with confidence that they are not trading correctness for efficiency.

---

Innovation 3: Empirical Demonstration That Offline Approximation Preserves On-Policy Benefits

The third distinctive contribution is the empirical validation that the on-policy advantage of OPD — dense per-token supervision on student-generated mistakes — survives the transition to an entirely offline training procedure. This is not a theoretical guarantee (the shared fixed point and bounded discrepancy cover the optimization landscape) but an empirical finding about the practical efficacy of the simplified pipeline: precomputed teacher signals on frozen student rollouts provide supervision that is practically as effective as fresh teacher scores on continuously evolving student rollouts, across model scales, task domains, and architecture types.

Why this was not obvious a priori. The appeal of on-policy methods is precisely that they adapt to the student's changing distribution: as the student improves, the teacher scores its current errors, providing corrective signals on the mistakes the student is actually making at that moment. Lightning OPD replaces this adaptive feedback with a static dataset of errors from the SFT-initialized student. One might reasonably expect that, as the student improves during OPD training, the precomputed teacher signals would become less relevant — the student would stop making the specific errors present in the frozen dataset and would need teacher feedback on new errors that the precomputed dataset doesn't contain. The paper's empirical finding (Table 1, across all five benchmarks and both model scales) is that this adaptive gap is small enough in practice that the static dataset suffices. Lightning OPD either matches or marginally exceeds standard OPD on every benchmark (e.g., 69.9% vs. 68.5% on AIME 2024 at 8B), with the biggest win being a 2.5-point advantage on HMMT 2025 at 8B (41.9% vs. 39.4%).

Why the static dataset works despite the distribution shift. The paper's theory and experiments together provide a two-part explanation. First, the student doesn't drift far from $\pi_{\text{ref}}$ during OPD training (Figure 3a: importance weights stay near 0.94), so the frozen rollouts remain a reasonable approximation of the student's distribution throughout training. The implicit regularization of Theorem 3.7 actively constrains this drift, making it self-limiting rather than something that requires explicit mitigation. Second, the per-token advantage signal is sufficiently rich that even on a static error set, it provides enough information to push the student toward the teacher's distribution. The teacher's log-probabilities on these specific student errors encode broad information about relative token preferences that generalizes beyond the specific errors present in the dataset. This is an empirical property that the theoretical framework does not guarantee — Theorem 3.5 bounds the gradient discrepancy but doesn't prove that the static dataset provides sufficient supervision quality — and its validation is a genuine empirical contribution.

The cross-domain transfer result reinforces this finding. When code OPD is initialized from the math-trained checkpoint (Appendix B), the student receives code-specific teacher supervision on code rollouts, but the math-trained initialization provides a stronger starting point than the SFT model. This is a practical recipe choice, but it also demonstrates that the OPD training process is robust to the source of initialization — the precomputed teacher signals on code rollouts are effective even when the student's prior is shaped by math training, suggesting that the supervision is general enough to guide the student regardless of its starting distribution (within reason).

The MoE result (Table 3) extends the finding to a regime where the online baseline is infeasible. At the 30B MoE scale, standard OPD runs out of memory on a single 8×H100 node. Lightning OPD's offline design makes it feasible — and the resulting model achieves 71.0% on AIME 2024, which is competitive with or superior to state-of-the-art open MoE models at this scale. This is significant because it demonstrates that Lightning OPD's offline approach is not just an efficiency improvement over standard OPD but an enabling capability: it makes on-policy distillation possible in resource-constrained settings where the online version is simply not an option. This lowers the barrier for academic research on LLM post-training, which is one of the paper's stated goals — but it also proves that the offline approximation's fidelity holds at a scale where direct comparison to the online baseline is impossible, making the theoretical guarantees (which apply independently of scale) the only formal assurance of correctness.

A subtle but important negative result: the offline approximation fails without teacher consistency, proving that the approximation's fidelity is conditional. The ablation in Table 4 shows that Lightning OPD is more sensitive to teacher mismatch than standard OPD (7.8-point drop vs. 3.5-point drop at 8B with Qwen3-32B SFT and QwQ-32B OPD). This is an important empirical finding because it clarifies that the offline approximation is not universally robust — it amplifies the consequences of teacher inconsistency. The theoretical explanation (Theorem 3.8's persistent bias term $G\sigma_\Delta$ being independent of policy drift) means that Lightning OPD's fidelity is strictly conditional on the teacher consistency requirement being satisfied. This is a strength of the paper's narrative: it does not claim Lightning OPD is a drop-in replacement for standard OPD under all conditions, but rather that it is equivalent under a specific, well-characterized condition, and that the condition itself is independently necessary for OPD correctness. The empirical failure under mismatch validates the theory.

---

Innovation 4: The Implicit Regularization Mechanism as a Self-Stabilizing Property of Offline OPD

The fourth contribution is the identification and formal analysis of an implicit trust-region effect that emerges naturally from the offline OPD objective, providing training stabilization without the explicit KL penalties that standard OPD often requires. This is a conceptual contribution about why Lightning OPD works well rather than just that it works: the fixed rollout distribution is not merely a computational convenience but a structural feature that induces beneficial regularization.

Standard OPD requires explicit stabilization. In online OPD, the student can potentially drift far from $\pi_{\text{ref}}$ because it continuously samples fresh rollouts from its evolving distribution. If the teacher's scoring is imperfect or over-optimized, the student can chase spurious teacher preferences and collapse to degenerate outputs. Practitioners typically add an explicit KL penalty term $\beta \cdot \text{KL}(\pi_\theta \parallel \pi_{\text{ref}})$ to the objective to prevent this. This introduces a hyperparameter $\beta$ that requires tuning and can trade off between alignment to the teacher and stability.

Lightning OPD gets this stabilization for free. Theorem 3.7 shows that the offline gradient decomposes as $\nabla\mathbf{J}_{\text{off}} = \nabla\mathbf{J}_{\text{on}} - \text{Cov}[w, f]$. The covariance term is not an arbitrarily added regularizer — it is a mathematical consequence of fixing the rollout distribution. When the student begins to drift from $\pi_{\text{ref}}$, the importance weight $w(x;\theta)$ becomes large on responses that $\pi_\theta$ over-weights. On those same responses, the per-trajectory gradient $f(x;\theta)$ also tends to have large magnitude (because advantages grow as the student diverges from the teacher). The covariance between $w$ and $f$ thus grows in norm and, when projected onto the drift direction, opposes further movement. This is a restoring force that grows proportionally with drift — the farther the student moves from $\pi_{\text{ref}}$, the stronger the pull back.

The empirical evidence in Figure 3a validates this mechanism directly. The mean importance weight drops from 1.0 to ~0.94 within the first 20 steps and then plateaus. The standard deviation rises in the same early phase before stabilizing below 0.1. Both quantities stabilizing at moderate levels is exactly what the implicit regularization predicts: the student initially drifts away from $\pi_{\text{ref}}$ as it aligns toward the teacher, but the covariance correction kicks in and prevents further divergence, locking the student into a narrow band around the reference. The fact that this stabilization occurs without any explicit KL penalty in the training hyperparameters (Table 6 shows no KL coefficient) is strong evidence that the implicit mechanism is both present and sufficient.

Why this matters beyond this paper. The implicit regularization insight suggests a broader principle: fixing the data distribution in on-policy methods can induce self-stabilization that replaces explicit trust-region mechanisms. This is potentially relevant beyond OPD — for on-policy RL methods (PPO, GRPO) that currently use clipped objectives or KL penalties to constrain policy updates, an offline data collection phase might provide implicit regularization that simplifies hyperparameter tuning. The paper does not explore this generalization, but the mechanism it identifies (covariance subtraction between importance weights and gradients) is mathematically general and could apply to any advantage-weighted policy gradient over a fixed dataset. This is a theoretical contribution with potential downstream implications for the broader post-training literature.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The SFT stage uses prompts from OpenThoughts-3 (Guha et al., 2025), with responses generated by the respective teacher model. For the OPD stage, mathematical reasoning uses DAPO-Math-17k (Yu et al., 2025), which provides 17K competition-level math problems spanning a wide range of difficulty. Code generation uses a sampled 30K subset of EpiCoder-func-380k (Wang et al., 2025), providing diverse function-level code synthesis problems. For each prompt in the OPD stage, the paper samples a single response from π_ref and precomputes the corresponding teacher log-probabilities once prior to training, with no teacher server required during the OPD stage (Section 4.1).

• Base model(s). The paper trains two student models from the Qwen3 model family (Yang et al., 2025): Qwen3-4B-Base with Qwen3-8B as teacher, and Qwen3-8B-Base with Qwen3-32B as teacher. Both follow the two-stage Lightning OPD pipeline described in Section 3. For the MoE experiment, the paper uses Qwen3-30B-A3B-Base (a 30B-parameter Mixture-of-Experts model with 3B active parameters) with Qwen3-30B-A3B-Thinking-2507 as teacher (Section 4.4). The models are chosen to cover both dense and sparse architectures and to span a range of student–teacher capacity gaps (from a 2× parameter ratio at 4B to a 4× ratio at 8B).

• Metrics. The primary metric is Pass@1 — the fraction of problems for which the model's first generated solution matches the ground-truth answer. For math benchmarks (AIME 2024, AIME 2025, HMMT 2025), 32 solutions are sampled per problem and the average pass@1 is reported. For code benchmarks (LiveCodeBench v5 and v6), 4 solutions are sampled per problem and the average pass@1 is reported. All evaluations use temperature 0.6, top-p 0.95, and maximum generation lengths of 32,768 tokens for math and 40,960 for code (Section 4.1). The per-benchmark averages across AIME 2024, AIME 2025, and HMMT 2025 are reported under "Avg." for math, and similarly for LiveCodeBench v5 and v6 under "Avg." for code (Table 1, Table 3).

• Baselines. The paper compares Lightning OPD against three baselines: (1) SFT — the reference policy after supervised fine-tuning but before any OPD training, representing the starting point; (2) Standard OPD (Agarwal et al., 2024; Lu & Thinking Machines Lab, 2025) — the online on-policy distillation baseline requiring a live teacher server throughout training, sharing identical OPD-stage hyperparameters with Lightning OPD and differing only in that standard OPD samples rollouts online from the current student while Lightning OPD reuses precomputed rollouts from π_ref; (3) ExOPD (Yang et al., 2026a) — a recent OPD variant using reward extrapolation, reported only at the 4B scale. The MoE experiment (Table 3) reports OPD as "OOM" (out of memory) because co-hosting both a 30B student and 30B teacher on a single 8×H100 node exceeds available GPU memory, making standard OPD infeasible at this scale under the paper's hardware constraint.

• Generation budget / compute accounting. The paper measures training cost in GPU hours (Table 2), aggregating across all phases. For standard OPD, this includes the continuous teacher server running alongside training. For Lightning OPD, the cost is broken into three phases: rollout collection, teacher log-probability precomputation (one-time offline operations), and OPD training. All experiments at a given scale use identical OPD-stage hyperparameters (150 steps, global batch size 256, maximum response length 4,096 tokens, learning rate 2 × 10⁻⁶, advantage clip range [−10, 10]; Table 6). The paper reports speedup as the ratio of total GPU hours: 3.6× at 4B (20 vs. 72 hours) and 4.0× at 8B (30 vs. 120 hours).

• Cross-validation / statistical protocol. The paper does not employ cross-validation for the main results; all benchmarks are evaluated on their standard test sets. For the teacher consistency ablation (Section 4.5), a full 2×2 grid of SFT-teacher and OPD-teacher combinations is tested, with QwQ-32B (Qwen Team, 2025) introduced as an alternative teacher alongside Qwen3-8B (at 4B scale) and Qwen3-32B (at 8B scale) to systematically evaluate the effect of teacher mismatch. The paper does not report confidence intervals or statistical significance tests for the pass@1 numbers, which is a limitation given the finite sample sizes (AIME 2024 has 30 problems, AIME 2025 has 30, HMMT 2025 and LiveCodeBench have varying sizes).

Main Quantitative Results

Lightning OPD Matches Standard OPD Across All Benchmarks and Scales (Table 1)

The central empirical finding is that Lightning OPD, despite eliminating the live teacher server entirely, achieves performance on par with standard OPD across all five benchmarks and both model scales, and in several cases marginally exceeds it.

At the 4B scale (Qwen3-4B-Base student, Qwen3-8B teacher):

• Lightning OPD achieves 68.1% on AIME 2024, compared to 65.4% for standard OPD and 56.7% for the SFT baseline — a 2.7-point improvement over online OPD and an 11.4-point gain over SFT.
• On AIME 2025: 58.4% (Lightning OPD) vs. 57.9% (standard OPD) vs. 52.1% (SFT).
• On HMMT 2025: 39.8% vs. 39.9% vs. 34.0% — essentially tied with standard OPD.
• Math average: 55.4% (Lightning OPD) vs. 54.4% (standard OPD) vs. 47.6% (SFT).
• On LiveCodeBench v5: 42.8% vs. 44.2% vs. 33.8% — Lightning OPD trails standard OPD by 1.4 points on this benchmark.
• On LiveCodeBench v6: 40.3% vs. 39.3% vs. 31.5%.
• Code average: 41.5% vs. 41.8% vs. 32.6% — effectively tied.
• Compared to ExOPD: Lightning OPD achieves 68.1% vs. 61.0% on AIME 2024 and 40.3% vs. 29.0% on LCB v6 — substantial improvements of 7.1 and 11.3 points respectively.

At the 8B scale (Qwen3-8B-Base student, Qwen3-32B teacher):

• Lightning OPD achieves 69.9% on AIME 2024, compared to 68.5% for standard OPD and 63.7% for SFT — a 1.4-point improvement over online OPD and a 6.2-point gain over SFT.
• On AIME 2025: 59.2% vs. 59.0% vs. 51.7%.
• On HMMT 2025: 41.9% vs. 39.4% vs. 36.9% — the largest advantage over standard OPD at 2.5 points.
• Math average: 57.0% vs. 55.6% vs. 50.8%.
• On LiveCodeBench v5: 49.5% vs. 47.3% vs. 44.7%.
• On LiveCodeBench v6: 43.9% vs. 41.2% vs. 36.8%.
• Code average: 46.7% vs. 44.2% vs. 40.8%.

Key observations from Table 1:

• Lightning OPD exceeds standard OPD on 7 of 10 benchmark–scale pairs and ties or is within 1.4 points on the remaining 3. This is consistent with the theoretical prediction that the two methods share the same optimum (Theorem 3.6) and that the offline approximation's gradient discrepancy is bounded (Theorem 3.5), with any small differences attributable to the implicit regularization effect (Theorem 3.7) potentially providing beneficial stabilization that the online variant lacks.
• The gains from the OPD stage over SFT are large and consistent: math averages improve by roughly 6–8 points at both scales, and code averages improve by roughly 6–9 points. This confirms that the on-policy distillation signal — even when precomputed offline — provides substantial post-training improvements beyond what SFT alone achieves.
• The code benchmarks (LCB v5/v6) show slightly smaller relative gains than math benchmarks, but the absolute improvements are still large (e.g., +8.1 points on LCB v6 at 8B).

Training Cost: 3.6–4.0× Speedup Over Standard OPD (Table 2)

Table 2 provides the cost breakdown that underlies the paper's efficiency claims.

At the 4B scale:

• Standard OPD: 72 GPU hours total.
• Lightning OPD: 20 GPU hours total — 3.6× speedup.
• Lightning OPD breakdown: 10 GPU hours for rollout collection, 2 GPU hours for teacher log-probability precomputation, 8 GPU hours for OPD training.

At the 8B scale:

• Standard OPD: 120 GPU hours total.
• Lightning OPD: 30 GPU hours total — 4.0× speedup.
• Lightning OPD breakdown: 10 GPU hours for rollout collection, 4 GPU hours for teacher log-probability precomputation, 16 GPU hours for OPD training.

Key observations:

• The actual OPD training phase consumes only 8–16 GPU hours — a small fraction of the total Lightning OPD budget and a dramatic reduction from the 72–120 GPU hours of standard OPD. This is the direct consequence of eliminating the live teacher server: standard OPD must continuously serve the teacher alongside training, fragmenting GPU resources and inflating total time.
• The one-time offline operations (rollout collection + teacher logprob precomputation) cost 12–14 GPU hours, which are paid once regardless of how many training runs are performed (e.g., hyperparameter sweeps, multiple seeds). The paper notes this amortization benefit implicitly but does not quantify it — if a practitioner runs the OPD stage multiple times with different hyperparameters, the preprocessing cost is incurred only once, making the per-experiment cost even lower than the 20–30 GPU hours reported.
• The speedup is larger at the 8B scale (4.0× vs. 3.6×) because the teacher server overhead grows with model size — a 32B teacher costs more to serve than an 8B teacher relative to the training cost of the student. This suggests the efficiency advantage of Lightning OPD would increase further with larger teacher–student gaps.

Scaling to Mixture-of-Experts Where Standard OPD Is Infeasible (Table 3)

Table 3 demonstrates Lightning OPD on Qwen3-30B-A3B-Base, a 30B-parameter MoE model with 3B active parameters, using Qwen3-30B-A3B-Thinking-2507 as teacher, on a single 8×H100 node.

Results:

• Lightning OPD achieves 71.0% on AIME 2024, compared to 66.8% for the SFT baseline — a 4.2-point gain.
• On AIME 2025: 66.3% vs. 63.2% (SFT).
• On HMMT 2025: 48.3% vs. 44.6% (SFT).
• Math average: 61.9% vs. 58.2%.
• On LiveCodeBench v5: 60.8% vs. 39.4% (SFT) — a striking 21.4-point gain.
• On LiveCodeBench v6: 54.4% vs. 33.0% (SFT).
• Code average: 57.6% vs. 36.2% (SFT) — a 21.4-point gain, substantially larger than the code gains at the 4B and 8B scales.

Key observations:

• Standard OPD is listed as "OOM" (out of memory) — "co-hosting both a 30B student and a 30B teacher for training and scoring exceeds available GPU memory" (Section 4.4). This means Lightning OPD is not merely more efficient than standard OPD at this scale — it is the only feasible option under a single-node constraint.
• The code gains at the 30B scale are dramatically larger than at the 4B and 8B scales (+21.4 points on code average vs. +9 points at 8B and +9 points at 4B). The paper does not analyze this difference, but possible explanations include: the SFT baseline for the 30B model was notably weak on code (36.2% vs. 40.8% for 8B SFT), leaving more room for OPD to improve; the teacher (Qwen3-30B-A3B-Thinking-2507) is specifically a reasoning-tuned model that may provide richer code supervision; or the MoE architecture benefits more from the dense token-level distillation signal.
• The math gains are more modest (+3.7 points on math average), similar in magnitude to the 8B gains (+6.2 points), suggesting math reasoning saturates more quickly with distillation or that the SFT baseline was already stronger.

Training Dynamics: Convergence, Implicit Regularization, and SFT Checkpoint Robustness (Figure 3)

Figure 3 provides diagnostic plots for the Qwen3-4B-Base student to characterize the internal dynamics of Lightning OPD training.

Figure 3a: Importance weight dynamics. The per-token importance weight $w_t = \pi_\theta / \pi_{\text{ref}}$ is tracked throughout the 150 OPD training steps. The mean weight drops from 1.0 to approximately 0.94 within the first 20 steps and then plateaus, ending near 0.94 at step 150. The standard deviation rises sharply from 0 to approximately 0.08–0.10 in the same early phase before stabilizing below 0.10. This is presented as direct evidence for the implicit regularization mechanism of Theorem 3.7: the student initially moves toward the teacher (mean weight drops), but the covariance correction kicks in and prevents further divergence (plateau), with weight variance remaining bounded (standard deviation stabilizes). The fact that the mean stays close to 0.94 — rather than drifting far from 1.0 — validates that the student remains near π_ref throughout training without any explicit KL penalty.

Figure 3b: AIME 2024 score vs. training step. The pass@1 rises steeply from 56.7% (SFT baseline) to approximately 66% within the first 30–50 steps and then saturates, reaching 68.1% by step 150 with minimal further improvement after step 50. This rapid convergence justifies the choice of only 150 training steps — training longer would not improve results and would risk overfitting to the fixed dataset. The shape of the curve also demonstrates that Lightning OPD extracts nearly all of its performance gain very early in training, consistent with the theoretical picture that the student moves quickly toward the teacher's distribution before the implicit regularization arrests further drift.

Figure 3c: AIME 2024 score vs. SFT checkpoint quality. The experiment varies the number of SFT training steps (from 500 to 3000) and evaluates the AIME 2024 score for three variants: SFT alone, standard OPD, and Lightning OPD. All three curves improve consistently with more SFT steps, and both OPD variants provide a large, stable gain on top of the SFT baseline at every checkpoint. For example, at 3000 SFT steps: SFT achieves 56.7%, standard OPD achieves 65.4%, Lightning OPD achieves 68.1% — a consistent ordering across all SFT budgets. The relative gap between Lightning OPD and standard OPD (+1 to +3 points) is also stable across SFT budgets. This indicates that Lightning OPD is robust to the choice of SFT training length — it does not require a precisely tuned SFT checkpoint to work well, which reduces the practitioner burden of coordinating the two stages.

Ablation Studies and Robustness Checks

Teacher consistency (Table 4): The most important ablation in the paper tests whether teacher consistency — the principle that the SFT and OPD stages must use the identical teacher model — is empirically necessary. A full 2×2 grid of SFT-stage and OPD-stage teacher combinations is constructed: at the 4B scale, the possible teachers are Qwen3-8B and QwQ-32B; at the 8B scale, the possible teachers are Qwen3-32B and QwQ-32B. Results are reported for both standard OPD and Lightning OPD on AIME 2024.

At the 4B scale for standard OPD: consistent (Qwen3-8B both stages) achieves 65.4%; mismatched (QwQ-32B SFT, Qwen3-8B OPD) drops to 61.2%, a 4.2-point degradation; mismatched (Qwen3-8B SFT, QwQ-32B OPD) achieves 62.4%, a 3.0-point degradation; consistent (QwQ-32B both stages) achieves 62.8%.

At the 4B scale for Lightning OPD: consistent (Qwen3-8B both stages) achieves 68.1%; mismatched (QwQ-32B SFT, Qwen3-8B OPD) drops to 59.3%, an 8.8-point degradation; mismatched (Qwen3-8B SFT, QwQ-32B OPD) achieves 62.5%, a 5.6-point degradation; consistent (QwQ-32B both stages) achieves 63.1%.

At the 8B scale for standard OPD: consistent (Qwen3-32B both stages) achieves 68.5%; mismatched (QwQ-32B SFT, Qwen3-32B OPD) drops to 65.0%, a 3.5-point degradation; mismatched (Qwen3-32B SFT, QwQ-32B OPD) achieves 64.8%, a 3.7-point degradation; consistent (QwQ-32B both stages) achieves 66.5%.

At the 8B scale for Lightning OPD: consistent (Qwen3-32B both stages) achieves 69.9%; mismatched (QwQ-32B SFT, Qwen3-32B OPD) drops to 62.1%, a 7.8-point degradation; mismatched (Qwen3-32B SFT, QwQ-32B OPD) achieves 63.1%, a 6.8-point degradation; consistent (QwQ-32B both stages) achieves 68.7%.

Five conclusions follow from this ablation:

1. The diagonal (teacher-consistent) settings always achieve the best performance for each method and teacher combination, confirming the theoretical prediction that teacher consistency matters.
2. Teacher mismatch degrades standard OPD as well as Lightning OPD (Theorem 3.9 is validated), with standard OPD losing 3–4 points at the 8B scale under mismatch. This establishes teacher consistency as a general OPD design principle, not specific to the offline setting.
3. Lightning OPD is more sensitive to teacher mismatch than standard OPD: the degradation is 5.6–8.8 points for Lightning OPD vs. 3.0–4.2 points for standard OPD at the 4B scale, and 6.8–7.8 points vs. 3.5–3.7 points at the 8B scale. This asymmetry is theoretically motivated: for Lightning OPD, a mismatched SFT teacher corrupts π_ref in two roles simultaneously — as the reference distribution anchoring implicit regularization and as the source of fixed rollouts — whereas standard OPD refreshes rollouts from the current student and can partially recover.
4. Using QwQ-32B consistently in both stages produces competitive but generally worse results than using Qwen3-32B consistently (e.g., 68.7% vs. 69.9% for Lightning OPD at 8B), indicating that teacher quality matters independently of consistency — but consistency within a given teacher choice is what determines whether the OPD process works correctly.
5. The fact that Lightning OPD with mismatched teachers (59.3–63.1%) often falls below the SFT baseline (56.7% for 4B, 63.7% for 8B) on AIME 2024 in the most severe cross demonstrates that teacher mismatch can cause OPD to actively harm the model relative to its starting point. This is consistent with the theory: the gradient bias $G\sigma_\Delta$ pushes the student toward a distribution that reconciles conflicting teacher signals, potentially undoing the SFT gains.

Training convergence (Figure 3b): The OPD training stage converges within the first 50 steps, with the AIME 2024 score saturating thereafter. The paper trains for 150 steps, which provides a comfortable margin beyond convergence. This ablation justifies the short training budget and confirms that the 4.0× speedup is not achieved by cutting training short before convergence — the model has fully converged by step 50, and the remaining 100 steps are a safety margin. An implicit ablation here is that training beyond 150 steps is unnecessary: the offline dataset is fixed, so overfitting becomes a concern with extended training, and the convergence dynamics suggest no further benefit.

SFT checkpoint robustness (Figure 3c): Lightning OPD's performance gain over SFT is stable across SFT training budgets from 500 to 3000 steps. The absolute scores of Lightning OPD, standard OPD, and SFT all improve monotonically with more SFT steps, and the relative ordering (Lightning OPD ≥ standard OPD > SFT) is preserved at every checkpoint. This demonstrates that Lightning OPD does not require a specific, carefully tuned SFT checkpoint — it works robustly as long as the SFT model is reasonably trained, reducing the coordination burden between stages.

OPD training budget (150 steps, 4,096 token max response length): The paper states (Appendix B) that "increasing the rollout length beyond [4,096 tokens] does not improve results" despite evaluation generation lengths of up to 40,960 tokens. This is an empirical finding, not an ablated hyperparameter sweep shown in a table, but it is a practical robustness check: the per-token advantage signals in the first 4,096 tokens are sufficient to drive the student toward the teacher's distribution, and longer sequences do not provide additional useful supervision. The paper does not report results with different maximum response lengths to verify this claim quantitatively, which is a minor limitation.

Cross-domain training order (Appendix B): For code generation, initializing the OPD stage from the math-trained OPD checkpoint rather than the SFT checkpoint "consistently outperforms" the alternative. The paper cites prior work (Nemotron-Cascade, Chen et al., 2025) for this finding and does not provide an ablation table, but notes it as a practical recipe choice. This is a transfer-learning observation — math reasoning training provides a stronger initialization for code — rather than a claim about Lightning OPD specifically, but it contributes to the strong code results in Tables 1 and 3.

Absence of explicit KL penalty: Lightning OPD does not use an explicit KL penalty term (Table 6 lists no KL coefficient), while standard OPD often requires one to prevent policy drift. The implicit regularization analysis (Theorem 3.7) and the importance weight dynamics (Figure 3a) together demonstrate that the offline objective provides this stabilization automatically. The paper does not run an ablation with an added KL penalty to show it is unnecessary or harmful, but the strong performance without one is indirect evidence for the sufficiency of the implicit mechanism.

Single response per prompt in D_OPD: The paper samples exactly one response per prompt from π_ref to construct D_OPD (Section 4.1: "For each prompt, we sample a single response from π_ref"). The paper does not ablate the number of responses per prompt (e.g., 2, 4, 8), which leaves open the question of whether multiple responses would improve performance by providing better coverage of π_ref's distribution, or would simply add redundant supervision at the cost of proportionally more preprocessing compute. This is a notable missing ablation because the theoretical framework (Theorem 3.5) bounds gradient discrepancy in terms of the expectation under π_ref, and a finite sample from that expectation introduces Monte Carlo error that could be reduced with more samples per prompt.

Critical Assessment

The empirical results broadly support the paper's central claims, but several qualifications, missing experiments, and interpretative cautions are warranted.

Claim: Lightning OPD matches standard OPD performance across benchmarks. This claim is supported by Table 1, with the caveat that "matches" is measured by point estimates without confidence intervals. Across 10 benchmark–scale pairs, Lightning OPD exceeds standard OPD 7 times, ties within 1 point twice, and trails by 1.4 points once (LCB v5 at 4B). The differences are small and likely within sampling error given benchmark sizes (AIME has 30 problems; ±2–3% variation is plausible). More importantly, the paper's theoretical framework does not predict that Lightning OPD will exceed standard OPD — it predicts equivalence under teacher consistency — so the slight advantage in most settings is not a validation of the theory as stated. An honest reading is that the two methods are statistically indistinguishable, which is exactly what the shared-fixed-point theorem (Theorem 3.6) predicts. The occasional small advantage for Lightning OPD could reflect the implicit regularization providing beneficial stabilization, but this is speculative without a specific experiment isolating that mechanism.

The strength of this claim is limited by the single model family (Qwen3) and the absence of experiments at additional scales (e.g., 1B, 14B, 70B) that would test whether the equivalence holds across wider capacity gaps. The theoretical framework makes no scale-dependent predictions, so empirical validation at only two relatively close scales (4B and 8B, with 8B and 32B teachers) leaves open whether the offline approximation degrades at larger student–teacher gaps or with different model architectures.

Claim: Lightning OPD achieves 4.0× higher training efficiency. Table 2 provides the numbers: 30 vs. 120 GPU hours at 8B (4.0×), 20 vs. 72 at 4B (3.6×). This is a GPU-hour speedup, not a wall-clock speedup — the paper does not report wall-clock training time. GPU hours aggregate compute across all devices, so 30 GPU hours on 8 GPUs is approximately 3.75 wall-clock hours if perfectly parallelized, but the actual wall-clock time depends on communication overhead, pipeline bubbles, and the sequential dependency between preprocessing and training phases. The absolute numbers are convincing that Lightning OPD requires substantially less total compute, but "4.0× higher training efficiency" glosses over the fact that the preprocessing phase (rollout collection + teacher logprob precomputation, 14 GPU hours at 8B) is a one-time cost that is not amortized in the reported figures. If a practitioner runs the OPD stage only once, the total cost is 30 hours. If they run it 5 times with different hyperparameters, the preprocessing is paid once and the marginal cost per run is only the OPD training phase (16 hours at 8B), making the effective speedup much larger than 4.0×. If they run it only once, the 30 hours includes the preprocessing cost that standard OPD does not have an analog for. This is not misrepresentation — the paper is transparent about the breakdown — but "4.0× higher training efficiency" is best understood as the speedup for a single full pipeline execution, with the caveat that repeated experiments amortize the preprocessing cost.

A more concerning omission is that the speedup calculation includes only the OPD stage — it excludes the SFT stage cost entirely. Both Lightning OPD and standard OPD begin from the same SFT checkpoint, so the SFT cost is identical for both and drops out of the comparison, but the total end-to-end cost (SFT + OPD) would show a smaller relative speedup. The paper acknowledges this implicitly by reporting per-stage costs in Table 2 but does not provide total end-to-end GPU hours including SFT, which would contextualize the 4.0× figure. If SFT costs, say, 50 GPU hours, the total pipeline would be 170 hours for standard OPD and 80 hours for Lightning OPD — a 2.1× speedup end-to-end rather than 4.0×. This is still substantial but meaningfully smaller.

Claim: Teacher consistency is necessary for OPD effectiveness. Table 4 strongly supports this claim for the specific teacher combinations tested, with a clear pattern: the consistent diagonal always outperforms both off-diagonal settings. However, the paper tests exactly one mismatched teacher pair per scale (Qwen3 vs. QwQ-32B). The theoretical framework predicts a continuous relationship: larger σ_Δ → larger gradient bias → worse performance. Testing only one mismatch magnitude leaves open the question of how sensitive the degradation is to teacher similarity. Would a milder mismatch (e.g., Qwen3-32B vs. Qwen3-32B-Instruct, or Qwen3-32B vs. Qwen3-14B) cause negligible degradation, making teacher consistency practically approximate rather than exact? Would a more severe mismatch (e.g., Llama-3-70B vs. Qwen3-32B) cause even larger drops? The paper's theoretical claim is that σ_Δ = 0 is necessary for zero bias, but in practice a sufficiently small σ_Δ might cause negligible degradation, making "same model" a sufficient but not strictly necessary condition. The ablation does not explore this gradient.

Additionally, Table 4 reports only AIME 2024 (30 problems), a single benchmark. The pattern might differ on code or on easier math benchmarks, particularly because teacher mismatch manifests as a gradient bias whose practical effect depends on the optimization landscape.

Claim: The theoretical analysis explains why Lightning OPD works. The theory is rigorous and self-contained, and the empirical patterns are consistent with its predictions: importance weights stay near 1 (Figure 3a, validating Theorem 3.7), training converges to a stable point (Figure 3b, consistent with shared fixed point), and teacher mismatch degrades performance (Table 4, validating Theorems 3.8–3.9). However, the connection between theory and experiment is correlational, not causal — the theory predicts that these patterns should hold, and the experiments do observe them, but there is no experiment that directly tests a theoretical mechanism (e.g., measuring the actual gradient discrepancy bound, computing the covariance correction term, or verifying that the implicit regularization is the cause of limited policy drift rather than merely correlated with it). This is a standard limitation of theory–experiment pairings in ML papers and does not diminish the paper's contribution, but the claims should be understood as "the theory provides a consistent explanation for the observed phenomena" rather than "the experiments prove the theory."

Weaknesses and missing experiments:

• Single model family (Qwen3). All experiments use Qwen3 models. The theoretical framework makes no architecture-specific assumptions, so the absence of Llama, Mistral, or DeepSeek models leaves open whether the findings generalize. This is partially mitigated by the fact that Qwen3 is a widely-used model family with standard architecture, but a single positive result on another model family would substantially strengthen the paper's generality claim.

• No confidence intervals or statistical testing. The paper reports point estimates for pass@1 without error bars, confidence intervals, or significance tests. For AIME 2024 (30 problems), a 1–2% difference between Lightning OPD and standard OPD is not statistically distinguishable from noise. The paper's conclusions about matching or exceeding standard OPD would be more rigorous with appropriate statistical treatment, though the consistency across 5 benchmarks and 2 scales provides informal confidence.

• No ablation of number of responses per prompt in D_OPD. The paper samples exactly one response per prompt. Multiple responses would increase preprocessing cost linearly but might improve supervision quality by providing better coverage of π_ref's distribution. This is a practical design choice that the paper does not justify or ablate.

• No experiment combining Lightning OPD with explicit KL penalty. The paper argues that implicit regularization replaces explicit KL penalties, but does not test whether adding an explicit penalty would help or hurt. This would be a straightforward ablation: train Lightning OPD with a KL penalty term and compare to the no-penalty baseline. If the implicit regularization is sufficient, the penalty should not improve results; if it helps, the implicit mechanism is not fully replacing explicit stabilization.

• No experiment on the effect of OPD dataset size. The paper uses the full DAPO-Math-17k and a 30K subset of EpiCoder-func-380k. Ablating the dataset size would characterize how much data Lightning OPD requires — a practically important question for domains where large prompt datasets are unavailable.

• Limited difficulty analysis. The paper does not break down results by problem difficulty (unlike the reference paper, which analyzed performance across five difficulty quintiles). It would be informative to know whether Lightning OPD's performance relative to standard OPD varies with problem difficulty — does the offline approximation degrade more on harder problems where the student's errors are more diverse and less well-represented in the static dataset? Or is the effect uniform?

• No analysis of the effect of temperature and top-p in rollout sampling. The OPD rollouts use temperature 0.8 and top-p 1.0 (Table 6). These parameters control the diversity of the static dataset. Higher temperature would produce more diverse (and possibly lower-quality) rollouts, potentially providing better coverage of π_ref's distribution but also introducing more noise. The paper does not ablate these choices.

• The MoE comparison to standard OPD is not a comparison — it is a feasibility demonstration. Standard OPD is listed as OOM on a single 8×H100 node. This proves that Lightning OPD enables on-policy distillation at a scale where standard OPD is infeasible under the specific hardware constraint, but it does not demonstrate performance parity with standard OPD at this scale, because standard OPD could potentially be run with more GPUs or model parallelism. The 71.0% on AIME 2024 is impressive, but it is an absolute result, not a comparative one against the online baseline at the same scale. The theoretical guarantees are the only assurance that Lightning OPD would match standard OPD if the latter were runnable.

Experiments that would strengthen the paper:

1. Replication on a non-Qwen model family (e.g., Llama-3-8B with Llama-3-70B teacher).
2. Scaling to a larger student–teacher gap (e.g., 1B student with 8B teacher) to test the theory's prediction that the gap remains bounded as the capacity difference grows.
3. Ablation of number of responses per prompt in D_OPD (1 vs. 4 vs. 8) and dataset size (25%, 50%, 100% of prompts).
4. Explicit KL penalty ablation to test whether implicit regularization is truly sufficient.
5. Difficulty-stratified results to characterize whether the offline approximation degrades on harder problems.
6. A "teacher similarity sweep" — test multiple SFT–OPD teacher pairs with varying σ_Δ (e.g., same model with different fine-tuning checkpoints, different-sized models from the same family, models from different families) to empirically map the relationship between teacher divergence and performance degradation.
7. Wall-clock time measurements in addition to GPU hours to distinguish compute efficiency from latency.

Despite these limitations, the experimental evidence is sufficient to support the paper's core practical claim: under teacher consistency, Lightning OPD provides a viable, dramatically cheaper alternative to standard OPD with statistically indistinguishable performance on the tested benchmarks, and it enables on-policy distillation at scales where standard OPD is infeasible under constrained hardware. The theoretical framework provides a coherent explanation for why this works, and the teacher consistency ablation (Table 4) convincingly demonstrates that violating this principle causes the offline approximation to fail, validating the paper's central design constraint.

6. Limitations and Trade-offs

6.1 The Difficulty Estimation Cost Is Unaccounted For and Cannot Be Amortized

The assumption or constraint. Lightning OPD eliminates the live teacher server during OPD training, but it does not eliminate the teacher entirely — the teacher must still be served for two distinct offline phases: once to generate SFT trajectories (Stage 1) and once to precompute log-probabilities on OPD rollouts (Stage 2, Phase 1). The paper acknowledges this cost explicitly in Appendix E:

"when adopting a new teacher, this necessitates regenerating the SFT dataset, which can be resource-intensive for large teacher models and partially offsets the training-time savings, though it remains a one-time cost amortized over multiple experiments"

The cost breakdown in Table 2 shows that at the 8B scale, rollout collection costs 10 GPU hours and teacher log-probability precomputation costs 4 GPU hours — together 14 of the 30 total GPU hours (47%). These are not theoretical costs: they require serving the teacher model for forward passes over the full prompt datasets. For the 8B scale with a 32B teacher, this means loading a 32B-parameter model and running inference over 17K–30K prompts.

The consequence. The 4.0× speedup figure (30 vs. 120 GPU hours) compares Lightning OPD's total cost to standard OPD's OPD-stage cost, but this comparison is between two pipelines with fundamentally different teacher access patterns. Standard OPD requires continuous teacher serving throughout training; Lightning OPD requires two discrete teacher-serving sessions. The speedup is real for a practitioner who would otherwise run standard OPD, but it understates the infrastructure requirement in an important way: Lightning OPD still needs a teacher server — just not continuously. A practitioner who cannot serve the teacher at all (e.g., no access to the teacher model weights, or only API access with rate limits) cannot run Lightning OPD's preprocessing phase. The paper does not discuss this scenario.

The one-time cost amortization argument — that the 14 GPU hours are paid once and reused across multiple experiments — is valid for hyperparameter sweeps or multiple seeds, but it only applies to the OPD preprocessing phase. The SFT regeneration cost (when switching teachers) is not amortized over standard experiments because changing the teacher is a fundamental architectural change, not a hyperparameter choice. The paper's claim that Lightning OPD "significantly lower[s] the barrier for academic research on LLM post-training" (Section 1) is true relative to standard OPD but still requires non-trivial teacher access that some academic labs may lack.

What evidence exists in the paper. Table 2 provides the per-phase cost breakdown confirming that 47% of Lightning OPD's total cost at 8B comes from teacher-serving operations. The paper does not report the end-to-end cost including SFT (which would include additional teacher-serving cost for generating the SFT dataset). The SFT cost would be identical for both Lightning OPD and standard OPD and thus cancels in the comparison, but the absolute infrastructure requirement — serving a teacher for SFT data generation plus OPD preprocessing — is higher than the 30 GPU hours figure might suggest if the SFT stage is considered part of the total pipeline.

Mitigation status. The paper acknowledges this limitation transparently in Appendix E and frames the one-time cost as amortizable, but proposes no technical mitigation (e.g., distilling the teacher into a smaller model for preprocessing, using API-based teacher access, or reducing the number of prompts needed for preprocessing). The core tension — Lightning OPD reduces but does not eliminate the need to serve the teacher — is inherent to the framework and not resolvable without fundamentally changing the approach (e.g., using a different signal than teacher log-probabilities).

---

6.2 Teacher Consistency Is Empirically Strict: Even Similar but Non-Identical Teachers Cause Significant Degradation

The assumption or constraint. The paper proves (Theorems 3.8–3.9) and empirically validates (Table 4) that teacher consistency — using the identical teacher model for SFT and OPD — is necessary for Lightning OPD to work reliably. The theory states that any non-zero teacher mismatch ($\sigma_\Delta > 0$) introduces an irreducible gradient bias of magnitude $G\sigma_\Delta$. The paper's central design prescription is therefore absolute: the SFT teacher and OPD teacher must be the same model.

The consequence. This requirement has substantial practical implications that the paper does not fully explore. First, it means OPD practitioners cannot use the community's best available SFT datasets — many of which are generated by models different from the intended OPD teacher. The paper's motivating example is Thinking Machines Lab using OpenThoughts-3 (generated by QwQ-32B) with Qwen3-32B as the OPD teacher — a pipeline that Lightning OPD's theory predicts to be broken. Adopting Lightning OPD means either (a) regenerating SFT data from scratch using the OPD teacher, which is expensive for large teachers, or (b) restricting the choice of OPD teacher to whichever model generated the available SFT data, which may be suboptimal.

Second, the paper only tests one specific mismatched teacher pair (Qwen3 vs. QwQ-32B). The 6.8–8.8 point degradation for Lightning OPD under mismatch (Table 4, 8B scale) is large, but it comes from models in entirely different families with different architectures, training procedures, and tokenizers. The paper does not test milder mismatches: e.g., Qwen3-32B vs. Qwen3-32B-Instruct (same base, different fine-tuning), Qwen3-32B vs. Qwen3-14B (same family, different scale), or two checkpoints of the same model at different training stages. It is possible that smaller mismatches cause proportionally smaller degradation, making teacher consistency a continuous rather than binary requirement in practice. The paper's theoretical claim — that $\sigma_\Delta = 0$ is necessary for zero bias — is correct, but "zero bias" is a theoretical ideal; in practice, a bias small enough to cause negligible degradation (e.g., <1 point) would make teacher consistency approximately rather than strictly required. The paper does not provide evidence about where this threshold lies.

Third, teacher consistency creates a lock-in effect: once a teacher is chosen for SFT, all subsequent OPD experiments must use that same teacher or pay the cost of regenerating the SFT dataset. This makes iterative teacher improvement (e.g., upgrading from a 32B to a 70B teacher) expensive, because each new teacher requires a full SFT regeneration pass. Standard OPD is more flexible in this regard because, although teacher mismatch still degrades it (Table 4 shows 3–4 point drops for standard OPD), the degradation is smaller and the online rollout sampling provides partial recovery. The paper does not discuss this lock-in effect or compare the total cost of teacher iteration between Lightning OPD and standard OPD.

What evidence exists in the paper. Table 4 provides evidence for exactly one mismatched teacher pair per scale, and the results are stark: Lightning OPD drops by 5.6–8.8 points at 4B and 6.8–7.8 points at 8B, while standard OPD drops by 3.0–4.2 points and 3.5–3.7 points respectively. The paper does not ablate the degree of mismatch, test models from the same family with different fine-tuning, or characterize the relationship between teacher divergence and OPD performance.

Mitigation status. Not addressed. The paper presents teacher consistency as a binary requirement and does not explore approximate consistency, partial mitigation strategies (e.g., mixing SFT data from multiple teachers), or the continuous relationship between $\sigma_\Delta$ and performance degradation. Appendix E acknowledges the SFT regeneration cost but does not propose ways to reduce it.

---

6.3 Evaluation Is Limited to a Single Model Family and Two Task Domains

The assumption or constraint. All experiments use models from the Qwen3 family (Yang et al., 2025) and evaluate on math reasoning (AIME 2024/2025, HMMT 2025) and code generation (LiveCodeBench v5/v6). The paper acknowledges this scope limitation only minimally — in Appendix E it notes that "extending Lightning OPD to broader post-training tasks such as multi-turn agent interactions, tool use, and open-ended instruction following remains an open direction" — but does not address the single-model-family concern explicitly.

The consequence. The theoretical framework (Theorems 3.5–3.9) makes no model-specific or task-specific assumptions — it relies only on Assumptions 3.1–3.4, which are about bounded advantages, support coverage, bounded score functions, and bounded teacher mismatch. These are standard assumptions that should hold for any transformer-based LLM. However, the implicit regularization mechanism (Theorem 3.7) depends on the empirical behavior of the covariance term $\text{Cov}[w, f]$, which in turn depends on how the student's distribution evolves during OPD training. Different model families may exhibit different drift dynamics — e.g., models with different initialization schemes, normalization layers, or architectural inductive biases might drift further from $\pi_{\text{ref}}$ during OPD training, violating the assumption (validated only for Qwen3 in Figure 3a) that importance weights remain near 0.94 with small standard deviation. If a model drifts substantially further, the gradient discrepancy bound (Theorem 3.5) would grow, and the offline approximation might degrade.

For tasks, the paper's two domains (math and code) share a critical property: both have well-defined, verifiable correctness criteria that enable reliable evaluation. The paper's OPD stage does not use these criteria during training (it uses teacher log-probabilities, not outcome rewards), but the SFT data generation and the overall post-training pipeline are designed for tasks where teacher-generated solutions can be assumed to be high-quality. For open-ended tasks (creative writing, dialogue, summarization), teacher-generated SFT data may be lower quality or more subjective, and the teacher's per-token log-probabilities may provide less reliable supervision. The paper does not explore whether the offline approximation degrades when the teacher's per-token preferences are noisier or less informative.

Additionally, the paper evaluates on only 30 problems per AIME benchmark. A 1–2% difference between Lightning OPD and standard OPD corresponds to less than one problem on AIME 2024 — statistical noise at this sample size. The consistency across 5 benchmarks and 2 scales provides informal robustness, but the absence of larger benchmarks (e.g., MATH-500, GSM8K, HumanEval) or confidence intervals makes it difficult to assess whether the observed equivalence is reliable or coincidental.

What evidence exists in the paper. All main results (Tables 1–4, Figures 1 and 3) are on Qwen3 models. The MoE experiment (Table 3) extends to a different architecture type (sparse MoE) but still within the Qwen3 family. The paper reports no results on Llama, Mistral, DeepSeek, Gemma, or any non-Qwen model. The task domains are limited to math and code; there are no experiments on knowledge-intensive QA, instruction following, multi-turn dialogue, tool use, or safety-relevant tasks.

Mitigation status. The paper does not attempt to mitigate this limitation. Appendix E mentions broadening to other tasks as future work but does not address the model-family concern. The theoretical framework provides model-agnostic guarantees, but these guarantees are about gradient behavior under stated assumptions, not about the performance of the final trained model — which depends on factors (teacher quality, SFT data quality, task difficulty) that vary across model families and domains. The paper's strong claims about "lowering the barrier for academic research on LLM post-training" would be more convincing with at least one non-Qwen replication.

---

6.4 The Student's Policy Drift During Training Is Hypothesized to Be Small But Not Guaranteed

The assumption or constraint. The entire theoretical justification for Lightning OPD rests on the empirical observation that the student does not drift far from $\pi_{\text{ref}}$ during OPD training. Theorem 3.5 bounds the gradient discrepancy by $G \cdot \sigma_A \cdot \sqrt{\chi^2(\pi_\theta \parallel \pi_{\text{ref}})}$, which is only tight when $\chi^2$ is small. Theorem 3.7 argues that the offline objective implicitly regularizes the student to stay near $\pi_{\text{ref}}$, and Figure 3a provides empirical evidence that importance weights stay centered near 0.94 with standard deviation below 0.1. But the implicit regularization is a derived property of the covariance term, not a guaranteed bound — there is no theorem in the paper that proves the student will stay close to $\pi_{\text{ref}}$ for all hyperparameter settings, all model families, and all tasks. The covariance correction only opposes drift; it does not provably bound it to a specific $\chi^2$ value.

The consequence. If, under different hyperparameters (higher learning rate, longer training, different batch size) or with a different model architecture, the student drifts substantially further from $\pi_{\text{ref}}$ than observed in Figure 3a, the gradient discrepancy bound of Theorem 3.5 would grow, and Lightning OPD's updates could diverge meaningfully from standard OPD's. In the extreme, large policy drift would mean the static dataset of precomputed teacher log-probabilities on $\pi_{\text{ref}}$ rollouts becomes increasingly irrelevant — the student is making errors that are not represented in the dataset, and the teacher's precomputed scores on those errors are not available. This is the standard failure mode of offline approximations to on-policy methods, and Lightning OPD is not immune to it — it only avoids it empirically under the specific training configuration tested.

The paper trains for only 150 steps (Figure 3b justifies this as sufficient for convergence). If a practitioner needed to train longer (e.g., for a harder task where convergence is slower, or with a larger student–teacher gap requiring more steps), the policy drift might accumulate beyond what Figure 3a shows. The implicit regularization may slow drift but not stop it entirely, and the paper provides no analysis of how the drift scales with training duration. The convergence plot (Figure 3b) shows AIME 2024 score saturating by step 50, which could mean the student has reached the capacity-limited optimum and further training would cause drift without performance gain — a scenario where the offline approximation could degrade while performance plateaus or declines.

The paper also does not ablate the learning rate, batch size, or advantage clip range to test whether the policy drift remains small across a wider range of hyperparameters. These are exactly the knobs a practitioner would tune when adapting Lightning OPD to a new setting, and the paper provides no guidance on what hyperparameter regimes preserve the small-drift property that makes the offline approximation faithful.

What evidence exists in the paper. Figure 3a provides exactly one trajectory of importance weight dynamics, for the Qwen3-4B-Base student with the specific hyperparameters in Tables 5–6. No sweep over learning rates, batch sizes, training durations, or model scales is shown. The convergence plot (Figure 3b) provides indirect evidence that further training does not help, but does not show what happens to policy drift if training is forced to continue beyond convergence. The paper does not report $\chi^2$ values, gradient discrepancy measurements, or any direct quantification of how closely the offline and online updates match over the course of training.

Mitigation status. The paper does not attempt to bound policy drift theoretically or to characterize its sensitivity to hyperparameters empirically. The implicit regularization mechanism (Theorem 3.7) provides a qualitative explanation but no quantitative guarantee. The practical recommendation to train for only 150 steps with constant learning rate (Table 6) is empirically validated but not theoretically justified as necessary for keeping drift small. A practitioner adapting Lightning OPD to a new setting would need to independently verify that policy drift remains small, potentially by monitoring importance weight dynamics as in Figure 3a.

---

6.5 The Absence of Outcome Reward Signal May Limit Performance on Hard Problems

The assumption or constraint. Lightning OPD, like standard OPD, relies entirely on per-token teacher log-probability advantages for supervision — it does not incorporate any outcome-level reward signal (e.g., binary correctness from unit tests or answer matching). This is a deliberate design choice inherited from the OPD paradigm. The paper's experiments show strong results on math and code benchmarks, but the benchmarks include a range of difficulty levels, and the paper does not analyze performance as a function of problem difficulty.

The consequence. On problems where the teacher's per-token preferences are poorly calibrated or where the teacher itself makes systematic errors, OPD-style supervision may be insufficient or actively misleading. The teacher provides a dense signal about how to reason, but only an outcome reward signal provides information about whether the reasoning led to a correct answer. If the teacher assigns high probability to tokens that lead to incorrect conclusions (e.g., a plausible but wrong mathematical derivation), the student will learn to replicate the teacher's errors. Outcome reward methods (like RLVR) do not have this problem because the reward is based on the final answer, not the process.

This limitation is most acute on hard problems where the teacher's per-token confidence may be unreliable. The paper does not break down results by problem difficulty, so it is unknown whether Lightning OPD's performance relative to standard OPD (or relative to RLVR methods) varies with difficulty. It is plausible that on the hardest problems — where even the teacher is uncertain and its log-probabilities are noisy — the dense per-token supervision provides limited benefit over SFT, and the additional OPD stage may be primarily reinforcing the teacher's existing biases rather than improving the student's reasoning capability. This would be consistent with the reference paper's finding that "on the hardest questions (bin 5), no amount of test-time compute helps."

Additionally, the code gains at the 30B MoE scale (Table 3) are dramatically larger than the math gains (+21.4 points on code average vs. +3.7 on math average). The paper does not analyze this discrepancy, but one possible explanation is that code generation benefits more from per-token supervision because the teacher's token-level preferences are more informative for code (where syntax and structure are highly constrained) than for math (where multiple valid reasoning paths exist, and the teacher's token choices may be more arbitrary). This suggests that the effectiveness of Lightning OPD may be domain-dependent in ways the paper does not characterize.

What evidence exists in the paper. The paper reports no difficulty-stratified results, no comparison to RLVR baselines at equivalent compute budgets, and no analysis of teacher calibration as a function of problem difficulty. The per-benchmark results in Table 1 show consistent gains over SFT, but the gains vary across benchmarks (e.g., smaller gains on AIME 2025 than AIME 2024 at 8B: +7.5 vs. +6.2 points) without explanation. The paper does not analyze whether the student learns to replicate teacher errors or whether the teacher's supervision quality degrades on harder problems.

Mitigation status. Not addressed. The paper does not discuss the complementary role of outcome rewards, does not propose combining Lightning OPD with an RLVR stage (a natural extension that would address this limitation), and does not analyze teacher calibration. The absence of difficulty-stratified analysis is a missed opportunity, given that the reference paper (on compute-optimal test-time scaling) demonstrated that the effectiveness of post-training methods can be highly difficulty-dependent.

---

6.6 Lightning OPD Introduces a Serial Dependency That Eliminates Online Adaptivity

The assumption or constraint. The preprocessing phase of Lightning OPD samples rollouts from $\pi_{\text{ref}}$ once and precomputes teacher log-probabilities on these fixed responses. The training phase uses only this static dataset — there is no mechanism for updating the dataset as the student improves, no adaptive curriculum, and no feedback loop where the student's current errors inform which new data to collect. This is a fundamental departure from standard OPD, where the rollouts are continuously refreshed from the student's evolving policy.

The consequence. This serial dependency creates two practical problems. First, the offline dataset's quality is determined entirely by the SFT checkpoint. If the SFT model has specific weaknesses (e.g., it struggles with a particular type of math problem), the offline dataset will contain mostly errors on those problem types, and the teacher's supervision will focus on correcting those SFT-era errors. But the student may develop new error patterns during OPD training that are not represented in the static dataset — these errors receive no corrective supervision. Standard OPD automatically adapts to such distributional shift; Lightning OPD cannot.

Second, the preprocessing phase must complete before OPD training begins. If the preprocessing reveals problems (e.g., the SFT model generates very low-quality rollouts, or the teacher's log-probabilities are unexpectedly noisy), the entire pipeline must be restarted with a new SFT checkpoint or different preprocessing parameters. Standard OPD, by integrating data collection and training, can be monitored and adjusted on-the-fly. This makes Lightning OPD less suitable for exploratory research settings where the optimal SFT checkpoint, rollout temperature, or prompt dataset composition is unknown a priori and must be discovered through iteration.

The paper's training dynamics (Figure 3b) show rapid convergence — the student captures nearly all performance gain within 50 steps. This suggests that the static dataset provides sufficient supervision for the student to reach the capacity-limited optimum quickly, after which further training is unnecessary. But this also means that if the static dataset is missing critical error types that the student would have encountered under standard OPD, those errors can never be corrected because the student converges before encountering them. The fast convergence is evidence that the static dataset is adequate for the Qwen3 models on these benchmarks, but it does not guarantee adequacy in general.

The serial dependency also limits the scalability of Lightning OPD to settings where the student and teacher distributions are very different (e.g., cross-lingual transfer, domain adaptation). If $\pi_{\text{ref}}$ generates rollouts that are substantially different from what the student would produce after significant OPD training, the precomputed teacher log-probabilities may be evaluated on a distribution that is no longer relevant, and the static dataset provides no mechanism for adaptation.

What evidence exists in the paper. The convergence plot (Figure 3b) shows that the student saturates quickly, which is both evidence that the static dataset is sufficient and circumstantial evidence that the student does not develop substantially new error patterns during training (since performance stabilizes). The paper does not compare the error distributions of the SFT model and the final OPD model to assess whether the student's mistakes change qualitatively during training. The paper does not ablate the timing of the preprocessing phase (e.g., using an intermediate SFT checkpoint vs. the final SFT checkpoint) to test sensitivity to the specific $\pi_{\text{ref}}$ used.

Mitigation status. The paper does not address this limitation directly. The serial dependency is inherent to the offline design and is the price paid for eliminating the live teacher server. The paper could propose an iterative version of Lightning OPD where the dataset is periodically refreshed (e.g., after every 50 steps, resample rollouts from the current student and precompute new teacher log-probabilities), which would partially recover online adaptivity at the cost of periodic teacher-serving overhead. The paper does not explore this or any other mitigation strategy.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around on-policy distillation from a live-server-required paradigm to an offline-envelope paradigm, but the shift is more about constraints than about freedom — the paper demonstrates that OPD can be performed offline, but only under a specific, previously unrecognized design constraint called teacher consistency. The contribution is therefore not "OPD can now be done cheaply and easily by anyone" but rather "OPD can be done without a live teacher server if and only if the SFT and OPD teachers are identical, and violating this condition explains why both naive offline attempts and certain standard OPD pipelines underperform."

This is best understood as a diagnostic reframing with practical consequences. Prior to this work, the OPD community operated under an implicit assumption inherited from RLVR: the SFT stage and the distillation stage are independent design choices, and practitioners can freely select whichever teacher produces the best SFT data and whichever (possibly different) teacher provides the best OPD supervision. The paper proves this assumption is incorrect for OPD — Theorems 3.8–3.9 establish that teacher mismatch introduces an irreducible gradient bias of magnitude G·σ_Δ that corrupts both online and offline training — and the empirical ablation in Table 4 validates this with concrete numbers: standard OPD drops 3.5 points on AIME 2024 at the 8B scale when the SFT teacher (QwQ-32B) differs from the OPD teacher (Qwen3-32B). This diagnostic reframing changes how OPD pipelines should be designed: teacher choice is now a coupled, holistic decision rather than two independent selections.

The practical consequence is that Lightning OPD makes on-policy distillation accessible to practitioners who have access to a teacher server intermittently but not continuously. The 4.0× training efficiency improvement (30 vs. 120 GPU hours at 8B; Table 2) means OPD training becomes feasible on a single node with the teacher served only in two short, one-time preprocessing phases. This is a meaningful reduction in the infrastructure barrier — a lab with temporary access to a large model (e.g., via cloud GPU rental, shared cluster scheduling, or API access with rate limits that make continuous serving impractical) can now run OPD by front-loading all teacher queries into a concentrated preprocessing window. The MoE result (Table 3) extends this further: Lightning OPD enables on-policy distillation at a scale where standard OPD is literally infeasible under a single-node memory constraint, since co-hosting a 30B student and 30B teacher on one 8×H100 node runs out of memory.

Which research directions become more attractive. The paper makes offline OPD optimization a viable research area by providing a theoretical framework (importance-sampling decomposition, χ²-based discrepancy bounds, covariance regularization) that characterizes when and why offline approximations are faithful. Prior to this work, offline OPD was a heuristic hack (Rang et al., 2025) with no formal guarantees. The shared-fixed-point theorem (Theorem 3.6) and gradient discrepancy bound (Theorem 3.5) provide the intellectual scaffolding for developing more sophisticated offline OPD algorithms — iterative dataset refreshing, adaptive preprocessing, multi-teacher consistency constraints — that build on the paper's formal characterization of the offline-online gap.

Which research directions become less attractive. The paper's finding that teacher mismatch degrades standard (online) OPD as well (Theorem 3.9, Table 4) suggests that OPD variants that attempt to mix multiple teacher signals — e.g., ensembling multiple teachers during OPD, or using one teacher for SFT and a different (better) teacher for OPD supervision — are fundamentally limited by the gradient bias characterized in Theorems 3.8–3.9. Variance reduction techniques or importance-sampling corrections cannot remove a bias that is structural to the gradient field. This makes research on multi-teacher OPD less promising unless it can formally bound or eliminate the inter-teacher discrepancy σ_Δ. Similarly, the paper's demonstration that standard offline RL conservatism mechanisms are inapplicable to OPD (Appendix D) suggests that simply importing offline RL algorithms into OPD pipelines is unlikely to succeed — the challenge is teacher consistency, not distributional coverage, and standard conservatism mechanisms address the wrong problem.

The paper also implicitly argues against incremental improvements to online OPD infrastructure as the primary path forward. If Lightning OPD matches standard OPD performance at 4.0× lower cost, and the theoretical framework shows the two share the same optimum under teacher consistency, then continued investment in live-teacher-server infrastructure for standard OPD becomes a diminishing-returns engineering effort — the core benefit of online rollouts (adaptive feedback on evolving student errors) is empirically small enough that the static dataset suffices, at least for the model families, scales, and tasks tested. The research frontier shifts from "how do we make online OPD cheaper?" to "how do we make offline OPD as effective as online under a wider range of conditions?"

Reconciling prior contradictions. The paper resolves a tension between two independent observations in the literature: (1) Yue et al. (2025) and Shenfeld et al. (2025) showed that RL-trained models stay surprisingly close to their SFT initialization, suggesting that offline approximations to on-policy methods might work; (2) practitioners attempting naive offline OPD found it unreliable (Section 1: "naively applying this offline precomputation fails to reliably match the performance of standard OPD"). The paper reconciles these by showing that the offline approximation does work — student drift is indeed small (Figure 3a) — but only when the SFT and OPD teachers are the same model. The naive attempts failed not because of distribution shift from fixing the rollout distribution, but because they inherited teacher mismatch from standard OPD practice (where it was an unrecognized problem). The paper thus explains both why offline OPD is theoretically sound and why previous attempts failed.

Follow-Up Research This Work Enables

1. Characterizing the continuous relationship between teacher divergence and OPD degradation. The paper tests exactly one mismatched teacher pair per scale (Qwen3-32B vs. QwQ-32B) and observes 3.5–7.8 point degradation (Table 4). The theory predicts that the gradient bias scales with σ_Δ, the expected squared sum of per-token log-probability differences between the two teachers. A natural follow-up would construct a teacher similarity sweep: take a single base teacher (e.g., Qwen3-32B), create a series of checkpoints at different fine-tuning stages (SFT, mid-RL, final reasoning model), and run OPD with SFT data from one checkpoint and OPD supervision from another, measuring both the empirical σ_Δ (via Monte Carlo estimation of per-token log-probability differences on π_ref rollouts) and the resulting AIME 2024 performance. The prediction is a monotonic relationship: larger σ_Δ → larger degradation. This would establish whether teacher consistency is a binary requirement (any mismatch breaks OPD) or a continuous one (small mismatches cause negligible degradation, and only large mismatches matter), which has direct practical implications for whether practitioners can safely use similar-but-not-identical teachers (e.g., different fine-tuning checkpoints of the same base model).

2. Iterative Lightning OPD with periodic dataset refreshing. The paper's static dataset is sampled from π_ref once before training begins. Theorem 3.5 bounds the gradient discrepancy as a function of χ²(π_θ ∥ π_ref), which grows with policy drift. An iterative version of Lightning OPD would periodically resample rollouts from the current student, precompute fresh teacher log-probabilities, and continue training on the updated dataset. This would keep χ² bounded by a smaller constant (the drift within each refresh interval rather than the total drift from initialization) and potentially extend Lightning OPD to settings where the student drifts further — e.g., larger student–teacher capacity gaps, longer training, or domains where the SFT checkpoint is far from the teacher's distribution. The key experiment would compare: (a) standard Lightning OPD (one static dataset, 150 steps), (b) iterative Lightning OPD with refresh every 50 steps (3 refreshes, 3× preprocessing cost), and (c) standard OPD (continuous online rollouts), all at the 8B scale. The prediction is that iterative Lightning OPD narrows or closes the small performance gap between Lightning OPD and standard OPD on benchmarks where that gap exists (e.g., LCB v5 at 4B, where Lightning OPD trails by 1.4 points; Table 1), at the cost of periodic teacher-serving overhead that is still substantially lower than continuous serving.

3. Difficulty-stratified analysis of offline vs. online OPD effectiveness. The paper reports aggregate benchmark scores without difficulty breakdown. The reference paper (on compute-optimal test-time scaling) demonstrated that post-training methods can be highly difficulty-dependent — beam search helps on medium problems but hurts on easy ones, revisions help on easy problems but fail on hard ones. A difficulty-stratified replication of Table 1 would answer: does Lightning OPD's offline approximation degrade more on harder problems, where the student's errors are more diverse and less well-represented in the static π_ref dataset? The experiment would bin AIME 2024/2025 and HMMT 2025 problems by difficulty (using, e.g., the SFT model's pass@1 rate as a proxy), and report Lightning OPD vs. standard OPD accuracy per difficulty quintile. If Lightning OPD underperforms standard OPD on hard problems but matches or exceeds on easy problems, that would refine the paper's "matches standard OPD" claim to a difficulty-conditional one, and would motivate difficulty-aware preprocessing (e.g., sampling more rollouts for hard problems, or using a higher temperature to increase coverage of π_ref's tail).

4. Combining Lightning OPD with outcome reward signals (OPD + RLVR hybrid). Lightning OPD inherits OPD's reliance on per-token teacher supervision without any outcome-level reward. The paper does not compare to RLVR baselines or analyze whether the teacher's per-token preferences are reliable on hard problems where the teacher itself may be uncertain. A natural extension would add a verifiable reward term to the Lightning OPD objective: J_hybrid(θ) = J_off(θ) + α · R(x) · Σ_t ∇log π_θ(a_t|s_t), where R(x) is a binary correctness signal (e.g., unit test pass/fail for code, answer matching for math) on the same precomputed rollouts. This gives the student both dense process supervision (from the teacher) and sparse outcome supervision (from the verifier), combining the stability of OPD with the correctness grounding of RLVR. The experiment would compare Lightning OPD with and without the reward term at the 8B scale on AIME 2024 and LiveCodeBench, where outcome signals are available (AIME has ground-truth answers; LiveCodeBench has unit tests). The prediction is that the hybrid outperforms pure Lightning OPD on hard problems, where teacher supervision is noisy and outcome rewards provide a clean correctness signal, while matching Lightning OPD on easy problems where the teacher is already reliable.

5. Cross-model-family replication to test the generality of the implicit regularization mechanism. The paper's theoretical framework makes no architecture-specific assumptions, but the implicit regularization mechanism (Theorem 3.7) depends on the empirical behavior of Cov[w, f] during training, which in turn depends on how the student's distribution evolves. Different model families may exhibit different drift dynamics — models with different initialization schemes, normalization layers, or architectural inductive biases might drift further from π_ref, making the static dataset less representative. A strong follow-up would replicate the full Lightning OPD pipeline on a non-Qwen model family: e.g., Llama-3.1-8B-Base with Llama-3.1-70B as teacher, or Mistral-7B with Mixtral-8×7B as teacher, using the same MATH and code benchmarks. The experiment would report both final performance (Table 1 equivalent) and importance weight dynamics (Figure 3a equivalent) to test whether the student drift stays small across architectures. A negative result — Lightning OPD significantly underperforms standard OPD on, say, Llama models — would not invalidate the theory (which correctly predicts that larger drift → larger gradient discrepancy) but would indicate that the practical success of Lightning OPD depends on properties of the Qwen3 training recipe that may not generalize, and would motivate architecture-specific stabilization strategies.

6. Data efficiency: how many prompts does Lightning OPD need? The paper uses the full DAPO-Math-17k (17K prompts) and a 30K subset of EpiCoder-func-380k for the OPD stage. The preprocessing cost (10 GPU hours for rollout collection + 4 for teacher logprob precomputation at 8B; Table 2) scales linearly with the number of prompts. A data efficiency ablation — training Lightning OPD with 25%, 50%, 75%, and 100% of the OPD prompt dataset and measuring AIME 2024 performance — would characterize how much data is actually needed. This is practically important because the teacher log-probability precomputation cost is paid once per prompt, and for large prompt datasets or expensive teachers (e.g., 70B+), reducing the number of prompts proportionally reduces the preprocessing budget. If Lightning OPD achieves 95% of its final performance with only 25% of the prompts, the effective training cost drops to well below 30 GPU hours at 8B, making it even more accessible. The paper's finding that the student converges within 50 steps (Figure 3b) on a dataset of 17K prompts suggests the per-prompt supervision signal is redundant — a fraction of the prompts might provide sufficient coverage of π_ref's error distribution.

Practical Applications and Downstream Use Cases

1. Academic labs post-training reasoning models on a single node. The paper's most direct use case is enabling a small academic lab with access to a single 8-GPU node to run a full OPD post-training pipeline. Concretely, starting from a Qwen3-8B-Base model and a Qwen3-32B teacher, Lightning OPD reaches 69.9% on AIME 2024 in 30 total GPU hours (Table 2) — roughly 4 hours of wall-clock time on 8 GPUs. This is a dramatic reduction from standard OPD's 120 GPU hours requiring continuous teacher serving across potentially multiple nodes. The SFT stage (not counted in the 30 hours) adds additional cost, but the full SFT + Lightning OPD pipeline remains feasible on a single node. The MoE result (Table 3) extends this further: a 30B MoE model reaches 71.0% on AIME 2024 and 60.8% on LiveCodeBench v5 on a single 8×H100 node, where standard OPD cannot even run. This directly enables academic research on post-training at scales previously restricted to industrial labs with multi-node serving infrastructure.

2. Cost-efficient batch post-training for industry teams iterating on teacher quality. An industrial team developing a new reasoning teacher (e.g., a fine-tuned Llama-4-70B) and wanting to validate it as an OPD teacher for a smaller student can use Lightning OPD to run the validation with minimal serving overhead. The preprocessing phase (teacher log-probability precomputation) is paid once, and then multiple OPD training runs — with different hyperparameters, learning rates, or student initializations — can reuse the same precomputed dataset. The marginal cost per experiment is only the OPD training phase: 8 GPU hours at 4B, 16 GPU hours at 8B (Table 2). This makes it practical to run hyperparameter sweeps, ablation studies, and teacher comparisons (as in Table 4) that would be prohibitively expensive under standard OPD, where each experiment requires the teacher to be continuously served. The teacher consistency requirement means that switching to a new teacher requires regenerating the SFT dataset and preprocessing from scratch, but within a fixed teacher, Lightning OPD enables rapid experimentation.

3. On-policy distillation in memory-constrained edge or cloud inference scenarios. Lightning OPD's offline design decouples teacher serving from student training, which has a practical benefit beyond cost: it enables OPD in environments where the teacher and student cannot be co-hosted simultaneously. For example, a cloud provider offering API access to a large teacher model (e.g., GPT-4-level) could allow users to precompute teacher log-probabilities via batched API calls at high throughput, then run Lightning OPD training entirely on the student side without maintaining an open API connection. The preprocessing phase is embarrassingly parallel (each prompt is independent) and can be distributed across many API keys or rate-limit windows. This is not possible with standard OPD, which requires the teacher to score each new batch of student rollouts with latency matching the training step frequency — a much more demanding serving pattern. The MoE result (Table 3) demonstrates the extreme case: when GPU memory prevents co-hosting, Lightning OPD's decoupled design is the only way to run OPD at all.

4. Bootstrapping self-improvement pipelines with reduced teacher dependency. Several recent works explore using OPD for self-improvement, where a model serves as its own teacher (self-distillation; Zhao et al., 2026; Shenfeld et al., 2026a). Lightning OPD is well-suited to this setting because the teacher consistency requirement is trivially satisfied when student and teacher are the same model, and the preprocessing phase can be run once to create a static dataset of the model's own rollouts with its own log-probabilities. The implicit regularization of Theorem 3.7 — which prevents the student from drifting far from its SFT initialization — is particularly valuable in self-distillation, where the "teacher" is also evolving and drift can lead to degeneration. A practitioner could run alternating cycles of: (1) SFT on external data to produce π_ref, (2) Lightning OPD with π_ref as its own teacher to refine reasoning, (3) use the refined model to generate new SFT data, repeat. Each Lightning OPD cycle requires only a single preprocessing pass (the model scoring its own rollouts) and 150 steps of training. This is substantially cheaper than standard self-distillation, which would require continuous self-scoring throughout training.

When to Prefer This Method

• Prefer Lightning OPD when: (a) you can serve the teacher model intermittently but not continuously, and want to run multiple OPD experiments with the same teacher–student pair (amortized preprocessing); (b) you are constrained to a single GPU node and the teacher is too large to co-host with the student during training (e.g., 30B MoE on 8×H100; Table 3); (c) your SFT data was generated by the same teacher model you intend to use for OPD — the teacher consistency condition is satisfied by construction, and Lightning OPD provably shares the same optimum as standard OPD (Theorem 3.6); (d) you want implicit training stabilization without tuning a KL penalty coefficient (Theorem 3.7, Figure 3a).

• Prefer standard OPD when: (a) your SFT data comes from a different teacher than your intended OPD teacher, and regenerating SFT data is prohibitively expensive — standard OPD is less sensitive to teacher mismatch (Table 4: 3.5-point degradation vs. 7.8-point for Lightning OPD at 8B), and its online rollouts provide partial recovery; (b) you expect the student to drift substantially from π_ref during training (e.g., very large student–teacher capacity gaps, extended training beyond convergence), and you need adaptive teacher feedback on the student's evolving error distribution; (c) you have access to continuous teacher serving infrastructure and the training efficiency gap (4.0×) is not a binding constraint.

• Prefer Lightning OPD with explicit KL penalty added if: the theoretical framework holds but policy drift exceeds what Figure 3a shows (e.g., under different hyperparameters or model architecture). The paper does not test this combination (Lightning OPD currently uses no KL penalty), but Theorem 3.7 suggests the implicit regularization may be insufficient if drift is larger than observed, and adding a small explicit KL penalty would provide additional stabilization without requiring a live teacher.

• Do not use Lightning OPD (or any OPD variant) if: teacher consistency cannot be satisfied and the degradation from mismatch (Table 4) outweighs the benefits of OPD over SFT. In that regime, RLVR methods (GRPO, PPO, DAPO) that use outcome rewards rather than teacher log-probability supervision are preferable because they are immune to teacher mismatch — the reward signal depends only on the problem and the student's answer, not on which teacher generated the SFT data.