Learning to Foresee: Unveiling the Unlocking Efficiency of On-Policy Distillation

ArXiv: 2605.11739

🎯 Pitch

On-policy distillation doesn't just train faster due to denser supervision—it locks onto the final update direction within the first 10% of training, unlike RL which wanders. The authors exploit this early directional foresight with a simple extrapolation trick to slash training time by 3× without sacrificing performance.

---

1. Executive Summary

This paper studies the parameter-level mechanisms underlying the training efficiency of On-Policy Distillation (OPD) compared to Reinforcement Learning (RL) for post-training large language models, using models ranging from 1.5B to 32B parameters (Qwen2.5, Qwen3) and RL algorithms including PPO, GRPO, and DAPO. The authors identify two properties that constitute OPD's "foresight": Functional Redundancy Avoidance at the module-allocation level (OPD suppresses low-utility updates in embedding and peripheral layers while concentrating updates on reasoning-critical middle-layer MLPs) and Early Low-Rank Lock-in at the update-direction level (OPD's dominant singular subspaces align with the final update subspace early in training, with a checkpoint at only 10% training progress recovering approximately 80% of final performance after norm scaling). Building on these insights, the paper proposes EffOPD, a plug-and-play acceleration method that extrapolates along OPD's early-stabilized update directions with adaptive step-size selection via lightweight validation, achieving an average training acceleration of 3× while maintaining comparable final performance across both code and math benchmarks—establishing that OPD's efficiency advantage arises from early directional stabilization and compact parameter allocation rather than merely from denser supervision signals.

2. Context and Motivation

The Core Problem: We Don't Understand WHY On-Policy Distillation Is More Efficient

The fundamental question this paper tackles is deceptively simple: on-policy distillation (OPD) consistently trains language models faster than reinforcement learning (RL) for post-training reasoning improvements—but nobody knows exactly why. This is not merely an academic curiosity. Across the contemporary LLM landscape—from DeepSeek-V4 to Qwen3 to MiMo-V2-Flash—OPD has emerged as a go-to paradigm for efficiently improving model capabilities after pretraining, often achieving RL-comparable or superior reasoning performance in a fraction of the training steps. Yet the field's understanding of this advantage has remained stubbornly superficial, stuck at the level of "denser supervision helps" without any mechanistic account of what happens inside the model's parameters during training.

This gap matters for several concrete reasons that the paper's framing makes clear:

Post-training is becoming the dominant cost in LLM development. As pretraining scales continue to push toward trillion-parameter models, the cost of post-training—RL fine-tuning, distillation, alignment—has grown to represent an increasingly large fraction of total development budgets. Yet unlike pretraining, where scaling laws (Hoffmann et al., 2022) provide principled guidance for allocating compute between model size and data quantity, post-training efficiency remains largely a matter of empirical trial-and-error. Understanding the parameter-level mechanisms that make one method more efficient than another opens the door to principled optimization rather than ad-hoc recipe engineering.

The denser-supervision explanation is insufficient and arguably misleading. Existing studies (He et al., 2026; Yue et al., 2025) attribute OPD's advantage to the fact that it provides a learning signal at every token position—the per-token KL divergence from the teacher—while RL from verifiable rewards (RLVR) provides a sparse reward only at sequence completion. This explanation is optimization-centric: it says OPD converges faster because it has less variance and more signal. But as the paper demonstrates through its parameter-efficiency analysis (Section 2, Figure 2), the story is more subtle. Under the same update norm constraint, OPD achieves substantially higher reasoning gains than RL, meaning that not all parameter updates are created equal—RL accumulates a non-trivial fraction of updates that contribute to the norm budget but provide limited task improvement. A purely optimization-centric account (denser signal → faster convergence) cannot explain this structural difference in update quality.

Without mechanistic understanding, acceleration methods are blind. Prior work on accelerating LLM post-training—such as AlphaOPD (Cai et al., 2025) and ExOPD (Yang et al., 2026a), both of which the paper compares against in Section 4—has largely relied on heuristics or fixed extrapolation strategies. These methods treat the optimization trajectory as a black box to be sped up, rather than understanding why certain directions are productive and others are not. If OPD's efficiency stems from specific geometric properties of its parameter updates—say, early convergence to a low-dimensional subspace aligned with the final solution—then acceleration methods can explicitly exploit those properties rather than guessing.

The training-inference gap in OPD understanding limits deployment decisions. Practitioners choosing between RL and OPD for post-training currently rely on empirical comparisons that are expensive to run and may not transfer across model scales, architectures, or task domains. A mechanistic account of OPD's efficiency would provide generalizable principles for predicting when OPD will outperform RL, what hyperparameters matter, and how to combine the two paradigms. For instance, the paper's finding that OPD suppresses updates in low-utility modules while RL does not (Section 2.2) suggests a concrete diagnostic: if your RL-trained model shows large parameter changes in embedding layers or bottom transformer layers, you may be spending compute on functionally irrelevant updates.

Conflicting Signals in the Literature

The paper is motivated by a subtle tension in prior findings about distillation and RL for language models:

On one side, OPD consistently works well. Qwen3 (Yang et al., 2025) demonstrates that OPD achieves "substantially higher training efficiency than RLVR." MiMo-V2-Flash (Xiao et al., 2026) and DeepSeek-V4 (DeepSeek-AI, 2026) both use multi-task on-policy distillation to integrate capabilities from multiple teacher models into smaller students. Li et al. (2026) systematically investigate OPD training dynamics and identify necessary conditions for success. The empirical track record is strong: OPD is practical, efficient, and increasingly deployed in production systems.

On the other side, the theoretical picture is murky. Fu et al. (2026) prove that token-level OPD is biased relative to the sequence-level reverse-KL objective—the very objective it's trying to optimize—although it has a tighter variance bound (O(T²) versus O(T⁴) for sequence-level methods). This means OPD is, in a formal sense, solving a different optimization problem than intended. Yang et al. (2026b) establish a theoretical equivalence between token-level distillation and RLVR under certain conditions, but these conditions may not hold in practice. Kim et al. (2026) raise the uncomfortable question of why self-distillation sometimes degrades reasoning capability despite providing dense supervision.

These findings are not contradictory, but they paint a picture of a method that works in practice for reasons we don't fully understand—a classic "engineering success, scientific mystery" scenario. The denser-supervision story is too coarse to explain why OPD sometimes outperforms RL even when both have converged (not just faster, but better), or why OPD's parameter updates look structurally different from RL's under the microscope.

The RL side has its own mysteries. Recent work on RL for LLMs has uncovered surprising regularities: AlphaOPD (Cai et al., 2025) finds that RL-induced parameter updates exhibit Rank-1 dominance and approximately linear dynamics, enabling 3× training acceleration. Cui et al. (2025b) identify "entropy collapse" in RL training, where rapid early convergence causes the model to lose exploratory capacity. These findings suggest that RL, too, has structure that is not captured by standard optimization narratives—but that structure appears to be different from OPD's structure, and the difference is what this paper seeks to characterize.

Where Existing Approaches Fall Short

The paper identifies specific limitations in prior work along three axes:

1. Optimization-level explanations ignore parameter-level reality. The standard account—OPD works because token-level KL provides dense, low-variance gradients—is a statement about the loss landscape, not about what actually happens to the model's weights. It cannot explain why OPD achieves higher per-norm efficiency (Section 2.2), why OPD concentrates updates in specific modules and layers while suppressing others (Section 2.2, Figure 3), or why OPD's update directions stabilize early in training (Section 3.3). These are parameter-dynamic phenomena that require a parameter-dynamic explanation.

2. No framework exists for comparing update structures across training paradigms. Prior mechanistic interpretability work (Meng et al., 2023; Geva et al., 2021) has characterized the functional roles of different Transformer modules—MLPs as key-value memories, middle layers as reasoning-relevant, etc.—but this work has not been connected to the question of which post-training method more efficiently navigates the model's existing functional structure. There is no established methodology for comparing the quality of parameter updates across different training algorithms beyond aggregate metrics like test accuracy or training loss.

3. Acceleration methods for OPD are heuristic rather than principle-driven. AlphaOPD (Cai et al., 2025) accelerates RL training by exploiting Rank-1 linear dynamics, but its applicability to OPD is unclear because the paper shows that OPD's dynamics differ qualitatively from RL's (more low-rank concentration, earlier stabilization, different module allocation). ExOPD (Yang et al., 2026a) uses fixed extrapolation along the update direction, but fixed-step extrapolation ignores the fact that the optimal extrapolation magnitude depends on how stable the current direction is—a quantity that changes throughout training and differs across modules. Without understanding why extrapolation works for OPD, these methods risk either under-extrapolating (leaving efficiency on the table) or over-extrapolating (degrading performance by moving too far along an imperfectly aligned direction).

How This Paper Positions Itself

The paper positions itself at the intersection of mechanistic interpretability and training efficiency, offering what amounts to a parameter-dynamics theory of OPD's advantage. Rather than proposing a new training algorithm (though EffOPD is a concrete contribution), the primary contribution is a framework for understanding what makes OPD updates different from RL updates and why those differences lead to faster, more parameter-efficient training.

The central thesis is that OPD's advantage is not merely about optimization speed—it's about update quality. The paper argues that OPD exhibits a form of "foresight" that manifests in two measurable, parameter-level properties:

• Functional Redundancy Avoidance (Property 1): OPD identifies low-marginal-utility modules early and suppresses updates to them, concentrating its limited update budget on modules where parameter changes actually improve reasoning performance. RL, by contrast, disperses updates more broadly, including substantial parameter changes in modules with limited functional contribution to the task.

• Early Low-Rank Lock-in (Property 2): OPD's parameter updates exhibit stronger spectral concentration (higher singular value decay, lower effective rank), and the dominant singular subspaces align with the final update subspace much earlier in training. This means OPD essentially "chooses a direction" early and then amplifies along that direction, while RL spends more of its training budget exploring and correcting its trajectory.

These properties are not just descriptive—they are predictive of where acceleration can work. Because OPD's update directions stabilize early, one can safely extrapolate along those directions to reach the same destination with fewer steps, which is exactly what EffOPD does (Section 4).

The paper draws a direct connection to the emerging literature on low-rank training dynamics. Recent work by Cai et al. (2025) showed that RL exhibits Rank-1 dominance and linear dynamics, enabling acceleration via subspace prediction. Chen et al. (2026) train a predictor to forecast subsequent optimization subspaces from early Rank-1 subspaces. This paper extends that line of inquiry by asking: why does OPD exhibit even stronger low-rank structure than RL? The answer, the paper argues, lies in the nature of the supervision signal: dense token-level KL divergence from a fixed teacher concentrates the driving force of optimization into a low-dimensional subspace aligned with the teacher's behavioral distribution, whereas sparse reward signals in RL produce more diffuse, higher-variance gradients that explore a broader set of parameter directions.

The paper explicitly frames this as filling a gap between theory and practice. Song and Zheng (2026) provide a unified f-divergence framework for OPD, grounding it in rigorous statistical theory. This paper provides the complementary empirical characterization: what that theory implies about parameter dynamics, and how those dynamics can be exploited for acceleration. The local geometric analysis in Appendix F.5 formalizes this connection by showing that OPD, under a linearization around the base model, reduces to a convex quadratic minimization with a driving term $b = \mathbb{E}_c[J_c^\top F_c r_c]$ whose low-rank structure is inherited from the teacher-student residual $r_c$. This provides a theoretical bridge between the optimization-level explanation (dense supervision → faster convergence) and the parameter-level observations (low-rank updates, early lock-in).

In summary, the paper argues that the missing piece in understanding OPD's efficiency is not better loss functions or training recipes—it's a mechanistic account of how OPD navigates the parameter landscape differently from RL, and an acceleration method that exploits those differences. The contribution is both scientific (explaining a practical phenomenon) and engineering (building a faster training method on that explanation), with the broader goal of moving post-training from an art of trial-and-error toward a science grounded in parameter dynamics.

3. Technical Approach

3.1 Reader Orientation

This paper develops a diagnostic framework and acceleration method for On-Policy Distillation (OPD) that explains why OPD trains language models more efficiently than Reinforcement Learning (RL) and exploits that explanation to speed up training. The core idea is that OPD exhibits two parameter-level properties—Functional Redundancy Avoidance (it suppresses wasted updates in low-utility modules) and Early Low-Rank Lock-in (it stabilizes effective update directions early in training)—and that by explicitly extrapolating along these early-stabilized directions with adaptive validation, one can achieve roughly 3× training acceleration without sacrificing final performance.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components:

1. Comparative Analysis Pipeline (Sections 2–3) — takes fully-trained RL and OPD checkpoints from identical base model initializations and decomposes their parameter updates $\Delta W = W_{\text{trained}} - W_{\text{base}}$ using module-wise norm comparisons, sliding-window interventions, singular value decomposition, subspace alignment, and norm-scaling experiments to reveal structural differences between the two training paradigms.

2. The Two Properties — Functional Redundancy Avoidance (Property 1): at the module-allocation level, OPD identifies low-marginal-utility parameter regions and concentrates updates on reasoning-critical intermediate MLP layers; Early Low-Rank Lock-in (Property 2): at the update-direction level, OPD's parameter updates exhibit stronger low-rank concentration and align their dominant singular subspaces with the final update subspace much earlier than RL.

3. Theoretical Framework (Appendix F.5) — a local linearization of the OPD objective around the base model that shows OPD reduces approximately to convex quadratic minimization $\frac{1}{2}\Delta\theta^\top A\Delta\theta - b^\top\Delta\theta$ where $A = \mathbb{E}_c[J_c^\top F_c J_c]$ is a Gram matrix of Jacobian- and Fisher-weighted sensitivities and $b = \mathbb{E}_c[J_c^\top F_c r_c]$ is the driving term inheriting low-rank structure from the teacher-student residual $r_c$. This provides a theoretical bridge between the empirical properties and the structure of the distillation objective.

4. EffOPD Accelerator (Section 4) — exploits Property 2 by triggering extrapolation searches at exponentially spaced checkpoints $t = 2^n$. For each checkpoint, EffOPD estimates the local update direction $\Delta_n = W_{2^n} - W_{2^{n-1}}$, generates five candidate extrapolated models along that direction with increasing step sizes (coefficients $2^k$ for $k = 1,...,5$), evaluates each on a lightweight validation set of 50 randomly sampled training examples, and accepts candidates that improve validation performance. This requires no additional trainable modules, no complex hyperparameter tuning, and is orthogonal to other acceleration techniques.

Information flows as follows: a shared base model $W_{\text{base}}$ is fine-tuned separately via RL (PPO/GRPO/DAPO) and OPD (reverse KL on on-policy samples) to produce final checkpoints → the analysis pipeline computes $\Delta W_{\text{RL}}$ and $\Delta W_{\text{OPD}}$ and compares their module-wise norms, spectral decompositions, subspace trajectories, and norm-scaling behavior → the two properties are identified and validated across model scales (1.5B to 32B) and RL algorithms → the theoretical framework in Appendix F.5 explains why the OPD objective induces these properties → EffOPD uses the early-lock-in property to accelerate training by extrapolating along early-stabilized directions.

3.3 Roadmap for the Deep Dive

• First, the comparative analysis setting: how RL and OPD checkpoints are produced from shared initializations, what models and algorithms are used, and how $\Delta W$ is defined — because all subsequent analyses depend on comparing these update matrices under controlled conditions.
• Second, Functional Redundancy Avoidance (Property 1): the norm-efficiency experiments, the sliding-window intervention methodology, and the finding that OPD suppresses updates in low-utility modules while concentrating on reasoning-critical middle-layer MLPs — because this establishes the modular-level manifestation of OPD's foresight.
• Third, Early Low-Rank Lock-in (Property 2): the four spectral metrics, the Top-k%/Bottom-k% subspace functional partition experiments, the subspace evolution and alignment analysis, and the critical norm-scaling experiment showing ~80% performance recovery at 10% training progress — because this establishes the geometric and temporal manifestation of OPD's foresight and directly motivates EffOPD.
• Fourth, EffOPD acceleration method: the extrapolation search procedure, the exponential checkpoint schedule, the adaptive candidate acceptance mechanism, and the lightweight validation design — because this is the practical synthesis of both properties into a deployable acceleration technique.
• Fifth, the local geometric theory (Appendix F.5) connecting Properties 1 and 2 to the structure of the OPD objective — because this provides the theoretical bridge between the empirical observations and the acceleration method.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a mechanistic analysis paper with a practical acceleration method derived from the analysis. The core idea is that OPD's training efficiency advantage over RL is not merely due to denser supervision (as prior work assumed) but to specific parameter-dynamic properties that can be measured, theoretically grounded, and exploited for acceleration.

---

Comparative Analysis Setting: RL and OPD from Shared Initializations

To enable meaningful comparison between RL and OPD parameter updates, the paper carefully controls the initialization and training pipeline. All comparisons start from a shared base model $W_{\text{base}}$, and the parameter update for each method is defined as:

$\Delta W_{\text{RL/OPD}} = W_{\text{RL/OPD}} - W_{\text{base}}$

where $W_{\text{RL/OPD}}$ is the final trained checkpoint and $W_{\text{base}}$ is the shared starting point.

What it computes: the total accumulated parameter displacement during training, represented as a matrix of the same shape as the weight matrices in the Transformer (MLP and attention projection matrices). This matrix captures both the direction and magnitude of optimization.

Why this form: defining $\Delta W$ as a simple difference against the shared base model ensures that any structural differences between RL and OPD updates are attributable to the training algorithm, not to different starting points. This is essential because if RL and OPD were initialized from different checkpoints, any observed differences in update structure could be confounded by differences in the base model's parameter values.

Models and scales: the paper conducts experiments across Qwen2.5 and Qwen3 model families at scales of 1.5B, 4B, 7B, 8B, 14B, and 32B parameters. This includes:

• Pretrained base models (e.g., Qwen3-8B-Base, Qwen2.5-7B)
• SFT-tuned models (e.g., Qwen2.5-1.5B-Deepseek, a DeepSeek-R1-distilled SFT checkpoint)
• Thinking-series models (Qwen3 variants with explicit chain-of-thought reasoning modes)

RL algorithms considered: three representative policy gradient methods:

• PPO (Proximal Policy Optimization) — uses clipped importance sampling with a learned value function baseline
• GRPO (Group Relative Policy Optimization) — a variant that estimates advantages from group-relative comparisons rather than a value function
• DAPO (a more recent method from Yu et al., 2025) — uses clip-higher ratios ($\epsilon_{\text{low}} = 0.2$, $\epsilon_{\text{high}} = 0.28$), dynamic sampling, token-level policy gradient loss, and removes KL divergence terms entirely from the reward

For both RL and OPD, training uses the Verl framework with shared core configuration: maximum prompt length 2,048 tokens, maximum response length 20,480 tokens (total budget 22,528 tokens for RL), mini-batch of 32 samples with 16 gradient accumulation steps (effective batch size 512 under Float16), learning rate $1 \times 10^{-6}$ with warmup, and gradient clipping at 1.0. Each prompt generates $n = 16$ outputs during rollout for RL.

OPD training specifics: the student model generates its own trajectories on-policy (temperature 1.0, top-p 1.0) and is trained to minimize the reverse KL divergence between student and teacher distributions on these student-generated sequences. Key OPD hyperparameters: maximum prompt length 2,048 tokens, maximum response length 16,384 tokens (total budget 18,432 tokens), prompt batch size 1,024 with $n = 1$ output per prompt (unlike RL's $n = 16$), learning rate $1 \times 10^{-6}$ without warmup, 3 training epochs. The training command (fully specified in Appendix D.2) uses only_reverse_kl_advantages=True, use_kl_loss=True with kl_loss_coef=0, and kl_loss_type=low_var_kl.

Teacher model selection: for all OPD student models, the capability-aligned teacher is the RL-tuned version of the same base model. For Qwen3-8B-Base, both Qwen3-8B-PPO and Qwen3-14B-Base-DAPO are used as teachers to ensure conclusions generalize across teacher scales. This design choice is deliberate: it ensures that the teacher possesses capabilities beyond the base model (having been RL-fine-tuned), so OPD's task is genuinely about acquiring new reasoning abilities, not merely replicating existing ones.

Training data: experiments cover both code generation (Eurus-RL-Code) and mathematical reasoning (DeepMath-103K, MATH-12K), with evaluation on seven benchmarks: Codeforces, Taco, AIME24, AIME25, AIME26, MINERVA, and GPQA. This multi-domain, multi-benchmark design ensures the identified properties are not artifacts of a specific task distribution.

Why this setup matters: by comparing RL and OPD across multiple model scales, base model types (pretrained vs. SFT vs. thinking), RL algorithms, teacher scales, and task domains, the paper establishes that the identified properties—Functional Redundancy Avoidance and Early Low-Rank Lock-in—are robust, intrinsic characteristics of OPD dynamics rather than artifacts of a particular experimental configuration.

---

Functional Redundancy Avoidance (Property 1): Modular-Level Foresight

This property captures OPD's ability to concentrate its limited parameter-update budget on modules where changes actually improve reasoning performance, while suppressing wasted updates in modules with low marginal utility. The paper establishes this through three linked experiments.

Experiment 1: Norm-scaled update efficiency at final checkpoints. The paper first asks: if we take the full update direction from the final checkpoint and merely change its magnitude, how much performance do we get? Formally, for a scaling factor $\alpha \in [0, 1]$, the paper evaluates models of the form:

$W_{\text{base}} + \alpha \cdot \Delta W_{\text{RL/OPD}}$

where $\alpha$ controls the fraction of the full update norm that is applied. At $\alpha = 0$, the model is the base model; at $\alpha = 1$, it is the fully trained checkpoint.

What it computes: the reasoning accuracy of the model as a function of how much of the final parameter displacement is applied, keeping the update direction fixed. This isolates the quality of the update direction from its magnitude.

Key result (Figure 2a): when updates from RL and OPD are scaled to the same Frobenius norm, OPD achieves substantially higher reasoning gains than RL. For example, on Qwen3-8B-Base, at $\alpha = 0.5$ (half the full update norm), OPD recovers roughly 75% of its final performance while RL recovers only about 40%. This means $\Delta W_{\text{RL}}$ contains a non-negligible fraction of components weakly correlated with task performance—they contribute to the update norm (making $\|\Delta W_{\text{RL}}\|_F$ large) but provide limited reasoning improvement. In contrast, OPD's update carries a greater fraction of task-relevant signal per unit of norm.

Why this form: the linear scaling $W_{\text{base}} + \alpha\Delta W$ preserves the direction of the update while controlling magnitude—if RL's direction were equally effective as OPD's, the two curves would overlap.

Experiment 2: Update norm vs. accuracy throughout the training trajectory. The paper then asks when these weakly task-correlated components emerge during RL training. It tracks intermediate checkpoints from early to final training steps for both RL and OPD.

Key result (Figure 2b): OPD consistently requires smaller parameter updates (lower $\|\Delta W\|_F$) than RL to achieve the same reasoning accuracy. Moreover, OPD achieves rapid accuracy improvement with relatively small increases in update norm, whereas RL improves more slowly under comparable update magnitudes. For example, on Qwen3-8B-Base with DAPO vs. OPD, reaching 20% accuracy on MATH500 requires $\|\Delta W_{\text{OPD}}\|_F \approx 2$ (arbitrary units) while RL requires $\|\Delta W_{\text{RL}}\|_F \approx 8$. This difference persists across the entire training trajectory, not just at the end, ruling out the hypothesis that RL's redundancy is merely late-stage saturation.

Experiment 3: Locating the redundant modules via sliding-window intervention. To identify where in the network RL accumulates redundant updates, the paper uses a sliding-window intervention methodology adapted from mechanistic interpretability work (Meng et al., 2023; Cai et al., 2024).

Intervention procedure: for a Transformer with $L$ layers, define a sliding window centered at layer $l$:

$W_l = \{i \in \mathbb{Z} \mid \max(1, l - 8) \leq i \leq \min(L, l + 8)\}$

This window spans up to 17 consecutive layers (8 on each side plus the center) and is truncated near model boundaries. For each window position $l$, two types of intervened models are constructed:

For MLP intervention:

$W^{(\text{interv})}_{\text{MLP},l} = \begin{cases} W^{(i,\text{MLP})}_{\text{base}} + \Delta W^{(i,\text{MLP})}_{\text{RL/OPD}}, & i \in W_l \\ W^{(i,\text{MLP})}_{\text{base}}, & i \notin W_l \end{cases}$

All attention parameters remain fixed at their base model values.

For Attention intervention, the same pattern is applied but with attention parameters updated within the window and MLP parameters fixed.

What it computes: the marginal contribution of parameter updates in each layer window to overall reasoning performance. By sliding the window across all valid positions $l = 1, 2, ..., L-8$ and evaluating on MATH500, the method builds a sensitivity map showing which layers' updates matter most.

Key findings (Figure 3b, Figure 10b):

1. MLP modules are overall more sensitive to reasoning-related updates than attention modules. Interventions that inject OPD or RL MLP updates into a window consistently produce larger accuracy gains than interventions that inject attention updates into the same window.

2. The sensitivity curve is inverted-U-shaped across layers: interventions in middle layers yield the largest accuracy gains, while interventions in bottom and top layers yield smaller gains. This is consistent across both MLP and attention modules, and aligns with prior findings that middle-layer MLPs are more critical for factual and relational reasoning (Meng et al., 2023; Geva et al., 2021).

3. RL and OPD exhibit highly consistent intervention sensitivity distributions—the shapes of their sensitivity curves are nearly identical. This means OPD and RL do not rely on fundamentally different functional pathways; they optimize along the model's existing key functional structures.

4. The critical difference is in layer-wise update norms. RL introduces substantially larger parameter changes in the low-sensitivity bottom and top layers (and in embedding layers, as shown in Figure 3a). Since these peripheral layers contribute relatively little to reasoning performance when their updates are injected in isolation, their large update norms in RL represent redundant updates—parameter changes that cost norm budget but provide limited functional return. In contrast, OPD suppresses parameter changes in these low-sensitivity regions and concentrates updates more strongly in middle-layer modules with higher functional contributions.

Embedding layer analysis (Figure 3a): replacing the embedding layer of RL or OPD models with the base model's embedding layer has negligible impact on reasoning performance on both AIME26 and MATH500. This confirms that embedding updates contribute minimally to reasoning gains. However, OPD still exhibits smaller embedding shifts than RL (Table 3, Figure 11), maintaining higher cosine similarity to base representations—suggesting OPD enforces compactness even in functionally peripheral regions.

Cross-scale and cross-algorithm validation (Figures 8, 9, 10): the norm-efficiency advantage of OPD over RL holds across all model scales from 1.5B to 32B (Figure 8) and across all RL algorithms (PPO, GRPO, DAPO) (Figure 9). The sliding-window findings replicate across different base model types and datasets (Figure 10). This robustness indicates that Property 1 is an intrinsic characteristic of OPD dynamics, not an artifact of specific model sizes or RL implementations.

Summary of Property 1: OPD exhibits modular-level foresight by forming a compact, task-relevant update pattern early in training, suppressing redundant parameter changes in low-marginal-utility regions (embedding layers, bottom/top transformer layers), and concentrating its limited update norm budget on reasoning-critical intermediate MLP modules. This is not because OPD learns an entirely new functional structure—it follows the same sensitivity pattern as RL—but because it more accurately distinguishes high-benefit from low-benefit parameter regions and avoids wasting updates on the latter.

---

Early Low-Rank Lock-in (Property 2): Geometric/Temporal Foresight

Property 2 concerns the intrinsic organization of OPD's parameter updates from a geometric perspective: OPD concentrates update energy into a small number of dominant directions, and these directions align with the final solution much earlier in training than RL's do. The paper validates this through a sequence of experiments moving from static spectral structure to temporal subspace evolution to functional effectiveness of early directions.

Step 1: Spectral concentration metrics. The paper performs Singular Value Decomposition (SVD) on the update matrix $\Delta W = U\Sigma V^\top$, where $\Sigma = \text{diag}(\sigma_1, ..., \sigma_r)$ contains the singular values in descending order. Four complementary geometric metrics are computed:

1. Spectral Norm: $\|\Delta W\|_2 = \sigma_1$, the largest singular value, capturing the magnitude along the dominant direction.

2. Spectral-to-Frobenius Norm Ratio:

$\rho = \frac{\sigma_1}{\sqrt{\sum_{j=1}^r \sigma_j^2}}$

where $\sigma_1$ is the leading singular value and the denominator is the Frobenius norm $\|\Delta W\|_F$. This ratio quantifies how much of the total update energy is concentrated in the top singular direction. A value near 1 indicates near-rank-1 structure; smaller values indicate more dispersed energy.

3. Effective Rank (Roy & Vetterli, 2007):

$r_{\text{eff}} = \exp\left(-\sum_{i=1}^r \bar{\sigma}_i \log \bar{\sigma}_i\right)$

where $\bar{\sigma}_i = \sigma_i / \sum_{j=1}^r \sigma_j$ are the normalized singular values. This is the exponential of the spectral entropy. A smaller effective rank means the singular value spectrum decays rapidly—most energy is in few directions. A larger effective rank means a more uniform, diffuse spectrum.

4. Top-1% Subspace Norm Ratio:

$R_{\text{Top-1\%}} = \frac{\|\Delta W_k\|_F}{\|\Delta W\|_F} = \frac{\sqrt{\sum_{i=1}^k \sigma_i^2}}{\sqrt{\sum_{j=1}^r \sigma_j^2}}$

where $k = \lceil r / 100 \rceil$ is the number of singular components in the top 1% of the rank, and $\Delta W_k = U_{:,1:k} \Sigma_{1:k} V_{:,1:k}^\top$ is the rank-$k$ approximation. This measures what fraction of total update energy is captured by the top 1% of singular directions. A value near 1 means extreme concentration.

What these metrics compute together: a multi-faceted characterization of how "low-rank" the update matrix is. The spectral norm and spectral/Frobenius ratio capture dominance of the leading direction; the effective rank and Top-1% ratio capture concentration across the spectrum.

Results (Table 1): across all model scales (1.5B to 14B), OPD consistently exhibits stronger low-rank structure than RL. For the 8B model:

• Spectral norm: 0.005 (OPD) vs. 0.004 (RL) — OPD's leading direction is slightly stronger despite having smaller total norm
• Spectral/Frobenius ratio: 36.8% (OPD) vs. 32.7% (RL) — OPD concentrates more energy in the top direction
• Effective rank: 2341 (OPD) vs. 2754 (RL) — OPD's spectrum decays faster (lower entropy)
• Top-1% subspace norm ratio: 94.7% (OPD) vs. 88.5% (RL) — OPD captures nearly 95% of update energy in just 1% of singular directions

These metrics are averaged over all MLP and attention matrices in the model.

Why this matters: the spectral concentration metrics establish that OPD updates are structurally different from RL updates at the matrix level—they are more compressible, more directionally focused. But compression alone could be good or bad; the next experiment tests whether the compressed directions are actually effective for the task.

Step 2: Functional partition of the update spectrum. To determine whether OPD's spectral concentration represents useful compression (energy concentrated in productive directions) or merely arbitrary compression, the paper separates the update into principal and tail subspaces and measures their individual contributions to reasoning performance.

Top-k% subspace experiment (Figure 4a): construct a truncated approximation $\Delta W_{\text{Top-}k\%}$ using only the top $k\%$ of singular components, then rescale its Frobenius norm to equalize between RL and OPD. Apply this low-rank update to the base model and evaluate reasoning performance.

• Both RL and OPD recover over 95% of full-model reasoning performance using only 10% of the rank, confirming that the top subspace is the primary carrier of task-relevant signal.
• OPD consistently outperforms RL across all rank levels under equal norm budgets. For example, on Qwen2.5-1.5B-Deepseek at 10% rank: OPD recovers ~42% accuracy vs. ~38% for RL. This gap persists across model scales and rank thresholds.
• This means OPD's principal directions inherently encode more effective update signals than RL's, even when given the same norm budget—the quality of OPD's compressed directions is higher, not just the degree of compression.

Bottom-k% subspace experiment (Figure 4b): construct $\Delta W_{\text{Bottom-}k\%}$ using the last $k\%$ of singular components and apply it without norm scaling (preserving the original training magnitudes) to measure the marginal utility of tail directions.

• Tail subspaces provide only limited performance recovery for both RL and OPD. On Qwen2.5-1.5B-Deepseek: retaining only 10% of the principal subspace increases accuracy from 23.33% (base) to 40.3%, while preserving 50% of the tail subspace achieves only ~30% accuracy despite using 5× more rank budget.
• RL exhibits a slight advantage over OPD in tail directions. However, this marginal benefit comes with a large norm cost: the Frobenius norm of RL's $\Delta W_{\text{Bottom-}50\%}$ is approximately 1.6 to 2.5 times that of OPD's, while the corresponding performance gain remains limited.

Interpretation: RL distributes more update energy into tail directions (higher effective rank, lower Top-1% ratio), but these tail directions have low marginal utility for reasoning. This is the spectral-level manifestation of Functional Redundancy Avoidance (Property 1): just as RL wastes update norm on low-utility modules, it also wastes update norm on low-utility singular directions. OPD concentrates energy in the principal subspace that actually drives performance gains.

Step 3: Subspace evolution trajectory analysis. The above experiments use final-checkpoint update matrices (static analysis). The paper next examines when during training these dominant subspaces stabilize—the temporal dimension of foresight.

t-SNE visualization (Figure 5a, Appendix Figures 15–28): the Top-1 singular subspace $U_1$ from each training checkpoint for each module is projected to 2D via t-SNE. RL trajectories exhibit larger variations and more dispersed patterns across checkpoints, while OPD trajectories are more compact and smoother—suggesting greater directional stability during training.

Quantitative subspace alignment (Figure 5b): for each Top-$k$ subspace ($k = 1,...,20$) at each training step, compute cosine similarity with the corresponding subspace at the final checkpoint, then average over $k$. OPD consistently exhibits stronger alignment with its final subspaces than RL across all evaluated ranks. This difference is particularly pronounced in early training (0–30% progress), where OPD's alignment is already high (~0.85–0.95 for intermediate layers) while RL's is substantially lower and more fluctuating.

Cosine similarity heatmap (Figure 12): comparing $U_1$ at the first step with $U_1$ at the final step for each module, OPD shows high similarity (reaching up to 0.9 in intermediate layers), while RL shows consistently low similarity across modules. This confirms that OPD's dominant subspaces are already well-aligned with their final state from the very beginning of training.

PCA trajectory concentration (Figure 13): performing PCA on the checkpoint-wise representation vectors and computing the cumulative variance explained by the first two principal components (EVR₀:₂), OPD consistently achieves substantially higher values than RL. This quantitatively confirms that OPD trajectories are more concentrated in a low-dimensional subspace, while RL trajectories distribute variation across more directions.

Why trajectory concentration matters: if the optimization path is confined to a low-dimensional, stable subspace from early training, then once that subspace is identified, subsequent optimization is largely "amplification along known directions" rather than "exploration plus correction." This is exactly the condition under which extrapolation-based acceleration (EffOPD) can work safely.

Step 4: Directional quality vs. magnitude development—the critical norm-scaling experiment. The subspace alignment analysis shows early directional stabilization, but it does not tell us whether early directions are functionally sufficient—i.e., if we gave early checkpoints the same magnitude as final checkpoints, would they work? This is the experiment that directly motivates EffOPD.

Norm-scaling intervention (Figure 5c): for each intermediate OPD checkpoint, preserve the update direction within each module (the normalized parameter displacement), but rescale its Frobenius norm to match that of the corresponding module in the final checkpoint. Formally:

$\Delta W_{\text{scaled}} = \Delta W_{\text{early}} + \Delta W_{\text{early}} \times \beta \cdot \frac{\|\Delta W_{\text{final}}\|_F - \|\Delta W_{\text{early}}\|_F}{\|\Delta W_{\text{early}}\|_F}$

where $\beta$ is a scaling coefficient. At $\beta = 0$, the update is unchanged; at $\beta = 1$, the scaled norm matches the final checkpoint's norm.

What it computes: how much of the final performance can be recovered by simply amplifying early update directions to the final magnitude, without any change in direction. This decomposes the training progress into a directional component (does the early checkpoint point the right way?) and a magnitude component (is the step size along that direction sufficient?).

Key result (Figure 5c): a checkpoint at only 10% training progress recovers approximately 80% of the final model's performance after module-wise norm scaling. The KL divergence between the rescaled checkpoint's output distribution and the teacher's distribution also drops substantially, indicating closer behavioral alignment with the teacher.

Scaling coefficient sweep (Figure 14): varying $\beta$ from 0 to beyond 1.0 reveals:

• Performance increases with $\beta$ up to $\beta \approx 0.8$, then plateaus in the range $\beta \in [0.8, 1.2]$
• At $\beta > 1.2$, performance begins to degrade, suggesting that excessive amplification amplifies noise or irrelevant components
• KL divergence follows the inverse pattern: decreasing monotonically through the optimal range, then rising at extreme $\beta$

What this proves: early OPD checkpoints already possess task-relevant update directions aligned with the final solution. The main limitation preventing them from achieving final performance is insufficient update magnitude along those directions, not poor directional quality. Subsequent training primarily accumulates magnitude rather than making large directional adjustments—a form of "momentum along a pre-determined path."

Summary of Property 2: OPD's parameter updates exhibit strong low-rank concentration (higher spectral/Frobenius ratio, lower effective rank, higher Top-1% subspace norm ratio), with this concentration reflecting functional compactness (principal directions carry more task-relevant signal per unit norm than RL's). The dominant update subspaces align with the final update subspace very early in training and remain stable thereafter, with subsequent optimization primarily amplifying magnitude along these pre-established directions. This "directional lock-in" is what the paper calls foresight: OPD essentially determines "which direction to go" almost immediately and then commits to it, while RL spends more of its training budget exploring and correcting its trajectory.

Connection to Property 1: Property 2 provides the geometric mechanism underlying Property 1's modular-level observations. By locking into efficient low-rank directions early, OPD reduces its reliance on redundant exploration and correction, avoids overlearning irrelevant information, and can afford to suppress updates in low-utility modules—because the relevant signal is already concentrated in the principal subspace of the critical modules.

---

Local Geometric Theory of OPD Dynamics (Appendix F.5)

The paper provides a theoretical framework that explains why OPD induces Properties 1 and 2, grounding the empirical observations in the structure of the OPD objective. The analysis is local (linearizing around the base model) and applies to early training when parameter displacements are small.

Setup and linearization: define token context $c = (x, y_{<t})$ as the prompt plus previously generated tokens. Let $z_\theta(c) \in \mathbb{R}^V$ be the student logits, $z^\star(c)$ the fixed teacher logits, $\theta_0$ the base model parameters, and $\Delta\theta = \theta - \theta_0$ the displacement. Linearizing:

$z_\theta(c) \approx z_0(c) + J_c \Delta\theta$

where $z_0(c) = z_{\theta_0}(c)$ and $J_c = \frac{\partial z_\theta(c)}{\partial \theta}|_{\theta=\theta_0} \in \mathbb{R}^{V \times \dim(\theta)}$ is the Jacobian of logits with respect to parameters at the base model.

Define the teacher-student logit residual at the base model:

$r_c = z^\star(c) - z_0(c)$

This residual captures how the teacher's token predictions differ from the base model's. In distillation, the teacher typically refines predictions on a relatively small set of important tokens (key reasoning steps, answer tokens, formatting tokens), so $r_c$ tends to be sparse or low-dimensional in its effective support.

Then the logit discrepancy becomes:

$z_\theta(c) - z^\star(c) \approx J_c \Delta\theta - r_c$

Local quadratic approximation of the OPD objective: the OPD objective minimizes reverse KL divergence:

$\mathcal{L}_{\text{OPD}}(\theta) = \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi_\theta(\cdot|x)} \left[ D_{\text{KL}}(\pi_\theta(\cdot|x, y_{<t}) \| \pi^\star(\cdot|x, y_{<t})) \right]$

When the student and teacher distributions are close (as is typical in early training when starting from a base model similar to the teacher), the KL divergence admits a second-order Taylor expansion in logit space. Let $p_z = \text{softmax}(z)$ and $F_c = \text{diag}(p_0(c)) - p_0(c)p_0(c)^\top$ be the Fisher information matrix evaluated at the base model distribution. Then:

$D_{\text{KL}}(p_\theta \| p^\star) \approx \frac{1}{2}(z_\theta(c) - z^\star(c))^\top F_c (z_\theta(c) - z^\star(c))$

Substituting the linearized logit discrepancy:

$\mathcal{L}_{\text{OPD}}(\Delta\theta) \approx \frac{1}{2}\mathbb{E}_c\left[(J_c\Delta\theta - r_c)^\top F_c (J_c\Delta\theta - r_c)\right]$

Expanding the quadratic yields:

$\mathcal{L}_{\text{OPD}}(\Delta\theta) = \frac{1}{2}\Delta\theta^\top \underbrace{\mathbb{E}_c[J_c^\top F_c J_c]}_{=:A} \Delta\theta - \Delta\theta^\top \underbrace{\mathbb{E}_c[J_c^\top F_c r_c]}_{=:b} + \text{const}$

where $A$ is a Gram matrix capturing parameter sensitivities weighted by the Fisher information (a positive semidefinite matrix), and $b$ is the driving term that pushes parameters toward the teacher distribution.

What this quadratic form means: locally, OPD approximates convex quadratic minimization—the optimization landscape near the base model is bowl-shaped, with a unique minimum determined by $A$ and $b$. The gradient is:

$g(\Delta\theta) = A\Delta\theta - b$

Gradient descent dynamics and closed-form solution: gradient descent with step size $\eta$ gives:

$\Delta\theta_{s+1} = (I - \eta A)\Delta\theta_s + \eta b$

Starting from $\Delta\theta_0 = 0$, the unrolled update at step $s$ is:

$\Delta\theta_s = \eta \sum_{j=0}^{s-1} (I - \eta A)^j b$

This is a geometric series of matrices. Assuming $A$ is diagonalizable as $A = U\Lambda U^\top$ with $\Lambda = \text{diag}(\lambda_1, ..., \lambda_d)$ and $\lambda_1 \geq \lambda_2 \geq ...$, and $b = U\beta$ with $\beta_i = \langle b, u_i \rangle$, the update has the spectral form:

$\Delta\theta_s = \sum_{i: \lambda_i > 0} \frac{1 - (1 - \eta\lambda_i)^s}{\lambda_i} \beta_i u_i$

What this spectral form reveals:

1. Each eigen-direction $u_i$ is activated proportionally to $\beta_i / \lambda_i$, where $\beta_i$ is the projection of the driving term $b$ onto that direction, and $\lambda_i$ is the local curvature.

2. The factor $1 - (1 - \eta\lambda_i)^s$ grows more rapidly for larger $\lambda_i$, meaning directions with high sensitivity (where small parameter changes cause large logit changes) saturate early.

3. If $\beta_i \approx 0$ for many directions (i.e., the driving term $b$ is concentrated in a low-dimensional subspace), then the effective update remains confined to that subspace throughout training—this is the origin of Early Low-Rank Lock-in (Property 2).

Why $b$ is low-rank in practice: recall $b = \mathbb{E}_c[J_c^\top F_c r_c]$. The residual $r_c$—the teacher-base logit difference—is concentrated on a relatively small set of functionally important token positions (key reasoning tokens, answer tokens). The Fisher matrix $F_c$ reweights according to the local geometry of the output distribution but does not fundamentally change the support of the signal. Although $J_c$ itself can be high-rank, the composition $J_c^\top F_c r_c$ projects this concentrated residual back into parameter space, inheriting its low-dimensional structure. After averaging over contexts, $b$ tends to concentrate on parameter directions that most strongly affect those critical token predictions.

Module-wise suppression (Functional Redundancy Avoidance): decomposing parameters into modules $m = 1, ..., M$, the driving term for module $m$ is:

$b_m = \mathbb{E}_c[J_{c,m}^\top F_c r_c]$

where $J_{c,m}$ is the Jacobian restricted to module $m$. If $b_m \approx 0$ (module $m$ is weakly coupled with the teacher residual), and cross-module coupling $A_{mn}$ is small for $m \neq n$ compared to $A_{mm}$, then the optimal update for that module is approximately zero: $\Delta\theta_m^\star \approx A_{mm}^{-1} b_m \approx 0$. This is the mechanism underlying Functional Redundancy Avoidance (Property 1): modules that do not help match the teacher residual receive negligible updates.

Comparison with RL dynamics: the RL gradient (e.g., PPO) for a trajectory of length $T$ is:

$g_{\text{RL}} = \mathbb{E}\left[\sum_{t=1}^T A_t J_{c_t}^\top (e_{y_t} - p_\theta(\cdot|c_t))\right]$

where $A_t$ is an advantage estimate. Key differences:

• Density of signal: OPD's $r_c$ is non-zero at many token positions, while RL's $A_t$ is typically zero for most tokens in sparse-reward settings, making RL gradients noisier.
• Directional concentration: OPD's $b$ inherits the low-rank structure of $r_c$, while RL's driving term involves $e_{y_t} - p_\theta(\cdot|c_t)$, a random vector with full support in vocabulary space, leading to less concentrated, more diffuse updates.
• Gradient covariance: the OPD gradient has lower variance because $r_c$ is a deterministic function of the fixed teacher, while RL's advantage $A_t$ is a high-variance estimate depending on trajectory-level rewards.

Why this theory matters: it provides a unified mechanistic explanation for both empirical properties, bridging the gap between the optimization-level account ("dense supervision helps") and the parameter-level observations. Properties 1 and 2 are not coincidental or ad-hoc—they are direct consequences of the structure of the OPD objective when linearized around the base model.

---

EffOPD: Accelerating OPD via Directional Extrapolation (Section 4)

Building directly on Property 2 (early directional lock-in confirmed by the norm-scaling experiment), EffOPD exploits the fact that OPD establishes stable, final-aligned update directions early in training. The core idea: if the direction is already correct at 10% training progress, why take 100% of the steps to reach the destination? Instead, extrapolate further along the known-good direction and validate that performance still improves.

Checkpoint schedule: EffOPD triggers extrapolation at exponentially spaced checkpoints $t = 2^n$ with $n$ starting from 0. So extrapolation is performed at $t = 1, 2, 4, 8, 16, ...$. The exponential spacing is crucial: it concentrates extrapolation attempts in early training when directional lock-in is being established, while avoiding excessive validation overhead in later training when the model is near convergence.

Local update direction estimation: for the first checkpoint ($t = 1$), the local update direction is the displacement from initialization:

$\Delta_0 = W_1 - W_0$

For subsequent checkpoints with $n \geq 1$, the direction uses the displacement between the current exponential checkpoint and the previous one:

$\Delta_n = W_{2^n} - W_{2^{n-1}}$

What it computes: $\Delta_n$ captures the accumulated parameter evolution between two adjacent exponential checkpoints. Because OPD update directions remain relatively stable during training (Property 2), this displacement serves as a local approximation of the direction subsequent updates will take.

Why this form: using the displacement between exponential checkpoints (rather than, say, the instantaneous gradient at the current step) provides a smoothed, accumulated direction that is less sensitive to per-step noise. It also naturally adapts to the timescale of the exponential schedule: each $\Delta_n$ spans a geometrically growing interval, so the extrapolation leverages proportionally larger-scale directional signal as training progresses.

Candidate generation: EffOPD generates five candidate parameter sets from $W_{2^n}$ along $\Delta_n$ with increasing extrapolation magnitudes. For $k = 1, 2, 3, 4, 5$:

$\widetilde{W}_{n,k} = W_{2^n} + 2k \cdot \Delta_n$

What it computes: each candidate extends the current parameters along the estimated update direction by a factor of $2k$. At $k = 1$, the candidate is $W_{2^n} + 2\Delta_n$ (twice the distance between exponential checkpoints); at $k = 5$, it is $W_{2^n} + 10\Delta_n$ (ten times that distance). The coefficient $2k$ (rather than, say, a multiplicative factor $\gamma^k$) provides a linear progression of step sizes up to 10× the base displacement, covering a range from modest to aggressive extrapolation.

Why five candidates with these coefficients: the linear spacing $2, 4, 6, 8, 10$ provides a systematic sweep of extrapolation magnitudes. This is simpler than binary search and ensures that at least one candidate is likely to be in the effective range. The maximum factor of 10 is sufficient given the observed stability: the norm-scaling experiment (Figure 14) shows performance is robust in $\beta \in [0.8, 1.2]$ relative to the final norm, so extrapolating up to 10× the inter-checkpoint displacement is well within the stable regime.

Lightweight validation: EffOPD randomly samples 50 examples from the training set to form a validation set $\mathcal{D}_v$. This is deliberately much smaller than the number of sequences generated per step in vanilla OPD (which uses a prompt batch size of 1,024 with $n = 1$ output per prompt, generating 1,024 sequences per step). Let $V_{\mathcal{D}_v}(\cdot)$ denote the validation function, which evaluates the model on these 50 examples (the paper uses accuracy as the validation metric).

Why only 50 examples: the validation set is designed to be lightweight enough that the validation overhead is small compared to the training time saved by extrapolation. Fifty examples provide enough signal for a binary decision—"is this extrapolated model better or worse than the current one?"—without requiring the precision of a full evaluation. The ablation in Figure 7b confirms that validation set difficulty (easy vs. medium vs. hard problems) does not critically affect EffOPD's effectiveness, suggesting the validation mainly serves to verify that the current update direction remains effective rather than to provide precise supervision.

Adaptive acceptance mechanism: EffOPD initializes the accepted parameters as $W^{\text{acc}} = W_{2^n}$ (the current checkpoint without extrapolation) and its validation score as $v^{\text{acc}} = V_{\mathcal{D}_v}(W_{2^n})$. It then evaluates candidates sequentially for $k = 1, 2, 3, 4, 5$:

$\text{If } V_{\mathcal{D}_v}(\widetilde{W}_{n,k}) \geq v^{\text{acc}}, \text{ then: } W^{\text{acc}} \leftarrow \widetilde{W}_{n,k}, \quad v^{\text{acc}} \leftarrow V_{\mathcal{D}_v}(\widetilde{W}_{n,k})$

If a candidate fails to improve validation performance, the search terminates immediately without evaluating larger $k$ values. The final accepted parameters at checkpoint $2^n$ are:

$W_{2^n}^{\text{EffOPD}} = W^{\text{acc}}$

What happens step-by-step at an exponential checkpoint $2^n$:

1. Compute $\Delta_n = W_{2^n} - W_{2^{n-1}}$ (or $W_1 - W_0$ for $n = 0$)
2. Set baseline: $W^{\text{acc}} = W_{2^n}$, evaluate $V_{\mathcal{D}_v}(W_{2^n})$
3. For $k = 1$: construct $\widetilde{W}_{n,1} = W_{2^n} + 2\Delta_n$, evaluate $V_{\mathcal{D}_v}(\widetilde{W}_{n,1})$. If better than baseline, accept and continue to $k = 2$. If not, stop and keep $W_{2^n}$.
4. For $k = 2$: construct $\widetilde{W}_{n,2} = W_{2^n} + 4\Delta_n$, evaluate, accept if improved, else stop.
5. Continue through $k = 5$ or until a candidate is rejected.
6. Set $W_{2^n}^{\text{EffOPD}} = W^{\text{acc}}$ and resume OPD training from this point.

Why immediate termination on failure: if a candidate with step size $2k$ degrades validation performance, any larger step size ($2(k+1)$, etc.) along the same direction is likely to degrade performance even more severely. The monotonicity assumption—that validation score as a function of step size is unimodal—is reasonable given the norm-scaling results in Figure 14, which show a single plateau followed by degradation at extreme magnitudes. Terminating early avoids wasting computation on evaluating candidates that are almost certainly worse.

Degeneration to vanilla OPD: if the very first candidate ($k = 1$, step size $2\Delta_n$) fails to improve validation performance, EffOPD keeps $W_{2^n}$ unchanged and effectively degenerates to vanilla OPD for that checkpoint. This ensures EffOPD never makes things worse than the baseline—it only extrapolates when the extrapolation is validated to help.

Computational overhead: the overhead of EffOPD consists of (1) evaluating up to 5 candidate models on 50 validation examples each, at exponentially spaced checkpoints. Since the number of exponential checkpoints up to training step $T$ is $\log_2 T$, and vanilla OPD processes 1,024 training examples per step, the total validation overhead is $O(50 \cdot \log_2 T)$ evaluations versus $O(1024 \cdot T)$ training examples—a negligible fraction of total compute. The ablation in Figure 7c confirms that EffOPD achieves better performance under the same wall-clock time budget, meaning the gain from exploiting early-stage update directions outweighs the validation cost.

Key results (Figure 6): across model scales from 1.5B to 32B on both code and math tasks:

• EffOPD typically begins to converge within about 10 training steps, compared with 30–40 steps for vanilla OPD—a >3× speedup in terms of steps to convergence.
• EffOPD often reaches a higher performance upper bound than vanilla OPD, possibly because prolonged vanilla training can cause over-optimization and semantic drift.
• Compared with AlphaOPD and ExOPD (which use fixed extrapolation strategies), EffOPD's adaptive step-size selection via validation feedback leads to more stable acceleration, particularly in early training where fixed-step methods risk over-extrapolation.
• The early-stage advantage is especially pronounced on Qwen3-4B-Non-Thinking, where EffOPD attains strong reasoning performance by the 4th step.

Ablation studies (Figure 7):

• Learning rate sensitivity (Figure 7a): vanilla OPD with larger learning rates accelerates early convergence but causes noticeable oscillations and instability. EffOPD uses lightweight validation to filter out overly aggressive extrapolation steps, improving training stability even at higher learning rates.
• Validation set difficulty (Figure 7b): varying $\mathcal{D}_v$ from easy to hard problems does not significantly change EffOPD's effectiveness. The validation metric ("extrapolation accuracy") remains similar across difficulty levels, confirming that the validation's role is directional sanity-checking rather than precise optimization.
• Wall-clock time comparison (Figure 7c): despite validation overhead, EffOPD converges faster in actual training time than vanilla OPD, confirming net efficiency gain.

Why EffOPD works: the method is a direct synthesis of Properties 1 and 2. Property 2 guarantees that OPD's update directions are stable and final-aligned early in training—so extrapolating along $\Delta_n$ is extrapolating along a direction that genuinely points toward the final solution. Property 1 guarantees that the updates are concentrated in the right modules—so extrapolation amplifies task-relevant signal rather than amplifying noise in low-utility regions. The adaptive validation with immediate termination ensures that the extrapolation never overshoots into the degradation regime ($\beta > 1.2$ in Figure 14 terms) while capturing the available efficiency gains.

4. Key Insights and Innovations

Innovation 1: Reframing OPD's Efficiency from an Optimization-Level Story to a Parameter-Dynamics Mechanism

The dominant explanation for OPD's training efficiency prior to this work was purely optimization-centric: OPD converges faster than RL because token-level KL divergence from a fixed teacher provides dense, low-variance gradients, while RL from verifiable rewards provides a sparse signal only at sequence completion (He et al., 2026; Yue et al., 2025). This is true as far as it goes, but it is fundamentally an account of the loss landscape — it says the optimizer sees a better-shaped surface — not an account of what actually happens to the model's weights.

The paper's fundamental conceptual move is to shift the unit of analysis from the loss function to the parameter update matrix itself. Rather than asking "how good is the gradient signal?", the paper asks "what is the structure of $\Delta W$ and where in the network does it concentrate?" This reframing is significant because it reveals that OPD's advantage is not merely about faster convergence (getting to the same place quicker) but about different convergence (arriving at a structurally different solution with less wasted parameter movement). The finding that OPD achieves higher reasoning gains than RL under the same update norm constraint (Figure 2a) cannot be explained by gradient variance alone — it requires that RL's parameter updates contain components with low marginal utility for the task, components that contribute to the update magnitude but not to performance. This is a statement about update quality, not update speed.

What makes this reframing distinctive is that it provides a diagnostic framework rather than just a single finding. The paper doesn't merely claim "OPD is more efficient"; it develops a methodology for localizing where efficiency gains occur (sliding-window interventions), quantifying the geometric structure of updates (spectral metrics), and validating the functional significance of that structure (norm-scaling experiments, subspace partition experiments). This transforms OPD analysis from a black-box comparison of training curves into a mechanistic investigation of parameter dynamics.

Compared to prior work on OPD (Song & Zheng, 2026; Li et al., 2026; Yang et al., 2026b), which focused on theoretical frameworks, training recipes, or necessary conditions for success, this paper provides the first systematic empirical characterization of what OPD updates actually look like under the hood, and why those structural properties explain its practical advantage. It is a fundamental shift from "OPD works because of better optimization" to "OPD works because it produces structurally different parameter updates — updates that are more compact, better targeted, and directionally stabilized earlier." This is a new causal model of OPD's efficiency, not just a new measurement.

---

Innovation 2: The Concept of "Foresight" as a Unifying Diagnostic Principle for Training Dynamics

The paper introduces "foresight" as a meta-level concept that unifies two seemingly distinct empirical observations — Functional Redundancy Avoidance (modular targeting) and Early Low-Rank Lock-in (directional concentration) — under a single interpretive framework. This is more than terminological convenience; it's a conceptual innovation that changes how one thinks about what makes a training algorithm effective.

The foresight metaphor captures a specific temporal property of OPD dynamics: the algorithm behaves as if it knows, very early in training, which parameter regions and which update directions will ultimately matter for the final solution. It doesn't waste steps exploring "wrong mountains" (Figure 1a). This is not to anthropomorphize the algorithm — the paper grounds this in the structure of the OPD objective in Appendix F.5 — but the metaphor serves as a powerful diagnostic lens. It recasts training efficiency not as a question of "how fast does loss decrease?" but as a question of "how quickly does the optimizer commit to effective directions and how much of its budget is wasted on exploration?"

Prior work on training dynamics for LLMs (Cai et al., 2025; Chen et al., 2026) had identified low-rank structure and trajectory predictability in RL, but framed these as properties to be exploited for acceleration without asking why they arise or how they differ between training paradigms. The foresight concept makes a comparative claim: OPD exhibits more foresight than RL, and this difference — not just the presence of low-rank structure per se — explains OPD's efficiency advantage. The paper operationalizes "foresight" through measurable quantities: how early the dominant subspaces align with the final checkpoint (Figure 5b), how much performance can be recovered from an early checkpoint through pure magnitude scaling (Figure 5c), and how concentrated the update energy is in the principal subspace (Table 1).

The significance of this innovation extends beyond OPD. It suggests a new framework for evaluating and designing training algorithms: rather than asking "does this algorithm achieve good final performance?", one can ask "how much foresight does this algorithm exhibit?" An algorithm with high foresight (early commitment to effective directions, minimal wasted exploration) should be more amenable to acceleration, more robust to hyperparameter choices, and more interpretable in its training trajectory. This reframes the goal of post-training optimization from "find the lowest loss" to "find the most direct path to an effective solution" — a conceptual shift with practical implications for algorithm design that the paper begins to explore with EffOPD but that clearly extends to future work.

---

Innovation 3: Empirical Discovery That OPD's Early-Stage Update Directions Are Functionally Sufficient — Not Just Well-Aligned

The normative interpretation of early-training checkpoints in the field is that they are incomplete — the model hasn't converged, the directions are noisy, and performance is low because the parameters are far from optimal. The finding that an OPD checkpoint at only 10% training progress recovers approximately 80% of final performance after module-wise norm scaling (Figure 5c) challenges this assumption in a way that has deep implications for how we think about the temporal structure of OPD training.

What makes this finding distinctive is the experimental decomposition it enables. By separately manipulating update direction (preserved) and update magnitude (scaled to final-norm-equivalent), the paper demonstrates that the primary bottleneck preventing early OPD checkpoints from achieving final performance is not directional quality — it's insufficient magnitude along already-good directions. This is not merely "early checkpoints are correlated with the final solution" (which could be a trivial consequence of gradient descent following a smooth path). It is the much stronger claim that the subspace of effective updates is largely determined very early in training, and subsequent optimization primarily amplifies within that subspace rather than discovering new directions or correcting old ones. The plateau in the scaling coefficient sweep (Figure 14) — performance saturates at $\beta \approx 0.8$ and degrades only at extreme $\beta > 1.2$ — reinforces that the early subspace is not just correlated with the final solution but is functionally equivalent to it up to a magnitude factor.

This finding contrasts sharply with the implicit model of training that underlies most optimization narratives. In that model, early training is exploration — the optimizer probes various directions, evaluates their quality through gradient signals, and gradually homes in on a good region. The OPD picture that emerges from this paper is closer to directional prespecification: the optimizer essentially commits to a direction almost immediately (within the first 10% of steps) and then walks along that path. The exploration phase is compressed into a tiny fraction of total training.

Prior work on low-rank training dynamics (Cai et al., 2025) had observed that RL exhibits Rank-1 dominance and near-linear dynamics, but did not test whether early checkpoints were functionally sufficient — only that their directions were predictable. The norm-scaling experiment in this paper is a novel diagnostic that tests for functional sufficiency, not just geometric alignment, and the answer it provides for OPD is striking. This has direct practical consequences for acceleration (EffOPD exploits it), but it also raises fundamental questions: why is the OPD driving term $b = \mathbb{E}_c[J_c^\top F_c r_c]$ (Appendix F.5) sufficiently informative that the correct subspace can be identified from a tiny fraction of the training data? What properties of the teacher-student residual $r_c$ make this possible? These questions point toward a theory of distillation that goes beyond the "dense supervision" account to characterize the informativeness of per-token teacher signals for parameter-space navigation.

---

Innovation 4: A New Geometric Theory Bridging Optimization-Level and Parameter-Level Accounts of Distillation

The paper's local geometric theory (Appendix F.5) is not merely an appendage to the empirical results — it is a conceptual innovation that provides the theoretical bridge between why OPD works (optimization-level) and how OPD's parameter updates look (parameter-level). Prior theoretical work on OPD (Fu et al., 2026; Song & Zheng, 2026) characterized the objective function — its bias relative to sequence-level KL, its variance bounds, its equivalence to RLVR under certain conditions — but did not connect these properties to the structure of the resulting parameter updates. This paper's theory fills that gap.

The key move is linearizing the OPD objective around the base model and showing that it reduces to convex quadratic minimization $\frac{1}{2}\Delta\theta^\top A \Delta\theta - b^\top\Delta\theta$. This is a standard technique (neural tangent kernel style), but what makes it innovative is the interpretation of the resulting terms. The driving term $b = \mathbb{E}_c[J_c^\top F_c r_c]$ is not just "the gradient at initialization" — it is a projection of the teacher-student residual $r_c$ back into parameter space, weighted by the Fisher information $F_c$ and the Jacobian $J_c$. The low-rank structure of updates (Property 2) is explained by the fact that $r_c$ — the difference between teacher and base model logits — is itself concentrated on a relatively small set of functionally important tokens (key reasoning steps, answer tokens). The module-wise suppression (Property 1) is explained by the fact that modules with $b_m \approx 0$ — those weakly coupled to the teacher residual — receive negligible updates.

What distinguishes this from a standard "OPD is a quadratic bowl" analysis is that it answers why the bowl is shaped the way it is in terms of properties of the teacher, the base model, and the token-level KL objective. The spectral decomposition $\Delta\theta_s = \sum_i \frac{1 - (1 - \eta\lambda_i)^s}{\lambda_i} \beta_i u_i$ shows exactly how the interplay between curvature $\lambda_i$ and residual projection $\beta_i$ determines which directions activate and when. This connects the relatively abstract "dense supervision" story to concrete, measurable properties of parameter space: the effective rank of $A$, the spectral concentration of $b$, and the alignment between $b$ and the top eigenspace of $A$.

The comparison with RL dynamics is particularly insightful. The RL gradient $g_{\text{RL}} = \mathbb{E}[\sum_t A_t J_{c_t}^\top (e_{y_t} - p_\theta)]$ involves a driving term that is not a fixed residual from a teacher but a stochastic advantage-weighted difference between one-hot token vectors and the model's own distribution. This term has full support in vocabulary space and high variance (especially in sparse-reward settings), explaining why RL updates are more diffuse, less low-rank, and less directionally stable than OPD's. The theory thus provides a unified explanation for all the empirical properties — Property 1, Property 2, and their differences from RL — in terms of the structure of the training objective, not in terms of ad-hoc explanations.

The significance of this theoretical contribution is that it converts OPD efficiency from a "mystery solved by empiricism" to a "prediction derived from a model." It tells us what to expect when we change the teacher, the base model, or the task distribution: if the teacher-student residual $r_c$ is less concentrated (e.g., the teacher provides signal on many diverse tokens), the low-rank structure may weaken. If the Fisher $F_c$ is more uniform across vocabulary items, the concentration may shift. These are testable predictions that the theory enables but that pure empiricism cannot.

This is a fundamental contribution to the theory of knowledge distillation for language models. Previous theoretical work focused on statistical properties (bias, variance, equivalence to other objectives). This paper provides a geometric theory that connects those statistical properties to the structure of parameter updates — essentially, a bridge between the statistics of the loss and the geometry of optimization. It is not a final theory (it is local, linearized, and assumes small displacements), but it establishes a framework that future work can extend to the non-linear, non-local regime.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The experiments use two training datasets: Eurus-RL-Code (Cui et al., 2025a) for code generation and DeepMath-103K (Yang et al., 2026a) for mathematical reasoning, with additional experiments on MATH-12K (Lightman et al., 2023). Evaluation is conducted on seven benchmarks spanning both code and math: Codeforces, Taco (Liu et al., 2023), AIME24, AIME25, AIME26, MINERVA, and GPQA (Ye et al., 2025). The MATH-12K split follows Lightman et al. (2023) with 12,000 training questions; for the sliding-window intervention analysis, evaluation uses MATH500, a 500-question subset of MATH.

• Base model(s). The paper uses models from the Qwen2.5 and Qwen3 families (Qwen et al., 2025; Yang et al., 2025) at scales spanning 1.5B, 4B, 7B, 8B, 14B, and 32B parameters. These include pretrained base models (e.g., Qwen3-8B-Base), SFT-tuned models (e.g., Qwen2.5-1.5B-Deepseek, a DeepSeek-R1-distilled checkpoint), and Thinking-series models. The authors state they chose these because they are "representative of the capabilities of many contemporary LLMs" and span the regime where post-training efficiency differences between RL and OPD are practically meaningful. For the RL baselines, three representative algorithms are used: PPO, GRPO (DeepSeek-AI et al., 2025), and DAPO (Yu et al., 2025). For all OPD experiments, the teacher is the RL-tuned version of the same base model, ensuring the teacher possesses capabilities the base model lacks. For Qwen3-8B-Base, both Qwen3-8B-PPO and Qwen3-14B-Base-DAPO are used as teachers to test generality across teacher scales (Table 2).

• Metrics. The paper uses several distinct metrics depending on the analysis:

  • Reasoning accuracy on benchmark datasets (pass@1 on MATH500, accuracy on AIME variants, Codeforces, etc.), graded using the official MATH grading function.
  • Parameter update norm $\|\Delta W\|_F$, the Frobenius norm of the difference between trained and base model weight matrices, aggregated across all MLP and attention modules, used to measure the total magnitude of parameter change.
  • Spectral metrics (Section 3.1, Table 1): Spectral Norm ($\sigma_1$), Spectral-to-Frobenius Norm Ratio ($\sigma_1 / \|\Delta W\|_F$), Effective Rank (exponential of spectral entropy), and Top-1% Subspace Norm Ratio ($\|\Delta W_k\|_F / \|\Delta W\|_F$ where $k = \lceil r/100 \rceil$), all computed from the SVD of $\Delta W$ and averaged across all MLP and attention matrices.
  • Cosine similarity between dominant subspaces (Top-k singular vectors) at different training checkpoints, used to measure directional alignment and stability.
  • KL divergence between student and teacher output distributions, used in the norm-scaling experiments to measure distributional alignment (Figure 5c, Figure 14b).
  • Extrapolation accuracy on the lightweight validation set $\mathcal{D}_v$, used as the acceptance criterion in EffOPD (Figure 7b).

• Baselines. The paper compares EffOPD against:

  • Vanilla OPD: standard on-policy distillation with reverse KL divergence, trained for 3 epochs with the settings detailed in Appendix D.2 (prompt batch size 1,024, $n = 1$ output per prompt, learning rate $1 \times 10^{-6}$ without warmup).
  • AlphaOPD (Cai et al., 2025): a method that accelerates training by exploiting Rank-1 dominance and linear dynamics, originally developed for RL but applied here to OPD. Uses fixed extrapolation based on subspace prediction.
  • ExOPD (Yang et al., 2026a): a method that performs reward extrapolation with fixed step sizes for on-policy distillation. Both AlphaOPD and ExOPD use fixed extrapolation strategies, unlike EffOPD's adaptive step-size selection via validation feedback.

• Generation budget / compute accounting. For the comparative analysis of RL vs. OPD, the paper measures parameter efficiency using $\|\Delta W\|_F$ as the primary resource metric — the total magnitude of parameter change — rather than training steps or wall-clock time, because this directly captures how much the model's weights have moved from the base initialization. For the EffOPD acceleration experiments (Section 4), the primary metrics are training steps to convergence and wall-clock training time (Figure 7c), with the validation overhead of EffOPD (50 examples per exponential checkpoint) explicitly accounted for. The generation budget per training step for vanilla OPD is 1,024 sequences (prompt batch size 1,024 × $n = 1$); for RL it is 512 sequences (effective batch size) × $n = 16$ outputs per prompt = 8,192 sequences. EffOPD adds approximately $50 \times \log_2 T$ validation examples, where $T$ is the total training steps — a negligible fraction of total compute.

• Cross-validation / statistical protocol. No formal cross-validation is reported. The analysis uses the entire training trajectory (all checkpoints saved at regular intervals of 1–2 steps) for the subspace evolution and scaling experiments. For EffOPD, the validation set $\mathcal{D}_v$ of 50 randomly sampled examples is drawn from the training set once; the ablation in Figure 7b tests robustness to different validation set compositions (easy/medium/hard problems) but does not use multiple random splits. The paper reports that each intervened model in the sliding-window analysis is evaluated using four independent forward passes with results averaged to reduce evaluation noise, but no confidence intervals or standard deviations are reported for the main figures.

---

Main Quantitative Results

The experimental results are organized into two major phases: (1) the comparative analysis establishing Properties 1 and 2 (Sections 2–3), and (2) the EffOPD acceleration results (Section 4). We treat these in sequence.

Property 1: Functional Redundancy Avoidance (Section 2 Results)

Headline finding: OPD achieves higher reasoning performance per unit of parameter update norm than RL, and this advantage stems from suppressing updates in low-utility modules while concentrating updates in reasoning-critical middle-layer MLPs.

Norm-scaled update efficiency. Figure 2 reports the relationship between parameter update magnitude and reasoning accuracy. At the final checkpoint (Figure 2a), when OPD and RL updates are scaled to the same Frobenius norm (via the linear scaling $W_{\text{base}} + \alpha \Delta W$ with $\alpha \in [0, 1]$), OPD achieves substantially higher reasoning gains. For example, on Qwen3-8B-Base, at $\alpha = 0.5$ (half the full update norm), OPD recovers roughly 75% of its final performance while RL recovers only about 40%. On Qwen3-4B-Non-Thinking, OPD achieves approximately 35% accuracy at the same norm level where RL achieves approximately 20%.

Across the training trajectory (Figure 2b), OPD consistently requires smaller $\|\Delta W\|_F$ to reach any given accuracy threshold. For Qwen3-8B-Base with DAPO vs. OPD, reaching 20% accuracy on MATH500 requires an OPD update norm of approximately 2 (arbitrary units reported in the paper's normalization) while RL requires approximately 8 — a ~4× difference in parameter movement for equivalent performance.

Cross-scale and cross-algorithm validation. Figure 8 extends the scaling analysis to final checkpoints for models from 1.5B to 32B. The pattern holds across all scales: OPD consistently achieves reasoning performance comparable to or exceeding RL while requiring substantially smaller parameter update norms. Figure 9 demonstrates the same advantage across all three RL algorithms (PPO, GRPO, DAPO), confirming that the efficiency gain arises from structural properties of OPD updates rather than the choice of the underlying RL method.

Locating redundant updates — embedding layers. Figure 3a reports the effect of replacing embedding layers. On AIME26, replacing the embedding layer of an RL-trained model with the base model's embedding layer changes accuracy negligibly — the embedding contributes little to reasoning gains despite having large update norms in some models. On MATH500, the same intervention on OPD models produces similarly negligible impact. Table 3 quantifies the distributional shift: OPD-trained models maintain higher cosine similarity between their token embeddings and the base model's embeddings than RL-trained models (e.g., on Qwen3-8B-Base: OPD similarity 0.975 vs. RL similarity 0.942 for DAPO-trained models). Figure 11 visualizes these embedding shifts via t-SNE, showing that OPD's embedding changes are more compact and less dispersed than RL's.

Locating redundant updates — sliding-window intervention. Figure 3b reports the layer-wise analysis on Qwen3-8B-Base. The bar plot (left axis) shows that RL introduces substantially larger update norms in bottom layers (layers 0–5) and top layers (layers 30–35) compared to OPD. The line plot (right axis) shows the reasoning accuracy when OPD updates are injected into a sliding window of 17 layers centered at each position — the curve is inverted-U-shaped, with interventions in middle layers (roughly layers 10–25) producing the largest gains (up to ~30% accuracy) and interventions in bottom/top layers producing significantly smaller gains (below ~15%). Figure 10b reports the mirror analysis for RL-trained models: the sensitivity curve has the same inverted-U shape, but RL's update norms in low-sensitivity regions are 2–4× larger than OPD's. In the MLP modules specifically, OPD concentrates updates in layers 12–24 where the marginal utility is highest, while RL spreads substantial updates into layers 2–8 and 28–34 where the marginal utility is low. The attention modules show a similar but less pronounced pattern. Figure 10a confirms that the embedding-replacement null effect replicates on MATH500 across multiple model configurations.

Interpretation. The paper argues this demonstrates Property 1: OPD exhibits "foresight" by identifying low-utility modules and suppressing their updates early, concentrating its limited norm budget on modules where changes actually improve reasoning. The fact that RL and OPD share the same sensitivity distribution (the curves have the same shape) but differ in how they allocate update magnitude across layers is the key evidence that OPD's advantage is about allocation efficiency, not about discovering different functional pathways.

---

Property 2: Early Low-Rank Lock-in (Section 3 Results)

Headline finding: OPD's parameter updates exhibit stronger low-rank concentration than RL's from early in training, and the dominant subspaces align with the final update subspace much earlier — a checkpoint at only 10% training progress recovers approximately 80% of final performance after norm scaling.

Spectral concentration — static analysis. Table 1 reports the four spectral metrics averaged across all MLP and attention matrices for models at 1.5B, 4B, 8B, and 14B scales. Across all scales, OPD exhibits stronger low-rank structure than RL. For the 8B model:

• Spectral norm: 0.005 (OPD) vs. 0.004 (RL)
• Spectral-to-Frobenius norm ratio: 36.8% (OPD) vs. 32.7% (RL) — OPD concentrates 12.5% more energy in the leading direction
• Effective rank: 2341 (OPD) vs. 2754 (RL) — OPD's spectrum decays faster (413 fewer effective dimensions)
• Top-1% subspace norm ratio: 94.7% (OPD) vs. 88.5% (RL) — OPD captures 94.7% of update energy in just 1% of singular directions

These differences are consistent across scales: at 1.5B, the Top-1% ratio is 92.3% (OPD) vs. 78.1% (RL); at 14B, 94.5% (OPD) vs. 81.2% (RL). The gap in spectral concentration is largest at the extremes of the scale range (14–16 percentage points) and narrowest at 8B (~6 percentage points).

Functional partition — Top-k% subspace quality. Figure 4a reports the reasoning accuracy achieved by the Top-k% subspace approximation at various rank budgets, with norm equalized between RL and OPD. On Qwen2.5-1.5B-Deepseek, using only 10% of singular components: OPD recovers approximately 42% accuracy vs. approximately 38% for RL — a consistent 3–4 percentage point advantage. Both methods recover over 95% of their full-model performance at 10% rank, confirming that the principal subspace is the primary carrier of task-relevant signal. The OPD advantage persists across rank levels from 1% to 20% and across model scales. The paper interprets this as evidence that OPD's principal directions inherently encode more effective update signals than RL's under equal norm budgets — it's not just that OPD is more compressed, but that the compression preserves higher-quality directions.

Functional partition — Bottom-k% subspace marginal utility. Figure 4b reports the performance and norm cost of the tail subspace. On Qwen2.5-1.5B-Deepseek, the Top-10% subspace increases accuracy from 23.33% (base) to 40.3%, while the Bottom-50% subspace achieves only approximately 30% accuracy — despite using 5× more rank budget. The norm of RL's tail subspace ($\Delta W_{\text{Bottom-50\%}}$) is approximately 1.6 to 2.5 times that of OPD's, but RL's corresponding performance gain in the tail is only marginally better than OPD's. The figure caption states this directly: "RL incurs significantly larger norm cost for marginal performance gains" in tail directions.

Subspace evolution — qualitative. Figure 5a presents t-SNE visualizations of the Top-1 subspace trajectory for RL (DAPO) and OPD training. The RL trajectory shows larger displacements and more irregular patterns across checkpoints, while OPD's trajectory is visibly more compact and smoother. Appendix Figures 15–28 provide the full layer-by-layer t-SNE visualizations for all modules (MLP up-projection, MLP gate, attention Q/K/V/O projections, and full attention) under both DAPO and OPD, consistently showing the more concentrated OPD pattern.

Subspace evolution — quantitative alignment. Figure 5b reports the cosine similarity between Top-k subspaces ($k = 1,...,20$, averaged) at intermediate checkpoints and at the final checkpoint. For Qwen3-8B-Base:

• OPD achieves cosine similarity of approximately 0.85–0.95 by 10% training progress for intermediate layers
• RL achieves only approximately 0.55–0.70 at 10% progress and shows larger fluctuations throughout training
• The gap is largest in early training (0–30% progress), where OPD's alignment is already high while RL's is still growing
• By 100% progress, both methods converge to similarity 1.0 by definition, but OPD's trajectory is substantially flatter

The cosine similarity heatmap in Figure 12 shows the alignment between $U_1$ at the first step and $U_1$ at the final step per module. OPD exhibits high similarity (reaching up to 0.9 in intermediate MLP layers), while RL shows consistently low values across all modules. This confirms that OPD locks into dominant directions almost immediately, while RL's dominant subspace undergoes substantial reorganization during training.

Subspace evolution — trajectory concentration. Figure 13 reports the cumulative variance explained by the first two principal components (EVR₀:₂) for checkpoint-wise representation trajectories. OPD consistently achieves higher EVR₀:₂ than RL across all modules. For example, in the MLP up-projection modules, OPD achieves EVR₀:₂ of approximately 0.75–0.85 (meaning 75–85% of trajectory variation is captured by a 2D plane) vs. approximately 0.50–0.65 for RL. This quantifies the qualitative observation from t-SNE: OPD trajectories are more strongly confined to a low-dimensional subspace.

The critical norm-scaling experiment. Figure 5c reports the performance of OPD intermediate checkpoints after module-wise norm scaling to match the final checkpoint's per-module Frobenius norm. On Qwen3-8B-Base:

• A checkpoint at 10% training progress recovers approximately 80% of the final model's MATH500 accuracy after norm scaling
• The KL divergence between student and teacher output distributions drops proportionally, indicating closer behavioral alignment with the teacher
• The accuracy and KL curves show rapid initial improvement followed by saturation

Figure 14 extends this with a sweep of the scaling coefficient $\beta$ (where $\beta = 0$ means no scaling and $\beta = 1$ means matching the final norm). On Qwen3-8B-Base:

• Accuracy increases monotonically from $\beta = 0$ to $\beta \approx 0.8$
• Accuracy plateaus in the range $\beta \in [0.8, 1.2]$ — meaning the early direction can support up to 120% of the final norm before degrading
• At $\beta > 1.2$, accuracy begins to decline, indicating that excessive amplification introduces noise or moves the model off the effective manifold
• KL divergence follows the inverse pattern: decreasing through the optimal range and rising at extreme $\beta$

The paper interprets this as evidence that early OPD checkpoints already possess task-relevant update directions of high quality, and the primary limitation is insufficient magnitude along those directions — subsequent training primarily accumulates magnitude rather than making large directional adjustments.

---

EffOPD Acceleration Results (Section 4 Results)

Headline finding: EffOPD achieves an average training acceleration of 3× over vanilla OPD across model scales from 1.5B to 32B on both code and math benchmarks, while maintaining comparable or better final performance.

Convergence speed. Figure 6 reports accuracy vs. training steps for EffOPD, vanilla OPD, AlphaOPD, and ExOPD across four model scales (1.5B, 4B, 14B, 32B) on code and math benchmarks. Key observations:

• On mathematical reasoning tasks (right column), EffOPD typically begins to converge within approximately 10 training steps, compared with 30–40 steps for vanilla OPD — yielding more than a 3× speedup in steps to convergence.
• On code generation tasks (left column), the acceleration is similarly pronounced: EffOPD reaches strong performance by step 10–15, while vanilla OPD requires 30+ steps.
• On Qwen3-4B-Non-Thinking (second row, right), EffOPD attains strong reasoning performance by the 4th step, while vanilla OPD is still in its initial ramp-up phase.
• EffOPD often reaches a higher final performance than vanilla OPD. For example, on Qwen2.5-32B math tasks, EffOPD achieves approximately 46% accuracy vs. approximately 42% for vanilla OPD at convergence.
• AlphaOPD and ExOPD show less consistent acceleration: their fixed extrapolation strategies sometimes accelerate (e.g., AlphaOPD on 1.5B shows early gains) but occasionally underperform vanilla OPD at convergence or show instability in mid-training.

Wall-clock time efficiency. Figure 7c directly compares accuracy vs. actual training time. Despite the additional validation overhead of evaluating up to 5 candidate models on 50 examples at each exponential checkpoint, EffOPD achieves better performance than vanilla OPD at every time point and converges to its final performance faster. The validation overhead is negligible relative to the training time saved by needing fewer steps.

Ablation: learning rate sensitivity. Figure 7a reports the effect of varying the learning rate on EffOPD and vanilla OPD. Vanilla OPD with larger learning rates accelerates early convergence but causes noticeable oscillations and performance instability — for example, at the highest learning rate tested, accuracy fluctuates by 5–8 percentage points between adjacent checkpoints. EffOPD achieves comparable early acceleration but with substantially smoother training curves, as the lightweight validation during extrapolation filters out overly aggressive steps. The paper frames this as evidence that EffOPD's adaptive acceptance mechanism provides implicit stabilization.

Ablation: validation set difficulty. Figure 7b reports the impact of validation set composition. The x-axis shows "Extrapolation Accuracy" — the accuracy of the model on $\mathcal{D}_v$ before training — and the paper tests validation sets of varying difficulty (easy, medium, hard, and mixed). The key result is that validation sets of different difficulty levels all provide consistent directional signals: the acceptance/rejection decisions are similar regardless of whether $\mathcal{D}_v$ consists of easy or hard problems. The paper interprets this as evidence that the validation step mainly serves to check whether the current update direction remains effective (a coarse binary signal) rather than providing precise supervision. The exact quantitative difference in rejection rates or acceptance counts across difficulty levels is not reported.

---

Ablation Studies and Robustness Checks

• Cross-scale robustness of norm-efficiency advantage (Figure 8). The paper tests the per-norm efficiency of OPD vs. RL at final checkpoints across 1.5B, 4B, 8B, and 32B models. At all scales, under the same Frobenius norm constraint, OPD achieves higher reasoning accuracy than RL. The advantage is not uniform: it is most pronounced at the 1.5B scale (where OPD achieves roughly double RL's accuracy at equivalent norm) and somewhat smaller at the 32B scale, but remains consistently positive. This rules out the possibility that Property 1 is an artifact of small models.

• Cross-algorithm robustness of norm-efficiency advantage (Figure 9). The paper tests OPD against three RL variants — PPO, GRPO, and DAPO — on the same base model (Qwen2.5-1.5B-Deepseek). OPD tracks the same accuracy-vs-norm curve regardless of which RL algorithm trained the teacher, and this curve consistently dominates the RL curves. This confirms that Property 1 is about OPD vs. the RL paradigm broadly, not about specific algorithmic choices within RL.

• Cross-dataset replication of sliding-window findings (Figure 10). The embedding replacement null effect (Figure 10a) and the sliding-window intervention patterns (Figure 10b) replicate on MATH500 across different RL algorithms and teacher configurations listed in Table 2 (Qwen2.5-1.5B-Deepseek with JustRL/GRPO, BroRL/PPO, and ProRL/DAPO; Qwen3-4B-Non-Thinking with Qwen-4B-GRPO; Qwen3-8B-Base with PPO and DAPO; Qwen3-14B-Base with DAPO). The inverted-U-shaped sensitivity curve and the OPD-vs-RL norm allocation difference are consistent across these diverse settings.

• Teacher scale generalization. In Table 2, the authors note that for Qwen3-8B-Base, both a same-scale teacher (Qwen3-8B-PPO) and a larger-scale teacher (Qwen3-14B-Base-DAPO) are used. The paper does not report a separate figure comparing results across teacher scales, but states in Section 2.1 that this ensures "the generality of our conclusions" — implying the properties hold regardless of teacher size.

• Spectral metrics — cross-scale consistency (Table 1). The four spectral metrics (Spectral Norm, Spectral/Frobenius Ratio, Effective Rank, Top-1% Subspace Norm Ratio) all show the same OPD > RL pattern across 1.5B, 4B, 8B, and 14B scales. The magnitude of the difference varies — the gap in Top-1% ratio is largest at 1.5B (92.3% vs. 78.1%, a 14.2 percentage point gap) and smallest at 8B (94.7% vs. 88.5%, a 6.2 percentage point gap) — but the direction is consistent. This is critical because spectral properties can be sensitive to model architecture and initialization; the cross-scale replication strengthens confidence that Property 2 is not an artifact of a particular model configuration.

• Norm-scaling coefficient sweep (Figure 14). The fan plot of accuracy vs. scaling coefficient $\beta$ at different training progress levels provides a robustness check on the norm-scaling experiment. The key finding — that $\beta \in [0.8, 1.2]$ is the optimal range across multiple checkpoint ages — holds consistently from 10% to 90% training progress. The fact that early checkpoints show the same optimal $\beta$ range as late checkpoints supports the claim that directional quality is established early and magnitude is the primary bottleneck. The corresponding KL divergence plot (Figure 14b) shows the same pattern inverted, confirming that the accuracy improvements correspond to genuine distributional alignment with the teacher, not just metric gaming.

• Negative result: EffOPD's adaptive step selection outperforms fixed extrapolation (Figure 6). The comparison with AlphaOPD and ExOPD serves as an ablation of the adaptive acceptance mechanism. Fixed-step methods (AlphaOPD, ExOPD) show inconsistent behavior: AlphaOPD provides some early acceleration on Qwen2.5-1.5B but plateaus below vanilla OPD on Qwen3-4B; ExOPD tracks vanilla OPD closely on some configurations but diverges on others. This negative result — that fixed extrapolation does not reliably accelerate OPD — is important because it validates the paper's claim that adaptive step-size selection is necessary, which in turn is motivated by the observation that directional stability is high but not perfect (over-extrapolation degrades performance at $\beta > 1.2$, Figure 14).

• PCA trajectory concentration (Figure 13). The EVR₀:₂ metric provides a quantitative, model-free summary of the t-SNE visualizations. The consistent OPD > RL gap confirms that the trajectory concentration is not an artifact of t-SNE's hyperparameters but reflects genuine low-dimensional structure in the checkpoint-wise representation vectors.

• Negative result: embedding updates contribute negligibly to reasoning (Figures 3a, 10a, Table 3). This is presented as a finding in its own right, but also serves as a negative control for the sliding-window methodology — if embedding intervention showed large effects, it would indicate that the method is sensitive to spurious correlations or that embeddings carry reasoning signal in ways the Transformer layers cannot compensate for. The null result strengthens the interpretability of the positive findings in the middle-layer interventions.

---

Critical Assessment

Does the paper genuinely demonstrate that OPD's efficiency stems from "foresight" rather than from denser supervision?

The paper's central argument is that OPD's efficiency advantage over RL is not merely about denser supervision (faster convergence) but about structurally different parameter updates — what the paper calls "foresight." The evidence for this is mixed but generally persuasive.

What the experiments actually demonstrate: The paper shows that at equal update norm, OPD achieves higher accuracy than RL (Figure 2a) and that RL allocates more update magnitude to low-utility modules (Figure 3b). These are genuine structural differences, not reducible to "OPD converges faster to the same solution." However, the paper does not provide a direct causal test of the claim that these structural differences cause the efficiency advantage. It could be that OPD's denser supervision enables it to converge faster, and the structural properties (low-rank updates, modular concentration) are consequences of getting closer to the optimum earlier, rather than causes of getting there efficiently. The norm-scaling experiment (Figure 5c) partially addresses this by showing that early directions are functionally equivalent to final directions, but it does not establish that because OPD updates are low-rank, it trains efficiently — only that low-rank structure and efficiency co-occur.

A stronger causal test would be to train an RL model with an auxiliary loss that explicitly penalizes updates in low-utility modules (e.g., an $L_2$ penalty on embedding and bottom/top layer parameter changes), and test whether this RL variant recovers OPD-like efficiency. Alternatively, adding noise to OPD's update directions (to artificially increase effective rank) and measuring whether this degrades efficiency would provide evidence that the low-rank structure is causally important. Neither experiment is reported.

Does Property 1 (Functional Redundancy Avoidance) genuinely reflect "foresight" or is it a post-hoc description of the obvious fact that a well-converged model has smaller updates in modules that don't matter?

The sliding-window intervention measures the marginal contribution of updates in each layer window, but this is a correlational analysis: it shows that layers with large updates tend to be layers where interventions produce large accuracy gains (for OPD), and that RL breaks this correlation by having large updates in layers where interventions don't help much. However, the paper does not demonstrate that OPD actively suppresses these low-utility updates through some foresight mechanism, as opposed to simply not needing to change those layers because the teacher signal is well-aligned with the base model. If the teacher and base model already agree on token predictions in bottom layers (because bottom layers encode general linguistic features that are largely shared), then OPD — which minimizes per-token KL divergence — would naturally produce small updates there. This would be a consequence of the distribution of the teacher signal, not an active "decision" by OPD to allocate updates efficiently.

The local geometric theory in Appendix F.5 partially addresses this: it shows that $b_m \approx 0$ for modules weakly coupled to $r_c$, meaning low-utility modules receive small driving forces. But the theory describes why OPD suppresses these updates (the teacher residual doesn't project onto those modules), not that OPD has a separate mechanism for identifying and suppressing them beyond what the objective function implies. The "foresight" framing may overstate the agency of the algorithm — it's more accurate to say that the OPD objective induces compact updates because the teacher signal is concentrated, and this happens to align with functional efficiency.

Are the spectral properties (Property 2) robust to the choice of SVD normalization?

The paper computes SVD on $\Delta W$ directly, but $\Delta W$ for different modules (MLP up-projection, attention Q/K/V/O) have different shapes and scales. The paper averages the four spectral metrics across all MLP and attention matrices (Table 1), which implicitly weights all modules equally regardless of their parameter count or their importance to the task. A module with a small $\Delta W$ but high spectral concentration would contribute equally to the average as a module with a large $\Delta W$ but moderate concentration. The paper does not report metric values per module or discuss whether the OPD-RL gap is driven by specific module types. If, for example, the gap is largest in attention output projections (which tend to be smaller matrices) and smallest in MLP up-projection (which tend to be larger), the practical significance of the spectral concentration finding would be more nuanced than the aggregate numbers suggest.

Further, the SVD is performed on each weight matrix independently, but parameter updates across modules are correlated — changing an attention output projection affects the input to the next layer's MLP. The per-matrix SVD analysis treats these as independent, potentially missing cross-module structure that might be important for understanding why OPD's updates are more efficient. A joint analysis (e.g., SVD on the concatenated update vector, or canonical correlation analysis between module updates) might reveal different patterns.

The norm-scaling experiment (Figure 5c) is the strongest single piece of evidence in the paper, but its interpretation requires careful qualification.

The finding that a 10% checkpoint recovers ~80% of final performance after norm scaling is striking, but it measures potential recoverability, not actual training dynamics. The scaling operation is applied module-wise: each module's update direction is preserved but its norm is independently rescaled to match the final checkpoint. This is a non-uniform scaling that effectively performs a per-module learning rate adjustment based on oracle knowledge of the final checkpoint's norms. It demonstrates that the directions are good, but not that a simple uniform extrapolation (which is what EffOPD does) would work — EffOPD extrapolates along the displacement between exponential checkpoints, which is a different direction than the path from the base model to the current checkpoint. The paper bridges this gap by noting that OPD's directions are stable (so the inter-checkpoint displacement approximates the direction to the final solution), but the quality of this approximation is not directly tested — the paper doesn't report, for example, the cosine similarity between $\Delta_n$ (EffOPD's extrapolation direction) and $\Delta W_{\text{final}} - \Delta W_{2^n}$ (the true remaining displacement to the final checkpoint).

The EffOPD validation set construction raises questions about practicality at scale.

EffOPD uses 50 randomly sampled examples from the training set as a validation set. This is small enough to be cheap, but the paper doesn't report the variance of the validation metric across different random samples of 50 examples. If the variance is high (which is plausible for mathematical reasoning problems where a single difficult problem can swing accuracy by 2 percentage points on a 50-example set), then the adaptive acceptance mechanism might make noisy decisions — accepting candidates that happened to score well on the 50 examples but would not generalize, or rejecting candidates that would have improved performance. Figure 7b shows that validation sets of different difficulties provide consistent directional signals, but "consistent" here means the acceptance/rejection patterns are similar, not that the validation metric has low variance. The paper does not report how often EffOPD accepts each candidate ($k = 1$ through $k = 5$), what fraction of exponential checkpoints result in extrapolation vs. degeneration to vanilla OPD, or whether the accepted extrapolation steps actually improved held-out benchmark performance or merely improved the validation metric.

Missing baselines and experiments.

• The paper compares EffOPD against AlphaOPD and ExOPD but not against a simple learning rate schedule baseline (e.g., cosine decay with warmup, or a step-wise schedule with larger initial learning rate). If a well-tuned learning rate schedule can achieve similar acceleration to EffOPD without the validation overhead, the practical value of EffOPD's adaptive mechanism would be diminished.

• The paper does not report statistical significance for any of its comparisons. The test set for the sliding-window analysis is MATH500 (500 questions per evaluation), so differences of 1–2 percentage points between interventions may be within the noise floor of the evaluation. Confidence intervals would help distinguish meaningful differences from sampling noise.

• The cross-validation for EffOPD's validation set is limited to one difficulty-level ablation (Figure 7b). A more thorough robustness check would vary the size of $\mathcal{D}_v$ (10, 20, 50, 100 examples) and measure how validation set size affects acceptance decisions and final performance. If 10 examples suffice, EffOPD is even cheaper than reported; if 50 is marginal, the method may be fragile.

• The paper does not test EffOPD in combination with other acceleration methods (e.g., learning rate warmup, gradient accumulation tuning, mixed-precision training optimizations). Since EffOPD is described as "orthogonally compatible," demonstrating that it provides additional gains on top of existing efficiency techniques would strengthen the practical case.

Generalization concerns.

All experiments use Qwen-family models and a specific OPD implementation (Verl framework, specific hyperparameters in Appendix D.2). The consistency across scales (1.5B–32B) and RL algorithms (PPO, GRPO, DAPO) provides some evidence of generality, but the paper does not test across model families (e.g., LLaMA, DeepSeek) or OPD implementations. The OPD training command in Appendix D.2 reveals that the paper uses kl_loss_coef=0 and kl_loss_type=low_var_kl — meaning the KL loss is computed but not added to the training objective. This is an unusual configuration (the paper states it follows Yang et al., 2026a) that may affect the generality of the findings. If other OPD implementations use a non-zero KL coefficient or a different KL estimator, the parameter-dynamic properties might differ.

The theory in Appendix F.5 is elegant but its quantitative predictions are not tested.

The local geometric theory predicts that the effective rank of $\Delta W$ should be related to the effective rank of $b$ (the driving term) and the spectral structure of $A$. The paper does not compute $b$ or $A$ empirically, does not verify that $b$ is actually low-rank (as the theory predicts), and does not test whether the spectral form $\Delta\theta_s = \sum_i \frac{1 - (1 - \eta\lambda_i)^s}{\lambda_i} \beta_i u_i$ fits the observed training trajectory. These would be direct tests of the theory's explanatory power but are computationally expensive (requiring Jacobian computation for large models). The paper acknowledges in Appendix C that the theory is "inherently local" and does "not fully capturing the global non-convex behavior." This is an honest limitation, but it means the theory should be viewed as a conceptual framework rather than a validated predictive model.

Overall assessment.

The paper's strongest contribution is the empirical demonstration that OPD and RL produce structurally different parameter updates, and that these differences are consistent across scales and algorithms. Property 1 (modular allocation efficiency) is well-supported by the norm-efficiency and sliding-window experiments, though the causal interpretation ("foresight") overstates what the correlational evidence can support. Property 2 (early low-rank lock-in) is well-supported by the spectral metrics and subspace alignment analysis, and the norm-scaling experiment is the most compelling single piece of evidence in the paper — it cleanly decomposes training progress into directional quality vs. magnitude development. EffOPD is a practical contribution with a clear motivation from the analysis, and the 3× acceleration claim is supported by the training trajectory comparisons in Figure 6, though the practical value depends on whether the acceleration holds in settings beyond the Qwen-VerL-OPD pipeline tested here. The theory in Appendix F.5 provides a useful conceptual bridge but is not empirically validated. The main gaps are: (1) no direct causal test of whether the identified properties cause efficiency, (2) no statistical significance reporting, (3) limited model family diversity, and (4) no test of whether simpler baselines (learning rate schedules) could achieve comparable acceleration without the adaptive validation mechanism.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Never Accounted For in Practical Terms

The assumption or constraint. Property 1 and Property 2 are identified through analyses that require fully trained RL and OPD checkpoints — meaning the entire training run must be completed before the structural properties of the updates can be characterized. The analyses compare final-checkpoint $\Delta W$ matrices, run sliding-window interventions requiring evaluation on MATH500 (500 questions × 4 forward passes per window position × $L$ layers, yielding thousands of evaluations), and perform SVD on all weight matrices. These are post-hoc diagnostic procedures, not in-training metrics. The paper acknowledges this implicitly in Appendix C:

"our theoretical analysis in Appendix is inherently local, characterizing OPD dynamics only in a neighborhood of the base model and therefore not fully capturing the global non-convex behavior of large-scale post-training."

The consequence. If a practitioner wants to determine whether their specific OPD training run exhibits Functional Redundancy Avoidance or Early Low-Rank Lock-in, they would need to complete the training run first (to get final checkpoints for comparison) or implement the full diagnostic suite on intermediate checkpoints — which requires running SVD on all weight matrices and evaluating on held-out benchmarks at each checkpoint. This is computationally expensive and provides diagnostic information only after the fact. More critically, it means there is no in-training signal that EffOPD could use to verify that Property 2 holds for the current training run before attempting extrapolation. If a training run deviates from the typical OPD pattern — for example, due to a different teacher, base model, or dataset — EffOPD's extrapolation could be applied in a regime where directional stability is weaker than the paper assumes, potentially degrading performance. The lightweight validation set (50 examples) provides a coarse acceptance/rejection signal, but this only detects when a specific extrapolation step is harmful; it does not diagnose whether the underlying property (early directional lock-in) holds for the current run.

What evidence exists in the paper. The paper demonstrates the properties across 1.5B–32B Qwen-family models, three RL algorithms, and two task domains — but all in the specific OPD setup described in Appendix D.2 (Verl framework, kl_loss_coef=0, specific hyperparameters). There is no experiment testing whether the properties generalize to substantively different OPD configurations (e.g., different KL estimators, nonzero KL coefficients, different teacher-student gaps, or non-Qwen architectures). The paper's ablation on validation set difficulty (Figure 7b) shows that EffOPD's acceptance mechanism is robust to the composition of the 50-example set, but this does not address the larger question of whether Property 2 holds in the first place for arbitrary OPD runs.

Mitigation status. Not addressed. The paper does not propose a lightweight in-training diagnostic for detecting whether Property 2 holds, nor does it test EffOPD's sensitivity to violations of the property. The extrapolation acceptance mechanism in EffOPD is a partial mitigation — it rejects harmful steps — but it is reactive rather than predictive. Future work on predicting directional stability from early-training statistics (e.g., gradient covariance, loss curvature) could address this gap.

---

Single Model Family and Single Training Framework Limit Generality Claims

The assumption or constraint. All experiments use models from the Qwen2.5 and Qwen3 families, and all training (both RL and OPD) uses the Verl framework with a specific OPD implementation detailed in Appendix D.2. The authors state in Section 2.1:

"We conduct experiments across models ranging from 1.5B to 32B parameters, including pretrained, SFT-tuned, and Thinking-series models (Qwen et al., 2025; Zhang et al., 2025c; Yang et al., 2025)."

While this covers a range of scales and base model types within the Qwen ecosystem, it does not test across model families (e.g., LLaMA, DeepSeek-V3, Gemma, Mistral) that have different architectural details, pretraining data distributions, and in-context learning behaviors.

The consequence. The identified properties may be partially artifacts of the Qwen architecture or the specific OPD implementation. For example, the finding that MLP modules are more sensitive to reasoning-related updates than attention modules (Section 2.2, Figure 3b) could depend on the specific ratio of MLP to attention parameters in Qwen models, or on the fact that Qwen models use a particular gating mechanism in their MLP layers. A model family with different architectural choices — e.g., DeepSeek's mixture-of-experts layers, or models with different relative dimensions for attention vs. MLP — might show different module-wise sensitivity patterns, potentially altering the practical implications of Property 1. Similarly, the OPD training command in Appendix D.2 reveals that the paper uses kl_loss_coef=0 (KL loss computed but not added to the objective) and kl_loss_type=low_var_kl. This is an unusual configuration that follows Yang et al. (2026a) but diverges from how OPD is implemented in other frameworks — if the low-rank lock-in property depends on the specific form of the KL estimator, the EffOPD acceleration may not transfer to other OPD implementations.

What evidence exists in the paper. The paper does not test any non-Qwen models. The cross-scale experiments (Figures 8, 9; Table 1) and cross-RL-algorithm experiments (Figure 9) demonstrate robustness within the Qwen+Verl ecosystem, but there is no out-of-distribution test. The authors do not claim their findings are universal — they describe the experiments as validating "the generality of our findings" across scales and algorithms (Section 2.1), which is true within the tested scope but does not constitute evidence of model-family generality.

Mitigation status. Not addressed. The paper does not discuss this as a limitation or propose experiments with other model families. The consistent pattern across 1.5B–32B scales within Qwen is suggestive of generality, but confirmatory, not conclusive, evidence. A practitioner using a non-Qwen model or a different OPD implementation would need to replicate the diagnostic analyses (swiding-window, spectral metrics, norm-scaling) to verify that the properties hold in their setting before deploying EffOPD.

---

The Norm-Scaling Experiment Does Not Directly Validate EffOPD's Extrapolation Direction

The assumption or constraint. EffOPD extrapolates along the inter-checkpoint displacement $\Delta_n = W_{2^n} - W_{2^{n-1}}$ (Section 4.1). The justification for this direction comes from Property 2 (Early Low-Rank Lock-in): OPD's update directions are stable and aligned with the final solution early in training, so the recent displacement should approximate the direction of future progress. The paper's strongest evidence for Property 2 is the norm-scaling experiment (Section 3.3, Figure 5c), which shows that an early checkpoint's direction — when its module-wise norms are rescaled to match the final checkpoint — recovers approximately 80% of final performance.

The consequence. The norm-scaling experiment rescales the direction from the base model to the early checkpoint ($\Delta W_{\text{early}}$). EffOPD extrapolates along the direction between two adjacent exponential checkpoints ($\Delta_n = W_{2^n} - W_{2^{n-1}}$). These are generally different vectors. The fact that $\Delta W_{\text{early}}$ points toward the final solution does not guarantee that $\Delta_n$ does — $\Delta_n$ is a local segment of the optimization path, which may meander even if the overall displacement from initialization is well-aligned with the destination. The paper does not directly test whether $\Delta_n$ is well-aligned with the remaining displacement to the final checkpoint ($\Delta W_{\text{final}} - \Delta W_{2^n}$). If the alignment is imperfect — as the subspace fluctuation in Figure 5b suggests it can be (OPD alignment fluctuates between ~0.80 and ~0.95) — then extrapolating along $\Delta_n$ may move the parameters in a direction that diverges from the final solution over larger step sizes.

What evidence exists in the paper. The subspace alignment analysis (Figure 5b) shows that dominant subspaces at intermediate checkpoints are well-aligned with the final subspace, with cosine similarity reaching ~0.85–0.95. However, this measures alignment between subspaces (sets of directions), not between the specific displacement vector $\Delta_n$ and the optimal remaining path. The trajectory t-SNE visualizations (Figures 5a, 15–28) show OPD trajectories are smoother and more compact than RL's, which is consistent with local directional stability but does not quantify how well $\Delta_n$ approximates the true remaining displacement. The paper does not report cosine similarity between $\Delta_n$ and $\Delta W_{\text{final}} - \Delta W_{2^n}$ at each exponential checkpoint.

Mitigation status. Partially addressed by EffOPD's adaptive validation mechanism. The immediate termination on validation failure (Section 4.1) provides a hard stop against extrapolating in a poor direction — if $\Delta_n$ is misaligned, the first candidate ($k = 1$, step size $2\Delta_n$) will likely degrade validation performance, and EffOPD will degenerate to vanilla OPD at that checkpoint. This prevents catastrophic failure but does not guarantee that EffOPD's extrapolation direction is optimal; it only guarantees that accepted steps do not harm a small validation set. The 3× acceleration (Figure 6) provides empirical evidence that $\Delta_n$ is a sufficiently good direction in the tested settings, but the gap between "sufficiently good" and "provably aligned" could matter in settings where the directional stability is lower than in the tested Qwen OPD runs.

---

The EffOPD Validation Set of 50 Examples May Have High Variance That Affects Acceptance Reliability

The assumption or constraint. EffOPD relies on a lightweight validation set $\mathcal{D}_v$ of 50 randomly sampled training examples to decide whether to accept extrapolated candidates (Section 4.1). The paper states:

"EffOPD randomly samples 50 examples from the training set to form a lightweight validation set $\mathcal{D}_v$, which is far smaller than the number of sentences generated per step in vanilla OPD."

The ablation in Figure 7b tests validation sets of different difficulty levels (easy, medium, hard, mixed) and concludes they provide "consistent directional signals." However, the paper does not report the variance of the validation metric across different random samples of 50 examples of the same difficulty.

The consequence. Mathematical reasoning problems have high variance in difficulty and model performance — a single difficult problem can swing pass@1 accuracy by 2 percentage points on a 50-example set. If the validation metric has high variance, EffOPD may make noisy acceptance decisions: accepting candidates that scored well on the particular 50 examples by chance but would not generalize, or rejecting candidates that would improve overall performance but happened to perform poorly on the small set. This manifests as either (a) over-extrapolation (accepting candidates that degrade held-out performance) or (b) under-extrapolation (prematurely degenerating to vanilla OPD when a larger step would have helped). Since EffOPD terminates the search immediately on the first rejection (Section 4.1), a single unlucky evaluation of $\widetilde{W}_{n,1}$ can prevent any extrapolation at that checkpoint, even if $\widetilde{W}_{n,2}$ or $\widetilde{W}_{n,3}$ would have been beneficial.

What evidence exists in the paper. Figure 7b shows that validation sets of different difficulty compositions all support EffOPD's performance — the final accuracy is similar regardless of $\mathcal{D}_v$ difficulty. However, this only tests the bias of the validation signal (whether it systematically favors or disfavors extrapolation), not its variance (how much the acceptance decision fluctuates across different random samples of the same difficulty). The paper does not report: how many exponential checkpoints result in accepted extrapolation vs. degeneration to vanilla OPD; the variance of validation accuracy across multiple random 50-example samples; or the per-candidate acceptance rate ($k = 1$ through $k = 5$). Without these, it is unclear whether EffOPD's acceleration stems from a few large, reliable extrapolation steps early in training (which would be robust to validation noise) or from many small, marginal steps (which would be vulnerable).

Mitigation status. Partially addressed. The fact that EffOPD achieves better performance than vanilla OPD across all tested configurations (Figure 6) suggests that validation noise, if present, is not catastrophic — net acceleration is achieved. However, the paper does not quantify the reliability of individual acceptance decisions, which limits a practitioner's ability to tune $\mathcal{D}_v$ size for their specific use case. A more informative ablation would sweep $\mathcal{D}_v$ size (e.g., 10, 20, 50, 100, 200 examples) and measure both the variance in acceptance decisions and the final performance, to determine the minimum reliable validation set size. The paper suggests this implicitly in the title of Section 4.2 ("50 examples") but does not test alternatives.

---

All Results Are on Code and Math Benchmarks with Clean Correctness Signals — No Evidence for Tasks Without Verifiable Answers

The assumption or constraint. All experiments use code generation (Eurus-RL-Code, evaluated on Codeforces and Taco) and mathematical reasoning (DeepMath-103K, MATH-12K, evaluated on AIME24/25/26, MINERVA, GPQA, MATH500). These are domains where correctness can be deterministically verified — unit tests pass or fail, mathematical answers are right or wrong. The RL baselines use verifiable rewards (RLVR), meaning the reward signal $r(x,y)$ is a binary correctness indicator. The OPD teacher is the RL-trained model, which was itself trained on verifiable rewards.

The consequence. The paper's entire analysis framework depends on the existence of a well-defined, deterministic correctness signal for two purposes: (1) to train the RL teacher that serves as the OPD teacher, and (2) to evaluate reasoning accuracy on held-out benchmarks for the diagnostic analyses (norm-efficiency curves, sliding-window interventions, subspace partition experiments). For tasks without such signals — open-ended dialogue, creative writing, summarization, translation, multi-step agent planning where "correctness" is ambiguous or multi-dimensional — it is unclear whether:

• The OPD teacher would exhibit the same structural properties (the teacher would need to be trained via RLHF with a learned reward model rather than RLVR with a verifiable reward, potentially changing the residual $r_c$ structure)
• The diagnostic analyses could be performed (what is "accuracy" for summarization quality?)
• EffOPD's validation metric would remain reliable (what metric would replace pass@1 accuracy on 50 examples?)

The local geometric theory in Appendix F.5 suggests that the low-rank structure of OPD updates depends on the concentration of the teacher-student residual $r_c$. If $r_c$ is less concentrated for open-ended tasks (because the teacher provides signal on many diverse tokens rather than a few key reasoning steps), Property 2 may weaken, and EffOPD's extrapolation may be less reliable.

What evidence exists in the paper. None. The paper does not discuss open-ended generation tasks or propose how the framework would extend to them. The limitation is implicit in the choice of benchmarks — the paper is transparent about what it tests, but does not acknowledge the scope restriction. This is a common limitation in reasoning-focused LLM post-training papers, but it is particularly relevant here because the paper's claims about "foresight" and "update efficiency" are grounded in a notion of task performance (accuracy) that may not have a natural analog in other domains.

Mitigation status. Not addressed. The paper does not mention this limitation or propose extensions to non-verifiable tasks. Future work would need to develop verifier-free or learned-verifier analogs of the diagnostic framework, and test whether the identified properties hold when the teacher is trained via RLHF rather than RLVR.

---

No Direct Causal Test That the Identified Properties Cause OPD's Efficiency — Only Correlational Evidence

The assumption or constraint. The paper establishes that OPD exhibits Functional Redundancy Avoidance and Early Low-Rank Lock-in, and that OPD is more training-efficient than RL. It argues that the properties explain the efficiency: OPD's foresight "establishes stable and highly aligned update directions early in training, enabling rapid convergence" (Section 1). However, the experimental design is fundamentally correlational: it measures properties of OPD and RL updates and observes that the method with better properties also trains faster, but does not intervene to test whether changing the properties changes the efficiency.

The consequence. An alternative hypothesis is that Property 1 and Property 2 are consequences of OPD's faster convergence, not causes of it. Under the "denser supervision" account that the paper argues against, OPD converges faster because it has lower-variance gradients. A model that converges faster will naturally show: (a) smaller total update norm at the same performance level (because it didn't need to wander as much), (b) fewer redundant updates (because it spent less time exploring), and (c) higher alignment between intermediate and final subspaces (because the path is shorter and straighter). In this alternative account, the structural properties are epiphenomena of efficient optimization, not mechanisms of it — and EffOPD works not because it exploits an intrinsic property of OPD, but because extrapolation is a generic speedup technique that would work for any sufficiently smooth optimization trajectory.

What evidence exists in the paper. The norm-scaling experiment (Figure 5c) partially distinguishes these accounts by showing that the directional quality of early OPD checkpoints is high — not just that they are close to the final checkpoint in Euclidean distance, but that their direction (when amplified) produces near-final performance. This is evidence that directional quality is established early, which is more consistent with the "foresight" account than the "smooth convergence" account. However, the paper does not test the alternative hypothesis directly — for example, by training an RL model with a denser reward signal (e.g., process rewards or step-level verification) and measuring whether it develops Property 2, or by adding directional noise to OPD updates and measuring whether efficiency degrades. The ablation of EffOPD against fixed-extrapolation methods (AlphaOPD, ExOPD) in Figure 6 shows that adaptive extrapolation outperforms fixed extrapolation, which is consistent with the need for direction-aware step sizes, but does not isolate whether OPD-specific directional properties (as opposed to generic smoothness) are the enabling factor.

Mitigation status. Partially addressed. The paper's theoretical framework (Appendix F.5) provides a mechanistic account of why Property 1 and Property 2 arise from the structure of the OPD objective — this is a form of causal reasoning (the objective causes the properties, which cause the efficiency). However, the theory is local and linearized, and its predictions are not empirically tested. The paper acknowledges this in Appendix C: "Our theoretical analysis... is inherently local, characterizing OPD dynamics only in a neighborhood of the base model and therefore not fully capturing the global non-convex behavior." The strongest causal test proposed — manipulating the objective to break the properties and measuring efficiency loss — is not performed. This is a substantial gap: without such an intervention, the paper has identified a mechanistically suggestive correlation but not a causal mechanism, and EffOPD's effectiveness could be explained by other factors (e.g., generic momentum-like extrapolation in a smooth loss landscape) that would not require the full "foresight" framework.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper reshapes how the field thinks about post-training efficiency for large language models by shifting the central question from "how fast does the loss decrease?" to "what is the structure of the parameter updates, and where in the network do they concentrate?" This is a methodological reframing rather than a paradigm shift — the paper does not replace OPD with a new algorithm, nor does it overturn the optimization-level account that dense supervision helps. Instead, it introduces a new diagnostic layer between the loss function and the training curve, providing a language for describing and measuring how a training algorithm navigates parameter space, not just how well it optimizes the objective.

The practical magnitude of this reframing is substantial in one specific way: it converts OPD acceleration from a heuristic art into a principled engineering discipline. Prior to this work, methods for speeding up OPD (AlphaOPD, ExOPD) relied on fixed extrapolation strategies justified by empirical observation that "the trajectory looks linear." This paper provides a mechanistic explanation for why the trajectory is linear enough to extrapolate (the norm-scaling experiment showing 80% performance recovery at 10% progress, Figure 5c), and when that linearity breaks down (over-extrapolation degrading performance at β > 1.2, Figure 14). The 3× acceleration from EffOPD is not the headline contribution — it is the validation that the diagnostic framework leads to better engineering decisions. A practitioner who understands Properties 1 and 2 can design their own acceleration strategy, tailored to their specific model and dataset, rather than blindly applying a fixed recipe.

The paper also performs a valuable reconciliation of conflicting signals in the post-training literature. Prior work contained a tension: OPD consistently outperforms RL in training efficiency (Qwen3, MiMo-V2-Flash, DeepSeek-V4 all use it), yet theoretical analyses showed that token-level OPD is biased relative to the sequence-level objective it approximates (Fu et al., 2026). How can a biased estimator produce better results? The paper's answer — implicit in its framework but not stated as a theorem — is that the bias helps: by inheriting the low-rank structure of the teacher-student residual r_c, OPD's gradient concentrates on parameter directions that matter for the final task, while RL's unbiased (but high-variance) gradient spreads updates across many directions, including low-utility ones. This reframes the bias-variance tradeoff in distillation: the "bias" of per-token KL is not an imperfection to be corrected but a feature that induces beneficial parameter-space concentration. This is a conceptual shift that should influence how future work designs distillation objectives — not by trying to eliminate bias, but by understanding which biases produce parameter-efficient updates.

Two research directions become more attractive as a result of this work:

• Mechanistic post-training diagnostics are now on the table as a legitimate subfield. The paper demonstrates that SVD-based spectral metrics, sliding-window interventions, and norm-scaling experiments — tools borrowed from mechanistic interpretability — can answer practical questions about training efficiency. This creates a bridge between two communities that have largely operated independently: the interpretability community studying what models know and the training community studying how to make models better. Future post-training papers can and should include parameter-dynamic analyses alongside standard training curves and benchmark scores.

• Low-rank acceleration methods now have a theoretical foundation specific to distillation. Prior work (Cai et al., 2025; Chen et al., 2026) demonstrated that RL trajectories are low-rank and can be accelerated, but did not explain why they are low-rank or whether the same acceleration logic applies to OPD. This paper shows that OPD is differently low-rank than RL — with stronger concentration and earlier lock-in — and that these differences arise from the structure of the teacher signal rather than generic properties of gradient descent. This means acceleration methods should be training-paradigm-aware: what works for RL may not be optimal for OPD (as the EffOPD vs. AlphaOPD comparison in Figure 6 shows), and vice versa.

Conversely, one research direction becomes less urgent: the search for ever-more-complex OPD loss functions that correct the token-level bias. The paper's evidence suggests that the bias is functionally beneficial, meaning that attempting to debias OPD — for example, by incorporating sequence-level corrections or importance sampling weights — may inadvertently destroy the low-rank concentration that makes OPD efficient. The field's energy might be better spent on understanding and amplifying the beneficial structure of existing objectives, rather than "fixing" them.

Follow-Up Research This Work Enables

1. In-training diagnostics for directional stability — a lightweight predictor of whether EffOPD will work. The paper's properties are identified post-hoc from fully trained checkpoints, but a practitioner deploying EffOPD needs to know during training whether extrapolation is safe. A concrete follow-up: at each exponential checkpoint t = 2^n, compute the cosine similarity between the current inter-checkpoint displacement Δ_n and the displacement from the previous exponential checkpoint Δ_{n-1}. If this similarity is below a threshold (say, below 0.7), Property 2 (directional stability) may be weak, and EffOPD should skip extrapolation at that checkpoint. This would require no additional evaluation — just storing Δ_{n-1} and computing a dot product — making it essentially free. The experiment would sweep thresholds on the cosine similarity and measure both the acceptance rate and final performance of EffOPD on the MATH and code benchmarks, comparing against the current EffOPD that always attempts extrapolation. A positive result (threshold-based skipping matches or exceeds full EffOPD while reducing wasted candidate evaluations) would provide a practical, deployable diagnostic. A negative result (cosine similarity is not predictive of extrapolation success) would suggest that the relevant stability occurs at the subspace level (not the displacement-vector level) and that more sophisticated in-training metrics are needed.

2. Stress-testing Property 1 by training RL with an anti-redundancy regularizer. The paper claims that OPD's efficiency advantage arises because it suppresses updates in low-utility modules while RL wastes norm on them. A direct causal test: train an RL model (e.g., DAPO on Qwen3-8B-Base) with an added L2 penalty on parameter updates in the modules identified as low-utility by the sliding-window analysis — specifically, embedding layers, bottom transformer layers (0–5), and top transformer layers (30–35). The penalty coefficient would be tuned to bring the per-module update norms of RL in line with those of OPD. If Property 1 is causal, this regularized RL should match OPD's per-norm efficiency (Figure 2a) and potentially converge faster than unregularized RL. The experiment would measure: (a) the accuracy-vs-norm curve of regularized RL compared to OPD and vanilla RL, (b) the spectral concentration metrics (Table 1) to see whether suppressing low-utility module updates also induces Property 2-style low-rank structure, and (c) whether EffOPD-like extrapolation can now be applied to regularized RL. A null result — regularized RL does not improve per-norm efficiency — would suggest that the module allocation pattern is a symptom of OPD's objective structure (the driving term b_m in Appendix F.5) rather than an independently manipulable cause of efficiency, and that simply suppressing RL's low-utility updates is insufficient without also providing the concentrated directional signal that OPD's teacher residual provides.

3. Replicating the properties on a non-Qwen model family with a different OPD implementation. The paper's exclusive use of Qwen models and the Verl framework with kl_loss_coef=0 is its most significant generality limitation. A concrete replication: reproduce the full diagnostic suite (norm-efficiency analysis, sliding-window interventions, spectral metrics, norm-scaling experiment) using LLaMA-3-8B as the base model, trained with OPD using the standard HuggingFace TRL library (which implements a different OPD variant from Verl's) and a DeepSeek-R1-distilled model as the teacher. The key measurements are: (a) whether the per-norm efficiency advantage of OPD over RL holds (Figure 2), (b) whether the spectral concentration gap (Table 1) is similar in magnitude, and (c) whether a 10% training checkpoint recovers ≥70% of final performance after norm scaling (Figure 5c). If all three replicate, the paper's framework has strong cross-family generality. If spectral concentration holds but norm-scaling recovery is lower (say, 50% rather than 80%), early directional lock-in may be Qwen-specific, perhaps due to architectural details like gated MLPs or pretraining data distribution, and EffOPD's acceleration would be correspondingly weaker. If neither property replicates, OPD's efficiency advantage in non-Qwen settings may have a different mechanistic basis entirely, and the "foresight" framework would need to be qualified as architecture-dependent.

4. Extending EffOPD to dynamic extrapolation with per-module step sizes. EffOPD's current design uses a single scalar step size 2k applied uniformly to all modules. However, Property 1 shows that modules differ in their marginal utility for reasoning, and Property 2 shows that the principal subspace is concentrated in middle-layer MLPs. This suggests that different modules may tolerate different extrapolation magnitudes — middle-layer MLPs, where directions are more stable and functionally relevant, might accept larger steps than bottom-layer embeddings, where updates are small and less consequential. A concrete extension: modify EffOPD to maintain per-module extrapolation coefficients k_m for each module m, initialized uniformly at k = 1 and updated independently based on per-module validation metrics. The validation set evaluation would track which modules' extrapolation steps improve vs. degrade performance, using a module-level ablation approach: for each candidate model W̃_{n,k} that fails validation, test whether reverting specific module groups (e.g., bottom layers) to W_{2^n} rescues performance; if so, those modules' k_m are capped at their current value while other modules continue to extrapolate. The experiment would measure whether per-module EffOPD achieves >3× acceleration or reaches higher final performance than uniform EffOPD on the MATH benchmarks. A positive result would provide a direct synthesis of Properties 1 and 2 into a single acceleration method. A negative result (per-module extrapolation does not outperform uniform) would suggest that module-level properties are less important for extrapolation than the aggregate directional stability, simplifying future acceleration designs.

5. Testing the theoretical prediction that teacher-student residual concentration predicts OPD update structure. The local geometric theory (Appendix F.5) predicts that the low-rank structure of ΔW arises from the concentration of the teacher-student residual r_c = z^*(c) - z_0(c). This is a testable causal claim: if you vary the concentration of the teacher signal while holding the base model and dataset fixed, the effective rank of ΔW should change predictably. A concrete experiment: train three OPD runs on the same base model (Qwen3-8B-Base) and dataset (DeepMath-103K) but with three different teachers: (a) the standard RL-tuned teacher (concentrated signal on reasoning tokens), (b) a teacher trained via standard SFT on the same dataset (less concentrated signal, more uniform across tokens), and (c) a teacher that is the base model itself (self-distillation, where r_c = 0 everywhere). Measure: (i) the effective rank and Top-1% subspace norm ratio of ΔW from each run (Table 1 metrics), (ii) the sparsity/entropy of the teacher-student residual r_c averaged over the training data, and (iii) the training efficiency (accuracy vs. steps). The prediction is that more concentrated residuals produce lower effective rank, higher Top-1% ratio, and faster convergence. Self-distillation (teacher = base model) should produce near-zero updates (since r_c ≈ 0, b ≈ 0), providing a baseline. If the prediction holds, this directly validates the causal chain: teacher signal structure → b concentration → low-rank updates → efficient training. If not, the theory's explanatory scope is limited, and other factors (e.g., the Hessian A in the quadratic approximation) dominate the update structure.

6. OPD-aware pruning: using Property 1's module sensitivity map to guide structured compression. The sliding-window intervention analysis (Figures 3b, 10b) provides a detailed map of which layers and modules contribute most to reasoning performance after OPD training. This is essentially a post-training sensitivity map that could guide structured pruning: if bottom-layer MLPs and attention modules show low marginal utility for reasoning, they are candidates for pruning with minimal performance loss. A concrete experiment: take an OPD-trained Qwen3-8B model and prune the bottom 6 and top 6 transformer layers (which the sliding-window analysis shows have the lowest marginal contributions), then fine-tune the pruned model briefly with OPD to recover any lost performance. Measure: (a) the accuracy of the pruned model before and after fine-tuning on MATH500 and AIME benchmarks, (b) the inference speedup and memory reduction from removing 12 layers (~33% of a 36-layer model), and (c) compare against a baseline pruning strategy that removes layers uniformly (e.g., every third layer) rather than using the OPD-specific sensitivity map. If OPD-aware pruning outperforms uniform pruning, it demonstrates that Property 1's diagnostic value extends beyond training efficiency to deployment efficiency — the same "foresight" that enables faster training also tells you which parts of the model you can safely discard. This would establish the OPD diagnostic framework as a general-purpose tool for model optimization, not just training acceleration.

Practical Applications and Downstream Use Cases

On-device deployment of reasoning models with limited post-training budgets. A practitioner developing a small reasoning model (1.5B–8B parameters) for on-device deployment faces a hard compute constraint: they can afford only a limited number of post-training steps before shipping. The paper's findings directly inform this decision in two ways. First, OPD is clearly preferable to RL: Figure 2b shows that OPD reaches equivalent reasoning accuracy with ~4× smaller parameter updates, meaning fewer training steps and less GPU time. Second, EffOPD compounds this advantage: Figure 6 shows that on Qwen3-4B-Non-Thinking, EffOPD reaches strong reasoning performance by step 4, while vanilla OPD requires 30+ steps — a ~7× reduction in steps to convergence at that scale. For a team with a fixed training budget of, say, 8 GPU-hours, the choice is between ~30 steps of vanilla OPD (partial convergence) and ~4 steps of EffOPD (near-full convergence). The concrete benefit is not just speed — it is the ability to reach a higher performance ceiling within the same budget.

Cost-efficient data generation for self-improvement pipelines. Many LLM training pipelines now use self-generated data for iterative improvement (STaR, ReST^EM, rejection sampling fine-tuning). In these pipelines, a model generates solution candidates, a verifier selects the correct ones, and the model is fine-tuned on its own successful outputs — then the cycle repeats. The bottleneck in such pipelines is often the generation phase: for each training prompt, the model must produce multiple candidate solutions (typically 8–64) to find at least one correct one, and this generation dominates the total compute cost. The paper's Property 1 and Property 2 suggest that OPD can dramatically reduce this cost: if the model is OPD-trained (rather than RL-trained), its generation quality improves faster per unit of compute (Figure 2b), meaning fewer candidates need to be generated per prompt to achieve the same correct-answer yield. Moreover, the OPD teacher can serve double duty — providing training signal and acting as a verifier via its token-level probabilities, eliminating the need for a separate reward model or execution environment in some settings. A concrete pipeline: use OPD with EffOPD to rapidly produce a strong base model, then use that model to generate high-quality training data for a smaller student, with the OPD teacher's per-token probabilities guiding which generated solutions are likely correct. The 3× training acceleration from EffOPD compounds with the reduced generation cost per iteration, potentially enabling self-improvement loops that were previously compute-prohibitive.

Verifier-free post-training for tasks where reward models are expensive or unavailable. In domains where deterministic correctness verification is impossible (creative writing, dialogue, summarization, translation), RL from verifiable rewards (RLVR) is inapplicable, and RL from human feedback (RLHF) requires expensive reward model training or human annotation. OPD sidesteps this entirely: it requires only a teacher model whose outputs are treated as "correct," with no need for an external reward signal. The paper's findings strengthen the case for OPD in these settings by showing that OPD's efficiency advantage is not dependent on the specific structure of verifiable rewards — it arises from the teacher-student residual r_c (Appendix F.5), which exists for any teacher, regardless of whether the teacher was trained with verifiable rewards or human preferences. A concrete application: a company with a strong proprietary model (the "teacher") wants to distill its dialogue capabilities into a smaller customer-facing model. OPD with EffOPD provides a recipe: use the large model to score the small model's on-policy dialogue responses at the token level, apply the EffOPD acceleration to complete the distillation in ~10 training steps rather than ~30–40, and deploy. The 50-example validation set for EffOPD can be drawn from the company's internal dialogue quality benchmarks, maintaining the "verifier-free" property. The paper's cross-scale results (Figure 8) suggest this works across large teacher-student gaps (tested up to teacher being ~14× larger), which is the typical distillation scenario.