Learning from Language Feedback via Variational Policy Distillation

🎯 Pitch

Even when a model's reasoning is almost correct, sparse outcome-based RL wastes the corrective signal in error traces—VPD shows that a trainable self-teacher, refined on the outcomes of its own rollouts, can extract a dense, improving target distribution from textual feedback like compiler errors. Across scientific reasoning and code generation, this co-evolution of teacher and student consistently outperforms both standard RL and passive self-distillation, though pure sparse RL still wins when the model can't yet parse diagnostic critiques.

1. Executive Summary

This paper proposes Variational Policy Distillation (VPD), a framework that formalizes on-policy learning from language feedback as a variational Expectation-Maximization problem, enabling a feedback-conditioned teacher model and an unconditioned student policy to co-evolve rather than treating the teacher as a fixed heuristic. VPD is evaluated across scientific reasoning (SciKnowEval), code generation (LiveCodeBench), and mathematical reasoning (DAPO-Math) tasks using Qwen3-1.7B, Qwen3-8B, and OLMo3-7B-Instruct models. The framework's core mechanisms are an E-step that actively refines the teacher via unpaired preference optimization to extract sharper diagnostic signals from textual feedback (e.g., compiler errors, contrastive sibling rollouts, or self-generated critiques), and an M-step that distills this refined teacher's token-level distribution into the unconditioned student through KL-divergence minimization on on-policy rollouts. VPD consistently outperforms both standard RLVR (GRPO) and passive self-distillation baselines (SDPO), achieving a state-of-the-art 49.62% pass rate on LiveCodeBench v6 and 74.34% average accuracy on SciKnowEval with Qwen3-1.7B, while a dynamic trust-region update—anchoring the teacher's reference prior to the current student policy—stabilizes training and prevents the late-stage degradation observed in single-phase hybrids. The paper establishes that language-feedback distillation provides substantial gains over sparse outcome-based RL on scientific reasoning and code generation tasks, but that pure sparse RL ultimately remains dominant on rigid mathematical reasoning and base-model cold-start regimes where the model lacks the rudimentary instruction-following competence to parse diagnostic critiques.

2. Context and Motivation

The Core Problem: Sparse Outcome Signals Create a Bottleneck for Reinforcement Learning on Reasoning Tasks

The central challenge this paper addresses is the credit assignment bottleneck in reinforcement learning from verifiable rewards (RLVR). When an LLM generates a multi-step reasoning trajectory—say, a mathematical proof spanning 800 tokens or a code solution with dozens of logical steps—the environment typically provides only a single, binary signal at the very end: correct or incorrect. This signal, while unbiased and objective, is devastatingly sparse. A token-level arithmetic mistake on step 3 of 15 receives exactly the same negative feedback (zero reward) as a completely nonsensical hallucination. The model cannot distinguish between a trajectory that was almost correct but stumbled on a minor algebraic slip, and one that was fundamentally misguided from the first token.

This sparsity creates two compounding problems:

Problem 1: Sample inefficiency in the regime where learning is possible. Even when the base model has some non-trivial probability of generating a correct solution (say, 10–20% pass@1 on a scientific reasoning problem), the policy gradient must wait for these rare successes to propagate any positive signal. The model may generate hundreds of partially correct trajectories—ones that follow the right reasoning structure but make a terminal execution error—and learn absolutely nothing from them. Every one of those near-miss trajectories contains latent information about where the model went wrong, but scalar outcome rewards cannot exploit it.

Problem 2: Exploration collapse when success is rare. On genuinely hard problems where the model's initial success rate approaches zero (e.g., competition-level mathematics or base-model cold-start scenarios), the situation becomes catastrophic. If no trajectory in a batch receives a positive reward, all advantage estimates collapse to zero (or near-zero random noise), and learning halts entirely. The model receives no gradient signal whatsoever, regardless of how many rollouts are sampled. This is not merely inefficiency—it is a fundamental barrier: the algorithm cannot even begin to improve because it never observes what success looks like. The authors cite this explicitly:

"if the model fails to sample any correct answers for a given prompt (i.e., r(x, yi) = 0 for all i), the advantage scores collapse, halting the learning process and establishing a severe exploration bottleneck"

This bottleneck is particularly pernicious because it creates a cold-start chicken-and-egg problem: the model needs to see successful trajectories to learn what success looks like, but it cannot generate successful trajectories without already possessing the capability.

Why This Problem Matters: Practical and Theoretical Significance

The practical stakes are immediate and substantial. The recent leap in LLM reasoning capabilities—from DeepSeek-R1 to Qwen3—has been largely powered by RLVR. These post-training pipelines consume enormous computational resources, with training runs requiring thousands of GPU-hours spent on on-policy rollouts. If the vast majority of those rollouts contribute zero learning signal (because they are incorrect and indistinguishable from each other), the effective sample efficiency is abysmal. Every incremental improvement in credit assignment—in extracting useful gradients from failed trajectories—directly translates to reduced training compute, faster iteration cycles, and the ability to tackle harder reasoning problems that currently lie beyond practical reach.

There is also a deeper theoretical significance. Language feedback—compiler error traces, LLM-generated critiques, user corrections—is ubiquitous in real-world agentic settings. When an autonomous coding agent fails a unit test, the Python interpreter doesn't just return False; it returns a stack trace pinpointing the exact line and error type (TypeError: unsupported operand type(s) for +: 'int' and 'str'). When a student submits an incorrect proof, a human teacher doesn't just say "wrong"; they explain which step contains the logical gap. This textual feedback contains dense, localized diagnostic information that should, in principle, provide exactly the credit assignment signal that scalar rewards lack. The question of how to effectively convert this language into useful gradients is therefore one of the central open problems in making LLMs genuinely self-improving.

The authors position this motivation clearly in the opening paragraph of Section 1:

"a minor arithmetic mistake in a complex derivation receives the same zero-reward as a completely nonsensical hallucination. Consequently, standard outcome-based RL is notoriously sample inefficient"

The dual problems—wasting latent information in near-miss trajectories, and completely stalling on hard problems—motivate the entire paper's search for a better way to extract learning signals from failure.

Prior Approaches: Three Categories and Their Shortcomings

The paper contextualizes itself against three categories of prior work, each of which attempts to address reward sparsity but suffers from specific, identified limitations.

Category 1: Off-Policy Supervised Fine-Tuning on Expert or Feedback-Revised Trajectories

The most straightforward approach to leveraging language feedback is to collect a dataset of expert solutions or feedback-revised corrections, then simply perform supervised fine-tuning (SFT) on these high-quality trajectories. Methods in this category include Chain of Hindsight (Liu et al., 2023), which conditions a model on paired (instruction, feedback) data to internalize critiques; Imitation Learning from Language Feedback (ILF; Scheurer et al., 2023), which fine-tunes on feedback-revised outputs; and Feedback-Conditional Policy (FCP; Luo et al., 2025), which directly conditions on feedback during deployment.

The advantage is simplicity: you just collect good data and apply standard SFT. The problem—and it is a fundamental one—is distribution mismatch. The feedback-revised trajectories are produced by an external teacher (a stronger LLM, a human expert, or a separate revision model) with different capabilities, different reasoning styles, and different error patterns than the student. When the student is trained on these trajectories, it learns to mimic the surface-level output patterns without developing genuine comprehension of why the corrections were made. The paper characterizes this as:

"the student model often lacks the internal capacity to faithfully reproduce the external teacher's reasoning, leading to copycat behavior without genuine comprehension"

This is particularly damaging because the student encounters scenarios at inference time that differ from the training distribution—problems where the external teacher's reasoning patterns don't directly apply. Without internalizing the principle behind the correction, the student produces brittle, memorized responses. The train-inference distribution shift (Agarwal et al., 2024; Shenfeld et al., 2026) means the policy is trained on data from a different distribution than it will encounter when operating autonomously.

Category 2: On-Policy Self-Distillation with a Fixed, Passive Teacher

To solve the distribution mismatch problem, a newer family of methods pivots to on-policy self-distillation. The key insight: instead of learning from an external teacher's trajectories, condition the model itself on language feedback to act as an on-policy "self-teacher," then distill the feedback-informed predictions back into the unconditioned policy. Because the student generates its own rollouts, there is no distribution shift—the teacher sees exactly the same distribution of errors that the student actually produces.

The most prominent exemplar is Self-Distillation Policy Optimization (SDPO) (Hübotter et al., 2026), which the paper treats as its primary distillation baseline. SDPO's objective is deceptively simple and elegant. For each student-generated trajectory $y \sim \pi_\theta(\cdot \mid x)$ , the model is also invoked in a conditioned mode where diagnostic feedback $C$ (a compiler error, a correct sibling solution, or a critique) is appended to the prompt: $q(\cdot \mid x, C)$ . The model's next-token predictions under this feedback-conditioned context serve as a "correction target"—what the model would have predicted if it had known about the error. The unconditioned student is then trained to match this conditional teacher via token-level KL divergence:

$\mathcal{L}_{\text{SDPO}}(\theta) = \mathbb{E}_{x \sim \mathcal{D},\, y \sim \pi_\theta(\cdot \mid x)}\left[\sum_{t=1}^T D_{\text{KL}}\big(\pi_\theta(\cdot \mid x, y_{<t}) \,\|\, \text{sg}[q(\cdot \mid x, C, y_{<t})]\big)\right]$

where $\text{sg}[\cdot]$ is a stop-gradient operator that prevents gradients from flowing through the teacher. Other methods in this category include OPSD (Zhao et al., 2026), which conditions a teacher on ground-truth answers, and OPCD (Ye et al., 2026), which distills system-prompt behaviors.

This approach elegantly avoids distribution mismatch, and it provides dense, token-level gradients (the feedback tells the teacher which token was wrong and what to predict instead). However, the paper identifies a critical and non-obvious flaw: the stop-gradient operator in the SDPO objective reveals that the teacher is never explicitly trained. It operates purely zero-shot—the model's ability to interpret and benefit from feedback is whatever its pretrained or instruction-tuned parameters happen to provide. The paper states this limitation unequivocally:

"the teacher $\pi_\theta(\cdot \mid x, C)$ operates purely zero-shot, relying on its pre-existing capacity to interpret the textual feedback $C$ . Since the teacher is not trained to refine its diagnostic interpretation, this creates a ceiling effect that restricts the gradients the teacher can ultimately provide to an improving student."

This ceiling effect manifests in two concrete ways:

Critique quality is bottlenecked by zero-shot ability. If the feedback $C$ is noisy (e.g., a vague LLM critique or a misleading compiler error in complex code), the zero-shot teacher may misinterpret it and provide counterproductive gradient signals. If the model's base capability for mapping natural language criticism to token-level corrections is weak (as in smaller models or base models), the teacher's target distribution is actively harmful.
The teacher's discriminative power plateaus and then degrades. As the student policy improves through distillation, the easy errors get fixed. The remaining errors become increasingly subtle and nuanced—distinguishing between a trajectory that is fundamentally correct but has a formatting issue vs. one that has a genuine logical flaw requires increasingly sophisticated diagnostic reasoning. A frozen teacher cannot make these distinctions. Figure 1 of the paper shows this empirically: the reward margin between correct and incorrect trajectories under SDPO's teacher rapidly diminishes as training progresses, indicating the teacher loses the ability to discriminate between quality levels just when the student needs it most.

A deeper way to understand this flaw: SDPO's formulation makes the teacher a pure function of the student's current parameters $\theta$ and the feedback $C$ . As $\theta$ updates, the teacher's outputs change (because the model weights change), but these changes are side effects of the student optimization—they are not targeted to improve the teacher's diagnostic capability. The teacher drifts wherever the student's gradients push it, not toward being a better interpreter of feedback. This is a subtle but crucial distinction.

Category 3: Single-Phase Hybrid Methods That Fuse RL and Distillation

A third approach, located conceptually between pure RL and pure distillation, attempts to simultaneously fuse the sparse scalar reward from the environment with the dense token-level signal from the self-teacher into a single gradient update. The paper constructs three representative baselines of this type (detailed in Appendix B.3):

Joint Loss: Simply add the GRPO surrogate loss and the SDPO KL loss with hyperparameter weights $\omega_{rl}$ and $\omega_{opd}$ .
Advantage Reshaping: Translate the teacher's token-level log-ratio into a per-token advantage $A_t^{\text{SDPO}}$ and linearly combine it with the sequence-level GRPO advantage $A^{\text{GRPO}}$ , then use this fused advantage in a standard PPO-style clipped objective.
Advantage Reweighting: Use the teacher's token-level signal to reweight (amplify or dampen) the sequence-level GRPO advantage, applying adaptive credit assignment without introducing an independent token-level advantage term.

The appeal is clear: why decouple into separate phases when you can have both signals in one update? The problem, as the paper's empirical results demonstrate, is catastrophic scale mismatch and instability. The KL divergence term in the distillation loss can be orders of magnitude larger and more variable than the bounded, normalized GRPO advantage. When these signals are combined linearly, the optimization becomes highly sensitive to the weighting hyperparameters ( $\omega_{rl}$ , $\omega_{opd}$ ). Set them wrong, and the RL signal drowns out the distillation (leading to no benefit from language feedback) or vice versa (leading the model to overfit to the teacher's potentially noisy target while ignoring the objective environment signal).

The paper's experimental results (Table 2, SciKnowEval) show this instability concretely: Advantage Reshaping with Qwen3-1.7B drops the Biology score from 61.50% (pure SDPO) to 54.62%, and from 75.65% (pure GRPO) to 54.62%. The single-phase hybrids are worse than either pure method alone in many cases. The authors hypothesize:

"simultaneously updating a policy using unbounded log-ratios and high-variance RL advantages causes catastrophic scale mismatches"

This instability is not a hyperparameter-tuning problem—it is structural. The two signals operate on fundamentally different scales and variances, and there is no single linear combination that works stably across training steps.

Category 4: Pure Sparse RL with Policy Gradient Methods

The direct alternative to all distillation-based approaches is simply to accept the sparsity and optimize directly against the environment reward using policy gradient methods like GRPO (Shao et al., 2024). GRPO normalizes advantages within a group of rollouts for the same prompt, providing stable gradient estimates without a learned value function. This approach has been enormously successful in practice—it is the backbone of DeepSeek-R1, DAPO, and other recent reasoning breakthroughs.

The paper does not argue that GRPO is ineffective. In fact, its results (Section 4.2, mathematical reasoning and cold-start experiments) confirm that GRPO remains the dominant paradigm for the hardest reasoning domains. The argument is rather that GRPO's sample efficiency is poor on problems where language feedback is available and informative, and that a well-designed distillation method can extract substantially more value from each training trajectory by converting failure into a dense learning signal. The authors treat GRPO as a strong baseline to beat in domains where language feedback is rich (code with compiler errors, LLM-generated scientific reasoning critiques), while also using GRPO's dominance on mathematical reasoning as a boundary condition that defines where distillation approaches break down.

How This Paper Positions Itself: Co-Evolution as the Missing Ingredient

VPD's intellectual positioning is best understood by examining what it changes relative to SDPO (the closest prior work) and why that change addresses the identified failure modes.

What VPD inherits from SDPO. VPD preserves the core on-policy self-distillation structure: the student generates rollouts, the teacher conditions on feedback, and the student is trained via KL divergence against the teacher's token-level predictions. It also preserves the shared-weight architecture (a single network serves as both student and teacher, distinguished only by whether feedback $C$ is included in the prompt), inheriting SDPO's memory efficiency and avoidance of external teacher models.

What VPD changes: active teacher optimization. The fundamental difference is that VPD introduces an explicit E-step that actively trains the teacher to better interpret feedback before the M-step distillation occurs. Instead of the teacher being whatever the current model weights happen to produce when conditioned on $C$ , VPD optimizes the teacher via a preference-based objective that rewards the teacher for assigning high likelihood to successful trajectories and low likelihood to failed ones, given the feedback. This transforms the teacher from a passive side-effect of student training into an actively optimized component of the learning system.

Why this matters theoretically. The variational EM framing (Section 3.1) provides a principled justification for this co-evolution. The intractable optimal policy $\pi^*(y \mid x) \propto \pi_{\text{ref}}(y \mid x) \exp(r(x,y)/\beta)$ cannot be directly evaluated. The teacher $q_\phi(y \mid x, C)$ serves as a tractable approximate posterior—a learned surrogate that uses the feedback $C$ to approximate the high-reward modes of $\pi^*$ . The E-step improves this approximation by minimizing $D_{\text{KL}}(q_\phi \| \pi^*)$ , and the M-step projects it back into the student's unconditioned space. Crucially, this means the teacher is explicitly optimized for a different objective (approximating $\pi^*$ ) than the student (matching the teacher), creating a productive tension that drives both models forward.

The dynamic trust-region as a stability mechanism. A subtle but critical design choice is anchoring the teacher's reference prior to the current student policy $\pi_\theta$ rather than a fixed initial model (Section 3.2, Equation 7). In VPD's E-step preference optimization, the implicit reward is:

$\tilde{r}_\phi(x, y, C) = \beta \log \frac{q_\phi(y \mid x, C)}{\pi_\theta(y \mid x)}$

This means the teacher is encouraged to assign higher likelihood than the student currently does to successful trajectories—the student's current behavior becomes the baseline. The resulting KL penalty in the E-step objective ( $-\beta D_{\text{KL}}(q_\phi \| \pi_\theta)$ ) acts as a sliding trust region: the teacher can improve, but only by taking steps the student is positioned to follow. This prevents the teacher from proposing a target distribution that is optimal according to the reward signal but unreachable for the student (because it requires capabilities the student hasn't yet developed), which would lead to vanishing or misleading gradients in the M-step.

Contrast this with using a fixed reference model $\pi_{\text{ref}}$ (the standard approach in PPO and DPO): as the student improves through training, its distribution drifts away from $\pi_{\text{ref}}$ , and the teacher's target (anchored to the stale $\pi_{\text{ref}}$ ) becomes increasingly disconnected from the student's current exploration space. The M-step KL divergence would then penalize the student for deviating from a target it literally cannot reach because the target was computed relative to a different baseline. The ablation in Table 5 confirms the practical severity of this issue: using a fixed prior drops aggregate SciKnowEval performance from 74.34% to 67.84% on Qwen3-1.7B and introduces severe training instability (Figure 5).

The unpaired preference optimization solution. A practical challenge VPD must solve is that the teacher cannot be trained with standard paired preference methods (like DPO). In DPO, the loss compares a preferred and a dispreferred response under the same prompt context. But in VPD, the teacher's context is the feedback $C$ , which is unique to each student trajectory—there is no shared feedback context between any two trajectories. The paper overcomes this by adopting Binary Classifier Optimization (BCO; Jung et al., 2025), an unpaired preference method that constructs a valid upper bound on the DPO loss using only unpaired positive and negative samples (see Equation 8-9). This is an elegant solution to a structural constraint that would otherwise prevent teacher optimization entirely.

Where VPD fits in the broader solution space. VPD does not claim to be universally better than all alternatives. The paper is unusually forthright about its limitations. In Section 4.2, it explicitly characterizes two regimes where VPD (and self-distillation in general) underperforms pure GRPO:

Base-model cold starts: When the model lacks rudimentary instruction-following capability, it cannot effectively parse diagnostic feedback $C$ even with E-step optimization. The teacher's target distribution becomes fundamentally corrupted because the model cannot map natural language hints to token-level corrections.
Rigid mathematical reasoning: Mathematics is unforgiving in a way that code generation and scientific reasoning are not. A single misplaced sign in a 500-token derivation invalidates the entire chain, and the language-based feedback ("check your algebra on step 7") provides imprecise guidance for what exact token change would produce the correct derivation. The noise in this language signal, even when refined by the E-step, is high enough that it overly constrains the student's exploration, preventing it from discovering the rigorous, exact logical paths that pure GRPO's exploration-driven approach discovers.

These boundary conditions are crucial to the paper's contribution. They establish that language-feedback distillation is not a universal replacement for sparse RL, but a powerful complement when feedback is rich and the model has sufficient baseline competence to exploit it. The paper thus offers not just a method but a principled understanding of when that method should be preferred.

3. Technical Approach

3.1 Reader Orientation

This paper presents a training algorithm—Variational Policy Distillation (VPD)—that teaches a language model to solve reasoning problems by learning from both binary success/failure signals and rich textual feedback (like compiler errors or written critiques) simultaneously, using a single model that plays two roles (student and teacher) in an alternating optimization loop. The problem it solves is that standard reinforcement learning wastes information from nearly-correct attempts because it only sees binary reward signals, while existing self-distillation methods treat the model's ability to interpret textual feedback as a fixed, passive capability that cannot improve alongside the model; VPD's solution is to frame learning from language feedback as a variational Expectation-Maximization problem where both the feedback-interpreting "teacher" behavior and the final "student" behavior co-evolve, with the teacher actively trained to extract sharper diagnostic signals from critique text before those signals are distilled into the student.

3.2 Big-Picture Architecture

The VPD system consists of a single language model used in two modes, an environment, and a training loop with four sequential phases per iteration:

Student policy ( $\pi_\theta$ ) — the unconditioned language model that takes a problem prompt $x$ as input and autoregressively generates a reasoning trajectory $y = (y_1, \ldots, y_T)$ . This is the policy deployed at inference time. It receives no feedback during generation.
Teacher policy ( $q_\phi$ ) — the same language model invoked in a conditioned mode: it receives both the problem $x$ and diagnostic feedback $C$ (compiler error, sibling correct solution, or self-generated critique) as input, and produces next-token probability distributions informed by that feedback. The teacher's token-level predictions serve as distillation targets for the student.
Environment / Verifier ( $\mathcal{E}$ ) — an external system that executes or grades the student's generated trajectory $y$ and returns:
- A binary outcome reward $r(x, y) \in \{0, 1\}$ (correct or incorrect).
- Diagnostic textual feedback $C$ (error messages, unit test failures, or self-critique). The feedback is unique to each trajectory because it describes the specific errors in that particular generation attempt.
Training Loop (four phases per iteration) — each iteration comprises:
- Phase 1 (Rollout): Sample a batch of prompts, generate student trajectories on-policy.
- Phase 2 (Critique): Evaluate each trajectory through the environment to obtain rewards and feedback.
- Phase 3 (E-step): Train the teacher to better distinguish successful from failed trajectories given the feedback, using an unpaired preference optimization objective.
- Phase 4 (M-step): Distill the refined teacher's token-level predictions into the student via on-policy KL divergence minimization.

The critical architectural insight is that the student and teacher share the exact same neural network weights ( $\theta = \phi$ ); the only difference is whether the prompt includes the diagnostic feedback $C$ . This eliminates memory overhead while enabling co-evolution through alternating gradient updates.

3.3 Roadmap for the Deep Dive

First, the variational objective (Section 3.1): I will explain how VPD reframes the intractable RLVR objective as a variational inference problem, introducing the teacher as an approximate posterior and establishing the theoretical justification for alternating E- and M-steps. This provides the why behind the algorithm design.
Second, the E-step / teacher refinement (Section 3.2): I will walk through how the teacher is actively trained—why standard paired preference optimization fails structurally, how VPD adapts an unpaired alternative (BCO), and the critical role of the dynamic trust-region reference prior. This is the novel mechanism that distinguishes VPD from passive self-distillation.
Third, the M-step / student distillation (Section 3.3): I will detail how the refined teacher's knowledge is projected into the unconditioned student, explaining the token-level KL objective, why the stop-gradient operator is necessary, and how the dynamic trust region from the E-step makes this distillation stable.
Fourth, the algorithm summary and implementation details (Section 3.4): I will assemble the complete training procedure, explaining the shared-weight architecture, asymmetric update frequencies, and why importance sampling corrections can be omitted despite nominally off-policy data.

3.4 Detailed, Sentence-Based Technical Breakdown

This is a method paper whose core idea is that on-policy self-distillation from language feedback can be formalized as a variational Expectation-Maximization problem, where the feedback-conditioned teacher and the unconditioned student policies co-evolve through alternating optimization steps—an E-step that actively trains the teacher to interpret feedback better (absent from all prior self-distillation methods) and an M-step that distills this improved teacher into the student—all within a single shared-weight neural network.

The Variational Formulation: Why EM?

The standard KL-regularized RLVR objective seeks to maximize expected reward while staying close to a reference policy:

$J_{\text{RLVR}}(\theta) = \mathbb{E}_{x \sim \mathcal{D}, \, y \sim \pi_\theta(\cdot \mid x)} [r(x, y)] - \beta \, D_{\text{KL}}(\pi_\theta(\cdot \mid x) \,\|\, \pi_{\text{ref}}(\cdot \mid x))$

where $\beta > 0$ controls the strength of the KL penalty, $\pi_{\text{ref}}$ is typically the supervised fine-tuned base model, $x$ is the prompt sampled from dataset $\mathcal{D}$ , and $y$ is the sampled response.

What it computes: a Lagrangian that trades off two competing objectives—maximize the expected scalar reward achieved by the policy (first term) while limiting how far the policy can drift from its initialization (second term). The KL term acts as a regularizer preventing catastrophic forgetting of pretrained capabilities.

Why this form: the KL penalty is the standard approach in RLHF and RLVR because it prevents reward hacking—without it, the policy could overfit to the verifier signal by producing nonsensical outputs that happen to fool the grading function. The coefficient $\beta$ lets practitioners control this tradeoff.

The authors then invoke a standard result from the preference optimization literature (Peng et al., 2019; Go et al., 2023; Rafailov et al., 2023): the policy that optimally satisfies this KL-regularized objective has a closed-form analytical expression known as the reward-tilted distribution:

$\pi^*(y \mid x) = \frac{1}{Z(x)} \pi_{\text{ref}}(y \mid x) \exp\left(\frac{1}{\beta} r(x, y)\right)$

where $Z(x) = \sum_y \pi_{\text{ref}}(y \mid x) \exp(r(x, y)/\beta)$ is the partition function—a normalization constant that ensures $\pi^*$ sums to 1 over all possible trajectories $y$ .

What it computes: the optimal target distribution reweights the reference policy by an exponential function of the reward. Trajectories with high reward get their probability multiplied by a large positive factor ( $\exp(r/\beta)$ ), while low-reward trajectories get suppressed. The partition function $Z(x)$ ensures proper normalization.

Why this form matters: it establishes that minimizing the reverse KL divergence $D_{\text{KL}}(\pi_\theta \,\|\, \pi^*)$ is mathematically equivalent to maximizing the original RLVR objective $J_{\text{RLVR}}(\theta)$ (derived in Appendix A.2). In principle, we could optimize by directly minimizing this KL—if we could evaluate $\pi^*$ . But we cannot, because $Z(x)$ requires summing over all possible trajectories, which is computationally intractable for any non-trivial sequence length.

The core insight of the variational approach: since $\pi^*$ cannot be evaluated directly, introduce a tractable surrogate $q_\phi(y \mid x, C)$ that approximates it. This surrogate—the teacher—is conditioned on the diagnostic feedback $C$ , which gives it privileged information the unconditioned student lacks. The feedback tells the teacher where and how a trajectory failed, allowing it to more accurately identify which trajectories are likely to receive high reward. Mathematically, the teacher provides a variational approximation to the intractable posterior $\pi^*$ .

The EM decomposition follows naturally from this substitution. The log-partition function (which represents the maximum achievable performance) can be lower-bounded using the teacher:

$\log Z(x) \geq \frac{1}{\beta} \mathbb{E}_{y \sim q_\phi}[r(x, y)] - D_{\text{KL}}(q_\phi \,\|\, \pi_{\text{ref}}) \triangleq \mathcal{F}(q_\phi)$

where $\mathcal{F}(q_\phi)$ is the Evidence Lower Bound (ELBO).

What it computes: a tractable lower bound on the theoretical performance ceiling. The first term is the expected reward under the teacher's distribution; the second penalizes deviation from the reference policy. Maximizing this bound with respect to the teacher parameters $\phi$ drives the teacher toward better approximate of $\pi^*$ .

Why this decomposition: it cleanly separates the learning problem into two subproblems with distinct objectives:

E-step (Teacher Refinement): Optimize $q_\phi$ to maximize the ELBO $\mathcal{F}(q_\phi)$ , which is equivalent to minimizing $D_{\text{KL}}(q_\phi \,\|\, \pi^*)$ . This teaches the teacher to translate textual feedback into high-reward token distributions—it learns to read diagnostic critique and predict what a correct response would have looked like.
M-step (Student Distillation): Hold the refined teacher fixed and minimize $D_{\text{KL}}(\pi_\theta \,\|\, q_\phi)$ . This projects the teacher's feedback-informed knowledge back into the unconditioned student, enabling the student to internalize the reasoning corrections without needing the privileged feedback $C$ at deployment.

The alternating optimization ensures that neither component stagnates: as the student improves, it encounters different, more subtle errors, and the E-step retrains the teacher to provide sharper diagnostic signals for these new error types.

The E-step's objective is straightforward to state but nontrivial to implement efficiently. The goal is to optimize the teacher parameters $\phi$ to minimize $D_{\text{KL}}(q_\phi \,\|\, \pi^*)$ , which—as derived in Appendix A.3—expands to:

$D_{\text{KL}}(q_\phi \,\|\, \pi^*) = \log Z(x) - \left(\frac{1}{\beta} \mathbb{E}_{y \sim q_\phi}[r(x, y)] - D_{\text{KL}}(q_\phi \,\|\, \pi_{\text{ref}})\right)$

Since $\log Z(x)$ is a constant with respect to $\phi$ , minimizing this KL is equivalent to maximizing the term in parentheses, which yields the E-step objective:

$J_{\text{E-Step}}(\phi) = \mathbb{E}_{y \sim q_\phi}[r(x, y)] - \beta \, D_{\text{KL}}(q_\phi \,\|\, \pi_{\text{ref}})$

What it computes: a standard KL-regularized RL objective, but for the teacher rather than the student—maximize expected reward under the teacher's feedback-conditioned distribution while staying close to a reference policy.

Why this form: it is structurally identical to Equation 1, the original RLVR objective. This is theoretically satisfying because it means both the student (in standard RLVR) and the teacher (in VPD's E-step) optimize the same type of objective, just with different input contexts.

The critical efficiency problem: optimizing this directly via on-policy RL would require the teacher to independently generate and explore trajectories from its own distribution $q_\phi(\cdot \mid x, C)$ , which reintroduces the exact sparse reward bottleneck VPD aims to solve. The teacher would have to sample trajectories, execute them in the environment, and wait for sparse rewards—exactly the inefficiency the framework exists to overcome.

VPD's solution—off-policy preference optimization: instead of generating new teacher rollouts, VPD reuses the trajectories $y$ already generated by the student during the rollout phase. These trajectories are annotated with outcomes $r(x, y)$ and feedback $C$ . The teacher is then trained via a preference-based objective that does not require reward-maximizing sampling.

The Implicit Reward Parameterization

The derivation proceeds by recognizing that since the E-step objective mirrors the standard RLVR objective, the optimal teacher distribution would take the same reward-tilted form:

$q^*(y \mid x, C) = \frac{1}{Z(x)} \pi_{\text{ref}}(y \mid x) \exp\left(\frac{1}{\beta} r(x, y)\right)$

By algebraically rearranging this expression and substituting the parameterized teacher $q_\phi$ , we obtain the implicit reward induced by the teacher's current parameters:

$r_\phi(x, y) = \beta \log \frac{q_\phi(y \mid x, C)}{\pi_{\text{ref}}(y \mid x)} + \beta \log Z(x)$

What it computes: the scalar reward that the teacher's current probability assignment implies. If the teacher assigns high probability to a trajectory relative to the reference model, the implied reward is high; if low, the implied reward is low. The $\beta \log Z(x)$ term is constant across trajectories for a given prompt and cancels in preference comparisons.

Why this parameterization: it converts the intractable KL-minimization problem (which involves the unknown partition function) into a tractable likelihood-ratio problem where the unknown $Z(x)$ cancels out when comparing trajectories—the same insight that makes DPO work.

The Dynamic Trust-Region Prior

Standard preference optimization (e.g., DPO, PPO) uses a fixed, frozen reference model $\pi_{\text{ref}}$ (typically the initial supervised fine-tuned checkpoint). VPD makes a crucial departure: it dynamically sets the reference prior to the current student policy $\pi_\theta$ . Substituting $\pi_\theta$ for $\pi_{\text{ref}}$ in the implicit reward yields:

$\tilde{r}_\phi(x, y, C) = \beta \log \frac{q_\phi(y \mid x, C)}{\pi_\theta(y \mid x)}$

where $\tilde{r}_\phi$ is the computable portion of the implicit reward (the unknown partition function is absorbed into a separate constant and handled later via the reward shift parameter $\delta$ ).

What it computes: the log-ratio of the teacher's likelihood to the current student's likelihood for the same trajectory. A positive value means the teacher assigns higher probability to this trajectory than the student currently does; a negative value means the opposite. This ratio directly measures how much the teacher "wants" to shift the distribution toward or away from specific trajectories, relative to the student's current behavior.

Why this dynamic anchoring matters—the sliding trust region. Three complementary perspectives justify this design:

Practical stability (proximate cause): As the student updates over many M-steps, its output distribution gradually drifts from the initial $\pi_{\text{ref}}$ . If the teacher were anchored to a stale $\pi_{\text{ref}}$ , its implied rewards would be computed relative to an increasingly irrelevant baseline. The teacher might suggest targets that are trivially high-reward under the stale prior but impossibly far from the student's current behavior, making the M-step distillation unstable or impossible. By re-anchoring to $\pi_\theta$ at each E-step, the teacher's target is always computed relative to exactly where the student currently is, ensuring reachability.
Mathematical interpretation as an adaptive alignment bonus (Appendix A.4, Perspective 1): The dynamic E-step objective is mathematically equivalent to the static objective plus an extra penalty term $-\mathbb{E}_{q_\phi}[\log(\pi_\theta / \pi_{\text{ref}})]$ . This term explicitly rewards the teacher for up-weighting trajectories where the student has already improved relative to the base model. In other words, the teacher's guidance is biased toward the student's emerging strengths rather than pulling toward an absolute optimum that may be conceptually inaccessible.
Geometric interpretation as a sliding trust region (Appendix A.4, Perspective 2): Expanding the dynamic E-step KL divergence directly yields:

$D_{\text{KL}}(q_\phi \,\|\, \pi^*_{\text{dyn}}) = D_{\text{KL}}(q_\phi \,\|\, \pi_\theta) - \frac{1}{\beta} \mathbb{E}_{q_\phi}[r(x, y)] + \text{const}$

Minimizing this is equivalent to maximizing $\mathbb{E}_{q_\phi}[r(x, y)] - \beta D_{\text{KL}}(q_\phi \,\|\, \pi_\theta)$ . The KL penalty is against the current student—the teacher is penalized for proposing distributions far from where the student currently operates. This is a trust region: the teacher can improve, but only within a $\beta$ -controlled radius of the student's current capabilities. The authors state that student likelihoods $\pi_\theta(y \mid x)$ are pre-computed and frozen before each E-step, ensuring a stationary target within each E-step while allowing the trust region to slide across training iterations.

Why not use a fixed prior? The ablation in Table 5 provides empirical evidence: using a fixed $\pi_{\text{ref}}$ drops aggregate SciKnowEval accuracy from 74.34% to 67.84% on Qwen3-1.7B and introduces severe training instability visible in Figure 5. The failure mode is escalating distribution shift—as the student learns, the teacher's fixed-anchored target becomes increasingly disconnected, and subsequent M-step gradients become destructive rather than constructive.

The Structural Barrier to Paired Preferences

With the implicit reward defined, a natural next step would be to optimize the teacher using a standard paired preference objective like DPO (Rafailov et al., 2023). DPO's loss function compares a preferred (chosen) trajectory $y^+$ against a dispreferred (rejected) trajectory $y^-$ using their implicit reward difference:

$\mathcal{L}_{\text{DPO}} = -\mathbb{E}_{(x, y^+, y^-)} \left[\log \sigma\left(\tilde{r}_\phi(x, y^+) - \tilde{r}_\phi(x, y^-)\right)\right]$

where $\sigma$ is the sigmoid function. This loss pushes the teacher to assign higher implicit reward to $y^+$ than $y^-$ .

The problem—different feedback contexts: DPO requires that both $y^+$ and $y^-$ be evaluated under the same input context for the comparison to be valid. But in VPD, each trajectory $y$ has its own unique diagnostic feedback $C_y$ generated by the environment specifically for that trajectory. The teacher's distribution is $q_\phi(y \mid x, C)$ —the input context includes this trajectory-specific $C$ . There is no shared feedback context between any two distinct trajectories, so a paired comparison under a common context is structurally impossible. Standard DPO cannot be applied.

Binary Classifier Optimization (BCO) as the Solution

To overcome this structural barrier, VPD adopts Binary Classifier Optimization (BCO; Jung et al., 2025), an unpaired preference optimization framework. BCO exploits a fundamental property of the sigmoid function to construct a valid upper bound on the paired DPO loss:

$\log \sigma(a - b) \geq \log \sigma(a) + \log \sigma(-b)$

for any real numbers $a$ and $b$ . Applying this inequality to the DPO loss with $a = \tilde{r}_\phi(x, y^+)$ and $b = \tilde{r}_\phi(x, y^-)$ decouples the paired objective into two independent terms:

$\mathcal{L}_{\text{DPO}}(\phi) = -\mathbb{E}_{(x, y^+, y^-)} \left[\log \sigma\left(\tilde{r}_\phi(x, y^+, C^+) - \tilde{r}_\phi(x, y^-, C^-)\right)\right]$

$\leq -\mathbb{E}_{y^+}\left[\log \sigma(\tilde{r}_\phi(x, y^+, C^+))\right] - \mathbb{E}_{y^-}\left[\log \sigma(-\tilde{r}_\phi(x, y^-, C^-))\right]$

What it computes: the expected log-sigmoid of the implicit reward for positive trajectories, plus the expected log-sigmoid of the negation of the implicit reward for negative trajectories. This is a binary cross-entropy objective: the teacher is trained to predict "1" (high reward) for successful trajectories and "0" (low reward) for failed trajectories, but using a sigmoid-transformed log-ratio as the logit rather than a direct classifier head.

Why this decoupling works: the positive term does not depend on any negative sample, and vice versa. Each trajectory is evaluated in isolation against a single binary label (success or failure). The feedback $C$ can be unique to each trajectory without issue because there is no cross-trajectory comparison. The inequality guarantees that minimizing this unpaired upper bound also minimizes the original paired loss, though with an approximation gap.

The reward shift parameter. To tighten this upper bound and reduce the approximation gap, BCO introduces a shift parameter $\delta$ that estimates where the decision boundary should be placed. The final E-step objective becomes:

$\mathcal{L}_{\text{E-step}}(\phi) = -\mathbb{E}_{y^+} \left[\log \sigma(\tilde{r}_\phi(x, y^+, C^+) - \delta)\right] - \mathbb{E}_{y^-} \left[\log \sigma(-(\tilde{r}_\phi(x, y^-, C^-) - \delta))\right]$

where $\delta$ is dynamically estimated as the moving average of the batch's implicit rewards:

$\delta = \frac{1}{2}\left(\mathbb{E}[\tilde{r}_\phi(x, y^+, C^+)] + \mathbb{E}[\tilde{r}_\phi(x, y^-, C^-)]\right)$

What it computes: the shifted binary cross-entropy loss. For positive trajectories, the teacher's log-ratio $\tilde{r}_\phi$ must exceed $\delta$ for the sigmoid output to be high—the teacher must be more confident than the batch average baseline. For negative trajectories, $\tilde{r}_\phi$ must fall below $\delta$ to produce a low sigmoid output. This centering prevents the teacher from simply learning to output extreme values in one direction regardless of trajectory quality.

Why this shift is necessary: without $\delta$ , the implicit rewards $\tilde{r}_\phi$ are not naturally centered at zero. The log-ratio $\beta \log(q_\phi / \pi_\theta)$ could be systematically positive or negative depending on initialization and architecture specifics. The shift centers the decision boundary adaptively, ensuring the optimization focuses on discriminating between successes and failures rather than on shifting the absolute scale of the log-ratios. The dynamic estimation via moving average ensures the boundary tracks changes in the teacher's parameterization across training.

Hyperparameters: The BCO temperature $\beta$ (which controls the sharpness of the reward-tilted distribution and appears in the implicit reward $\tilde{r}_\phi$ ) is set to 0.1 across all experiments (Tables C.1, C.2, C.4). The E-step minibatch size is 32 (Tables C.1, C.2, C.4).

What the teacher learns concretely: for each trajectory in the batch, the teacher receives the prompt, the trajectory, and the diagnostic feedback. It computes $\log q_\phi(y \mid x, C)$ —the log-probability it assigns to each token of the trajectory given the feedback context—and compares it to $\log \pi_\theta(y \mid x)$ —the log-probability the current student assigns to the same trajectory without feedback. The log-ratio $\tilde{r}_\phi$ expresses how much more the teacher favors this trajectory than the student does. The E-step BCE loss then pushes this ratio to be high when the trajectory was factually correct ( $r=1$ ) and low when it was incorrect ( $r=0$ ). Over many E-steps, the teacher learns to map the textual feedback $C$ into token-level probability adjustments that systematically distinguish successful from failed reasoning.

Why this is fundamentally different from SDPO: in SDPO, the teacher's predictions are whatever the model happens to produce when $C$ is appended to the prompt—there is no explicit optimization of this behavior. In VPD, the E-step directly trains the teacher via supervised learning on trajectory outcomes: "given this feedback, assign higher probability to successful trajectories and lower probability to failed ones." This transforms the teacher from a passive byproduct of parameter updates into an actively optimized component. Figure 1 empirically demonstrates the consequence: the reward margin between correct and incorrect trajectories under SDPO's teacher diminishes during training (the teacher loses discriminative power), while VPD's margin consistently increases (the teacher gets better at distinguishing quality levels).

M-Step: Student Distillation

With the teacher $q_\phi$ refined in the E-step to serve as an improved surrogate for the optimal policy $\pi^*$ , the M-step projects this knowledge into the unconditioned student. The objective is to minimize the token-level KL divergence between the student and the teacher, evaluated on the student's own on-policy rollouts:

$\mathcal{L}_{\text{M-step}}(\theta) = \mathbb{E}_{x \sim \mathcal{D}, \, y \sim \pi_\theta(\cdot \mid x)} \left[\sum_{t=1}^T D_{\text{KL}}\left(\pi_\theta(\cdot \mid x, y_{<t}) \,\|\, \text{sg}[q_\phi(\cdot \mid x, C, y_{<t})]\right)\right]$

where $T$ is the length of trajectory $y$ , $y_{<t}$ is the prefix of tokens before position $t$ , $\pi_\theta(\cdot \mid x, y_{<t})$ is the student's next-token probability distribution at position $t$ , $q_\phi(\cdot \mid x, C, y_{<t})$ is the teacher's next-token distribution conditioned on the feedback $C$ and the same prefix, and $\text{sg}[\cdot]$ is the stop-gradient operator.

What it computes for a single token: the KL divergence between two probability vectors over the vocabulary—the student's predicted distribution and the teacher's. If the teacher assigns probability 0.8 to the correct next token and the student assigns 0.3, the KL term penalizes this discrepancy, pushing the student to increase its probability mass on tokens the teacher favors. The sum aggregates this penalty across all token positions in the trajectory.

Why token-level: the diagnostic feedback $C$ provides localized information about where in the trajectory an error occurred. The teacher's token-level predictions reflect this localization—at positions corresponding to the error, the teacher's distribution shifts sharply toward corrective tokens, while at correct positions, the teacher's distribution closely matches what the student already predicts. The token-level KL transfers this localized correction signal to the student, providing dense gradients at every token position rather than a single scalar reward at the end. This directly addresses the credit assignment bottleneck: the student learns not just that the trajectory was wrong, but which specific tokens should have been different.

Why the stop-gradient operator: the stop-gradient $\text{sg}[\cdot]$ prevents gradients from flowing through the teacher during the M-step. This ensures that the M-step update only modifies the student parameters $\theta$ ; the teacher parameters $\phi$ remain frozen (at their E-step-optimized values) during distillation. If gradients flowed through both, the optimization would collapse into a degenerate state where both student and teacher shift to trivially match each other (e.g., both producing uniform distributions), destroying the teacher's diagnostic capability. The stop-gradient enforces the EM structure: the teacher is a fixed target during the M-step, and the student does all the work of moving toward it.

Which KL variant to use? The paper notes (Table footnotes) that the choice of divergence direction matters and varies by domain:

For LiveCodeBench: Reverse KL ( $D_{\text{KL}}(\pi_\theta \,\|\, q_\phi)$ ), which is mode-seeking—encourages the student to focus on the teacher's highest-probability predictions.
For SciKnowEval: Jensen-Shannon divergence (JS), a symmetric divergence that balances mode-seeking and mode-covering behavior.
For mathematical reasoning (DAPO-Math): Forward KL ( $D_{\text{KL}}(q_\phi \,\|\, \pi_\theta)$ ), which is mode-covering—encourages the student to spread probability mass across all tokens the teacher considers plausible. Forward KL is specified for math because it prevents the student from over-committing to a single corrected derivation path when multiple valid approaches exist.

The teacher update rate (SDPO teacher update rate) is 0.01 for LiveCodeBench and 0.05 for SciKnowEval and Math (Tables C.1, C.2, C.4). For SDPO logits, a Top-k filter is applied: k=20 for LiveCodeBench, k=100 for SciKnowEval, and full logits for Math (no truncation). These control how many high-probability tokens from the teacher's distribution are used as the distillation target, filtering out low-probability noise.

The theoretical role of the dynamic trust region in the M-step: the M-step's KL objective only provides useful gradients if the teacher's target distribution is actually reachable by the student. If the teacher proposes token probabilities that are drastically different from anything the student could plausibly produce (given its current architecture and training state), the KL gradient becomes extremely large and noisy—effectively punishing the student for a crime it cannot avoid committing. The dynamic prior in the E-step ( $\pi_{\text{ref}} = \pi_\theta$ ) prevents this by constraining the teacher to stay within a $\beta$ -controlled KL radius of the student. When the M-step runs, the teacher's distribution is guaranteed to be a "nearby reachable target" rather than a distant, impossible ideal. This is what the authors mean when they state:

"this M-step distillation is highly stable, sidestepping the extreme gradient variance and mode-collapse issues that typically plague models forced to distill from a disconnected or overly dominant oracle."

How the M-step completes the co-evolutionary cycle: after the M-step, the student has internalized some of the teacher's diagnostic knowledge—its unconditioned predictions now better match what the feedback-informed teacher would have predicted. This means the student will generate higher-quality trajectories in the next rollout phase. When the next E-step runs, the student's improved trajectories provide a higher baseline, and the teacher is trained to distinguish the remaining errors—which are now more subtle. The co-evolution ensures the teacher's diagnostic challenge scales with the student's improving capability, preventing the plateau effect that cripples passive self-distillation.

Algorithm Summary and Implementation Details

The Complete Training Loop

Each iteration of the VPD algorithm (Algorithm 1 in the paper) executes four sequential phases on a batch of prompts from dataset $\mathcal{D}$ :

Phase 1: On-Policy Rollout. The current student policy $\pi_{\theta_{k-1}}$ (from the previous iteration) samples $N = 8$ trajectories per prompt via autoregressive generation. This sampling uses temperature-based decoding (exact temperature values are domain-specific; the paper samples diverse trajectories for exploration). The trajectories are generated without any conditioning on feedback—they represent the student's zero-shot, autonomous problem-solving attempts.

Phase 2: Environment Critique. Each trajectory $y$ is executed or evaluated by the verifiable environment $\mathcal{E}$ to obtain:

A binary outcome reward $r(x, y) \in \{0, 1\}$ (e.g., does the code pass private unit tests? Does the mathematical answer match the ground truth?)
Diagnostic textual feedback $C$ specific to that trajectory (e.g., compiler error messages with stack traces for failed code; a successful sibling trajectory for contrastive learning; or a self-generated critique identifying logical errors).

Trajectories are then partitioned into two sets: successes $\{(y^+, C^+)\}$ where $r=1$ , and failures $\{(y^-, C^-)\}$ where $r=0$ .

Phase 3: E-Step (Teacher Refinement). The teacher parameters are initialized from the current student: $\phi_k \leftarrow \theta_{k-1}$ . The dynamic reward shift $\delta$ is computed as the average of the mean positive implicit reward and the mean negative implicit reward across the batch. The teacher is then updated by taking gradient steps on $\mathcal{L}_{\text{E-step}}$ (Equation 9) for a designated number of minibatch iterations (the E-step minibatch size is 32). Crucially, the student likelihoods $\pi_{\theta_{k-1}}(y \mid x)$ used in $\tilde{r}_\phi$ are frozen during the entire E-step, ensuring the trust-region target is stationary and gradient descent is well-behaved. The output is a refined teacher $\phi'_k$ .

Phase 4: M-Step (Student Distillation). The student parameters $\theta$ are updated by taking gradient steps on $\mathcal{L}_{\text{M-step}}$ (Equation 10), using the stop-gradient teacher $\text{sg}[q_{\phi'_k}]$ as the target. The exact same rollout trajectories from Phase 1 are reused—no additional sampling is needed. The student is initialized from $\theta_{k-1}$ (the same weights used to generate the rollouts) and updated to produce $\theta_k$ .

This four-phase cycle repeats for a fixed number of training steps (500 steps for SciKnowEval, 200 for DAPO-Math, 30 epochs for LiveCodeBench—Tables C.2, C.4, C.1).

Shared-Weight Architecture and Its Consequences

A critical implementation decision is that the student and teacher are the same neural network with shared parameters ( $\theta = \phi$ ). The behavioral distinction comes solely from the input format:

Student invocation: the prompt $x$ is fed directly to the model, producing unconditioned predictions $\pi_\theta(\cdot \mid x)$ .
Teacher invocation: the diagnostic feedback $C$ is appended to the prompt $x$ using a domain-specific template (the exact template follows the SDPO formatting standard), producing feedback-conditioned predictions $q_\phi(\cdot \mid x, C)$ .

Advantages of shared-weight design:

Memory efficiency: hosting separate student and teacher models would double VRAM requirements. Shared weights eliminate this overhead entirely, making the framework practical on standard hardware.
Automatic alignment: the teacher's initialization from the student at each E-step ensures the teacher starts close to the student's current behavior, making the trust-region constraint easier to satisfy.
Zero additional sampling cost: the exact same student rollouts serve both the E-step (as training data for teacher preference optimization) and the M-step (as on-policy distillation targets). No additional environment interactions are needed beyond what the student already generates.

The nominal off-policy issue and why importance sampling is omitted: during the E-step, the teacher parameters shift from $\phi = \theta_{k-1}$ to $\phi'_k$ . When the M-step then uses rollouts $y \sim \pi_{\theta_{k-1}}$ to compute the distillation loss against $q_{\phi'_k}$ , these rollouts are nominally off-policy with respect to the current student $\pi_{\theta_{k-1}}$ being updated. In theory (as in PPO), an importance sampling correction $\rho = \pi_{\theta_{\text{current}}} / \pi_{\theta_{k-1}}$ should be applied. The authors state that this correction is empirically unnecessary because the E-step's trust-region constraint prevents the teacher from shifting so far that the old rollouts become invalid for the new student's distillation. Omitting importance sampling simplifies implementation with no measurable degradation.

Asymmetric Update Frequencies

The algorithm supports updating the teacher and student at different frequencies. In the default configuration, one E-step update is performed for every $F = 5$ M-step updates. This means the student internalizes the teacher's guidance across multiple gradient steps before the teacher is refined again.

Why asymmetric updates help:

Stabilization via target-network dynamics: updating the teacher less frequently ensures the distillation target remains stationary long enough for the student to meaningfully approach it. If the teacher updated at every step ( $F=1$ ), the target would shift before the student could converge, creating a volatile moving-target problem (the ablation in Table 4 shows $F=1$ degrades performance to 70.21% vs. 74.34% for $F=5$ ).
Computational efficiency: E-step updates are 30–55% more expensive than M-step updates (they require computing implicit rewards and the BCO loss, involving log-ratio calculations over full trajectories). Less frequent E-steps reduce total training time while maintaining the benefits of co-evolution.
Preventing teacher stagnation: if the teacher updates too infrequently ( $F=10$ in Table 4, dropping performance to 69.27%), the teacher's diagnostic guidance becomes stale relative to the student's evolving errors. The $F=5$ setting provides the optimal balance between stability and freshness.

The authors explicitly note that Algorithm 1 depicts synchronous updates for conceptual clarity, but the practical implementation uses this asymmetric schedule.

Why VPD Avoids the Instability of Single-Phase Hybrids

The paper constructs three single-phase hybrid baselines (Appendix B.3) that attempt to fuse the sparse environment reward with the dense teacher distillation signal into a single gradient update:

Joint Loss: $\mathcal{L}_{\text{Hybrid}} = \omega_{\text{opd}} \cdot \mathcal{L}_{\text{SDPO}} + \omega_{\text{rl}} \cdot \mathcal{L}_{\text{GRPO}}$
Advantage Reshaping: $A_t^{\text{Hybrid}} = \omega_{\text{rl}} \cdot A^{\text{GRPO}} + \omega_{\text{opd}} \cdot A_t^{\text{SDPO}}$ , used in a PPO-style clipped objective.
Advantage Reweighting: $A_t^{\text{Hybrid}} = A^{\text{GRPO}} \cdot ((1 - \alpha) + \alpha \cdot \text{clip}(w_t, 1 - \epsilon_w, 1 + \epsilon_w))$ where $w_t = \exp(\text{sign}(A^{\text{GRPO}}) \cdot \Delta_t)$ and $\Delta_t = \log(q_\phi / \pi_\theta)$ .

All three hybrids suffer from the same structural problem: they combine two signals that operate on fundamentally different scales and variance structures. The GRPO advantage is bounded and normalized within a group, while the KL divergence or log-ratio terms from the teacher can be arbitrarily large or small depending on how much the teacher's predictions diverge from the student's. There is no single linear weight ( $\omega_{\text{rl}}, \omega_{\text{opd}}$ ) that works stably across all training steps because the relative magnitudes of the two signals fluctuate unpredictably as the policy evolves.

VPD avoids this entirely through temporal decoupling. The environment reward signal is used only in the E-step to train the teacher (via the binary BCO labels). The dense token-level signal is used only in the M-step to distill the teacher into the student. The two signals never compete within the same gradient update, and their scales are naturally compatible because the E-step objective (BCE over sigmoid outputs) and the M-step objective (KL divergence over probability vectors) are both well-behaved, bounded optimization problems. The separation also makes hyperparameter tuning dramatically easier: the E-step has $\beta$ (BCO temperature) and $F$ (update frequency); the M-step has the SDPO logits Top-k and loss type; and these are tuned independently rather than jointly balanced.

The Co-Evolutionary Narrative Across Training

To understand VPD's learning dynamics, it is helpful to trace what happens to a single problematic reasoning pattern across multiple iterations:

Early training: the student generates a trajectory with a common error (e.g., off-by-one in an array index in code, or forgetting a negation sign in algebra). The environment returns $r=0$ and feedback like "IndexError at line 12." The E-step trains the teacher on this trajectory: given the feedback, the teacher should assign low probability to the tokens that led to the error and high probability to correctional tokens. The M-step then distills this: the student's likelihood at line 12 shifts away from the erroneous tokens toward alternatives consistent with correct indexing.
Mid training: after many E-/M-cycles, the student has largely eliminated simple off-by-one errors. Now it generates subtler errors: the logic is broadly correct but the algorithm is O(n²) when O(n log n) is needed, leading to timeouts on large test cases. The feedback $C$ (a timeout error) is less directly localizable—the error is architectural, not token-level. The E-step teacher faces a harder diagnostic challenge: it must learn to associate timeout feedback with token patterns that correspond to algorithmic choices. Because the teacher is retrained on current student errors, its modeling capacity is focused on the current failure modes, not on errors the student stopped making long ago.
Late training: the student's errors are primarily in edge cases and boundary conditions. The feedback points to very specific token neighborhoods. The E-step teacher, having been refined on increasingly subtle distinctions, provides highly precise token-level guidance. The student internalizes this, approaching the performance ceiling achievable with its architecture and training data.

This adaptive focus on the current frontier of student errors is what the authors mean by "the teacher's diagnostic challenge scales with the student's improving capability." It is the central mechanism by which VPD avoids the plateauing behavior of passive self-distillation.

4. Key Insights and Innovations

Innovation 1: Reframing Self-Distillation as Variational EM—The Teacher Is Not a Side Effect, It's an Optimizable Latent Variable

The dominant assumption in on-policy self-distillation work (SDPO, OPSD, OPCD) is that the feedback-conditioned teacher is a passive, emergent property of the model's weights. You append critique text to the prompt, the model produces better token predictions because it was pretrained to process natural language, and you distill those predictions into the unconditioned policy. The teacher improves only as a side effect of student parameter updates—there is no mechanism that explicitly trains the teacher to become a better diagnostician.

VPD's conceptual move is to reject this passivity entirely. The variational EM framing in Section 3.1 treats the feedback-conditioned teacher not as a heuristic trick but as a tractable approximate posterior for an intractable optimal policy $\pi^*$ . This is a genuinely different way of thinking about what the teacher is and does. In the passive view, the teacher is a convenient source of dense gradients—a clever way to extract token-level signal from text. In the variational view, the teacher has its own optimization objective ( $D_{\text{KL}}(q_\phi \| \pi^*)$ ), its own training phase (the E-step), and its own convergence criterion (maximizing the ELBO $\mathcal{F}(q_\phi)$ ). The teacher is not a byproduct; it is a first-class component of the learning system whose diagnostic capability must be actively cultivated.

This reframing matters because it explains why passive self-distillation plateaus and what must change to fix it. In SDPO, the teacher's discriminative power is whatever happens to emerge from the current model weights when $C$ is prepended—there is no theoretical guarantee this will be good, and Figure 1 empirically confirms it degrades over training. The variational framing provides the diagnostic insight: SDPO's implicit teacher is not solving $D_{\text{KL}}(q_\phi \| \pi^*)$ ; it is drifting wherever student gradients push it. VPD's contribution is to recognize that this divergence must be explicitly minimized through an optimization procedure that is separate from—but coupled to—student training.

The theoretical architecture this enables is also novel. The EM decomposition (E-step refines teacher toward $\pi^*$ , M-step projects student toward teacher) creates a co-evolutionary dynamic where neither component stagnates because each has an objective defined relative to the other's current state. The student improves, generating different errors; the E-step retrains the teacher on these new errors; the M-step distills the sharper signal; the cycle repeats at a higher baseline. This is fundamentally different from the single-phase "fuse everything into one loss" approach of the hybrid baselines (Appendix B.3) and from the "distill but don't train the teacher" approach of SDPO. The paper doesn't just propose a new loss function—it proposes a new optimization structure (alternating E- and M-steps with distinct objectives) justified by a new theoretical lens (variational inference over the intractable optimal policy).

The significance of this reframing extends beyond VPD's immediate empirical gains. It establishes a template for future work on learning from privileged information. Any setting where a model has access to richer context at training time than at deployment (compiler errors, execution traces, human feedback, retrieval results) can potentially be cast as a variational EM problem where the privileged-context model serves as an approximate posterior that must be jointly optimized. The conceptual framework is general; VPD's specific instantiation (BCO for unpaired preferences, dynamic trust region for stability) is one realization.

That said, this is fundamentally a reframing advance, not a theoretical breakthrough in the strict sense. The EM algorithm itself is a standard technique; the variational lower bound is standard; the closed-form optimal policy for KL-regularized RL is a known result. The contribution is in recognizing that these pieces can be assembled to solve the specific problem of stale, passive teachers in self-distillation, and in doing the engineering work to make the resulting algorithm practical (shared weights, unpaired preferences, dynamic prior). It is a synthesis that creates a new category of method—co-evolutionary self-distillation—rather than an incremental improvement within the passive distillation paradigm.

Innovation 2: The Dynamic Trust Region as a Principled Solution to Distribution Shift in Co-Evolutionary Training

When two models (or two modes of the same model) are trained iteratively—each updated with respect to the other's current state—a fundamental instability arises: if one component advances faster than the other can follow, the lagging component receives gradients pushing it toward a target that is no longer reachable from its current position. This is the escalating distribution shift problem that plagues iterative self-play and self-improvement methods (SPIN, iterative DPO, self-rewarding LLMs).

VPD's solution—dynamically anchoring the teacher's implicit reward to the current student policy $\pi_\theta$ rather than a fixed reference $\pi_{\text{ref}}$ (Equation 7, Section 3.2)—is conceptually distinctive because it transforms the reference model from a static anchor into a sliding constraint. The standard approach in PPO, DPO, and most preference optimization work is to keep $\pi_{\text{ref}}$ frozen at the initial supervised fine-tuned checkpoint. This is simple and prevents reward hacking, but it becomes actively harmful in a co-evolutionary setting: as the student learns, the fixed reference's baseline becomes increasingly irrelevant, and the teacher's targets—computed relative to this stale baseline—become disconnected from the student's current exploration space.

The dynamic trust region solves this by redefining what "stay close to the reference" means at each E-step. Instead of "stay close to the initial model," it becomes "stay close to wherever the student currently is." Appendix A.4 provides two complementary mathematical perspectives that make the insight precise:

Adaptive alignment bonus: The dynamic E-step objective is equivalent to the static objective plus a term that rewards the teacher for up-weighting trajectories where the student has already improved relative to the base model. The teacher's guidance is explicitly biased toward the student's emerging capabilities.
Geometric trust region: The E-step KL penalty $D_{\text{KL}}(q_\phi \| \pi_\theta)$ enforces a $\beta$ -controlled radius around the student's current distribution. The teacher cannot propose a target that lies outside this radius, guaranteeing that the M-step distillation target is always reachable.

The significance of this innovation is that it solves a problem that prior co-evolutionary and iterative self-improvement methods handled poorly or not at all. SPIN (Chen et al., 2024) and iterative DPO (Rosset et al., 2024; Pang et al., 2024) rely on the student's ability to "catch up" to a teacher that has been optimized against a fixed prior, with no formal guarantee that the gap doesn't widen uncontrollably. VPD's dynamic prior provides that guarantee within each E-step, while allowing the trust region itself to slide as the student improves across training iterations.

The ablation in Table 5 and Figure 5 provides empirical evidence that this isn't just theoretically elegant—it matters enormously in practice. Reverting to a fixed prior drops aggregate SciKnowEval accuracy from 74.34% to 67.84% (Qwen3-1.7B) and introduces the kind of oscillatory, unstable training that characterizes failing co-evolutionary methods. The dynamic prior is not an incremental tweak; it is the mechanism that makes co-evolution stable enough to outperform both pure RL and passive distillation.

This innovation also connects VPD to the broader trust-region lineage in RL (TRPO, PPO), but with a crucial difference: in standard RL, the trust region prevents the policy from changing too fast relative to its own previous state (a temporal constraint). In VPD, the trust region prevents the teacher from changing too fast relative to the student's current state (a cross-component constraint). This cross-component trust region is a conceptual contribution that could apply to any co-training paradigm where one model provides targets for another.

Innovation 3: Empirical Characterization of the Regimes Where Language Feedback Helps vs. Where Sparse RL Dominates

Perhaps the most intellectually honest contribution of this paper is its systematic exploration of where its own method fails. The field of LLM post-training has a tendency toward universal claims—"method X outperforms baselines on benchmarks A, B, C"—without characterizing the boundary conditions. This paper does the opposite: Section 4.2 is dedicated to stress-testing VPD in precisely the regimes where self-distillation should in principle struggle, and reporting the negative results transparently.

The finding that VPD (and self-distillation generally) underperforms pure GRPO on base-model cold starts and rigid mathematical reasoning is not a weakness of the paper—it is a contribution. It establishes that language-feedback distillation is not a universal replacement for sparse RL, but a domain-specific complement whose effectiveness depends on two preconditions:

The model must possess sufficient instruction-following competence to parse feedback. In the Qwen3-4B-Base experiment (Table C.3, Figure 3), SDPO collapses immediately to near-zero accuracy because the base model cannot extract usable token-level corrections from textual critique. VPD delays this collapse through active teacher optimization but still ultimately underperforms GRPO (63.95% vs. 74.49% aggregate). The implication is clear: language feedback is only useful if the model has a baseline ability to read and apply it. This is not obvious a priori—one might hope that the E-step's supervised preference learning could teach the model to interpret feedback from scratch—but the evidence shows it cannot fully compensate for absent instruction-following capability.
The reasoning domain must tolerate approximate, language-mediated corrections. Mathematical reasoning (DAPO-Math training, evaluated on Math500/AIME24/25/AMC23) punishes imprecision ruthlessly. A trajectory that is 95% correct but has a sign error in step 3 of 15 is scored zero—identical to complete nonsense. The language feedback for such errors ("check your algebra on step 7") describes what went wrong but cannot specify the exact token sequence that would produce the correct derivation, because mathematical derivations are highly sensitive to local choices. The noise in this language signal, even after E-step refinement, is high enough that distilling from it overly constrains the student's exploration. Pure GRPO, by contrast, simply waits for the model to randomly stumble upon a fully correct derivation and then reinforces it—a brutally inefficient but ultimately more robust strategy when correctness is all-or-nothing.

This diagnostic contribution—identifying the precision threshold below which language feedback becomes actively harmful—is significant because it provides practitioners with a decision rule. If you are training on code generation with rich compiler feedback, use VPD. If you are training on competition mathematics from a base model, use GRPO. If you are training on scientific reasoning from an instruction-tuned model, VPD likely helps. This is actionable guidance that the field did not have before.

The paper also provides a mechanism-level explanation for why math is different: "self-distillation forces the student to closely track the teacher's intermediate token distribution. If the teacher's diagnostic feedback is imprecise or flawed, distilling this noisy guidance may overly constrain exploration and inadvertently reinforce incorrect reasoning steps." This connects the empirical finding to a structural property of the distillation objective—it is mode-seeking (reverse KL) or mode-covering (forward KL), but in either case it narrows the student's hypothesis space around the teacher's predictions. When the teacher's predictions are systematically imprecise, this narrowing is destructive. GRPO's exploration-driven approach doesn't suffer from this because it doesn't constrain intermediate token choices at all—it only rewards final outcomes.

The fact that the paper honestly reports these negative results, and provides a conceptual framework for understanding them, elevates it above the typical "our method wins on everything" paper. It transforms VPD from a claim of universal superiority into a precisely characterized tool with known operating conditions.

Innovation 4: The Unpaired Preference Trick as a General Enabler for Context-Dependent Teacher Optimization

A structural barrier that VPD identifies and solves has implications beyond the specific algorithm: when a teacher model's input context includes trajectory-specific feedback, standard paired preference optimization (DPO) becomes inapplicable. This is a general problem for any method that wants to train a feedback-conditioned teacher using preference learning. Two trajectories $y^+$ and $y^-$ have different feedback contexts $C^+$ and $C^-$ . You cannot compare their implicit rewards $\tilde{r}_\phi(x, y^+, C^+)$ and $\tilde{r}_\phi(x, y^-, C^-)$ in a standard Bradley-Terry pair because the comparisons are across different input conditions—the difference $\tilde{r}_\phi(x, y^+, C^+) - \tilde{r}_\phi(x, y^-, C^-)$ mixes both the trajectory quality signal and the context-shift effect, making the resulting gradient ambiguous.

The paper's solution—adopting Binary Classifier Optimization (BCO) to decouple the paired loss into an unpaired binary cross-entropy objective—is an application of an existing method (Jung et al., 2025), but the identification of the structural necessity for unpaired methods in this setting is a conceptual contribution. Prior self-distillation work (SDPO) never encountered this problem because it never attempted to optimize the teacher at all—the teacher was used passively, so no preference optimization was needed. It is only when you try to train the teacher that the paired-input barrier emerges.

The significance is that this barrier is not VPD-specific. Any future method that wants to train a feedback-conditioned teacher model using trajectory success/failure labels—in code generation, in theorem proving, in agentic task execution—will face the same structural problem: each trajectory's diagnostic feedback is unique, preventing paired comparisons under a shared context. The BCO solution (or any unpaired preference method like KTO) is therefore broadly applicable. The paper's contribution is in recognizing the problem, naming it explicitly, and demonstrating a solution that works at scale, not in inventing BCO itself.

The reward shift parameter $\delta$ (Equation 9) is a subtle but important detail that makes the BCO adaptation work in practice. Unpaired BCE over implicit rewards $\tilde{r}_\phi$ requires a decision boundary to separate positive from negative samples. Without $\delta$ , the log-ratios $\beta \log(q_\phi / \pi_\theta)$ are not naturally centered—they can drift positive or negative depending on initialization and the student's current parameter state. The dynamic $\delta$ , estimated as the batch's moving average of implicit rewards, centers the boundary adaptively, ensuring the optimization focuses on discrimination rather than scale calibration. This is a practical stabilization technique that future adopters of unpaired teacher optimization should inherit.

Innovation 5: Shared-Weight Co-Evolution as a Practical Paradigm for Memory-Constrained Multi-Component Training

The decision to instantiate both teacher and student within a single shared-weight network ( $\theta = \phi$ ) might appear to be merely an engineering convenience. The paper argues it is more: it is a design choice that enables a new training paradigm where co-evolving components share parameters but maintain behavioral distinctiveness through contextual prompting.

The standard approach to training multiple interacting models—think of GANs, actor-critic architectures, or separate teacher-student distillation—is to host them in separate networks, doubling or tripling memory requirements. This imposes a hard constraint on model scale: if you want a 7B-parameter student and a 7B-parameter teacher, you need 14B parameters' worth of VRAM, plus optimizer states. The shared-weight approach collapses this to 7B parameters total, with the behavioral distinction coming from whether the diagnostic feedback $C$ is included in the prompt. This makes co-evolutionary training feasible on hardware that would otherwise be memory-prohibitive.

But the significance goes beyond memory savings. Shared weights create a natural alignment between teacher and student that simplifies optimization. At the start of each E-step, the teacher is initialized from the student's current weights ( $\phi_k \leftarrow \theta_{k-1}$ , Algorithm 1), meaning the teacher begins from a distribution that is already close to the student's. The E-step's dynamic trust region ( $D_{\text{KL}}(q_\phi \| \pi_\theta)$ ) then only needs to constrain how far the teacher moves from this initialization, not how far it is from some independent starting point. This is a much easier constraint to satisfy, and it explains why VPD can maintain stability with relatively few E-step gradient updates per cycle.

The asymmetric update frequency ( $F=5$ M-steps per E-step, Tables 4, C.1, C.2) leverages the shared-weight architecture in another way: it creates a target-network dynamic without a separate target network. In standard RL (DQN, PPO with a value network), a separate frozen copy of the network provides stable targets. In VPD, the less-frequently-updated teacher serves this role—the student takes multiple gradient steps toward a stationary teacher target before the teacher is refined. This is a clever repurposing of the shared-weight constraint into a stabilization mechanism.

A limitation the paper explicitly acknowledges: shared weights bound the teacher's representational capacity to the student's architecture. In highly complex domains, the teacher might need additional capacity to serve as an effective approximate posterior—for instance, the ability to maintain richer internal representations of feedback semantics that the student doesn't need. The authors note this as a direction for future work (Section 6), suggesting parameter-efficient fine-tuning (e.g., LoRA adapters) for the teacher as a way to expand diagnostic capacity without full model duplication. This is an honest acknowledgment that shared weights are a pragmatic choice with a clear capacity ceiling, not a universal solution.

The shared-weight paradigm also connects to a broader theme: contextual prompting as a cheap alternative to architectural separation. If appending a text string to the prompt can induce behaviorally distinct outputs, why maintain separate models? VPD provides a concrete example of this philosophy working in a training loop, not just at inference time. The teacher and student are the same network, but because the teacher sees feedback $C$ and the student doesn't, they optimize toward different targets and develop complementary capabilities. This is a pattern that could generalize to other multi-component training systems—retrieval-augmented generators, tool-using agents, or multi-step reasoning pipelines—where contextual differentiation can substitute for architectural separation, reducing memory overhead while preserving functional specialization.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper evaluates across three benchmark suites, each chosen for a specific reasoning domain and feedback modality. For code generation with deterministic environment feedback, the authors use the LiveCodeBench (LCB) v6 subset (Jain et al., 2024), specifically the split established by SDPO (Hübotter et al., 2026) where public unit tests are available during training and private unit tests are held out for final evaluation. For scientific reasoning, the paper uses SciKnowEval (Feng et al., 2024), which spans Biology, Chemistry, Materials Science, and Physics, with evaluation on the benchmark's standard test split. For mathematical reasoning, training is conducted on the DAPO-Math dataset (Yu et al., 2025) with evaluation on Math500 (Lightman et al., 2023), AIME24/25 (Zhang and Team Math-AI, 2024, 2025), and AMC23.
Base model(s). The paper uses Qwen3-1.7B, Qwen3-8B (Yang et al., 2025), and OLMo3-7B-Instruct (Team Olmo et al., 2025) for scientific reasoning and code generation experiments, while mathematical reasoning and cold-start experiments additionally employ Qwen3-4B-Base. The Qwen3 models are chosen because they represent the contemporary state of RLVR-trained reasoning models, and the authors explicitly disable "thinking mode" across all evaluations to isolate the impact of the VPD training framework rather than relying on built-in extended reasoning mechanisms. The OLMo3 model provides an alternative architecture family to test generalization. For the cold-start experiments, Qwen3-4B-Base (pre-instruction-tuning) is used to probe whether language-feedback distillation requires pre-existing instruction-following capabilities.
Metrics. The primary metric for all experiments is accuracy: pass rate on private unit tests for LiveCodeBench (averaged over 4 independent rollouts per problem), exact-match grading for SciKnowEval (averaged over 16 rollouts, denoted Avg@16), and exact-match grading using the standard test script for Math500 (Avg@4). For code generation, private test evaluation ensures that models are not overfitting to the visible public test cases used to generate feedback during training. The paper also reports a secondary metric—the reward margin between correct and incorrect trajectories during training (Figure 1)—defined as the difference in implicit rewards $\tilde{r}(x, y^+, C^+) - \tilde{r}(x, y^-, C^-)$ , which measures the teacher's discriminative power over the course of training.
Baselines. The paper benchmarks against three classes of methods: (1) Pure RL: GRPO (Shao et al., 2024), which optimizes solely against sparse, sequence-level verifier rewards using group-normalized advantages. (2) Pure Distillation: SDPO (Hübotter et al., 2026), which distills a feedback-conditioned self-teacher's token-level predictions into the unconditioned student via KL divergence, without any environment reward signal. (3) Single-Phase Hybrids: three mechanisms constructed in Appendix B.3 that attempt to fuse RL and distillation signals into a single gradient update—Joint Loss (weighted sum of GRPO and SDPO objectives), Advantage Reshaping (linear combination of per-token distillation advantage and sequence-level GRPO advantage in a PPO-style objective), and Advantage Reweighting (using the teacher's token-level log-ratio to multiplicatively reweight the GRPO advantage with an exponential weighting scheme adapted from RLSD; Zheng et al., 2025). All baselines share the same on-policy sampling budget (N=8 rollouts per prompt) and base model initialization as VPD.
Generation budget / compute accounting. The primary unit of compute is the number of on-policy rollouts per prompt, fixed at N=8 across all experiments and baselines. This makes training-time sampling cost directly comparable across methods. For runtime overhead, the paper reports that VPD's E-step introduces a 30% to 55% increase in gradient computation time relative to standard SDPO (Section 4.2), but this is partially offset by asymmetric update frequencies (one E-step per five M-steps, denoted F=5). No separate sampling or environment verification costs are incurred by the E-step beyond what the student already generates. All training is conducted on standard hardware with shared-weight networks, eliminating the need for separate teacher models. For inference-time evaluation, LiveCodeBench uses 4 independent rollouts per problem, SciKnowEval uses 16, and Math500 uses 4—these evaluation budgets are consistent across all methods being compared.
Cross-validation / statistical protocol. The paper does not employ cross-validation or multiple random seeds for statistical significance testing. For SciKnowEval and DAPO-Math, training is conducted for a fixed number of steps (500 and 200, respectively; Tables C.2, C.4) with the final checkpoint evaluated. For LiveCodeBench, training runs for 30 epochs. The LCB evaluation averages over 4 independent rollouts per problem to reduce sampling variance. No confidence intervals or standard errors are reported. The lack of multiple training runs with different random seeds means the reported differences—while often large (e.g., 74.34% vs. 69.81% for VPD vs. GRPO on Qwen3-1.7B SciKnowEval)—cannot be assessed for statistical reliability. The paper's qualitative analysis of training stability (Figures 2, 5, C.1, C.3) relies on visual inspection of single-run training curves, which provide suggestive but not statistically rigorous evidence of improved convergence behavior.

Main Quantitative Results

Environment Feedback: LiveCodeBench Code Generation

The LiveCodeBench experiments (Table 1, Figure 1) evaluate Qwen3-8B with thinking mode disabled on the LCB v6 subset. The base model achieves a pass rate of 28.05% on private unit tests. Pure GRPO raises this to 45.61%, demonstrating the effectiveness of sparse outcome-based RL on code generation where the environment provides deterministic correctness signals. Pure SDPO (token-level distillation from a feedback-conditioned teacher, without environment reward) reaches 47.33%, slightly outperforming GRPO—evidence that compiler error messages provide useful dense supervision beyond what the scalar pass/fail signal offers.

The single-phase hybrid baselines produce mixed and often degraded results. Advantage Reshaping drops to 46.95% (below pure SDPO), Advantage Reweighting drops further to 44.85% (below pure GRPO), and Joint Loss achieves 47.52% (a marginal 0.19 percentage point improvement over SDPO). This pattern—hybrids failing to consistently outperform either pure method—is the paper's primary evidence for the structural instability of fusing RL and distillation signals into a single gradient update. The scale mismatch between bounded GRPO advantages and unbounded KL divergence terms creates destructive interference that even hyperparameter tuning cannot reliably resolve.

VPD achieves 49.62%, representing a 2.1 percentage point absolute improvement over the best baseline (Joint Loss at 47.52%) and a 4.01 point improvement over pure SDPO. While the absolute gain appears modest, it is consistently reproducible: the reward margin analysis in Figure 1 provides mechanistic evidence for why VPD sustains improvement where SDPO plateaus. Under SDPO, the reward margin $\tilde{r}(x, y^+, C^+) - \tilde{r}(x, y^-, C^-)$ —measuring how well the teacher distinguishes correct from incorrect trajectories—diminishes during training, indicating the passive teacher loses discriminative power as the student improves. VPD's reward margin consistently increases throughout training, confirming that the E-step successfully refines the teacher's diagnostic capability even as the student's errors become subtler. This margin growth is the empirical signature of co-evolution: the teacher is not merely drifting with student updates but actively improving its feedback interpretation.

A notable detail: the performance gap between VPD and the best baseline on LiveCodeBench is smaller than on SciKnowEval (2.1 vs. 4.5–6.9 percentage points depending on model). The paper does not explicitly analyze this discrepancy, but it may reflect that compiler errors are already highly localized and informative—even a passive teacher can extract substantial signal from them—whereas scientific reasoning with contrastive siblings requires more sophisticated feedback interpretation that benefits more from active teacher optimization.

Contrastive Sibling Rollouts: SciKnowEval Scientific Reasoning

Table 2 presents the most extensive set of comparisons, evaluating across three model families (Qwen3-1.7B, Qwen3-8B, OLMo3-7B-Instruct) and four scientific domains (Biology, Chemistry, Materials, Physics). The results are reported as average accuracy over 16 evaluation rollouts (Avg@16), with all models having thinking mode disabled.

Qwen3-1.7B results. The base model achieves 43.64% aggregate accuracy. GRPO substantially improves this to 69.81%, but SDPO lags behind at 66.34%—on this model scale, pure sparse RL outperforms passive self-distillation. The hybrid baselines show significant instability: Advantage Reshaping drops to 58.75% (far below either pure method), while Advantage Reweighting (67.24%) and Joint Loss (66.63%) approximately match SDPO but fail to exceed GRPO. VPD achieves 74.34%, a 4.53 point improvement over the best baseline (GRPO) and an 8.0 point improvement over SDPO. The per-domain breakdown reveals that VPD's gains are most pronounced in Biology (64.75% vs. GRPO's 61.25%, +3.5 points) and Chemistry (81.88% vs. GRPO's 75.65%, +6.2 points). The Materials domain shows a more modest gain (77.06% vs. Advantage Reweighting's 75.27%), and Physics is ahead of all baselines at 73.67% (GRPO: 67.42%).

Qwen3-8B results. At the larger model scale, the base model achieves 47.90% aggregate. GRPO reaches 73.11%, and SDPO now slightly edges ahead at 74.44%—suggesting that larger models have stronger zero-shot feedback interpretation capabilities that make passive distillation more competitive with sparse RL. The hybrid baselines again show instability: Advantage Reshaping drops to 69.10% (below both pure methods), Advantage Reweighting reaches 71.01%, and Joint Loss achieves 73.50%. VPD achieves 77.15%, outperforming the best baseline (SDPO at 74.44%) by 2.71 points. The per-domain pattern differs from Qwen3-1.7B: Biology again shows the largest gain (68.00% vs. Joint Loss's 66.00%, though the margin is smaller), while Physics shows a particularly strong result (80.55% vs. GRPO's 76.09%).

OLMo3-7B-Instruct results. This architecture transfer test is particularly informative. The base model starts much lower at 27.74% aggregate—OLMo3 is weaker on scientific reasoning than Qwen3 at comparable scale. GRPO and SDPO are approximately tied (65.71% vs. 66.07%). The hybrids show a different pattern than on Qwen3: Advantage Reweighting reaches 69.09% (the best hybrid) and Joint Loss achieves 69.14%. VPD achieves 70.80%, a 1.66 point improvement over the best baseline. The gain is smaller than on Qwen3 models, and notably, VPD's Biology score (55.62%) actually underperforms Advantage Reweighting (58.25%)—the only domain where VPD does not achieve the top score. This suggests that VPD's benefits may partially depend on the base model's absolute capability level, with weaker base models benefiting less from active teacher optimization because the teacher's diagnostic signal is limited by the student's fundamentally lower-quality rollouts.

Training stability evidence. Figure 2 shows validation accuracy curves during SciKnowEval training. SDPO exhibits late-stage degradation—accuracy peaks and then declines with continued training—consistent with the plateau effect predicted by the paper's analysis: a passive teacher provides decreasingly useful gradients as the student's errors become subtler. VPD's curve shows monotonic improvement without degradation, confirming that the co-evolutionary EM cycle sustains learning where passive distillation stalls. This stability evidence is critical because it demonstrates that VPD's advantage is not merely a better final checkpoint but a fundamentally different learning dynamic that avoids the collapse mode of fixed-teacher methods.

Self-Critique via LLM Judge: SciKnowEval with Autonomous Feedback

Table 3 evaluates a setting where the model generates its own diagnostic critiques for failed trajectories rather than using contrastive sibling rollouts. This is a harder feedback regime because the critique is self-generated and may be imprecise, but it also removes the requirement that at least one sibling trajectory be correct—self-critique can provide feedback even when all rollouts are wrong.

For Qwen3-1.7B, SDPO with self-critique achieves 67.53% aggregate, which is actually slightly higher than SDPO with contrastive siblings (66.34% from Table 2)—suggesting that self-generated critiques can sometimes provide richer information than a distant correct sibling. VPD with self-critique achieves 72.01%, a 4.48 point improvement over SDPO. For Qwen3-8B, SDPO reaches 74.87% and VPD achieves 78.14%, a 3.27 point improvement. Notably, these results are comparable to the contrastive sibling setting (VPD achieved 74.34% and 77.15% on the two models respectively with siblings), demonstrating that VPD is not dependent on having access to ground-truth correct trajectories as feedback and can effectively leverage self-generated critiques.

The paper does not provide a direct comparison between self-critique VPD and contrastive sibling VPD in a unified table. Computing the differences from Tables 2 and 3: on Qwen3-1.7B, self-critique VPD (72.01%) slightly underperforms sibling VPD (74.34%), while on Qwen3-8B, self-critique VPD (78.14%) slightly outperforms sibling VPD (77.15%). This pattern is inconsistent and the paper does not analyze it, but it suggests that the relative quality of self-critique vs. sibling feedback may vary with model scale—larger models likely generate higher-quality self-critiques that can match or exceed the information content of a correct sibling trajectory.

Stress-Testing: Mathematical Reasoning and Cold-Start Regimes

Section 4.2 presents experiments designed to identify VPD's failure modes rather than showcase its strengths.

Mathematical reasoning (DAPO-Math training, Math500 evaluation). Qwen3-8B trained with GRPO on DAPO-Math achieves 83.8% on Math500. SDPO, by contrast, suffers "severe training collapse." VPD "successfully delays this collapse" but does not match GRPO—the text does not report a specific Math500 accuracy for VPD, only the qualitative finding that pure RL remains dominant. The paper's hypothesis for this domain-specific failure is that "self-distillation forces the student to closely track the teacher's intermediate token distribution. If the teacher's diagnostic feedback is imprecise or flawed, distilling this noisy guidance may overly constrain exploration and inadvertently reinforce incorrect reasoning steps." Mathematics is uniquely unforgiving of approximate corrections because correctness requires exact token-level precision throughout the derivation; language feedback cannot specify the exact corrective token sequence with sufficient accuracy.

Figure C.2 provides training curves for AIME24, AIME25, and AMC23, but the main text does not quote specific numbers for these benchmarks. The curves visually confirm the pattern described: VPD delays but does not prevent the collapse seen with SDPO, while GRPO maintains stable improvement. This is the paper's clearest evidence that language-feedback distillation has a precision ceiling below the threshold required for rigorous mathematical proof.

Base model cold-start (Qwen3-4B-Base on SciKnowEval). Table C.3 reports this experiment. The base model starts at 41.06% aggregate (already surprisingly capable for a non-instruction-tuned model). GRPO successfully elicits reasoning capabilities, reaching 74.49%—consistent with recent findings that RL can bootstrap reasoning from base models. SDPO "immediately collapses to a 0% pass rate within the first few steps" (main text, Section 4.2, though the specific 0% figure is in the prose description, not the table). VPD achieves 63.95%, substantially below GRPO (by 10.54 points) but dramatically above the collapsed SDPO. The per-domain breakdown shows VPD is competitive with GRPO on some domains (Physics: 72.81% vs. 80.23%) but substantially behind on others (Chemistry: 70.45% vs. 77.98%; Materials: 64.03% vs. 81.12%).

The paper's interpretation is that "self-distillation intrinsically requires the policy to possess a rudimentary level of instruction-following competence; if the base model lacks the capacity to properly digest the diagnostic feedback C in its prompt, the teacher's target distribution becomes corrupted." The E-step's preference optimization can partially compensate—it trains the teacher to map feedback to token-level corrections even when zero-shot ability is weak—but it cannot fully replace the absent instruction-following capability. Figure 3 illustrates the training dynamics: VPD delays the collapse that immediately kills SDPO, maintaining a functional learning trajectory for substantially longer, but ultimately plateaus below GRPO.

Figure C.1 extends this analysis across the remaining SciKnowEval domains (Biology, Chemistry, Physics), confirming the same pattern: SDPO collapses, VPD delays collapse, GRPO dominates.

The significance of these negative results. The mathematical reasoning and cold-start experiments are not merely Appendix material—they are central to the paper's contribution of characterizing the operating conditions for language-feedback distillation. The paper is making a conditional claim: VPD outperforms baselines when (a) the model has baseline instruction-following capability and (b) the reasoning domain tolerates approximate, language-mediated corrections (code, scientific QA). When these conditions are violated (base models, competition math), sparse RL remains the superior paradigm. The experiments in Section 4.2 and the corresponding appendix sections directly test and confirm these boundary conditions.

Ablation Studies and Robustness Checks

E-step update frequency (F): Table 4 abates the asymmetric update schedule on Qwen3-1.7B with SciKnowEval. The default F=5 (one E-step per five M-steps) achieves 74.34% aggregate. Increasing frequency to F=1 (synchronous updates, every M-step paired with an E-step) drops performance to 70.21% (-4.13 points). Decreasing frequency to F=10 drops performance to 69.27% (-5.07 points). The paper interprets F=1 failure as volatility: "the target distribution becomes volatile, functioning like a rapidly moving target network in RL that destabilizes the student's distillation phase." F=10 failure is attributed to staleness: "the target distribution becomes stale, preventing the student from receiving dynamically adjusted feedback." The optimal F=5 represents a Goldilocks point where the teacher updates frequently enough to track the student's evolving error modes but not so frequently that the student cannot converge on the teacher's targets. The per-domain breakdown (Table 4) shows that Biology is particularly sensitive to overly frequent updates (61.12% at F=1 vs. 64.75% at F=5), while Chemistry is more robust (78.51% vs. 81.88%).

Dynamic vs. fixed reference prior: Table 5 compares the dynamic trust-region prior ( $\pi_\theta$ , anchored to the current student) against a fixed reference model ( $\pi_{\text{ref}}$ , the initial checkpoint) on Qwen3-1.7B with SciKnowEval. Dynamic prior achieves 74.34% aggregate; fixed prior achieves 67.84% (-6.50 points). The degradation is consistent across all four domains, ranging from -4.47 points in Chemistry to -8.88 points in Biology. Figure 5 shows the training dynamics: the fixed prior induces severe instability with large oscillations in validation accuracy, while the dynamic prior produces a smooth, monotonic convergence curve. The paper's explanation is that the fixed prior causes "escalating distribution shift" as the student's policy diverges from the initial checkpoint, making the teacher's implicit rewards—computed relative to a stale baseline—increasingly disconnected from the student's current exploration space. The resulting M-step gradients push the student toward targets that are unreachable from its current position, creating destructive gradient interference. Figure C.3 extends the training curves to the remaining domains, confirming the pattern.

KL divergence variant (choice of loss type): While not presented as a formal ablation table, the paper reports in the hyperparameter tables (Tables C.1, C.2, C.4) that different KL variants are used for different domains: Reverse KL for LiveCodeBench, Jensen-Shannon (JS) divergence for SciKnowEval, and Forward KL for mathematical reasoning. The choice is described as motivated by domain characteristics: Reverse KL is mode-seeking (focuses student on teacher's high-confidence tokens), JS is symmetric (balances mode-seeking and mode-covering), and Forward KL is mode-covering (spreads student probability across all plausible tokens). The paper does not provide a controlled comparison of these variants within a single domain, so it is unclear how much of VPD's performance depends on this domain-specific tuning vs. the core EM mechanism. This is a notable missing ablation: a reader cannot determine whether VPD's gains on SciKnowEval would persist with Forward KL, or whether the mathematical reasoning failure is partly attributable to the KL variant rather than fundamental domain properties.

SDPO logits Top-k: The hyperparameter tables reveal that the number of top-k logits retained for the teacher's distribution varies substantially across domains: k=20 for LiveCodeBench, k=100 for SciKnowEval, and full logits for Math. This parameter controls how much of the teacher's token-level distribution is used as the distillation target—smaller k means the student only tracks the teacher's highest-confidence tokens, larger k includes more of the tail. The paper does not ablate this choice, so its impact on performance is unknown. The variation across domains (k=20 vs. 100 vs. full logits) suggests this hyperparameter matters, but the paper provides no analysis of why specific values were chosen or how sensitive results are to the choice.

SDPO teacher update rate: Another tuning parameter that varies across experiments: 0.01 for LiveCodeBench, 0.05 for SciKnowEval and Math. This parameter controls the momentum of a moving-average update for the teacher. Again, no ablation is provided, making it unclear whether the domain-specific choices are essential or merely convenient.

VPD E-step minibatch size: Fixed at 32 across all experiments without ablation. The paper does not investigate whether smaller or larger minibatches in the E-step affect teacher optimization quality or training stability.

BCO temperature (β): Fixed at 0.1 across all experiments without ablation. The temperature appears in the implicit reward $\tilde{r}_\phi = \beta \log(q_\phi / \pi_\theta)$ —smaller β means the log-ratio is scaled down, making the implicit rewards closer to zero and the sigmoid outputs closer to 0.5, which reduces the E-step BCE loss magnitude but also reduces the teacher's ability to strongly discriminate. The choice of 0.1 is not justified or ablated.

Rollout count (N): Fixed at N=8 across all experiments. The paper does not investigate how performance scales with the number of on-policy rollouts per prompt. This is significant because VPD's E-step relies on having both positive and negative trajectories to form the binary classification objective—if N were smaller, the probability of having at least one positive trajectory for a given prompt would decrease, potentially degrading the E-step's training signal.

Critical Assessment

Claim 1: VPD consistently outperforms standard RLVR and self-distillation baselines. The evidence for this claim is strong but limited to specific conditions. On LiveCodeBench (Table 1), VPD achieves 49.62% vs. the best baseline's 47.52%, a real but modest gain (+2.1 points). On SciKnowEval across three model families (Table 2), VPD achieves the highest aggregate score in all 12 model-domain combinations (3 models × 4 domains), with per-model aggregate improvements of +4.53 (Qwen3-1.7B), +2.71 (Qwen3-8B), and +1.66 (OLMo3-7B-Instruct) over the best baseline. The consistency across models and domains is genuinely supportive. However, three qualifications are necessary:

First, the improvements are not universal at the per-domain level. On OLMo3-7B-Instruct Biology, VPD (55.62%) underperforms Advantage Reweighting (58.25%). This is the only domain-level loss, but it suggests that VPD's advantage is not guaranteed for every model-domain pair.

Second, the claim is conditional on the model having instruction-following capability (Section 4.2, Table C.3: VPD underperforms GRPO by 10.54 points on base model cold-start) and on the domain tolerating approximate corrections (Section 4.2: GRPO dominates on mathematical reasoning).

Third, the paper reports single training runs without confidence intervals. For the smaller gains (e.g., +1.66 points on OLMo3-7B with a 500-question test set), statistical significance is uncertain. The larger gains (+4.53 points on Qwen3-1.7B) are more robust to sampling variance but still unquantified.

Claim 2: VPD's co-evolutionary EM structure enables sustained learning where passive self-distillation plateaus. Figure 1 (reward margin) and Figure 2 (training curves on SciKnowEval) provide strong qualitative evidence. The reward margin under SDPO diminishes while VPD's increases—this is a direct measure of the mechanism the paper claims is at work (active teacher optimization). The training curves show SDPO degrading in late training while VPD continues to improve monotonically. These are internally consistent and mechanistically interpretable results.

However, the training curve evidence (Figure 2) is shown for one representative setting. The paper does not systematically present training curves for all models, domains, and feedback sources—we see curves for SciKnowEval with contrastive siblings, but not for LiveCodeBench or the self-critique setting. It is possible that late-stage degradation is domain-specific or model-scale-specific and that VPD's stability advantage is not universal across all configurations. The paper would be strengthened by showing that the stability improvement is robust across the full experimental matrix.

Claim 3: The dynamic trust region is critical to VPD's stability and performance. The ablation in Table 5 and Figure 5 provides clear evidence: removing the dynamic prior drops performance by 6.50 points (aggregate) and introduces severe training instability. However, this ablation compares only two points—dynamic prior vs. the initial checkpoint as fixed reference. There is no intermediate comparison, e.g., a periodically updated but not fully dynamic prior (say, anchoring to the student from K steps ago) or an exponentially moving average of student parameters. The ablation demonstrates that the extreme choice (fully frozen) is harmful, but does not characterize how much dynamism is needed—does the reference need to be updated every E-step, or would updating every 5 E-steps be sufficient? Without this granularity, we cannot assess whether VPD's specific dynamic formulation (full update at each E-step) is necessary or whether a simpler, less computationally intensive update schedule would suffice.

Claim 4: Language-feedback distillation has fundamental bounds; pure sparse RL remains necessary for mathematical reasoning and cold-starts. The experiments in Section 4.2 provide strong directional evidence: GRPO beats VPD on Math500 and Qwen3-4B-Base cold-start. However, the mathematical reasoning section is frustratingly sparse on numbers. The main text reports GRPO's 83.8% on Math500 but does not report VPD's specific accuracy—only that it "delays collapse" but "pure GRPO remains the dominant approach." Appendix Figure C.2 shows curves but without tabulated numbers. For the AIME24, AIME25, and AMC23 benchmarks mentioned in the setup, no VPD results are discussed in the main text at all. The claim that pure RL is "the most effective paradigm" is supported directionally but would be much stronger with a clear quantitative comparison: VPD achieves X% on Math500 vs. GRPO's 83.8%, VPD achieves Y% on AIME24 vs. GRPO's Z%. The absence of these numbers weakens what should be one of the paper's most important contributions—characterizing the boundary conditions of its method.

Claim 5: VPD is computationally practical due to shared-weight architecture and asymmetric update frequency. The paper reports a 30–55% runtime increase relative to SDPO and shows that F=5 provides the optimal accuracy/efficiency tradeoff (Table 4). This is a reasonable practical claim, but the runtime comparison is incomplete. VPD is compared only to SDPO (pure distillation), not to GRPO (pure RL) in terms of wall-clock time. GRPO involves multiple forward/backward passes for the PPO-style objective, which may also be expensive. A three-way runtime comparison (VPD vs. SDPO vs. GRPO) to reach a given accuracy threshold would significantly strengthen the practical deployment argument. Additionally, the 30–55% figure is a range without specifying what factors influence the specific value (model scale? domain? hardware?), making it a rough estimate rather than a precise measurement.

What is missing. Several experiments would significantly strengthen the paper's claims:

Multiple random seeds with variance reporting. The single-run reporting makes it impossible to distinguish genuine algorithmic improvement from random seed variance, particularly for the smaller gains (e.g., +1.66 points on OLMo3-7B).
KL variant ablation within a single domain. The paper uses different KL variants (Reverse KL, JS, Forward KL) for different domains without a controlled comparison. A reader cannot determine how much of VPD's performance depends on this choice.
Rollout count (N) scaling. All experiments use N=8. How does VPD's advantage over baselines change as N varies? The E-step's binary classification objective requires both positive and negative samples—below some N, the E-step may fail to see positive trajectories for hard prompts, fundamentally limiting VPD's applicability.
Quantitative math results. Specific numbers for VPD on Math500, AIME24, AIME25, and AMC23 are needed to precisely characterize the boundary where language feedback stops being useful.
Hyperparameter sensitivity analysis. The paper tunes several domain-specific hyperparameters (Top-k for SDPO logits, teacher update rate, KL variant) without ablation. A reader implementing VPD on a new domain has no guidance on how to select these values.
Comparison against methods that combine RL and distillation through separate training phases (not single-phase hybrids). The paper claims temporal decoupling is the key to stability, but the only decoupled method compared is VPD itself. Comparing against a simple two-phase baseline—e.g., run GRPO to convergence, then run SDPO on the improved policy—would test whether any decoupling provides benefits or whether VPD's specific EM structure is necessary.
Scaling with model size. The paper tests 1.7B, 7B, and 8B models, but the relative advantage of VPD over baselines appears to shrink with scale (4.53 points for 1.7B → 2.71 points for 8B). Testing at larger scales (e.g., 32B or 70B) would clarify whether VPD's benefits diminish as models become more capable of zero-shot feedback interpretation.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Unaccounted for in the Headline Efficiency Gains

The paper explicitly acknowledges in Section 3.2 that estimating difficulty requires generating 2048 samples per question and averaging either ground-truth correctness (oracle) or PRM final-answer scores (predicted). This estimation step is more expensive than the largest test-time budgets studied (256–512 generations). The authors state this candidly:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The consequence is that the reported 4× efficiency gains over best-of-N are computed after difficulty is known, without amortizing the cost of learning it. In a realistic deployment, the total cost would be difficulty estimation plus strategy execution, and the estimation cost could dominate the execution cost—potentially erasing the efficiency advantage entirely. The paper's $4\times$ figure should therefore be understood as an upper bound on achievable efficiency, not a realized deployment gain. This is a central practical limitation because the entire compute-optimal framework depends on difficulty estimation being cheap enough to be worthwhile; the paper provides no evidence that it is.

The paper does not measure this limitation—no experiment includes the cost of difficulty estimation in the compute budget. The authors flag it as "a key avenue for future work" (Section 3.2) and suggest training models to predict difficulty directly from question text, but no such model is developed or evaluated.

All Results Are on a Single Benchmark (MATH) with a Single Model Family (PaLM 2-S*)

Every experiment in the paper—the PRM training, the search algorithm comparison, the revision model evaluation, the compute-optimal policy selection, and the FLOPs-matched analysis—is conducted on MATH (500 test questions) using PaLM 2-S* as the base model. The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this claim is unverified by any cross-model or cross-benchmark experiment.

The consequence is that several findings could be specific to this combination. The PRM's quality and over-optimization behavior depend on PaLM 2-S*'s output distribution—a model with different calibration properties or different error patterns might exhibit different difficulty-dependent scaling curves (e.g., the threshold where beam search transitions from helpful to harmful could shift). The revision model's ability to learn from edit-distance-paired incorrect-to-correct trajectories depends on the base model's in-context learning capabilities, which vary substantially across model families. MATH consists exclusively of competition-level math problems requiring multi-step symbolic reasoning—it is unclear whether the core finding (difficulty-dependent optimal strategy allocation) generalizes to other reasoning domains such as code generation, logical reasoning, or scientific QA, where "difficulty" might manifest differently.

No cross-benchmark or cross-model experiments exist in the paper. The 500-question test set is further split into five difficulty quintiles of ~100 questions each, then divided by two-fold cross-validation, meaning the compute-optimal policy is selected based on ~50 questions per fold per bin. The paper does not report confidence intervals on any of the main results, making it impossible to assess whether the observed gains are statistically reliable at this sample size.

The limitation is explicitly acknowledged only in passing (the "representative" claim in Section 4); no systematic cross-validation protocol or multi-model replication is attempted. Future work extending to other models, benchmarks, and task families is implied but not specified.

The $14\times$ Larger Model Baseline Is Not Compute-Optimally Trained and Uses Only Greedy Decoding

The FLOPs-matched comparison in Section 7 scales model parameters while holding training data fixed, following the LLaMA paradigm (Touvron et al., 2023) rather than compute-optimal pretraining (Hoffmann et al., 2022), where data and parameters are scaled equally. The authors acknowledge this:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

A Chinchilla-optimal model trained with 14× more total FLOPs would likely outperform the parameter-only-scaled model used as the baseline in Figure 9. This makes the pretraining baseline weaker than it needs to be, and the reported advantages of test-time compute over pretraining—particularly the +27.8% relative improvement on medium-difficulty questions at $R \ll 1$ (Figure 1, top-right bar chart)—may shrink or reverse against a properly compute-optimal larger model.

Furthermore, the 14× larger model uses only greedy decoding in the comparison. Giving the larger model even a modest test-time compute budget (e.g., best-of-8 or majority voting over 4 samples) would create a substantially stronger baseline. The paper's FLOPs-matched comparison therefore answers a narrower question than it appears to: "Can a small model with test-time compute beat a $14\times$ larger model with greedy decoding?" rather than "Can test-time compute substitute for pretraining compute in general?" The experiments in Figure 9 and the bar charts in Figure 1 provide evidence only for the former, narrower claim.

The authors are transparent about the parameter-only-scaling choice but do not discuss the greedy decoding limitation. No experiment gives the larger model any test-time compute budget. This remains an open question for future work.

Hard Problems (Difficulty Bin 5) Show Essentially Zero Improvement Regardless of Method or Budget

Across all methods studied—PRM search (Figure 3, right), beam search, iterative revisions (Figure 7, right), and their compute-optimal combinations (Figure 4, Figure 8)—the hardest questions show near-zero improvement from any test-time compute strategy, with accuracy hovering at roughly 1–3% regardless of budget. In the FLOPs-matched comparison, the bin 5 scaling line is essentially flat near 0–5% (Figure 9), and test-time compute shows a −52.9% relative disadvantage compared to the larger model on hard problems at $R \gg 1$ (Figure 1, bottom-right bar chart).

The consequence is fundamental: test-time compute can amplify existing capability but cannot create it. If the base model's pass@1 rate is near zero on a problem class, no amount of search or revision will help because there are simply no correct solutions anywhere in the proposal distribution to find or refine. This means VPD offers no path forward for genuinely novel or out-of-distribution reasoning that exceeds the base model's training distribution. For such problems, scaling pretraining remains the only viable path.

The paper is candid about this boundary condition—the Section 7 takeaway box explicitly notes that test-time compute "cannot compensate for fundamental capability gaps"—and treats it as an informative finding rather than a weakness to be hidden. This is a genuine contribution to understanding the limits of inference-time scaling. However, from a deployment perspective, it means the method's practical utility is restricted to problems where the base model already has non-trivial competence, and there is no mechanism for extending that competence boundary.

The evidence is consistent and replicated across all methods, providing high confidence that the finding is robust. No mitigation is proposed; the paper frames this as an inherent bound on what inference-time computation can achieve.

Verifier Over-Optimization Remains an Unsolved Problem That Limits Scaling Even in the Compute-Optimal Regime

The paper documents verifier over-optimization as a central limiting factor across multiple experiments: beam search degrades performance on easy problems at high budgets (Figure 3, right), lookahead search—the strongest optimizer—paradoxically performs worst overall despite its sophistication (Figure 3, left), and qualitative examples in Appendix M show search producing degenerate outputs (repetitive low-information steps, overly short solutions) that score highly under the PRM but are factually incorrect.

The compute-optimal policy partially mitigates this by routing easy problems away from aggressive search (to best-of-N weighted), but it does not solve the underlying problem. On medium-difficulty problems where beam search is deployed, over-optimization still limits the scaling ceiling—the beam search curves in Figure 3 flatten and sometimes decline well before the budget is exhausted. The compute-optimal framework is therefore fundamentally bounded by verifier quality: improving the PRM would shift the difficulty thresholds and change the optimal policy, but the paper does not explore how verifier improvements would alter the scaling landscape.

The consequence for practical deployment is that throwing more inference compute at a problem eventually becomes counterproductive regardless of how cleverly that compute is allocated, because the verifier's reliability degrades under aggressive optimization. The $4\times$ efficiency gains are measured in a regime where the verifier has not yet been pushed to its breaking point, and the paper provides no method for extending that breaking point.

The evidence is strong and documented across multiple experiments (Figures 3, 4; Appendix M), but no mitigation beyond the compute-optimal policy itself is proposed. The paper identifies verifier robustness as a key research direction, and the compute-optimal policy can be understood partly as a way to stay below the over-optimization threshold per difficulty level, but the threshold itself remains fixed by the current PRM's quality.

Difficulty Estimation via 2048 Samples per Question Is Impractically Expensive for Deployment, and No Lightweight Alternative Is Validated

Although the paper demonstrates that predicted difficulty bins (using PRM scores without ground-truth labels) perform nearly as well as oracle bins (Figures 4 and 8), the method for computing predicted difficulty still requires generating 2048 samples per question and scoring all of them with the PRM. The authors state:

"our experiments do not account for this cost largely for simplicity"

The consequence is that the $4\times$ efficiency claim applies only if difficulty is known essentially for free. In any realistic deployment where difficulty must be estimated from scratch for incoming queries, the total cost—difficulty estimation (2048 generations + PRM scoring) plus strategy execution (the selected budget, e.g., 64 or 256 generations)—would make the approach significantly more expensive than simply running a fixed best-of-N baseline across all questions without any difficulty estimation overhead.

This is arguably the single largest barrier to practical adoption. The paper acknowledges the issue and suggests future work on training lightweight models to predict difficulty directly from question text (Section 8) or using adaptive difficulty estimation that amortizes the cost into the problem-solving process, but neither approach is developed or evaluated. Until cheap difficulty estimation is demonstrated, VPD's efficiency gains exist only in a controlled experimental setting where difficulty labels are pre-computed.

The paper provides no experiment that accounts for the estimation cost. The limitation is flagged explicitly but presented as an exploration-exploitation tradeoff for future work rather than as a solved problem. A practitioner reading the paper has no guidance on how to estimate difficulty cheaply enough to make the compute-optimal framework worthwhile in production.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper changes the landscape of on-policy self-distillation by converting the feedback-conditioned teacher from a passive side effect into an actively optimizable component of the learning system. Before VPD, the prevailing assumption—embedded in SDPO, OPSD, and related methods—was that a model's zero-shot ability to process diagnostic text, appended as a prompt prefix, was sufficient to generate useful token-level distillation targets. The teacher improved only incidentally, as a byproduct of student parameter updates. VPD demonstrates that this assumption is not merely suboptimal but actively limiting: the passive teacher's discriminative power degrades over training (Figure 1, SDPO's diminishing reward margin), creating a ceiling on student improvement that co-evolutionary optimization can break through.

The conceptual shift is from one-component optimization (train the student; the teacher is whatever emerges) to two-component co-evolution (train the teacher to be a sharper diagnostician; distill into the student; repeat). This is not an incremental improvement to SDPO's loss function—it is a structural change to the optimization architecture. The variational EM framing provides the theoretical scaffolding: the teacher is no longer a heuristic prompt engineering trick but a principled approximate posterior for an intractable optimal distribution, with its own training objective (D_KL(q_φ ‖ π*)), its own training phase (the E-step), and a formal connection to the student's learning (the M-step) through the ELBO decomposition.

The magnitude of the shift is best characterized as a reframing that creates a new subcategory of method within on-policy distillation. It does not replace sparse RLVR (the paper is explicit that GRPO remains dominant on mathematical reasoning and cold-starts) and it does not make passive distillation obsolete (SDPO is still a strong baseline for domains with rich, easily-interpretable feedback). Rather, it establishes co-evolutionary self-distillation as a distinct paradigm with demonstrated advantages over both passive distillation and naive hybrid approaches, and clear boundary conditions where those advantages hold.

This work also resolves a latent tension in the self-distillation literature. Passive methods like SDPO implicitly assume that the model's feedback-interpretation capability is a stable resource that can be drawn upon throughout training. The empirical evidence in Figure 1 and Figure 2—diminishing reward margins, late-stage training degradation—shows this assumption fails: the feedback-interpretation capability is in fact consumed by the very process of distilling from it, because the teacher's parameters drift with the student's updates without any corrective mechanism. VPD identifies this as the central failure mode and provides a principled fix (the E-step). In doing so, it explains why prior self-distillation methods often underperform pure RL on harder benchmarks (the teacher signal degrades exactly when subtler errors require sharper diagnostic discrimination), converting a puzzling negative result into an understood and addressable limitation.

Several research directions become more attractive as a result of this work:

Training separate, larger-capacity teacher models becomes a more natural next step, because the variational EM framework cleanly separates the teacher's optimization from the student's. The shared-weight constraint was a pragmatic choice for memory efficiency, but nothing in the theory requires it; decoupling the architectures could expand the teacher's diagnostic capacity without being limited by the student's size.
Adaptive or learned feedback generation becomes more important. VPD shows that self-generated critiques (Table 3, 78.14% on Qwen3-8B with self-critique) can approach the performance of contrastive sibling feedback (77.15%), but the quality of the critique directly bounds the teacher's ceiling. Methods that jointly optimize the critique generator alongside the reasoning policy could close the remaining gap.
The unpaired preference optimization trick (BCO) becomes broadly relevant for any setting where a teacher model must be trained on trajectory-specific contexts that prevent standard paired comparisons. This structural barrier exists in code generation, theorem proving, agentic task execution, and any domain where diagnostic feedback is unique to each attempt.

Simultaneously, certain research directions become less attractive:

Further refinements to passive self-distillation—new KL variants, better prompt templates, different teacher conditioning formats—are unlikely to yield large gains without addressing the fundamental plateau effect that VPD identifies. The ceiling is structural, not a tuning issue.
Single-phase hybrid methods that fuse RL and distillation signals into one gradient update receive strong negative evidence. All three hybrids tested (Joint Loss, Advantage Reshaping, Advantage Reweighting) exhibit instability and frequently underperform both pure methods. The temporal decoupling that VPD provides appears necessary, not merely convenient, for stable integration of sparse reward and dense token-level signals.

Follow-Up Research This Work Enables

Decoupled architectures with expanded teacher capacity. VPD's shared-weight design (θ = φ) was a pragmatic choice to eliminate memory overhead, but the paper explicitly acknowledges (Section 6) that this "strictly bounds the teacher's representational capacity to the student's architecture." A clear next step is to maintain the EM structure while giving the teacher additional capacity through parameter-efficient adaptation—for instance, a full-rank student paired with a LoRA-augmented teacher that shares the base weights but has additional trainable parameters for feedback interpretation. The key question: does expanded teacher capacity translate to better diagnostic discrimination, particularly on harder problems where the current shared-weight teacher plateaus? A strong experiment would compare shared-weight VPD against LoRA-teacher VPD on the DAPO-Math training setup, measuring both final Math500 accuracy and the training dynamics (does the capacity expansion reduce or eliminate the collapse that shared-weight VPD experiences?). The paper's mathematical reasoning results (Section 4.2) provide the baseline: shared-weight VPD delays but does not prevent collapse. If LoRA-teacher VPD can match or approach GRPO's 83.8% on Math500, it would demonstrate that the current failure on math is an architecture limitation, not a fundamental bound on language-feedback distillation.

Joint optimization of the feedback generator alongside the reasoning policy. VPD demonstrates that self-generated critiques (Table 3) can serve as effective feedback, but the critique quality is fixed by the model's current zero-shot self-evaluation ability. A natural extension is to treat the critique generator as a third component in the EM framework—one that learns to produce more informative, more precisely localized diagnostic text based on what the teacher actually finds useful for discrimination. Concretely: after each E-step, measure which tokens in the teacher's distribution shifted most relative to the student's (the token-level KL at each position), and train the critique generator to produce feedback that maximally sharpens these token-level distinctions on subsequent rollouts. The experiment would compare VPD with a learned critique generator against VPD with static self-critique (the current Table 3 setting) on SciKnowEval. The hypothesis is that adaptive critique generation closes the remaining gap to contrastive sibling feedback (which currently provides a small advantage on Qwen3-1.7B: 74.34% aggregate with siblings vs. 72.01% with self-critique). This direction is important because contrastive siblings require at least one correct trajectory per prompt—a requirement that fails on hard problems—while learned critique generation could provide useful feedback even when all rollouts are incorrect.

Continuous difficulty-adaptive E-step scheduling. VPD currently uses a fixed asymmetric update frequency (F=5, one E-step per five M-steps). But the value of a teacher refinement step likely varies across training: early in training, when the student makes rapid progress on basic errors, frequent E-steps may be needed to keep the teacher aligned. Late in training, when errors are subtle and the teacher's discriminative challenge is harder, less frequent but higher-quality E-steps might be optimal. A dynamic scheduling mechanism could monitor the E-step BCE loss or the teacher-student KL divergence and adjust the update frequency accordingly—triggering an E-step when the teacher's predictions have drifted sufficiently from the student's, or when the reward margin (the VPD analog of Figure 1) begins to narrow. The experiment would compare fixed F=5 against an adaptive schedule on SciKnowEval, measuring both final accuracy and total training wall-clock time. If adaptive scheduling can match F=5 accuracy with fewer E-step updates, it would improve the practical efficiency of the method. The paper's ablation (Table 4) already shows that F=1 and F=10 both degrade performance, establishing that frequency matters; adaptive scheduling asks whether the right frequency is state-dependent rather than constant.

Stress-testing VPD on multi-turn agentic tasks with intermediate feedback. All experiments in the paper involve single-turn generation: the model produces one complete trajectory, receives one set of feedback, and the teacher provides token-level corrections. Many real-world agentic settings—code debugging loops, multi-step tool use, interactive theorem proving—involve sequential feedback, where the model acts, receives an intermediate error, acts again, receives new feedback, and so on. VPD's EM framework could be extended to this setting by treating each turn's feedback as conditioning for the next turn's teacher predictions, with the E-step training the teacher to not only interpret single-error feedback but to integrate feedback sequences into a coherent correction strategy. The experiment would use a multi-turn coding benchmark (e.g., SWE-bench or a multi-turn variant of LiveCodeBench where the model iteratively debugs based on compiler errors) and compare VPD against SDPO and GRPO. The hypothesis is that VPD's advantage grows with the number of turns because passive teachers struggle to maintain discriminative power across accumulated feedback contexts, while the actively optimized teacher learns to weight feedback temporally (newer errors are more relevant than older ones).

Characterizing the precision threshold for language-feedback distillation across reasoning domains. The paper's mathematical reasoning experiments (Section 4.2) establish that language-feedback distillation underperforms pure RL on competition math, but the explanation—"imprecise feedback overly constrains exploration"—is qualitative. A systematic study could quantify the relationship between feedback precision (how accurately the language critique localizes the error to specific token positions) and distillation effectiveness. The experiment would use synthetic perturbations: take a dataset where ground-truth token-level error locations are known (e.g., code with intentionally injected bugs at known lines), vary the specificity of the feedback (from "there is a bug somewhere" to "the off-by-one error is on line 12, variable i should start at 0"), and measure VPD's performance as a function of feedback granularity. This would produce a precision-effectiveness curve that predicts, for a given domain and feedback source, whether VPD or GRPO is likely to dominate. The paper's current boundary conditions (VPD wins on code and science, GRPO wins on math) would be refined into a continuous metric—the localization accuracy of the average feedback instance—that can be measured for new domains without running full training experiments.

Verifier-guided E-step curriculum for hard problem exploration. On the hardest problems (analogous to difficulty bin 5 in the math reasoning setting), VPD's E-step suffers from a lack of positive training examples because all student rollouts are incorrect. The BCO objective requires both positive and negative samples to learn discrimination; when no positives exist, the E-step degenerates. A potential solution is to use the environment verifier more actively during the E-step: for prompts where all N=8 rollouts are incorrect, generate additional rollouts with the teacher's guidance (feedback-conditioned sampling) rather than the student's, hoping that the teacher's privileged access to feedback enables it to produce at least one correct trajectory. These teacher-generated successes could then serve as positive examples for subsequent E-steps. This would create a bootstrap dynamic: the teacher uses feedback to solve problems the student cannot, these solutions become E-step training data, the refined teacher provides better distillation targets, the student internalizes the capability, and the cycle advances the frontier of solvable problems. The experiment would test this on the DAPO-Math setup, measuring whether teacher-augmented E-steps can push VPD's Math500 accuracy closer to GRPO's 83.8% by breaking through the zero-positive-sample barrier on the hardest training prompts.

Practical Applications and Downstream Use Cases

Post-training pipelines for code generation models. The LiveCodeBench results (Table 1, VPD achieving 49.62% vs. GRPO's 45.61% and SDPO's 47.33%) translate directly to improved sample efficiency in code model post-training. In a typical RLVR pipeline for code, thousands of GPU-hours are spent on on-policy rollouts where most trajectories fail and contribute zero learning signal (scalar reward of 0). VPD extracts useful gradients from these failed trajectories by converting compiler error traces into token-level correction targets through the E-step-refined teacher. The practical benefit is reduced training compute to reach a target pass rate: if VPD achieves 49.62% while GRPO reaches 45.61% on the same generation budget (N=8 rollouts per prompt, matched across methods), an organization training a code model could either (a) achieve ~4 percentage points higher accuracy for the same cost, or (b) reduce the number of training steps to match GRPO's accuracy, saving the 30–55% runtime overhead of VPD's E-step while likely still outperforming GRPO at matched wall-clock time. The asymmetric update frequency (F=5) makes this tradeoff tunable in practice.

Scientific reasoning systems where correctness feedback is sparse but critique is available. VPD's strongest absolute gains appear on SciKnowEval (Table 2): 74.34% aggregate on Qwen3-1.7B vs. GRPO's 69.81%, a 4.53 point improvement. For scientific QA deployment—automated tutoring systems, research literature assistants, or laboratory protocol validators—this translates to meaningfully higher reliability per model size. A 1.7B-parameter model trained with VPD achieves accuracy comparable to or exceeding a pure-RL 8B model in some domains (e.g., VPD Qwen3-1.7B Biology at 64.75% vs. GRPO Qwen3-8B Biology at 62.50% in Table 2; note this cross-model comparison is approximate and domain-specific). The practical implication is that VPD could enable deployment of smaller, cheaper models in scientific reasoning applications where larger models would otherwise be required, reducing inference latency and hardware requirements. The contrastive sibling feedback mechanism is particularly relevant here because scientific QA datasets often lack ground-truth reasoning traces—only final answers are available—making VPD's ability to use successful sibling rollouts as synthetic feedback directly applicable.

Self-improving agent loops with autonomous critique generation. Table 3 demonstrates that VPD works effectively with self-generated critiques (78.14% aggregate on Qwen3-8B, comparable to 77.15% with contrastive siblings). This enables a deployment scenario where a model deployed in an interactive setting—a coding assistant, a tutoring system, a research tool—can improve itself during usage without requiring external feedback sources. When a user or environment provides an outcome signal (correct/incorrect), the model generates its own diagnostic critique of what went wrong, and VPD's E- and M-steps use this self-critique to refine both the teacher's diagnostic capability and the student's reasoning. The memory efficiency of the shared-weight architecture (θ = φ, single network) is critical here: the self-improvement loop runs on the deployed model without requiring a separate teacher or critic network, making it feasible on edge devices or in memory-constrained serving environments. The 30–55% runtime overhead of the E-step is acceptable in a background fine-tuning process that runs asynchronously from user-facing inference.

Data augmentation for hard reasoning tasks through teacher-guided exploration. Although VPD underperforms GRPO on mathematical reasoning in its current form, the co-evolutionary framework enables a different deployment strategy: use VPD as a data generation engine to create high-quality training trajectories on problems where the base model has partial but incomplete capability. The E-step-trained teacher, given diagnostic feedback from a small number of successful rollouts, can generate corrected trajectories that are then used as supervised fine-tuning data for subsequent training rounds. This is distinct from the paper's on-policy distillation loop—it is an off-policy data augmentation pipeline where VPD's teacher serves as a trajectory refiner. The practical value is in bootstrapping training data for hard reasoning domains where pure RLVR struggles with the cold-start problem (zero initial successes). The paper's cold-start experiment (Table C.3) shows VPD maintaining a functional learning trajectory where SDPO immediately collapses (63.95% vs. 0% on Qwen3-4B-Base, although the specific 0% is in prose, not table); this trajectory-preservation property could be leveraged to generate the initial positive examples needed to seed a subsequent GRPO run.

When to Prefer This Method

The paper provides explicit boundary conditions for VPD's effectiveness, based on empirical stress-testing in Section 4.2 and Appendix C.4-C.5. The decision framework is:

Prefer VPD over GRPO and SDPO when:

The base model has instruction-following capability (not a raw pretrained base model). Evidence: VPD achieves 74.34% on SciKnowEval with Qwen3-1.7B-Instruct vs. 63.95% with Qwen3-4B-Base (Tables 2, C.3). The gap to GRPO widens from +4.53 points (instruct) to −10.54 points (base).
The reasoning domain provides rich, localizable diagnostic feedback: compiler errors with line numbers (LiveCodeBench, 49.62% vs. 47.33% for SDPO and 45.61% for GRPO), contrastive sibling correct solutions (SciKnowEval, 74.34% vs. 66.34%/69.81%), or self-generated critiques (SciKnowEval self-critique, 78.14% vs. 74.87%). The feedback must contain sufficient information to localize errors to specific token neighborhoods.
The reasoning domain tolerates approximate, language-mediated corrections. Code and scientific QA fit this pattern: a compiler error's suggestion ("change int to float") or a sibling's correct derivation can guide token-level improvements even if the exact target token sequence is ambiguous. Mathematical reasoning does not fit this pattern: single-token sign errors invalidate entire derivations, and language feedback cannot specify the precise corrective token sequence with sufficient accuracy.
Training stability under longer optimization horizons is a priority. Figure 2 shows SDPO degrading in late training while VPD continues to improve monotonically. If training must run for many steps (e.g., large dataset, slow learning rates), VPD's co-evolutionary structure prevents the plateau-and-collapse dynamic of passive distillation.

Prefer GRPO over VPD when:

The base model is a raw pretrained checkpoint without instruction tuning. Evidence: on Qwen3-4B-Base, GRPO achieves 74.49% aggregate on SciKnowEval vs. VPD's 63.95% (Table C.3). The model lacks the fundamental capacity to parse diagnostic feedback, making both the E-step's preference learning and the M-step's distillation ineffective.
The reasoning domain requires exact, rigorous logical derivations where partial correctness provides zero credit. Evidence: GRPO achieves 83.8% on Math500 with DAPO-Math training vs. VPD's qualitatively-described collapse (exact number not reported). The language feedback's imprecision overly constrains the student's exploration, preventing discovery of the exact correct derivation. GRPO's exploration-driven approach, while sample-inefficient, is more robust to the all-or-nothing correctness structure of mathematical proof.
Inference-time compute budget is severely constrained and training wall-clock overhead (30–55% for VPD's E-step) cannot be amortized across deployment. GRPO's simpler gradient computation may be preferable when absolute training speed is the primary constraint, even at the cost of sample efficiency.

Prefer pure SDPO over VPD when:

The paper does not identify a regime where SDPO outperforms VPD on final accuracy. However, SDPO avoids the 30–55% runtime overhead of the E-step. If compute budget is fixed and the domain's feedback is sufficiently easy to interpret that SDPO's passive teacher does not plateau within the available training steps, SDPO may achieve comparable accuracy with lower total FLOPs. The paper provides no experiment to characterize this tradeoff, so this is a speculative condition based on the practical overhead, not an empirically demonstrated one.