ArXiv: 2605.11458
🎯 Pitch
Full teacher access to privileged reasoning chains hurts student learning when the solution far exceeds the student’s capability—yet all prior self-distillation methods default to it. ATESD learns to truncate the teacher’s view adaptively during training, and a 4B model trained this way matches the accuracy of a 32B model on competition math.
1. Executive Summary
This paper introduces Adaptive Teacher Exposure for Self-Distillation (ATESD), a method that treats how much privileged reasoning the teacher sees during on-policy self-distillation as a learnable training-time control variable rather than a fixed default. Working on the AIME 2024, AIME 2025, and HMMT 2025 competition-math benchmarks with Qwen3-{1.7B, 4B, 8B} models, ATESD uses a lightweight Beta-policy controller conditioned on compact training-state statistics to modulate the teacher exposure ratio (truncating the privileged chain-of-thought prefix to a fraction α while keeping the boxed answer), trained via REINFORCE with a discounted learning-progress reward that scores each held exposure decision by its effect on the student's future improvement over subsequent optimization steps rather than by immediate distillation loss changes. ATESD improves over the OPSD baseline by +0.95, +2.05, and +2.33 Average@12 points at 1.7B, 4B, and 8B scales respectively, with the 4B model reaching 65.65 Average@12, establishing that adaptive teacher exposure is an effective new axis for reasoning self-distillation—while the method's global-exposure controller proves that feedback-driven adaptation outperforms both the full-exposure default and the best fixed exposure, the coarse difficulty-bin diagnostics further reveal that the optimal exposure varies across easy, medium, and hard problems, motivating per-sample control as a natural extension.
2. Context and Motivation
The Core Problem: We Don't Question the Teacher's Information Level in Self-Distillation
The fundamental question this paper tackles is deceptively simple: during on-policy self-distillation for LLM reasoning, how much of the privileged reference solution should the teacher actually see? Every existing method in this family—from On-Policy Self-Distillation (OPSD) to its follow-ups—implicitly assumes the answer is "all of it." The teacher conditions on the complete ground-truth reasoning chain when providing token-level supervision to the student. This default has gone entirely unquestioned. The paper argues that this assumption has become an invisible bottleneck: it is neither justified by evidence nor, as it turns out, reliably optimal.
The paper formalizes this question through the concept of teacher-side exposure mismatch. When the teacher sees the full reference solution on a problem that is genuinely hard for the student, the resulting token-level targets can far exceed what the student can currently absorb. The student is generating its own on-policy rollout—a partial, potentially flawed attempt at reasoning—while the teacher scores that same rollout from a position of complete knowledge of the correct solution. The gap between these two perspectives produces supervision signals that are informative in theory but unusable in practice because the student lacks the capability to bridge the distance. Figure 1A illustrates this concretely: on a simple arithmetic problem (2 + 3), the teacher's privileged chain-of-thought ("count from 2 three times") stays firmly within the student's competence and distillation succeeds. On a quadratic equation (), the full privileged chain-of-thought jumps through factoring, discriminant calculation, and root formulas—steps that far exceed what the student can currently reproduce. The distillation targets become noise, not guidance.
This mismatch is the supervision-side analogue of the rollout mismatch that OPSD was designed to remove. OPSD's key insight was that the student should learn from its own sampled trajectories (on-policy rollouts) rather than a fixed offline dataset, because the latter creates a distribution gap between what the student is trained on and what it actually generates. The present paper extends this logic one level up: even when the student is generating on-policy, the teacher's information level creates a second, orthogonal mismatch. The student may be visiting the right trajectory for its current capability, but the teacher's privileged perspective makes that trajectory appear far more deficient than it actually is from the student's own learning standpoint.
Why This Problem Matters
The practical stakes are high because on-policy self-distillation has become the dominant recipe for post-training reasoning models, especially in regimes where reliable process-level verifiers remain prohibitively expensive to construct. As the paper notes in Section 1:
"On competition-level mathematical reasoning it is now the dominant route for lifting small open-weight reasoners onto the same accuracy frontier as much larger proprietary teachers from frontier labs."
When a method becomes this central to the post-training pipeline, every design choice matters. The teacher's exposure level is not a minor implementation detail—it directly controls the information content of the supervision signal that drives the entire distillation process.
The significance extends in three directions:
1. Efficiency of learning. If full exposure produces suboptimal supervision on hard problems (which dominate the training distribution for competition-level math), then the default configuration is systematically wasting training compute. The student spends optimization steps trying to absorb targets it cannot use, which slows convergence and potentially leaves achievable accuracy on the table. The paper's controlled fixed-exposure sweep (Figure 2) confirms this empirically: intermediate exposure (α = 0.5) consistently outperforms full exposure (α = 1.0) across seeds, meaning the default costs real accuracy points. This has immediate practical implications for anyone running self-distillation pipelines—switching to a well-chosen fixed exposure could improve results with no change to training budget, model architecture, or data.
2. A missing axis in the distillation design space. The broader literature on distillation has extensively addressed student-side mismatch through scheduled sampling, DAgger-style imitation, and importance reweighting—all mechanisms that adjust what the student sees or how it is trained. The teacher side has been treated as fixed. This leaves an entire dimension of the design space unexplored. When the paper introduces α as a continuous control variable for teacher exposure, it opens a new axis that previous work implicitly set to 1.0 and then optimized around. This is not a small number of additional hyperparameters; it is a qualitatively different kind of intervention—controlling the teacher's information rather than the student's training distribution—that could interact productively with existing student-side adaptations.
3. Reconciling an implicit contradiction in the OPSD philosophy. OPSD is built on the insight that the student and teacher should use the same model weights, with only the conditioning context differing. The student sees only the problem; the teacher sees the problem plus the reference solution. But this design embeds an unresolved tension: if the student's rollout distribution is meaningfully different from the reference solution (which it will be on any non-trivial problem), then the teacher is effectively scoring the student's tokens from a perspective the student cannot access. The per-token KL in Equation (1) compares two distributions that are conditioned on fundamentally different information. The paper's exposure mechanism resolves this tension by making the teacher's information level a tunable parameter: rather than always giving the teacher everything, give it only enough privileged context to provide useful guidance without exceeding the student's current absorptive capacity.
Where Prior Approaches Fall Short
The paper identifies specific limitations in existing work along several axes.
OPSD and its follow-ups fix the student side while leaving the teacher side unexamined. OPSD (Zhao et al., 2025) is the direct baseline. Its formulation in Equation (1) defines the teacher policy as —conditioned on the problem and the complete reference solution—and the student as —conditioned only on the problem. This architecture correctly eliminates student-side distribution mismatch by training on the student's own rollouts. But it introduces a new asymmetry: the teacher always sees the full regardless of the problem's difficulty or the student's current competence. The paper notes that this is "a default inherited by follow-up methods without justification" (Section 3.2). Subsequent work extending OPSD to diverse feedback formats, continual fine-tuning, reasoning compression, and RL hybrids maintains this full-exposure assumption without questioning whether it is optimal.
Student-side mismatch has been addressed, but the analogous teacher-side problem remains unexplored. The literature on scheduled sampling (Bengio et al., 2015), DAgger (Ross et al., 2011), and importance reweighting (Li et al., 2026; Yan et al., 2026) has developed sophisticated techniques for bringing the student's training distribution closer to its inference distribution. All of these methods modulate the student's view of the teacher or the student's own training data. None of them modulate the teacher's access to privileged information. The paper's framing makes this gap explicit: these methods fix the teacher while adjusting the student, whereas ATESD keeps the student's on-policy rollout protocol unchanged and adjusts only what the teacher conditions on.
Adaptive distillation has previously modulated the student's view of a fixed teacher, not the teacher's own information level. Existing adaptive distillation techniques operate on the student side: curriculum learning orders examples by difficulty (Bengio et al., 2009), dynamic temperature schedules adjust the distillation softmax (Li et al., 2023; Jin et al., 2025; Matsuyama et al., 2025; Islam et al., 2025), and stronger adaptive teachers tune their teaching strategy to student progress (Huang et al., 2025). In all cases, the teacher's knowledge is taken as a fixed, complete signal; adaptation occurs in how that signal is presented to or weighted for the student. ATESD introduces a different form of adaptation: it modulates the teacher's own information level—how much of the reference solution the teacher actually uses—rather than how the student receives the information.
Reinforcement learning for reasoning operates in a different paradigm. Methods like PPO (Schulman et al., 2017), DPO (Rafailov et al., 2023), DeepSeek-R1 (Guo et al., 2025), and DAPO (Yu et al., 2025) use verifiable rewards (e.g., whether the final answer is correct) to directly optimize the model's reasoning policy. These methods handle delayed effects through credit assignment mechanisms like Generalized Advantage Estimation or outcome-based reward propagation. However, they operate in a fundamentally different framework: the model is learning from scalar reward signals rather than from dense token-level teacher supervision. ATESD's challenge with delayed credit assignment—exposure choices affect the student only after subsequent optimization steps, and high-exposure decisions may not produce the largest immediate loss drops—is specific to the distillation setting where the teacher provides privileged token-level targets. The paper draws on the RL toolkit (REINFORCE with delayed rewards) but applies it to a problem that RL-based reasoning methods do not face because they lack a privileged teacher altogether.
No prior work treats teacher exposure as a learnable control variable. This is the paper's central claim to novelty. Across all distillation variants—off-policy, on-policy, adaptive, student-side corrected—the teacher's access to privileged information has been a fixed binary choice: either the teacher sees the full reference solution or it sees nothing (the answer-only extreme). The paper's fixed-exposure sweep (Figure 2) provides controlled evidence that this binary framing is inadequate: intermediate values of α produce different behaviors, the optimal value depends on both difficulty and training stage, and treating α as a global fixed constant—while better than the full-exposure default—still leaves performance on the table compared to learned adaptation.
How This Paper Positions Itself
The paper positions ATESD as a natural extension of the on-policy self-distillation lineage, not a departure from it. The architecture keeps the OPSD backbone intact—the student samples on-policy rollouts from the problem-only prompt, the teacher scores the same forced tokens, and the distillation objective remains a token-level forward KL. The only change is that the teacher's context is now modulated by a learned exposure parameter α rather than always including the full reference solution. This positioning is deliberate: by isolating the one change (teacher exposure), the paper can cleanly attribute any performance differences to the exposure control mechanism rather than to changes in the student rollout protocol, optimization procedure, or training data.
The paper also positions itself relative to the difficulty-aware training literature without requiring per-sample difficulty labels. The coarse difficulty-bin analysis in Figure 2C shows that "different learning regimes prefer different tested exposures"—easy problems favor high α (full exposure), medium problems favor intermediate α, and hard problems favor low α. This observation naturally suggests per-sample or difficulty-aware exposure control. However, the paper deliberately studies a simpler global training-state-level controller first, arguing that this establishes the viability of learned exposure without requiring the infrastructure for per-sample conditioning. The results motivate per-sample control as a natural extension (Section 6), but the present method demonstrates that even training-state-level adaptation—a single α for all samples within a hold window—already outperforms both the full-exposure default and the best fixed exposure. This is a strategically modest claim that sets up a clear research trajectory.
Importantly, the paper frames the teacher exposure problem as not simply "less information is sometimes better." The relationship is not monotonic. Low α avoids mismatch but provides weak supervision (the teacher has little privileged information to guide with). High α provides rich supervision but risks exceeding the student's absorptive capacity. The optimum lies in between, and it shifts with the student's capability. This framing distinguishes ATESD from a simple annealing schedule (start with high exposure, reduce it over time) or a fixed intermediate choice. The controller is learning a policy that conditions on the actual training state—EMA-smoothed loss, mismatch, exposure history, student self-confidence—to decide what exposure level the current student can best use. The learned Beta policy in Figure 4B confirms this: it concentrates in an interior regime rather than collapsing to either boundary, suggesting the controller is making state-dependent tradeoffs rather than converging to a trivial rule.
Finally, the paper explicitly scopes its contribution as introducing a new axis rather than claiming a universally superior method. The conclusion states that "validating ATESD on larger model scales, code generation, and scientific reasoning remains important," and the limitations discussion acknowledges that the current global-exposure controller is a first step toward per-sample control. The contribution is therefore the identification and systematization of teacher-side exposure mismatch, the demonstration that it can be addressed through learned control, and the evidence that this new axis meaningfully improves reasoning self-distillation—rather than a claim that ATESD in its current form is the optimal or final solution to the exposure problem.
3. Technical Approach
3.1 Reader Orientation
This paper builds a learned exposure controller that sits on top of an existing on-policy self-distillation pipeline (OPSD) and dynamically decides how much of the privileged reference solution the teacher model gets to see when providing token-level supervision. The system solves the problem that full teacher access to the reference solution—the unquestioned default in all prior self-distillation work—produces supervision targets that can exceed what the student can currently absorb on hard problems, and it does so by treating the teacher's information level as a continuous, training-state-conditioned control variable optimized through reinforcement learning with delayed rewards rather than as a fixed hyperparameter.
3.2 Big-Picture Architecture (Diagram in Words)
The ATESD system has four major components layered on top of the OPSD backbone:
-
OPSD Backbone (unchanged): A single language model plays both student and teacher roles. The student generates on-policy rollouts from the problem-only prompt. The teacher scores those exact same tokens but with access to privileged reference information. Token-level forward KL between teacher and student logits drives the distillation loss.
-
Exposure-Modulated Teacher: Replaces the full-reference teacher context with a truncated reference controlled by a continuous exposure fraction
$\alpha \in [0, 1]$. The truncation acts only on the reasoning prefix; the final boxed answer is always preserved. Low$\alpha$gives the teacher minimal privileged reasoning (just the answer); high$\alpha$recovers the standard full-reference OPSD teacher. -
Beta Exposure Controller: A lightweight neural policy
$\pi_\phi(\alpha \mid s_t)$that maps compact training-state statistics (EMA-smoothed loss, mismatch, exposure history, student self-confidence, probe-NLL) to a Beta distribution over$\alpha$. It samples one global exposure value for each hold window of$H$student updates. The Beta parameterization keeps the distribution unimodal and interpretable: the mean represents the preferred exposure level, and the concentration represents confidence. -
Closed-Loop Training Control: A two-timescale schedule where the controller updates slowly via REINFORCE with a delayed reward signal computed over a lookahead window of
$L$student optimization steps after each held action completes. The reward combines discounted learning progress (realized reductions in student distillation loss) with a teacher-grounded credit score, enabling the controller to learn exposure decisions based on their actual effect on future learning rather than their immediate one-step loss change.
Information flows as follows: the controller observes the current training state $s_t$ → samples an exposure $\alpha_t$ from its Beta policy → the exposure-modulated teacher truncates the reference solution to $\alpha_t$ and builds the teacher context → the student generates on-policy rollouts from the problem-only prompt → the teacher scores those rollouts from its $\alpha_t$-limited context → the student updates on the token-level KL loss for $H$ steps while $\alpha_t$ is held fixed → after an additional $L$ lookahead steps, the delayed reward $R(t)$ is computed → the controller updates its policy parameters $\phi$ via REINFORCE using centered and normalized advantages → the cycle repeats with a new $\alpha_{t+H}$.
3.3 Roadmap for the Deep Dive
- First, the formalization of teacher exposure as a continuous variable (Equation 2, Equation 3) and the mismatch metric it enables (Equation 4), since understanding what
$\alpha$controls and why controlling it matters is the foundation. - Second, the exposure-modulated teacher (Equation 5, Equation 6), which shows exactly how
$\alpha_t$modifies the OPSD teacher context and distillation objective without changing the student rollout protocol. - Third, the Beta controller architecture (Equation 7), including the exact training-state statistics used, the MLP-to-Beta mapping, and the design choices that make the controller lightweight, interpretable, and constrained to unimodal distributions.
- Fourth, the closed-loop training control mechanism (Equation 8, Equation 9), which is the most complex component: the two-timescale schedule, the discounted learning-progress reward, the teacher-grounded credit score, and the REINFORCE update with entropy cap.
- Fifth, how these components integrate into the training loop—the hold-window protocol, the lookahead credit assignment, and why this temporal structure is necessary given the delayed effects of exposure decisions in on-policy distillation.
3.4 Detailed, Sentence-Based Technical Breakdown
This is primarily a method paper that introduces a new training-time control mechanism for on-policy self-distillation. The core idea is that teacher exposure—how much of the privileged reference solution the teacher conditions on when scoring student rollouts—should be treated as a learnable control variable rather than a fixed default, and that a lightweight Beta-policy controller trained with delayed learning-progress rewards can learn to set this exposure adaptively based on the training state.
Teacher Exposure as a Continuous Variable
The paper's first conceptual move is to formalize teacher exposure as a continuous variable rather than a binary choice. Prior work implicitly treats $\alpha$ as either 1.0 (full reference in the teacher context) or 0.0 (answer-only supervision). The paper argues this binary framing is inadequate because it cannot express intermediate levels of privileged information, and the fixed-exposure sweep in Figure 2 shows that intermediate values can outperform both extremes. The continuous formalization makes $\alpha$ something that can be optimized, adapted during training, and conditioned on state—the three properties that make learned control possible.
The truncation operator. Given a reference solution $y^\star = (y^\star_{\text{reason}}, y^\star_{\text{answer}})$ consisting of a reasoning trace followed by a boxed final answer, the paper defines an exposure-dependent truncation:
where $\alpha \in [0, 1]$ is the exposure fraction, $|y^\star_{\text{reason}}|$ is the total number of tokens (or characters—the paper uses length generically) in the reasoning trace, and $\lfloor \cdot \rfloor$ denotes integer floor.
What it computes: the operator takes the reasoning prefix and keeps only the first $\lfloor \alpha \cdot |y^\star_{\text{reason}}| \rfloor$ tokens, discarding the remainder, while always appending the final boxed answer $y^\star_{\text{answer}}$ in full. The output $\tilde{y}^\star_\alpha$ is a valid but truncated reference that the teacher can condition on. At $\alpha = 1$, the reasoning is complete; at $\alpha = 0$, only the answer remains; at $\alpha = 0.5$, the teacher sees the first half of the reasoning plus the answer.
Why this form: truncating from the prefix respects the causal structure of chain-of-thought reasoning—earlier steps naturally scaffold later ones—and keeps the teacher's privileged information monotonic in $\alpha$ (more $\alpha$ always means strictly more context, never different context). Preserving the answer regardless of $\alpha$ ensures that the teacher always knows the correct final outcome, which provides a minimal anchor for the $\alpha = 0$ case (answer-only supervision) while isolating the effect of $\alpha$ entirely to how much reasoning detail the teacher sees. An alternative would be to drop the answer at low $\alpha$, but this would conflate "less reasoning" with "no outcome signal," making it impossible to distinguish exposure control from simply removing supervision. Another alternative would be to sample random subsets of reasoning steps rather than a contiguous prefix, but this would destroy the logical flow of the chain-of-thought and produce incoherent teacher contexts.
The exposure-modulated teacher. Given the truncation operator, the teacher policy at exposure level $\alpha$ is defined as:
where $p_\theta$ is the base language model (the same weights used for both teacher and student), $x$ is the problem text, and $y^\star$ is the full reference solution.
What it computes: the teacher's logit distribution at each token position, conditioned on the problem and the $\alpha$-truncated reference. This is a simple drop-in replacement for the full-reference teacher in OPSD's Equation (1): instead of $p_T(\cdot \mid x, y^\star)$, we use $p^\alpha_T(\cdot \mid x, y^\star)$. The rest of the OPSD mechanism—student samples from $x$ alone, teacher and student logits are compared on the same forced tokens, KL drives the loss—is completely unchanged.
Why this form: defining $p^\alpha_T$ as a function of the same base model $p_\theta$ rather than a separate model keeps the self-distillation property intact (teacher and student share weights) and ensures that the only difference between $\alpha = 1$ and $\alpha < 1$ is the conditioning context, not model capacity or training. This is critical for clean attribution: any performance change from modulating $\alpha$ is due to the information the teacher receives, not due to the teacher being a different model.
The mismatch metric. To quantify how teacher exposure affects the supervision difficulty, the paper defines an expected per-token teacher–student mismatch:
where $\mathcal{S}$ is the training dataset of problem-reference pairs, $\hat{y} \sim p_S(\cdot \mid x)$ is an on-policy student rollout (generated from the problem alone), $|\hat{y}|$ is the length of the rollout, $\hat{y}_{<n}$ is the prefix up to (but not including) position $n$, $p^\alpha_T$ is the exposure-modulated teacher, and $p_S$ is the student distribution.
What it computes: the average per-token forward KL divergence between the teacher's $\alpha$-conditioned next-token distribution and the student's problem-only next-token distribution, taken along trajectories the student actually generates. This is precisely the loss that OPSD minimizes (Equation 1), but parameterized by $\alpha$ rather than fixed at $\alpha = 1$. $\mathcal{M}(\alpha)$ measures how difficult the teacher's targets are for the current student to match: higher mismatch means the teacher is predicting tokens that the student assigns low probability to on its own.
Why this form: forward KL $\text{KL}(p_T \| p_S)$ penalizes the student for placing low probability on tokens the teacher considers likely, which means it forces the student toward the teacher's distribution. This is the standard distillation direction: the student should learn to mimic the teacher's predictions, not the reverse. As $\alpha$ increases, $p^\alpha_T$ becomes sharper (more concentrated on reference-consistent tokens) because the teacher has more context about what the correct reasoning looks like, while $p_S$ remains unchanged (it never sees the reference). The KL therefore grows with $\alpha$, as confirmed by the monotonic mismatch growth in Figure 2B. This growth is not necessarily bad—more informative supervision should produce larger KL if the student hasn't yet absorbed it—but it becomes problematic when the gap exceeds what the student can close in a single optimization step, which is the exposure mismatch the paper identifies.
The Exposure-Modulated Teacher in Practice
The second component translates the abstract $\alpha$ into a concrete teacher context used during training. This is where ATESD interfaces with the OPSD pipeline: it replaces the teacher prompt construction while leaving the student sampling and loss computation unchanged.
Teacher context construction. Given a sampled exposure $\alpha_t$ at training step $t$, the teacher's input prompt is built as:
where $x$ is the problem text, $\tilde{y}^\star_{\alpha_t}$ is the $\alpha_t$-truncated reference solution (reasoning prefix up to fraction $\alpha_t$ plus the boxed answer), $\tau$ is a fixed transition instruction (e.g., "Now solve the problem:" or similar, which tells the model to switch from reading the reference to generating), and $[\cdot; \cdot; \cdot]$ denotes concatenation into a single prompt.
What it computes: the full teacher input string. At $\alpha_t = 1$, this is [problem; full reference; transition], which is exactly OPSD's teacher prompt. At $\alpha_t = 0$, it is [problem; answer only; transition]. At intermediate $\alpha_t$, the reasoning is partially shown. The key design choice is that the truncation applies only to the reasoning prefix; the answer is always preserved. This means the teacher always knows the correct final outcome, and $\alpha_t$ controls only how much of the reasoning process is revealed.
Why this form: concatenating $x$, $\tilde{y}^\star_{\alpha_t}$, and $\tau$ into a single prompt string is the simplest way to give the teacher privileged information, matching how OPSD constructs the teacher context. The transition instruction $\tau$ is necessary because without it, the model would simply continue the reference solution autoregressively rather than generating new reasoning for the student's rollout. The paper keeps this construction identical to OPSD's except for the truncation of $\tilde{y}^\star_{\alpha_t}$, isolating the effect of $\alpha_t$ from prompt engineering choices.
The ATESD distillation objective. The training loss for the student at exposure level $\alpha_t$ is:
where $\theta$ are the student (and teacher, since they share weights) model parameters, and all other symbols are as defined above.
What it computes: the expected per-token forward KL along on-policy student trajectories, with the teacher conditioned on the $\alpha_t$-truncated reference. Gradients flow only through $p_S$; the teacher $p^{\alpha_t}_T$ provides frozen targets at each step. This is mathematically identical to the OPSD objective (Equation 1) except for the substitution of $p^{\alpha_t}_T$ for $p_T$. When $\alpha_t = 1$, ATESD exactly recovers OPSD.
Why this form: keeping the loss function unchanged from OPSD except for the teacher context is a deliberate design choice. It means the exposure controller is the only added complexity; the student's learning dynamics are governed by the same token-level KL that OPSD uses, just with a different teacher signal at each hold window. This makes ATESD a drop-in modification: any pipeline that runs OPSD can switch to ATESD by replacing the teacher prompt construction with the $\alpha_t$-modulated version and adding the controller. The student optimizer, learning rate, LoRA configuration, and rollout protocol all remain identical.
The student rollout protocol is unchanged. Critically, the student always generates $\hat{y}$ from $p_S(\cdot \mid x)$—the problem-only prompt—regardless of $\alpha_t$. The exposure decision affects only the teacher's context during scoring, not what trajectories the student produces. This is important for two reasons. First, it isolates exposure as a teacher-side phenomenon: any effect of $\alpha_t$ on learning is mediated entirely through the teacher's targets, not through changes in what the student samples. Second, it keeps the on-policy property intact: the student is trained on its own rollouts, not on teacher-generated or reference-generated trajectories. The exposure controller modulates the difficulty of the supervision applied to those rollouts, not the rollouts themselves.
Token scoring is teacher-forced. During the forward pass, the student's sampled tokens $\hat{y}_{1:T}$ are fed as input to both the student and teacher models (teacher-forcing). The student model sees $[x; \hat{y}_{<n}]$ at position $n$ and produces logits $p_S(\cdot \mid x, \hat{y}_{<n})$. The teacher model sees $[q^{\alpha_t}_T; \hat{y}_{<n}]$ and produces logits $p^{\alpha_t}_T(\cdot \mid x, y^\star, \hat{y}_{<n})$. The KL is computed between these two distributions at each position. This means the teacher is evaluating the same tokens the student generated, but from a context enriched with (truncated) reference information. The mismatch measured by the KL is therefore the divergence between what the teacher predicts given privileged knowledge and what the student predicts given only the problem and its own preceding tokens—a direct measure of how much the student's internal reasoning differs from the reference-informed teacher's expectations.
Beta Exposure Controller Architecture
The controller is the component that learns what $\alpha$ to use. Rather than selecting from a discrete set of exposure levels (which would limit granularity and require a separate policy head per action) or directly outputting a scalar (which provides no uncertainty signal), the paper parameterizes the controller as a Beta distribution.
Training state representation. The controller conditions on a compact vector $s_t$ of training-state statistics. The paper lists six components (Section 4.2, with details implied by the architecture description):
- Global training progress (likely step count or fraction of total budget, normalized).
- Recent exposure EMA (exponentially weighted moving average of past
$\alpha$values). - Distillation loss EMA (smoothed
$\mathcal{L}_{\text{ATESD}}$). - Mismatch EMA (smoothed KL, tracking
$\mathcal{M}(\alpha)$during training). - Probe-NLL EMA (negative log-likelihood of reference tokens under the student, providing a signal of how well the student can currently predict the ground-truth solution).
- Batch-aggregated student self-confidence (likely the average of
$\max p_S(\cdot \mid x, \hat{y}_{<n})$over tokens, measuring how certain the student is about its own rollouts).
What these statistics capture: they provide a multi-faceted view of the training state that goes beyond simple loss values. The exposure EMA tells the controller what it has been doing recently. The loss and mismatch EMAs tell it how the student is responding. The probe-NLL tells it how close the student is to the reference. The self-confidence tells it how certain the student is (high confidence with high loss suggests overconfidence on wrong tokens; low confidence with low loss suggests useful uncertainty). Together, these statistics allow the controller to distinguish between states where the student is struggling (high mismatch, high loss, low probe-NLL) versus states where it is converging (low mismatch, low loss, high probe-NLL), and to adjust exposure accordingly.
Why these specific statistics: the paper chooses statistics that are (a) cheap to compute (EMAs require only running averages, and probe-NLL requires one extra forward pass or can be extracted from the distillation forward pass), (b) directly related to the exposure mismatch phenomenon (loss, mismatch, and probe-NLL all respond to $\alpha$ changes), and (c) provide complementary information (loss measures fit to teacher targets, mismatch measures distance from teacher, probe-NLL measures distance from ground truth, self-confidence measures student calibration). An alternative would be to feed raw model weights or gradients to the controller, but this would be computationally prohibitive and would create a controller with more parameters than the model being trained.
Policy parameterization. The controller is a 2-layer MLP that maps $s_t$ to the parameters of a Beta distribution:
where $f_\phi$ is the MLP with parameters $\phi$, softplus ensures the output is positive (since Beta concentration parameters must be positive), adding 1 ensures $a_t, b_t > 1$ (keeping the distribution unimodal—a Beta with parameters less than 1 is U-shaped and bimodal at the boundaries), $\alpha_t$ is the sampled exposure value, and $[\alpha_{\min}, \alpha_{\max}]$ is a safety clipping range (likely $[0, 1]$ or a slightly narrower interval to avoid degenerate edge cases).
What it computes: the controller takes in the training state, transforms it through two dense layers with nonlinearities, produces two positive scalars (the Beta concentration parameters), and samples a single $\alpha_t$ from the resulting Beta distribution. The mean of the Beta is $a_t / (a_t + b_t)$—the controller's preferred exposure level given the state. The variance is $a_t b_t / ((a_t + b_t)^2 (a_t + b_t + 1))$—inversely related to concentration: when $a_t + b_t$ is large, the distribution is tightly peaked around the mean (high confidence); when it is small (but still >2 due to the +1 offset), the distribution is broader (exploration).
Why the Beta distribution: the Beta is the natural choice for a continuous variable bounded in $[0, 1]$ because (a) it is a flexible two-parameter family that can express means anywhere in $(0, 1)$ with varying concentration, (b) it is a conjugate prior for Bernoulli/binomial likelihoods, making it the standard choice for modeling probabilities, and (c) its parameters have a clean interpretation (mean and concentration) that makes policy analysis straightforward (Figure 4B can be visualized directly). Alternatives considered and rejected: a Gaussian with a sigmoid squashing (less natural for bounded variables, can create problematic gradients near boundaries), a categorical distribution over discretized alpha values (loses the continuous structure and requires arbitrary binning), or a deterministic scalar output (no exploration, cannot learn stochastic policies that are necessary for REINFORCE).
The +1 offset and unimodality. The constraint $a_t, b_t > 1$ (achieved by the +1 after softplus) is important because it keeps the Beta distribution unimodal. A Beta(0.5, 0.5) is U-shaped, concentrating probability mass near 0 and 1, which would cause the controller to oscillate between extremes rather than learning an interior preference. A Beta(2, 2) is unimodal at 0.5. By enforcing $> 1$, the controller is restricted to distributions that have a single preferred exposure level with some exploration around it, which is a reasonable inductive bias for this problem (the optimal exposure is likely a single value at any given training state, not a bimodal choice between extremes).
Controller scale. The paper emphasizes that the controller is "intentionally small" (Section 5.1) and "lightweight" (Section 1). A 2-layer MLP with a compact input (6 statistics) and 2 outputs (the Beta parameters) likely has a few hundred to a few thousand parameters, negligible compared to the billions of parameters in the student model. This is important because (a) the controller's optimization overhead is minimal, (b) the small capacity forces the controller to learn a genuine training-state-conditioned policy rather than memorizing training dynamics, and (c) it keeps the method practical—the controller is not a second large model competing for compute.
Closed-Loop Training Control: The Two-Timescale Schedule
The controller cannot be trained with standard supervised learning because there is no ground-truth optimal $\alpha$ at each training step. Instead, the paper frames exposure control as a reinforcement learning problem: the controller is an agent that takes actions (exposure values) in an environment (the training process), and receives rewards (improvements in student learning). The challenge is that the effect of an exposure decision is not visible until after the student has optimized against the resulting teacher targets for several steps—a classic delayed credit assignment problem.
The hold-window protocol. The controller does not choose a new $\alpha$ at every student update. Instead, it operates on a slower timescale. At decision step $t_0$, the controller samples $\alpha_{t_0} \sim \pi_\phi(\cdot \mid s_{t_0})$ and holds this value fixed for the next $H$ student updates. During these $H$ steps, the student optimizes $\mathcal{L}_{\text{ATESD}}(\theta; \alpha_{t_0})$ against teacher targets generated with exposure $\alpha_{t_0}$. The student sees $H$ minibatches of on-policy rollouts, all scored by the same $\alpha_{t_0}$-modulated teacher, before the controller is allowed to change the exposure.
What this accomplishes: holding $\alpha$ fixed for $H$ steps creates a coherent "episode" during which the effect of the exposure decision can accumulate. If $\alpha$ changed every step, the student would be chasing a moving target, and it would be impossible to attribute any learning progress to a specific exposure choice. By batching $H$ student updates under a single $\alpha$, the controller's action has a measurable footprint: the change in student loss, mismatch, and probe-NLL over the $H$-step window is the direct consequence of that $\alpha$, plus the inherent noise of SGD. The paper does not specify $H$ numerically in the main text (it may be in the appendix), but the hold-window concept is standard in hierarchical RL and two-timescale optimization.
Why not per-step control: updating $\alpha$ every student step would couple the controller's learning dynamics too tightly to the student's noisy per-minibatch loss. The controller would overfit to short-term fluctuations and fail to learn the longer-term effects of exposure on learning progress. More practically, per-step control would require a reward signal at every step, which would force the use of myopic one-step loss changes (which Table 2A shows are substantially worse than delayed rewards).
The lookahead window for credit assignment. After the $H$-step hold window ends, the controller does not immediately receive its reward and update. Instead, it waits an additional $L$ student steps (the "lookahead window") before computing the reward. During these $L$ steps, the student continues training—possibly under a different $\alpha$ from a subsequent controller decision—and the effect of the held $\alpha_{t_0}$ ripples forward through subsequent student states.
What this accomplishes: the lookahead window allows the reward to capture delayed consequences of the exposure decision. A low $\alpha$ during the hold window might make current learning easy (low immediate loss) but leave the student unprepared for harder problems later (high future loss). A high $\alpha$ might cause struggle now but unlock faster progress later once the student catches up. The $L$-step delay means the reward at step $t_0$ is based on student performance from $t_0+1$ through $t_0+L$, not just $t_0+1$. The paper uses $L = 20$ for all main experiments (Section 5.1).
Why 20 steps: the choice of $L$ balances two tensions. Too short (e.g., $L = 1$, immediate one-step feedback, which Table 2A shows reaches only 52.22 Average@12) misses delayed effects. Too long (e.g., $L = 100$, the entire remaining training budget) creates a credit assignment problem so diffuse that the controller cannot distinguish good from bad decisions—every action taken early in training is "responsible" for everything that happens later. Intermediate $L = 20$ is long enough to capture the multi-step optimization dynamics that exposure controls (the student needs several updates to meaningfully incorporate new teacher targets) while short enough to maintain temporal credit locality.
Closed-Loop Training Control: The Discounted Learning-Progress Reward
The reward function is the most technically nuanced component of ATESD. It must solve the problem that "choices most beneficial to the student's future learning often do not yield the largest immediate KD-loss drop" (Section 1)—in reinforcement learning terms, the reward is not aligned with the immediate loss change. The paper designs a compound reward with two terms.
Discounted learning progress. The primary reward term rewards improvements in the student's distillation loss over the lookahead window:
where $t_0$ is the decision step, $L = 20$ is the lookahead horizon, $\gamma \in (0, 1]$ is a discount factor (likely 0.9–0.99, the paper does not specify the exact value), $\ell_{t}$ is the distillation loss $\mathcal{L}_{\text{ATESD}}$ after student step $t$, and $\max(0, \cdot)$ clips negative loss changes to zero.
What it computes: the discounted sum of positive reductions in the student's distillation loss over the $L$ steps following the hold window. Each term $\ell_{t+i-1} - \ell_{t+i}$ measures how much the loss decreased from one step to the next. If the loss increased ($\ell_{t+i-1} - \ell_{t+i} < 0$), that term contributes zero to the reward—the controller is not penalized for loss increases, only rewarded for loss decreases. The discount $\gamma^{i-1}$ weights earlier improvements more heavily than later ones, encoding a preference for exposure decisions that lead to rapid learning progress.
Why this form: the max(0,·) clipping is crucial. Without it, the reward would penalize the controller for loss increases, but loss increases are normal in SGD training (noisy minibatches, exploration, the student encountering hard examples). Penalizing the controller for stochastic loss fluctuations would create a noisy reward signal dominated by minibatch variance rather than genuine learning effects. By only rewarding positive progress, the controller learns to prefer exposure levels that tend to produce consistent improvements, implicitly averaging out the noise. The discount factor $\gamma$ is standard in RL: it encodes that sooner progress is more attributable to the recent exposure decision, while later progress is increasingly confounded by subsequent controller actions and other factors.
Teacher-grounded credit score. The second reward term anchors the controller's decisions to the teacher's ground-truth knowledge:
where $g_{t}$ is "the average log-probability assigned by the exposure-modulated teacher to verified reference tokens" at step $t$ (Section 4.3).
What it computes: a discounted average of the teacher's log-probability on reference (ground-truth) tokens over the lookahead window. Concretely, after student step $t$, the teacher—with whatever exposure $\alpha$ is currently in use—computes the log-probability it assigns to the tokens of the correct reference solution. If the teacher assigns high probability to the reference (i.e., the teacher's $\alpha$-conditioned predictions align with the ground truth), $g_t$ is high; if the teacher's predictions diverge from the reference, $g_t$ is low. The discounted average $G_{\text{gt}}$ measures whether the exposure level leads to a teacher that remains grounded in the correct solution.
Why this term exists: pure learning progress $G_{\text{lp}}$ could be gamed. A controller might learn to set $\alpha$ very low, making the teacher targets easy (close to the student's current distribution), which produces rapid loss decreases but provides no useful signal for solving hard problems. The teacher-grounded score prevents this by requiring that the teacher still predicts the reference solution well. If $\alpha$ is too low, the teacher lacks the privileged context to assign high probability to reference tokens, so $G_{\text{gt}}$ will be low even if $G_{\text{lp}}$ is high. The two terms together encode that good exposure balances two desiderata: the student should learn quickly (high $G_{\text{lp}}$), and the teacher should remain informative about the ground truth (high $G_{\text{gt}}$).
The combined reward. The final reward for a held exposure action is:
where $\lambda_{\text{gt}}$ is a hyperparameter controlling the weight of the teacher-grounded term relative to the learning progress term. The paper does not specify $\lambda_{\text{gt}}$ numerically in the main text, but its existence is implied by the formulation.
What this computes: a scalar reward signal summarizing the quality of the exposure decision made at step $t_0$. High $R(t_0)$ means the chosen $\alpha$ led to (a) consistent student improvement and (b) teacher predictions that remained faithful to the reference. Low $R(t_0)$ means either learning stalled or the teacher lost grounding.
Why this form, not reward shaping with mismatch penalty: a natural alternative reward would directly penalize high teacher–student mismatch, on the theory that mismatch above some threshold indicates unusable supervision. The paper explicitly rejects this (Section 4.3): "Teacher–student mismatch is used as controller state and diagnostic signal, not as a direct reward penalty in the main objective, because such a penalty would prefer low exposure simply for mechanically reducing KL against the student." This is a subtle but important point. If the reward included a penalty for mismatch, the controller would learn to always set $\alpha$ low—low exposure reduces KL trivially by giving the teacher less information, regardless of whether that helps the student learn. The current reward structure avoids this degenerate strategy by making learning progress (which requires the student to actually improve on held-out or future problems) and teacher grounding (which requires the teacher to predict the reference) the only reward components. Mismatch remains in the controller's state $s_t$ so the controller can observe it, but it is not directly optimized.
Closed-Loop Training Control: The REINFORCE Update
The controller is trained using the REINFORCE policy gradient algorithm (Williams, 1992), the simplest and most foundational method for training stochastic policies from scalar rewards.
Advantage normalization. Before applying the policy gradient, rewards from a batch of completed decisions are centered and normalized:
where $R_j$ is the reward for the $j$-th completed decision in the batch, $\bar{R}$ is the batch mean reward, $\text{Std}(R)$ is the batch standard deviation, and $\epsilon$ is a small constant for numerical stability.
What it computes: the advantage $A_j$ is a centered, scale-normalized version of the raw reward. Positive $A_j$ means the decision $\alpha_j$ was better than the average decision in the recent batch; negative $A_j$ means it was worse. Centering ensures that about half the decisions in each batch receive positive updates (reinforcing the policy toward those $\alpha$ values) and half receive negative updates (pushing the policy away). Normalizing by standard deviation keeps the policy gradient magnitude stable across training stages where raw reward magnitudes might change (e.g., early training has large loss drops, late training has small ones).
Why centering matters: without centering, REINFORCE with a Beta policy can exhibit a "rich get richer" dynamic where the policy collapses to a single action because every sampled $\alpha$ gets a positive update when rewards are positive on average, reducing variance until the policy becomes deterministic. Centering ensures that actions are judged relative to the current policy's expectation, maintaining exploration as long as some actions are genuinely better than others.
The policy gradient loss. The controller's loss function is:
where $B$ is the batch size (number of completed hold-window episodes), $A_j$ is the normalized advantage, $\log \pi_\phi(\alpha_j \mid s_j)$ is the log-probability of the sampled action under the current policy, $\mathcal{H}[\pi_\phi]$ is the entropy of the Beta policy, $\mathcal{H}_{\text{target}}$ is a target entropy threshold, and $c_t$ is a coefficient.
What it computes (first term): the standard REINFORCE loss. For each decision $j$, the gradient of $-\!A_j \log \pi_\phi(\alpha_j \mid s_j)$ with respect to $\phi$ increases the log-probability of $\alpha_j$ when $A_j > 0$ (good decision, reinforce it) and decreases it when $A_j < 0$ (bad decision, push away from it). The magnitude of the update is proportional to $|A_j|$: decisions that are much better or worse than average get stronger updates. The batch average $\frac{1}{B} \sum$ means the controller learns from multiple episodes simultaneously, reducing variance.
What it computes (second term): an entropy cap penalty. The entropy of a Beta($a$, $b$) distribution is a measure of its spread: high entropy means broad exploration (many $\alpha$ values are likely), low entropy means concentration (the policy is confident about a narrow range). The penalty $\max(0, \mathcal{H}[\pi_\phi] - \mathcal{H}_{\text{target}})^2$ is zero when entropy is below the target and quadratic when it exceeds the target. This means the penalty only activates when the policy is too random (over-exploring), pushing it to concentrate. It does not penalize low entropy, so the policy can become arbitrarily confident when the reward signal consistently favors a narrow exposure range.
Why this entropy penalty design: standard entropy bonuses in RL (adding $+\beta \mathcal{H}$ to the reward or loss) encourage exploration uniformly, which is useful when the reward landscape is unknown and the policy might prematurely collapse to a suboptimal action. But in ATESD's setting, the policy should eventually concentrate once it finds a good exposure region—persistent exploration at convergence would add noise to the teacher signal. The one-sided entropy cap achieves this: it allows the policy to concentrate freely when rewards are consistent (late training, stable regime) while preventing pathological over-exploration (early training, when the policy has no signal). The quadratic penalty $(\cdot)^2$ means small exceedances of the target are lightly penalized but large exceedances are strongly penalized, creating a soft constraint rather than a hard ceiling.
The REINFORCE update frequency. The controller is "updated only after held actions complete their lookahead windows" (Section 4.3). This means controller updates happen roughly every $H + L$ student steps, creating the two-timescale dynamic: the student updates frequently (every step) on the distillation loss, while the controller updates infrequently on the cumulative effect of its decisions.
Why REINFORCE rather than a critic-based method: REINFORCE is the simplest policy gradient algorithm—it requires only the ability to sample actions and compute their log-probabilities, with no need for a learned value function (critic). For ATESD's setting, this simplicity is a virtue: the controller is already a small MLP, and adding a value function head would increase complexity without clear benefit given the relatively low variance of the training-state-conditioned policy (the state $s_t$ is deterministic given the training history, so variance comes only from the stochastic action sampling and minibatch noise in the student). More sophisticated algorithms like PPO or A2C would add hyperparameters (clipping thresholds, value loss coefficients) and training instability risks without addressing a clear failure mode of REINFORCE in this context.
Integration: The Full Training Loop
The four components—OPSD backbone, exposure-modulated teacher, Beta controller, and closed-loop training control—integrate into a single training loop with nested timescales.
Step-by-step training dynamics:
-
Initialization: The student model
$\theta$is initialized from a pretrained instruct-tuned checkpoint (Qwen3-1.7B, 4B, or 8B). The controller$\phi$is initialized randomly (small MLP weights) or with a heuristic that starts the Beta mean near 0.5–0.7. The OPSD training data, LoRA configuration, and optimizer are set up identically to the baseline. -
Controller decision phase (every
$H + L$student steps): The controller observes the current training state$s_t$, computes$(a_t, b_t) = 1 + \text{softplus}(f_\phi(s_t))$, samples$\alpha_t \sim \text{Beta}(a_t, b_t)$, and clips to$[\alpha_{\min}, \alpha_{\max}]$. This$\alpha_t$is stored as the active exposure for the upcoming hold window. -
Hold window (next
$H$student steps): For each of the next$H$student optimization steps:- Sample a minibatch of problems from the training set.
- For each problem, the student generates an on-policy rollout
$\hat{y} \sim p_S(\cdot \mid x)$(problem-only prompt, standard sampling parameters). - The teacher prompt is constructed as
$[x; \text{truncate}(y^\star, \alpha_t); \tau]$. - Teacher-force the student's rollout tokens through both the student and teacher models.
- Compute the per-token forward KL
$\text{KL}(p^{\alpha_t}_T \| p_S)$along the rollout. - Update the student parameters
$\theta$using the OPSD optimizer (gradients only through$p_S$). - Track the distillation loss
$\ell_t$, mismatch$\mathcal{M}(\alpha_t)$, and probe-NLL for controller state updates. The exposure$\alpha_t$does not change during these$H$steps.
-
Lookahead window (next
$L$student steps): The hold window ends. The controller may or may not sample a new$\alpha_{t+H}$for the next hold window (depending on implementation—the paper implies a continuous cycle where each completed episode yields a controller update, and a new episode starts immediately). During these$L$steps, the student continues training (possibly under the new$\alpha$). The loss values$\ell_{t+H}, \ldots, \ell_{t+H+L}$and teacher-grounded scores$g_{t+H}, \ldots, g_{t+H+L}$are recorded. -
Reward computation and controller update: At step
$t_0 + H + L$, the reward for the decision made at$t_0$is computed:$G_{\text{lp}}(t_0) = \sum_{i=1}^{L} \gamma^{i-1} \max(0, \ell_{t_0+H+i-1} - \ell_{t_0+H+i})$$G_{\text{gt}}(t_0) = \frac{\sum_{i=1}^{L} \gamma^{i-1} g_{t_0+H+i}}{\sum_{i=1}^{L} \gamma^{i-1}}$$R(t_0) = G_{\text{lp}}(t_0) + \lambda_{\text{gt}} G_{\text{gt}}(t_0)$The episode$(s_{t_0}, \alpha_{t_0}, R(t_0))$is added to a controller replay buffer. Once enough episodes have accumulated (batch size$B$), the advantages are computed, and the controller parameters$\phi$are updated via one or a few gradient steps on$\mathcal{L}_{\text{ctrl}}$.
-
Repeat: The cycle continues for the full training budget (100 student update steps in the paper's experiments, matching the OPSD budget).
Why this temporal structure: the nested timescales (student updates every step, controller updates every $H + L$ steps) decouple the fast optimization dynamics (student fitting to teacher targets) from the slow meta-optimization (controller learning what exposure produces good student progress). If the controller updated every step, it would chase its own tail: a controller update changes $\alpha$, which changes the teacher targets, which changes the student loss, which changes the controller state, triggering another controller update. The hold-and-lookahead structure gives each exposure decision enough time to have a measurable effect before the controller reacts to it, making the credit assignment tractable.
The 100-step budget and checkpoint selection. The paper follows the OPSD protocol exactly: all post-training methods use 100 on-policy distillation steps, and the best checkpoint within this budget (measured by Average@12 on the evaluation benchmarks) is reported. This means the controller must learn a useful policy within 100 steps—a very tight budget. The controller likely makes only a handful of decisions over the entire training run (if $H + L$ is, say, 25 steps, the controller updates roughly 4 times). This is feasible because (a) the training state evolves smoothly, so a policy learned from early episodes can generalize to later ones, (b) the Beta parameterization provides a strong inductive bias (unimodal, bounded), and (c) the REINFORCE updates with normalized advantages can make meaningful parameter changes in few steps.
Summary of Design Choices and Their Justifications
- Continuous
$\alpha$over discrete levels: avoids arbitrary binning, allows smooth policy gradients, and captures the continuous nature of "how much reasoning to reveal." - Prefix truncation over random subset: preserves the causal, scaffolding structure of chain-of-thought reasoning—early steps naturally lead to later ones, so truncating from the end is semantically coherent while random subsets would be nonsensical.
- Answer always preserved: isolates exposure control to reasoning detail while keeping the teacher grounded in the correct outcome, preventing degenerate
$\alpha = 0$from meaning "no supervision at all." - Beta distribution over Gaussian-softmax or categorical: natural support on
$[0, 1]$, interpretable parameters (mean and concentration), standard in the probability modeling literature, and avoids boundary gradient issues. - Unimodality constraint (
$a, b > 1$): prevents the policy from oscillating between extremes, encoding the prior that the optimal exposure is a single value at each training state rather than a mixture of extremes. - Hold-window protocol: creates coherent episodes for credit assignment by letting the student's response to a single
$\alpha$accumulate over multiple optimization steps, making the reward signal less noisy. - Lookahead window of 20 steps: long enough to capture delayed effects of exposure on student learning, short enough to maintain temporal credit locality and avoid conflating different controller decisions.
- Learning progress reward (
$G_{\text{lp}}$): directly measures what we care about—whether the student is improving—rather than using a proxy like mismatch or loss magnitude that can be gamed. - Max(0,·) clipping in
$G_{\text{lp}}$: avoids penalizing the controller for stochastic loss increases, focusing the reward signal on consistent improvements. - Teacher-grounded credit (
$G_{\text{gt}}$): prevents the controller from learning to set$\alpha$low simply because low exposure makes the teacher targets trivially easy to match—the teacher must still predict the reference solution. - Mismatch as state, not reward: avoids the degenerate strategy of always reducing
$\alpha$to mechanically lower KL; the controller can observe mismatch to inform its decisions but is not rewarded for minimizing it directly. - REINFORCE over actor-critic: simplicity (no learned value function) with acceptable variance given the compact state space and Beta action space; avoids introducing a second learned component that could itself be unstable.
- Entropy cap over entropy bonus: allows the policy to concentrate at convergence (reducing noise in the teacher signal) while preventing pathological over-exploration early in training; the one-sided penalty is a soft constraint that doesn't force the policy to maintain exploration when it has converged.
- Global exposure over per-sample: deliberately scoped as a simpler first step; the training-state-level controller establishes that adaptive exposure is beneficial without requiring per-sample difficulty estimation, and the difficulty-bin analysis motivates per-sample control as future work.
- Same student rollout protocol as OPSD: isolates the effect of exposure control to the teacher side, ensuring that any performance difference is attributable to the learned
$\alpha$rather than to changes in sampling, prompting, or optimization.
4. Key Insights and Innovations
Innovation 1: Teacher-Side Exposure Mismatch as a New Diagnostic Category
The paper's most fundamental conceptual contribution is the identification and formalization of teacher-side exposure mismatch as a distinct phenomenon orthogonal to the student-side mismatch that prior work addressed. Before this paper, the self-distillation literature recognized exactly one type of mismatch: the distribution gap between the student's training trajectories and its inference-time behavior. OPSD and its lineage solved this by making rollouts on-policy—the student trains on its own sampled outputs. ATESD's diagnostic insight is that fixing the student side reveals a second, previously invisible mismatch operating at the teacher level.
This is not an incremental refinement. It is a category shift in how we think about what makes privileged supervision difficult to absorb. Prior work implicitly treated the teacher as an oracle whose information was uniformly valuable: the more privileged context, the better the supervision. The paper's controlled fixed-exposure sweep (Figure 2) provides empirical evidence that this oracle assumption fails systematically. The finding that intermediate exposure (α = 0.5) outperforms full exposure (α = 1.0) across multiple seeds is genuinely surprising under the oracle model—it means that withholding privileged information from the teacher produces better student outcomes than giving the teacher everything. This cannot be explained by student-side mechanisms (the student rollout protocol is identical across all α values—Section 3.2 explicitly isolates this variable), which forces the conclusion that the teacher's information level is itself a supervision-quality parameter with a non-monotonic relationship to student learning.
The formalization via $\mathcal{M}(\alpha)$ (Equation 4) gives the concept analytical traction beyond the empirical sweep. By defining teacher–student mismatch as the expected per-token KL between the α-conditioned teacher and the problem-only student along on-policy trajectories, the paper creates a measurable quantity that (a) grows monotonically with α (verified in Figure 2B), (b) isolates the teacher-side contribution (since $p_S$ is held constant across α), and (c) provides a diagnostic that can be tracked during training (it becomes one of the controller's state inputs) without requiring ground-truth performance evaluation. This transforms exposure mismatch from a vague intuition ("too much information might confuse the student") into an operationalized concept that can be measured, predicted, and controlled.
The significance extends beyond ATESD itself. The paper's framing positions teacher-side exposure as a general design axis that all distillation methods implicitly set to a fixed value. Future work on self-distillation—whether on-policy or off-policy, whether using KL or alternative losses, whether for reasoning or other domains—must now contend with the question "what should the teacher's information level be?" rather than assuming the answer is "everything." This is analogous to how the recognition of student-side mismatch changed distillation from a static offline process to an adaptive on-policy one; identifying the teacher-side analogue opens a parallel trajectory.
Innovation 2: Reframing Teacher Exposure as a Learnable Control Variable Rather Than a Hyperparameter
The paper's second conceptual move is to redefine what kind of thing teacher exposure is. In all prior work, exposure was a fixed hyperparameter—chosen once before training begins and held constant throughout, usually at the implicit default of 1.0. ATESD reclassifies it as a training-time control variable: something that should change during training in response to the student's evolving capability, and whose value should be learned from feedback rather than set by human intuition.
This reframing matters because it shifts the optimization problem. If exposure is a hyperparameter, the search space is a single scalar (or a discrete grid, as in Figure 2), and the optimization is done once through grid search or tuning. But the fixed-exposure sweep reveals that no single α is optimal across all conditions: Figure 2C shows that easy, medium, and hard problems prefer different exposure levels under even a coarse grid. This implies that a global static α—even the best one found by expensive sweep—is a compromise that leaves performance on the table for every difficulty tier except the one it happens to match. The reframing to a control variable converts this from a hyperparameter-tuning problem (find the single best α) into a policy-learning problem (learn $\pi(\alpha \mid \text{state})$ that selects the right α for the right condition).
The Beta policy parameterization (Equation 7) is central to this reframing. By modeling the controller's output as a distribution over α rather than a point estimate, the paper treats exposure selection as decision-making under uncertainty—the controller expresses both a preference (the Beta mean) and a confidence (the Beta concentration). This is qualitatively different from learning a deterministic α schedule (e.g., an annealing function from high to low exposure). A schedule encodes a fixed assumption about how training should evolve; a policy conditions on what training actually looks like. The learned Beta distribution in Figure 4B demonstrates this difference empirically: rather than following a monotonic trajectory toward either boundary, the policy concentrates in an interior region that varies with training state. The controller is not learning that "exposure should decrease over time"; it is learning that "when mismatch is high and student confidence is low, use intermediate exposure; when mismatch stabilizes, adjust accordingly." This state-conditioned behavior cannot be replicated by any fixed schedule, no matter how carefully tuned.
The comparison to prior adaptive distillation work sharpens this innovation. Methods like dynamic temperature scheduling (Li et al., 2023; Matsuyama et al., 2025) or curriculum-based example ordering (Bengio et al., 2009) adapt how the student receives the teacher's signal—the temperature softens or sharpens the same targets, the curriculum reorders the same examples. ATESD adapts what information the teacher signal contains in the first place. This is a more fundamental intervention because it changes the target distribution $p_T^\alpha$ itself, not just how the student is optimized against it. The teacher-grounded credit score $G_{\text{gt}}$ in the reward function (Equation 8) makes this distinction concrete: it verifies that the α-chosen teacher still predicts the reference solution, ensuring that the adaptation modulates information content without breaking the connection to ground truth.
Innovation 3: Delayed Credit Assignment for Meta-Optimization in On-Policy Distillation
The third innovation is methodological but has conceptual implications beyond ATESD. The paper identifies a credit assignment problem specific to meta-optimization in on-policy distillation: when a controller chooses a parameter (exposure α) that affects the teacher's targets, the effect on student learning is not visible in the immediate loss change. High-exposure decisions may produce large immediate KL (making them appear "bad" to a myopic reward) but unlock faster learning later; low-exposure decisions may produce small immediate KL (appearing "good") but stall progress. The paper's solution—a two-timescale schedule with a discounted learning-progress reward computed over a lookahead window—is an instance of a broader principle: meta-optimization in self-play systems requires credit assignment over the timescale of the inner optimizer's dynamics, not the outer controller's decision interval.
The ablation in Table 2A provides controlled evidence for this claim. Immediate one-step feedback (which is what a naive implementation would use—reward the controller based on the loss change at the next student step) reaches only 52.22 Average@12 on AIME 2024. Simply introducing short-horizon delayed credit raises this to 56.11. Extending to discounted lookahead further improves to 58.06. Adding the teacher-grounded term reaches 59.17. The gap between immediate and delayed feedback is 6.95 points—larger than the gap between ATESD and the OPSD baseline (1.97 points on this benchmark at 1.7B). This means that how you credit the controller's decisions is as consequential as whether you use a controller at all. The credit assignment structure is not an implementation detail; it is the enabling mechanism.
This finding has implications for any system that learns to modulate its own training process. The fundamental challenge is that the controller operates on a slower timescale than the student, and the student's response to a controller action unfolds over multiple optimization steps. A single-step reward conflates the controller's action quality with the student's stochastic minibatch noise and transient optimization dynamics. The lookahead window (L = 20 steps) effectively applies a low-pass filter: by summing discounted progress over many steps, the reward averages out high-frequency noise while retaining the medium-frequency signal of genuine learning improvement. The max(0,·) clipping on individual loss changes further filters the signal by only rewarding positive progress, treating loss increases as uninformative noise rather than negative evidence about the controller's decision. This is a specific design choice motivated by the on-policy distillation setting—SGD noise causes frequent loss fluctuations that would dominate the reward if negative changes were penalized—but the principle of filtering the reward signal to match the timescale of the inner optimizer's dynamics generalizes.
The teacher-grounded credit score $G_{\text{gt}}$ adds a second filtering mechanism: it prevents the controller from learning a degenerate policy that maximizes learning progress by minimizing the teacher's information content. Without $G_{\text{gt}}$, the controller could set α very low, making the teacher targets trivially close to the student's current distribution, producing steady but meaningless loss decreases (the student matches the teacher effortlessly but learns nothing about the reference solution). $G_{\text{gt}}$ gates this by requiring that the α-chosen teacher still assigns high probability to reference tokens—a constraint that can only be satisfied when α is high enough for the teacher to actually predict the correct solution. The two reward terms together encode the dual objective: learn quickly (progress) and learn the right thing (grounding). This dual-reward structure is not specific to exposure control; it applies to any meta-optimization where the controller can reduce the difficulty of the inner optimization problem by removing useful information from the supervision signal.
Innovation 4: Evidence That Full Teacher Exposure Is Not Only Suboptimal But Systematically Counterproductive
While Innovations 1–3 describe the conceptual framework, the fourth innovation is an empirical finding with significant implications for practice: full teacher exposure is not merely suboptimal in a "could be better" sense—it actively degrades performance relative to intermediate exposure, and the degradation mechanism (monotonic mismatch growth) is systematic rather than anecdotal.
The fixed-exposure sweep in Figure 2 makes this case with unusual clarity for an ML paper. Three independent seeds all show the same pattern: accuracy peaks at α = 0.5 and declines at α = 1.0. This is not a noisy result where full exposure is sometimes best; it is a consistent finding that the default used by every prior self-distillation method is measurably worse than a simple intermediate alternative. Figure 2B provides the mechanistic explanation: both mismatch proxies (on-policy KD loss tail and top-1 disagreement) grow monotonically with α. More exposure always means more mismatch; the question is only whether the mismatch is productive (the student can bridge the gap and learn) or counterproductive (the gap exceeds the student's current capacity, producing targets that are noise rather than guidance). The accuracy curve shows that the transition from productive to counterproductive happens somewhere between α = 0.5 and α = 1.0 for this model and dataset combination, meaning the full-exposure default operates deep in the counterproductive regime.
This finding is significant because it challenges a deeply held intuition in the distillation literature: that more information from the teacher is always better. That intuition is correct in the limit of an infinitely capable student that can absorb any supervision signal, but it breaks down when the student has finite capacity and the teacher's privileged information creates targets that require capabilities the student has not yet developed. The paper's framing of this as a "supervision-side analogue of rollout mismatch" is apt: just as training on off-policy trajectories creates targets the student would never generate, training with full-exposure teacher targets creates supervision the student cannot yet use. Both are forms of distribution mismatch, operating at different levels of the distillation pipeline.
The difficulty-bin analysis in Figure 2C adds critical nuance that prevents oversimplification. Easy problems prefer high α, medium problems prefer intermediate α, and hard problems prefer low α (under the tested grid). This means the optimal exposure is not universally "less than full"—it depends on the relationship between problem difficulty and student capability. Full exposure is appropriate when the student can already produce reasoning close to the reference (easy problems, early success); it becomes counterproductive when the reference reasoning exceeds what the student can meaningfully engage with. This difficulty-dependence is what motivates treating exposure as adaptive rather than fixed, and it explains why a single static α (even the best one, α = 0.5) leaves performance on the table: it must compromise between the different optimal values for easy, medium, and hard problems within the training distribution.
This finding also provides a retrospective explanation for patterns in prior work. OPSD's strong performance despite using full exposure suggests that on-policy rollouts partially mitigate the exposure mismatch problem—when the student generates its own trajectories, the worst-case misalignment between student outputs and teacher targets is bounded because the student's rollouts are at least within its own capability range. But the mitigation is incomplete, as ATESD's improvements over OPSD demonstrate. This implies that the exposure mismatch problem is most severe in off-policy distillation settings (where the student is trained on reference trajectories it cannot generate), which aligns with the observation that on-policy methods reliably outperform off-policy ones for reasoning tasks. The exposure lens provides a unified explanation: on-policy rollouts reduce student-side mismatch; adaptive exposure reduces teacher-side mismatch; both address different aspects of the same underlying problem—the capability gap between what the student can do and what the supervision assumes it can do.
5. Experimental Analysis
Evaluation Methodology
-
Dataset. All experiments use the OpenThoughts mathematical reasoning corpus (Guha et al., 2025) for training, following the OPSD protocol. Evaluation is on three competition-level mathematical reasoning benchmarks: AIME 2024, AIME 2025, and HMMT 2025. The paper does not report exact dataset sizes, but these are standard benchmarks with known question counts (AIME has 30 questions per year, HMMT typically has 30–40). The training corpus is the same mathematical reasoning dataset used by Zhao et al. (2025) in the original OPSD paper, ensuring comparability of results.
-
Base model(s). The paper uses the Qwen3 model family (Qwen Team, 2025) at three scales: Qwen3-1.7B, Qwen3-4B, and Qwen3-8B, all in their instruct-tuned variants. These models are chosen because they represent a range of capability levels (from small and relatively weak on competition math at 1.7B to moderately capable at 8B) and because Qwen3 is a contemporary open-weight model family that serves as a strong baseline for post-training reasoning research. The paper states that all post-training methods use the same 100-step on-policy distillation budget, matching the OPSD reporting protocol exactly for fair comparison.
-
Metrics. The primary metric is Average@12, defined as the mean accuracy over 12 independently sampled completions per problem under the OPSD sampling protocol. Accuracy is binary per problem—the model's final answer must match the ground-truth answer exactly. For each benchmark, the best Average@12 score across checkpoints within the 100-step training budget is reported, following the within-budget checkpoint selection convention established by Zhao et al. (2025). The "Average" column in Table 1 is the arithmetic mean of Average@12 across the three benchmarks, providing a single summary metric per model scale.
-
Baselines. Four baselines are reported in Table 1, with numbers taken directly from Zhao et al. (2025) to match the original reporting convention:
- Base (Instruct): the unmodified Qwen3 instruct-tuned model without any post-training, serving as the lower bound for what post-training adds.
- + SFT: supervised fine-tuning on the training corpus (Ouyang et al., 2022), representing the standard non-distillation fine-tuning approach.
- + GRPO: Group Relative Policy Optimization (Shao et al., 2024), a reinforcement learning method with verifiable rewards—this is the RL baseline that does not use privileged teacher supervision.
- + OPSD: On-Policy Self-Distillation (Zhao et al., 2025), the direct baseline that ATESD builds on top of, which uses the full-reference teacher with α = 1.0. The paper reports ATESD with mean ± standard error across multiple seeds (three seeds, based on the error bars and the fixed-exposure sweep in Figure 2 which uses three seeds), though the exact number of seeds for the main results is not stated explicitly in the text—the ± values in Table 1 imply at least two runs.
-
Generation budget / compute accounting. The paper measures compute in terms of the number of on-policy distillation steps, holding this constant at 100 steps across all methods (Section 5.1). This is a training-compute metric rather than an inference-compute metric. Within the 100-step budget, methods may differ in how they use each step—ATESD adds the controller overhead, but the paper argues this is negligible (a 2-layer MLP with a few hundred to a few thousand parameters, compared to the billions of parameters in the student model). The controller's REINFORCE updates happen infrequently (roughly every H + L steps, likely every 20–30 steps if H is small and L = 20), adding minimal wall-clock time. No FLOPs-matched comparison across methods is performed—the fairness argument rests on equal training steps and identical student optimizer configuration. At inference time, all methods use the same sampling protocol (12 completions per problem under the OPSD sampling configuration), so inference cost is identical across baselines and ATESD.
-
Cross-validation / statistical protocol. The paper does not employ explicit cross-validation. The 100-step within-budget checkpoint selection convention serves as a form of temporal holdout: the best checkpoint within the training budget is selected based on Average@12 on the evaluation benchmarks, and that score is reported. This means the evaluation is on the same benchmarks used to select checkpoints, which could introduce optimistic bias—the reported scores are the maximum within a window of checkpoints evaluated on the same test set. However, this is the identical protocol used by OPSD, and the paper adopts it deliberately for comparability. The error bars (± values) in Table 1 indicate reporting of mean and standard error across multiple training runs, providing a measure of seed-to-seed variance. The fixed-exposure sweep in Figure 2 explicitly uses 3 seeds with mean ± s.e.m. reported, establishing the statistical variation expected under this setup.
Main Quantitative Results
ATESD vs. Baselines Across Model Scales
The primary comparison is in Table 1, which reports Average@12 on AIME 2024, AIME 2025, and HMMT 2025 across three model scales. The headline finding is that ATESD achieves the highest average performance at every scale under the OPSD reporting protocol:
Qwen3-1.7B: ATESD reaches 44.35 Average across the three benchmarks, improving over OPSD (43.4) by +0.95 points. The per-benchmark breakdown shows ATESD at 59.17 ± 0.8 on AIME 2024 (vs. OPSD's 57.2, +1.97), 44.72 ± 0.28 on AIME 2025 (vs. 43.9, +0.82), and 29.17 ± 1.37 on HMMT 2025 (vs. 29.2, essentially tied at −0.03). The AIME improvements are the main drivers; HMMT is within noise. GRPO reaches 37.7 Average, and SFT underperforms the base instruct model (35.8 vs. 37.1), which the OPSD paper also observed and attributed to SFT's known limitations on hard reasoning tasks without on-policy or privileged supervision.
Qwen3-4B: ATESD reaches 65.65 Average—the strongest result for this model scale. This represents a +2.05 point improvement over OPSD (63.6) and a +2.95 point improvement over GRPO (62.7). On AIME 2024, ATESD scores 78.06 ± 0.43 (vs. OPSD 76.4, +1.66); on AIME 2025, 71.39 ± 0.48 (vs. 68.3, +3.09); on HMMT 2025, 47.50 ± 0.89 (vs. 46.1, +1.40). ATESD improves OPSD on all three benchmarks at this scale, with the largest absolute gain on AIME 2025. The gap between ATESD and GRPO is notably larger at 4B (+2.95) than at 1.7B (+0.4, comparing the Average column for ATESD vs. GRPO: 44.35 - 37.7 = 6.65, though this includes the base model gap—the more relevant comparison is ATESD minus OPSD versus OPSD minus GRPO, showing that ATESD extracts more from the distillation paradigm than OPSD alone does over RL).
Qwen3-8B: ATESD reaches 67.13 Average, improving over OPSD (64.8) by +2.33 points and over GRPO (64.0) by +3.13 points. On AIME 2024: 80.56 ± 0.70 (vs. OPSD 77.8, +2.76); on AIME 2025: 72.50 ± 0.32 (vs. 70.8, +1.70); on HMMT 2025: 48.33 ± 0.28 (vs. 45.8, +2.53). The gain is largest on HMMT 2025 in relative terms (+5.5% relative improvement over OPSD), though the paper does not compute relative percentages.
Scale-dependent pattern. The improvement over OPSD grows with model scale: +0.95 at 1.7B, +2.05 at 4B, +2.33 at 8B. The paper interprets this in Section 5.2: exposure control becomes more valuable when "the student has enough capacity to exploit privileged teacher context but still needs that context to be regulated." At 1.7B, the student's capacity is more limited, so the headroom between useful privileged information and what the student can absorb is narrower—even the full-exposure teacher provides targets that are relatively close to the student's ceiling, leaving less room for exposure control to help. At 4B and 8B, the student can benefit from richer supervision but can also be harmed by supervision that exceeds its current level, making adaptive exposure more impactful.
Comparison to GRPO. ATESD's advantage over GRPO—the RL baseline without privileged teacher supervision—is substantial and consistent: roughly +6.7 Average at 1.7B (44.35 vs. 37.7), +2.95 at 4B (65.65 vs. 62.7), and +3.13 at 8B (67.13 vs. 64.0). This gap represents the combined benefit of (a) privileged teacher supervision (the OPSD advantage over GRPO) plus (b) adaptive exposure control (the ATESD advantage over OPSD). The diminishing relative gap at larger scales (the 4B gap is smaller than the 1.7B gap) is driven by the base model's improving capability—at 8B, the instruct model already reaches 61.8 Average, and GRPO pushes this to 64.0, meaning the room for distillation to add value shrinks as the base model approaches ceiling performance on these benchmarks.
Exposure Control Reduces Mismatch on Positive Trajectories (Figure 4A)
Figure 4A presents a controlled diagnostic: fixing the problem, the student rollout, and the scoring positions, the paper compares the token-level teacher–student KL under full exposure (α = 1.0) versus under the ATESD-chosen exposure (α = 0.3, reproduced from a training run). On a positive trajectory—one where the student is already producing a useful continuation—exposure control substantially reduces supervision mismatch:
- Mean KL drops from 0.0136 (full exposure) to 0.0061 (α = 0.3), a reduction of 55%.
- Max KL drops from 0.2432 to 0.0645, a reduction of 73%.
- The largest spike at token position 26 drops from 0.2432 to 0.0098, a reduction of 96%—essentially eliminating a major mismatch event.
This is a mechanistically important result because it isolates the teacher-side effect. The student rollout is identical in both conditions—the same tokens, same positions, same student model. The only difference is how much privileged reasoning the teacher sees. The reduction in KL therefore cannot be attributed to the student generating an easier trajectory or to any change in the student's sampling distribution. It comes purely from the teacher having less privileged context, which makes its token-level predictions closer to what the student naturally produces on a trajectory that is already headed toward the correct answer.
The paper frames this as evidence that "full privileged exposure can over-constrain positive trajectories"—when the student is already on the right track, the full-reference teacher sees a much more complete and structured reasoning path and expects the student to match it token-by-token, creating large KL at positions where the student's phrasing or intermediate steps diverge from the reference even though they are substantively correct or on the right path. Reducing exposure relaxes this over-constraint, making the same useful trajectory easier to distill.
Critically, this result does not claim that lower mismatch is universally better. On negative trajectories (where the student is generating incorrect reasoning), higher exposure might be beneficial because the teacher needs more privileged context to provide corrective signal. The paper does not show the equivalent plot for a negative trajectory, which limits the interpretation to positive trajectories only—a gap noted in the Critical Assessment below.
Learned Exposure Policy Converges to Interior Distribution (Figure 4B)
Figure 4B visualizes the evolution of the Beta exposure policy over training steps. Three patterns are visible:
-
Early training (roughly steps 0–30): The Beta distribution is broad, with substantial probability mass across much of the [0, 1] range. The controller is exploring different exposure levels to gather reward signal.
-
Mid-to-late training (roughly steps 40–80): The distribution concentrates, with the mode shifting to an interior value (roughly 0.3–0.5 based on the density plot). The concentration increases (the distribution becomes narrower), indicating the controller is gaining confidence about which exposure region is productive given the current training state.
-
No collapse to boundaries: The distribution does not collapse to α ≈ 0 (answer-only) or α ≈ 1 (full exposure). It converges to an interior mode, consistent with the fixed-exposure sweep finding that intermediate values outperform extremes. If the controller had learned a trivial rule (e.g., "always use low α to make loss decrease quickly" or "always use high α because that's the default"), the policy would collapse toward one boundary. The interior concentration suggests the controller is making state-dependent tradeoffs rather than rediscovering a fixed heuristic.
The paper argues this is important evidence for its main claim: "if exposure control merely rediscovered a trivial always-low or always-full rule, the learned policy would collapse toward one boundary and reduce to another fixed heuristic. Instead, the mass remains in a usable middle regime." The evolving concentration also shows that the controller is not just injecting noise—it starts broad (exploration), then narrows (exploitation as it identifies productive exposure ranges), consistent with the entropy cap design that allows concentration at convergence.
Delayed Credit Assignment Is Necessary for Learning Exposure (Table 2A)
Table 2A ablates the credit assignment mechanism by progressively adding the components of ATESD's delayed reward, all evaluated on AIME 2024 Average@12 at the 1.7B scale:
- Immediate one-step feedback: 52.22 Average@12. This is the naive approach—reward the controller based on the loss change at the very next student step. Performance is substantially below ATESD (59.17) and even below the best fixed exposure (57.44, from Table 2B).
- Short-horizon delayed credit: 56.11. This introduces a delay but without the lookahead discounting and teacher-grounded term. It already recovers most of the gap to the best fixed exposure, demonstrating that delayed credit alone is a significant improvement over myopic feedback.
- Discounted lookahead (learning progress only, no teacher-grounded term): 58.06. Adding γ-discounting to the learning progress sum over the lookahead window further improves performance, bringing it close to ATESD's full reward.
- Full delayed reward (ATESD, with teacher-grounded score): 59.17. The addition of
G_gtprovides the final increment, reaching the full ATESD performance.
The most striking gap is between immediate one-step (52.22) and short-horizon delayed (56.11)—a difference of 3.89 points, which is larger than the final gap between the best fixed exposure and ATESD (1.73 points). This means the choice of credit assignment mechanism matters more than whether exposure is learned or fixed. Immediate feedback is actively harmful: it performs worse than simply using the best fixed α (57.44) because the controller learns to chase one-step loss decreases rather than making decisions that benefit learning over longer horizons.
The paper interprets this progression as evidence that "delayed credit [is] an enabling mechanism for learning exposure, not as an incidental implementation detail." The structure of the reward matters more than the decision to learn exposure at all—without proper credit assignment, learned exposure underperforms even a fixed heuristic.
Learned Exposure Outperforms Fixed and Uncontrolled Alternatives (Table 2B)
Table 2B compares different exposure policies on AIME 2024 Average@12 at 1.7B:
- OPSD full exposure (α = 1.0): 57.20. This is the baseline that ATESD is designed to improve upon.
- Best fixed α = 0.5: 57.44. The optimal value from the fixed-exposure sweep in Figure 2A, providing a +0.24 improvement over the OPSD default. This is a minimal gain, confirming that simply switching to a better fixed α provides only marginal benefit over the full-exposure default at this scale.
- Stochastic exposure (uncontrolled random α): 54.94. This performs substantially worse than any fixed α, showing that merely adding stochasticity to the teacher context—without training-state-conditioned adaptation—is harmful. Random exposure introduces noise into the supervision signal without the benefit of matching exposure to training state.
- Learned policy (ATESD): 59.17. This outperforms the best fixed exposure by +1.73 points and OPSD full exposure by +1.97 points, demonstrating that the gain comes from feedback-driven adaptation to training state, not from merely avoiding full exposure or injecting noise.
The pairwise comparisons rule out two alternative explanations for ATESD's improvement:
- It's not just finding a better fixed α: the best fixed α (57.44) is 1.73 points below the learned policy. If the learned policy were simply converging to the optimal fixed α, it would match the best fixed α performance; the gap shows it is doing something that a static choice cannot replicate—namely, adapting α as training progresses.
- It's not just beneficial noise: uncontrolled stochastic exposure (54.94) performs worse than any fixed α (including the full-exposure OPSD default at 57.20). This means that the controller is not helpful merely because it varies α; the variation must be informed by training state and reward feedback to be beneficial. Random variation in the teacher's privileged context degrades performance by introducing unpredictable supervision quality.
The paper frames this as evidence that "the useful signal is feedback-driven adaptation of α to the training state, consistent with the learned Beta-policy evolution in Figure 4." The controller is not a post-hoc selection mechanism that picks the best fixed α after observing training outcomes; it is continuously adjusting α based on the training state, which allows it to provide different exposure levels at different stages of training—something no fixed α can do.
Ablation Studies and Robustness Checks
Ablation of credit assignment structure (Table 2A): As described above, progressively removing the delayed components of the reward function—the lookahead window, the discounting, the teacher-grounded term—systematically degrades performance from 59.17 (full ATESD) to 52.22 (immediate one-step). This ablation has three important implications that are not fully spelled out in the paper but are directly inferable from the numbers:
First, the teacher-grounded term G_gt provides a modest but consistent benefit (+1.11 over discounted lookahead without it, comparing 59.17 to 58.06). This confirms that without G_gt, the controller has some tendency to choose α values that reduce the teacher's informativeness about the reference solution—the learning-progress-only reward can be partially gamed. However, the fact that discounted lookahead without G_gt still reaches 58.06 (above the best fixed α) suggests this gaming is partial, not catastrophic—pure learning progress tracking already provides substantial signal for good exposure decisions.
Second, the discount factor γ (comparing short-horizon delayed at 56.11 to discounted lookahead at 58.06) provides roughly 2 points of improvement. The discount weights earlier progress more heavily, which aligns credit more closely with the controller's recent decision and reduces the influence of learning dynamics caused by subsequent controller actions. This supports the paper's claim that temporal credit locality (reward should be concentrated shortly after the action, not diluted across the full remainder of training) is important for learning.
Third, the gap between immediate (52.22) and short-horizon delayed (56.11) is the largest single jump in the ablation, suggesting that the fundamental problem with myopic credit is not the lack of discounting or teacher grounding, but the complete absence of a temporal window—the controller simply cannot see the consequences of its actions in a single step. This is the most robust finding in the ablation.
Ablation of exposure policy type (Table 2B): As described above, this ablation compares fixed, stochastic, and learned policies. An additional interpretation not fully explored in the paper: the best fixed α (α = 0.5, scoring 57.44) is only marginally better than full exposure (α = 1.0, scoring 57.20)—a +0.24 difference that may be within seed-to-seed noise given the error bars on ATESD (e.g., ±0.8 on AIME 2024 at 1.7B). This means that for this model scale and benchmark, the optimal static α provides almost no benefit over the OPSD default. The benefit of ATESD (+1.97 over OPSD) therefore comes almost entirely from adaptivity—changing α during training—rather than from the improved average α level. This is a stronger claim than the paper makes explicitly: if true, it means that even a carefully tuned α schedule (e.g., an annealing function from high to low α) would not match ATESD unless it perfectly replicated the training-state-conditioned choices, which is unlikely without a learned controller.
Positive-trajectory mismatch reduction (Figure 4A): This is primarily a mechanistic diagnostic, not an ablation. However, it implicitly ablates the effect of α on supervision mismatch by holding all other variables constant. The finding that α = 0.3 dramatically reduces KL spikes compared to α = 1.0 on the same trajectory demonstrates that exposure—independent of student sampling, problem difficulty, or training stage—directly controls the mismatch magnitude. This provides causal evidence for the paper's claim that high exposure on positive trajectories produces "avoidable teacher-context mismatch rather than a need for stronger correction."
Learned policy evolution (Figure 4B): This visualization serves as a robustness check that the controller is actually learning a nontrivial policy rather than degenerating. The key robustness check is that the policy does not collapse to α ≈ 0 or α ≈ 1, which would indicate that the controller had learned a degenerate strategy (minimize α to reduce mismatch and boost learning progress, or maximize α to boost teacher grounding). The interior concentration is evidence that the dual-reward structure successfully balances these competing pressures. However, the paper does not show what happens if either reward term is removed—i.e., does the policy collapse to a boundary when only G_lp or only G_gt is used? This would be a cleaner demonstration that both terms are necessary for interior convergence, and its absence is a limitation.
Scale consistency of improvement: The paper does not present this as a formal ablation, but the per-scale results in Table 1 serve as an implicit robustness check. The improvement over OPSD is positive at all three model sizes (1.7B, 4B, 8B) and across all three benchmarks (with the minor exception of HMMT at 1.7B, where the improvement is essentially zero at −0.03). The consistency of improvement across scales addresses the concern that exposure control might only help at a specific model capacity: the monotonic increase in benefit with scale (+0.95 → +2.05 → +2.33 Average points) suggests a systematic relationship between model capability and the value of adaptive exposure, rather than a fluke at a single scale.
Missing ablation: effect of hold window length H. The paper describes the hold-window protocol but does not report an ablation over H (the number of student steps for which α is held fixed). This is a significant omission because H controls the tradeoff between controller responsiveness (small H allows faster adaptation to changing training state) and credit assignment quality (large H gives each exposure decision more time to show effects). The optimal H likely interacts with the lookahead horizon L and the total training budget (100 steps). Without this ablation, it's unclear whether the reported results are sensitive to the specific H used, and whether practitioners adopting ATESD need to tune H for their setting.
Missing ablation: the +1 offset in the Beta parameterization (unimodality constraint). The paper describes the Beta controller as using a_t, b_t = 1 + softplus(f_φ(s_t)) to enforce a, b > 1 and hence unimodality. Whether this constraint matters—i.e., whether removing the +1 and allowing bimodal Beta distributions would change behavior—is not tested. If the controller naturally converges to unimodal behavior without the constraint, the +1 is an unnecessary restriction; if it oscillates between bimodal policies without it, the constraint is load-bearing. The ablation would clarify this.
Missing comparison: α annealing schedule vs. learned policy. The paper compares learned exposure to fixed exposure and random exposure but does not compare to a hand-designed α schedule (e.g., linear anneal from α = 1.0 to α = 0.3 over 100 steps, or a step function that switches from high to low α at a predetermined point). This is a meaningful missing baseline because a simple schedule could potentially capture the coarse trend of "high exposure when the student needs rich supervision early, lower exposure when the student is converging"—and if a schedule matches ATESD, the complexity of the learned controller would be unnecessary. The fact that the learned policy does not follow a simple monotonic trajectory (Figure 4B shows it concentrates in an interior regime rather than moving from one value to another) suggests a schedule would not replicate it, but this should be tested directly.
Missing ablation: controller input features. The paper lists six training-state statistics used as controller inputs but does not ablate which are necessary. Would a controller using only loss and mismatch EMAs perform similarly? Does removing probe-NLL or self-confidence degrade performance? Without feature ablation, it's unclear whether all six statistics are load-bearing or whether a simpler state representation would suffice. Given the paper's emphasis on the controller being "lightweight," a demonstration that even fewer features could work would strengthen the practicality argument.
Critical Assessment
Claim: "Full teacher exposure is suboptimal—intermediate exposure consistently outperforms it."
The fixed-exposure sweep in Figure 2A provides direct evidence for this claim at the 1.7B scale on AIME 2024. Three seeds all show the same pattern: accuracy at α = 0.5 exceeds accuracy at α = 1.0. The mean difference appears to be roughly 2–3 percentage points based on the y-axis scale (the exact values are not tabulated but are visually clear from the plot). However, this evidence has important boundary conditions that the paper does not fully acknowledge:
Single benchmark, single scale. The fixed-exposure sweep is only shown for AIME 2024 and Qwen3-1.7B. The paper does not report equivalent sweeps for AIME 2025, HMMT 2025, or the 4B and 8B models. Given that the main results in Table 1 show ATESD's benefit over OPSD increases with scale, it is plausible that the suboptimality of full exposure also varies with scale—at 1.7B, the limited student capacity might make intermediate exposure more strongly favored; at 8B, full exposure might be closer to optimal because the student can absorb more privileged information. The paper's claim that "full teacher exposure is suboptimal" is stated as a general finding, but the evidence is from one data point in model-scale × benchmark space. This does not invalidate the claim—the one data point is persuasive—but it means the generality of the claim rests on extrapolation from the ATESD results (which show improvement over full exposure at all scales) rather than direct sweep evidence at each scale.
Grid granularity. The sweep tests only five α values: 0, 0.25, 0.5, 0.75, and 1.0. The finding that α = 0.5 is the best among these five does not guarantee it is the global optimum. If the true optimum were at α = 0.35 or α = 0.6, the coarse grid might miss it and the gap between the best fixed α and full exposure might be larger or smaller than reported. This is a minor concern for the fixed-exposure analysis (the qualitative pattern is clear even with coarse bins) but matters for Table 2B, where "best fixed α = 0.5" at 57.44 is used as the static baseline—the true best fixed α could be slightly higher.
Seed count and error bars. Three seeds with mean ± s.e.m. is adequate for establishing the pattern direction (α = 0.5 > α = 1.0), but the error bars in Figure 2A appear tight enough that zero overlap between α = 0.5 and α = 1.0 would indicate statistical significance. Without explicit p-values or confidence intervals, the reader must infer significance from the visual separation of error bars.
Claim: "Teacher–student mismatch grows monotonically with α."
Figure 2B provides direct evidence: both proxies (on-policy KD loss tail and top-1 disagreement) increase monotonically with α. This is the most robust empirical finding in the paper—it follows directly from the formalization in Equation 4 and would be expected under any reasonable model of how privileged context affects teacher predictions. As α increases, the teacher sees more reasoning steps, which constrains its token-level predictions to be more specific (higher probability on reference-consistent tokens, lower probability on alternatives). The student sees none of this context, so its predictions remain unchanged. The KL must therefore increase with α, barring pathological cases where additional context reduces the teacher's certainty (which would require the reference reasoning to be internally contradictory or the model to have poor long-context utilization).
The claim is well-supported but the proxies used deserve scrutiny. "On-policy KD loss tail" is not defined precisely—it likely refers to the average KL over some subset of tokens (perhaps excluding the first few, or focusing on the tail of the distribution), but the exact computation is not specified. "Top-1 disagreement" likely measures the rate at which the teacher's argmax token differs from the student's argmax token, which is a discretized version of the KL trend. Both proxies are reasonable, but without formal definitions, the claim's precision is limited.
More importantly, the monotonic growth of mismatch does not by itself establish that high α is harmful—it only establishes that the teacher and student distributions diverge more. Whether this divergence is productive or counterproductive depends on whether the student can close the gap through gradient updates. The accuracy curve in Figure 2A provides the link: accuracy declines as α moves from 0.5 to 1.0, suggesting the additional mismatch in that range is counterproductive. But at very low α (moving from 0 to 0.25 to 0.5), mismatch increases while accuracy also increases, suggesting the additional mismatch in that range is productive. The relationship between mismatch and learning is therefore non-monotonic—some mismatch is necessary (the teacher must differ from the student to provide a learning signal), but too much is harmful. The paper acknowledges this implicitly by stating that "the exposure level should be chosen from training feedback instead of fixed by hand," but the claim of "monotonic mismatch growth" is sometimes presented as inherently problematic when it is actually the marginal benefit of additional mismatch that determines optimal α.
Claim: "ATESD consistently outperforms competitive self-distillation and RL baselines."
Table 1 provides support: ATESD achieves the highest Average score at all three model scales, improving over OPSD by +0.95 to +2.33 points and over GRPO by larger margins. "Consistently" is justified—the improvement is positive on 8 of 9 model-scale × benchmark combinations (the exception being HMMT at 1.7B, essentially tied). However, several qualifications are necessary:
The improvement over OPSD is modest at 1.7B (+0.95 Average). Given the error bars on ATESD at 1.7B (±0.23 Average, implying roughly ±0.4 standard deviation across the three benchmarks), the +0.95 gain is roughly 2–4 standard errors above zero—statistically meaningful but not overwhelmingly large. At this scale, a practitioner might reasonably ask whether the added complexity of the controller is justified by a sub-1-point gain. The paper's response would point to the larger gains at 4B and 8B (+2.05, +2.33) and to the fact that the controller is lightweight, but the 1.7B result alone does not make a compelling cost-benefit case.
The baseline numbers are taken from prior work, not re-run. The paper states: "Baseline numbers are from Zhao et al. [35]; ATESD is evaluated with the same within-100-step checkpoint selection convention." This is standard practice, but it means that any differences in infrastructure, random seeds, or subtle implementation details between the prior work's runs and the current paper's runs could confound the comparison. If Zhao et al. used a slightly different LoRA configuration, optimizer setting, or data preprocessing, the reported OPSD numbers might not exactly match what OPSD would achieve under ATESD's exact training setup. The paper mitigates this by claiming to match the OPSD protocol exactly, but without reproducing OPSD from scratch under identical conditions, this cannot be verified.
The evaluation benchmarks are used for checkpoint selection. The 100-step within-budget checkpoint selection convention picks the best checkpoint based on Average@12 on AIME 2024, AIME 2025, and HMMT 2025—the same benchmarks used for evaluation. This is not a train/test split; it's training-aware checkpoint selection on the test set. The reported numbers are therefore the maximum achievable within the training window, not the expected performance at a predetermined checkpoint. This inflates absolute scores for all methods (including baselines) but is most problematic if ATESD's learned controller causes it to have more variance in checkpoint quality—a method with higher variance would benefit more from picking the maximum, making the comparison unfair. The paper does not report checkpoint-to-checkpoint variance or the performance at a fixed checkpoint (e.g., step 100), which would be a cleaner measure.
No comparison to OPSD with the best fixed α. Table 2B shows that ATESD (59.17) outperforms the best fixed α (57.44) on AIME 2024 at 1.7B, but Table 1 only compares ATESD to OPSD with α = 1.0. A fairer baseline would be OPSD with α = 0.5 (the best fixed value) or with an oracle-tuned fixed α per benchmark. Since the full results in Table 1 likely use the same configuration as Table 2B (1.7B on AIME 2024 component matches), we can infer that OPSD with α = 0.5 would score higher than the reported OPSD (α = 1.0) baseline. The paper's choice to report only α = 1.0 in Table 1 makes the ATESD-to-OPSD gap appear larger than it would against the best static α. This is a standard practice—comparing against the canonical version of the baseline rather than a separately tuned variant—but it means the "improvement over OPSD" should be understood as "improvement over the default OPSD configuration" rather than "improvement over the best possible static OPSD."
Claim: "Different learning regimes prefer different exposures, motivating adaptive control."
Figure 2C provides coarse evidence: binning problems into easy, medium, and hard by difficulty (the binning method is not described in detail in the main text—it may derive from the base model's pass@1 rate or from some external difficulty metric), the best observed α among the five grid values is 1.0 for easy, 0.5 for medium, and 0.0 for hard. This is a 3 × 5 grid with what appears to be single-seed evaluation per cell (no error bars are shown in Figure 2C).
The evidence is suggestive but has significant limitations:
Coarse binning: Three difficulty bins is minimal, and it's unclear whether the bins are balanced in size or if "hard" contains very few problems (which would make the α = 0.0 finding noisy).
Grid search, not optimization: The "best observed" α is simply the maximum among five tested values. It does not establish that the true optimal α differs by difficulty—only that the best of five options differs. If the true optimum for all difficulty levels were α = 0.6 (but this was not in the grid), the apparent difficulty-dependence could be an artifact of the grid spacing interacting with different accuracy-vs-α curve shapes across difficulty bins.
Single model scale: Figure 2C is at 1.7B only. The difficulty-dependent pattern likely changes with model scale—at 8B, "hard" problems may behave more like "medium" problems at 1.7B because the 8B model has higher base capability, meaning the optimal α for "hard" problems might not be 0.0. Without scale sweeps, the difficulty-dependence finding is specific to the 1.7B setting.
Despite these limitations, the qualitative pattern is plausible and aligns with intuition: easy problems need little privileged guidance (the student already knows roughly what to do), medium problems benefit from partial guidance, and hard problems are so far beyond the student that full reasoning is unhelpful. The paper appropriately treats this as motivation for adaptive control rather than as definitive evidence of the optimal policy structure.
What the paper does not test but should
Per-sample exposure control. The difficulty-bin analysis in Figure 2C directly motivates per-sample α selection (different problems get different exposure), but ATESD uses a global α for all samples within a hold window. The paper acknowledges this as a limitation (Section 6), but a simple comparison—e.g., grouping samples by estimated difficulty and assigning group-specific α values based on the learned policy or a heuristic—would demonstrate whether the difficulty-dependence observed in the fixed-exposure sweep translates to practical per-sample gains. The absence of even a preliminary per-sample experiment leaves the strongest motivation for the paper (difficulty-dependent optimal exposure) underexploited in the method itself.
Generalization beyond competition math. All experiments are on AIME and HMMT benchmarks with the OpenThoughts mathematical reasoning corpus. The paper does not test on code generation (HumanEval, MBPP), scientific reasoning (GPQA, ARC), or general reasoning (MMLU, BBH). Self-distillation is widely used across domains, and the exposure mismatch phenomenon should logically apply wherever there is a capability gap between the teacher's privileged knowledge and the student's current skill. Without cross-domain evidence, it is unclear whether ATESD's benefits are specific to mathematical reasoning (where chain-of-thought structure is highly regular and privileged solutions follow predictable patterns) or general.
Interaction with training budget. All experiments use exactly 100 distillation steps. The paper does not test whether ATESD's benefit changes with training budget—would the advantage over OPSD grow, shrink, or stay constant if training ran for 200 or 500 steps? If exposure control primarily helps early in training (when the student is weak and needs carefully calibrated supervision) and becomes less important later (when the student has converged and can handle full exposure), then ATESD's benefit might diminish with longer training. Conversely, if exposure control helps throughout training by continually matching supervision to the student's evolving capability, the benefit might persist or grow. The 100-step budget is quite short by modern standards, and the dynamics could differ at scale.
Controller computational overhead. The paper claims the controller is "lightweight" and "intentionally small," but provides no wall-clock time measurements, FLOP counts, or memory overhead numbers. In practice, the controller adds: (a) the MLP forward pass to sample α, which is negligible, (b) the storage and computation of training-state EMAs, which is minor, (c) the REINFORCE backward pass on the controller parameters, which is also minor, and (d) the need to run teacher forward passes on reference tokens for the G_gt computation, which could be meaningful if it requires separate forward passes beyond the normal distillation computation. Without overhead measurements, the "lightweight" claim is qualitative.
The fixed-exposure sweep at 4B and 8B. The paper's central diagnostic (Figure 2) is only at 1.7B. Given that the main results show ATESD's benefit growing with scale, reproducing the fixed-exposure sweep at 4B or 8B would reveal whether the suboptimality of full exposure also grows with scale (which would explain the larger ATESD gains at larger models) or whether full exposure becomes less suboptimal at larger scales (which would make the ATESD gains more about adaptivity and less about simply avoiding full exposure). Either outcome would strengthen the paper's mechanistic narrative.
Direct evidence of the exposure mismatch mechanism during training. The paper defines $\mathcal{M}(\alpha)$ as the expected per-token KL (Equation 4) and shows in Figure 2B that mismatch proxies grow with α in a static evaluation. But it does not show how mismatch changes during ATESD training in response to the controller's decisions. Does the controller successfully keep mismatch within a productive range? Does mismatch spike when α is too high and the controller subsequently reduces α? Tracking $\mathcal{M}(\alpha_t)$ alongside the controller's α choices and student accuracy over training would provide direct evidence that the controller is responding to mismatch signals as intended, rather than learning an opaque policy that happens to improve final accuracy.
Multiple benchmark evaluation without checkpoint selection leakage. Using the same benchmarks for checkpoint selection and final evaluation is standard in this subfield (OPSD does it), but it means the reported numbers are optimistically biased. A cleaner protocol would hold out one benchmark (e.g., HMMT 2025) entirely from checkpoint selection, selecting the best checkpoint based on AIME 2024 + AIME 2025 only, and reporting HMMT 2025 as a transfer metric. This would test whether ATESD's improvements generalize to unseen benchmarks or are partially an artifact of overfitting the checkpoint selection to the evaluation set.
Summary of evidential support
The paper's strongest claims—that full teacher exposure is not reliably optimal (Figure 2A), that mismatch grows with exposure (Figure 2B), and that learned exposure control improves over the OPSD default (Table 1)—are supported by the reported experiments, with the caveats about the 1.7B specificity of the diagnostic sweep and the modest gain at the smallest scale.
The paper's deeper mechanistic claims—that the controller adapts to training state (rather than learning a fixed heuristic), that delayed credit is necessary (rather than merely helpful), and that exposure control reduces unproductive mismatch—are supported by the ablations in Figure 4 and Table 2, but the evidence is more indirect. The learned policy's interior concentration (Figure 4B) is consistent with state-conditioned adaptation but does not prove it—a policy that converges to a fixed intermediate α would show similar behavior. The delayed credit ablation (Table 2A) shows that delayed rewards outperform immediate feedback, but this could be because any smoothing of the noisy one-step loss signal helps, not because the controller is learning about delayed consequences per se. The positive-trajectory mismatch reduction (Figure 4A) shows that lower α reduces KL on a good trajectory, but does not show that this reduction causes better learning outcomes—it shows a plausible mechanism, not a demonstrated causal chain.
The most significant gap is between the motivation (difficulty-dependent optimal α, Figure 2C) and the method (global training-state-conditioned α). The evidence that exposure control matters is stronger than the evidence that ATESD's specific approach to exposure control is the right one. A simpler method—e.g., annealing α based on a hand-tuned schedule, or selecting α based on a problem-level difficulty heuristic—might achieve similar gains with less complexity. The paper does not test these alternatives, which means the contribution is better characterized as "establishing that adaptive exposure is an effective axis" rather than "establishing ATESD as the optimal method for controlling exposure." The paper's own framing in the conclusion—"establishing adaptive teacher exposure as an effective new axis for reasoning self-distillation"—is appropriately modest on this point.
6. Limitations and Trade-offs
Global Exposure Control Leaves Per-Sample Difficulty Differences Unexploited
The assumption or constraint. ATESD samples a single global exposure α_t for all training samples within a hold window, conditioning the controller only on aggregate training-state statistics. The paper acknowledges this explicitly in Section 6:
"The current controller operates at the global level, selecting a single α for all samples within a hold period. A natural extension is per-sample or difficulty-aware exposure control, where α is conditioned on problem difficulty or the student's confidence."
The controller's state s_t (Section 4.2) includes batch-aggregated quantities—EMA-smoothed loss, mismatch, exposure history, probe-NLL, and student self-confidence—none of which vary at the per-sample level. This means every problem in a minibatch receives the same truncated reference, regardless of whether the student could handle full reasoning on some problems and needs minimal guidance on others.
The consequence. The method directly contradicts its own motivating evidence. Figure 2C demonstrates that easy, medium, and hard problems prefer different exposure levels: easy problems benefit from full exposure (α = 1.0), medium problems from intermediate exposure (α = 0.5), and hard problems from minimal exposure (α = 0.0, the lowest tested). By using a single global α, ATESD forces all difficulty tiers within a hold window to share the same exposure, which must be a compromise. If the controller selects α ≈ 0.5, easy problems receive suboptimally weak supervision (the teacher withholds useful reasoning that the student could absorb), and hard problems may still receive more privileged context than they can productively use. The difficulty-bin analysis is presented as motivation for adaptive control, but the implemented controller cannot act on per-sample difficulty, leaving the most direct implication of Figure 2C unaddressed.
A concrete failure mode: in a minibatch containing both simple arithmetic problems (where the student would benefit from full reference reasoning to learn formatting and rigor) and complex competition problems (where full reference reasoning would produce unabsorbable targets), the global α must either over-expose the hard problems or under-expose the easy ones. Neither choice is optimal, and the controller's reward signal averages over both effects, potentially learning an α that is mediocre for all difficulty levels rather than optimal for any.
What evidence exists in the paper. The difficulty-bin analysis in Figure 2C is the direct evidence. On Qwen3-1.7B with AIME 2024, the best observed α among the five grid values differs by difficulty tier: 1.0 for easy, 0.5 for medium, and 0.0 for hard. The paper also notes in Section 3.2 that "the best observed grid value differs across easy, medium, and hard samples." However, no experiment tests whether a per-sample or difficulty-conditioned α controller would outperform the global controller. The paper reports only the global controller's results (Table 1) and does not include even a heuristic per-sample baseline (e.g., assigning α based on a coarse difficulty estimate). The gap between global and per-sample control is therefore unknown.
Mitigation status. The paper treats this as intentional scope limitation rather than a failure, stating that "the present method deliberately studies the simpler training-state-level controller first" (Section 6). The difficulty-bin analysis is offered as motivation for future per-sample work. No mitigation is attempted within ATESD itself—the global controller is the entirety of what is implemented and evaluated. The practical consequence for a practitioner is that ATESD leaves on the table whatever gains per-sample adaptation could provide, and the size of those gains is unmeasured. Since the fixed-exposure sweep (Figure 2A) shows a roughly 5–10 percentage point difference between α = 0.0 and α = 1.0 on AIME 2024 for the 1.7B model, the per-sample gap could be substantial if difficulty varies widely within the training distribution.
The Fixed-Exposure Diagnostic Sweep Is Only at the Smallest Model Scale
The assumption or constraint. The paper's central empirical motivation—that full teacher exposure is suboptimal and that mismatch grows monotonically with α—is established through a controlled fixed-exposure sweep on AIME 2024 using Qwen3-1.7B only (Figure 2). The paper does not report equivalent sweeps for Qwen3-4B, Qwen3-8B, or the other evaluation benchmarks. The main results in Table 1 then show ATESD improving over OPSD by +0.95, +2.05, and +2.33 Average points at 1.7B, 4B, and 8B respectively—a pattern where the benefit grows with scale.
The consequence. The paper's mechanistic narrative—that exposure control matters because full exposure produces unabsorbable targets (Figure 1A, Figure 2A–C)—is grounded in evidence from the model where ATESD provides its smallest benefit (1.7B). This creates an interpretive tension. At 4B and 8B, where ATESD's gains are larger, we do not know whether: (a) full exposure is even more suboptimal than at 1.7B, and the larger ATESD gains reflect a larger gap between the full-exposure default and the optimal α; (b) full exposure is less suboptimal at larger scales (because more capable students can absorb more privileged context), and ATESD's gains come primarily from training-state-conditioned adaptation rather than from simply being better than α = 1.0; or (c) some interaction between model capacity and the optimal α policy produces the observed scale-dependent pattern. Without fixed-exposure sweeps at 4B and 8B, the mechanism driving the scale-dependent gains is unidentified.
A practitioner cannot determine from the reported evidence whether they should expect ATESD to provide larger or smaller benefits on models larger than 8B. If hypothesis (a) is correct (more suboptimality at larger scales), the benefit might continue to grow. If hypothesis (b) is correct (less suboptimality, gains from adaptivity), the benefit might plateau or shrink as models become capable enough that full exposure is close to optimal. The paper's claim that exposure control "becomes more valuable when the student has enough capacity to exploit privileged teacher context but still needs that context to be regulated" (Section 5.2) is a post-hoc interpretation consistent with the pattern but not directly tested.
What evidence exists in the paper. The fixed-exposure sweep (Figure 2) is only at 1.7B. The main results (Table 1) show the ATESD-OPSD gap growing with scale, but these comparisons are against OPSD with α = 1.0 only—they do not reveal how the optimal fixed α or the shape of the accuracy-vs-α curve changes with scale. The paper reports no per-scale fixed-exposure data, no per-scale mismatch curves, and no per-scale difficulty-bin analysis. The scale-dependent mechanism is asserted based on the ATESD results, not tested through controlled sweeps at multiple scales.
Mitigation status. Not addressed. The paper does not acknowledge this as a limitation, and Section 6 ("Limitations and future directions") focuses on domain generalization and controller extensions rather than on the scale-specificity of the diagnostic evidence. A practitioner evaluating whether to adopt ATESD on a different model scale would need to run their own fixed-exposure sweep to determine whether the motivational pattern (suboptimality of full exposure, difficulty-dependent optimal α) holds in their setting. The paper provides no guidance for extrapolating the 1.7B diagnostic to other scales.
Difficulty Estimation Cost Is Unaccounted For in the Headline Comparisons
The assumption or constraint. While ATESD itself does not require explicit difficulty labels (the controller operates on training-state statistics, not per-sample difficulty estimates), the motivation and analysis rely heavily on difficulty-binned evaluations. Figure 2C requires binning problems into easy, medium, and hard tiers. The paper does not specify the binning method in the main text, but any reasonable approach—computing pass@1 from the base model using many samples, using an external difficulty metric, or using the PRM-style score from the teacher—requires computation that is not included in the 100-step distillation budget used for all method comparisons. Additionally, the fixed-exposure sweep in Figure 2A (testing five α values across three seeds) requires several times the compute of a single training run.
The consequence. The paper's primary empirical claim is that ATESD improves over OPSD within a fixed 100-step training budget. However, the design of ATESD—and the selection of its hyperparameters, reward structure, and controller architecture—depends on diagnostic experiments (the fixed-exposure sweep, the difficulty-bin analysis) whose compute cost is externalized from the headline comparison. If a practitioner wanted to apply ATESD to a new model, dataset, or domain, they would need to: (a) determine whether full exposure is suboptimal for their setting (requiring a fixed-exposure sweep comparable to Figure 2A), (b) determine whether the optimal α varies with difficulty (requiring a difficulty-bin analysis comparable to Figure 2C), (c) tune the controller hyperparameters (hold window H, lookahead L, reward weight λ_gt, entropy target, learning rate), and (d) potentially tune the controller input features. The total compute for this diagnostic and tuning phase could exceed the cost of multiple OPSD training runs.
The paper does not provide transferable guidance—e.g., "these controller hyperparameters work across model scales and benchmarks" or "the optimal fixed α at 1.7B predicts the optimal fixed α at 4B." Without such guidance, adoption requires re-running much of the paper's diagnostic pipeline. This is a practical limitation for practitioners who want a drop-in improvement over OPSD without the overhead of method-specific hyperparameter exploration.
What evidence exists in the paper. The paper implicitly acknowledges this by conducting the diagnostic sweep only at 1.7B (Figure 2), which is the cheapest scale to sweep. The controller configuration is standardized across scales in the main experiments (Section 5.1: "All main runs use the same lookahead horizon L = 20"), but the paper does not report whether this standardization required tuning or worked out-of-the-box. No ablation over controller hyperparameters (H, λ_gt, entropy target, MLP size, learning rate) is presented. The paper does not report the total compute used for all experiments including diagnostics, sweeps, and tuning.
Mitigation status. Not addressed as a limitation. The paper treats the fixed-exposure sweep as a one-time diagnostic that establishes the problem (Section 3.2) and the ATESD results as the solution (Section 5.2), without acknowledging the cost of bridging from diagnosis to solution in a new setting. Section 6 focuses on extensions (per-sample control, credit assignment improvements, domain generalization) rather than on the practical overhead of method adoption. A practitioner reading the paper would need to budget for significant diagnostic compute beyond the reported training budget.
The 100-Step Budget and Checkpoint Selection Protocol May Favor Methods with Higher Variance
The assumption or constraint. All methods in Table 1 are evaluated using the OPSD within-budget checkpoint selection convention: the best Average@12 score across checkpoints saved during the 100-step training budget is reported. This means the reported number is the maximum observed performance, not the expected performance at a predetermined checkpoint. The paper states this protocol is adopted "for fairness" and to match the OPSD baseline reporting.
The consequence. This evaluation protocol inflates absolute scores for all methods relative to fixed-checkpoint evaluation, but it can introduce systematic bias if methods differ in their checkpoint-to-checkpoint variance. A method whose performance oscillates widely during training will benefit more from selecting the maximum than a method with stable, monotonic improvement. ATESD introduces a controller that actively modulates the teacher's information level, creating training dynamics that are qualitatively different from OPSD's fixed-α optimization. The controller's exploration (sampling α from a Beta distribution) could increase performance variance across checkpoints relative to OPSD, because some controller decisions may lead to temporarily worse teacher targets that the student takes several steps to recover from. If ATESD has higher variance, the within-budget maximum selection would overstate ATESD's typical performance relative to OPSD.
The paper reports seed-to-seed error bars (± values) in Table 1, which capture variance across different training runs but do not capture within-run variance across checkpoints. A practitioner deploying ATESD cannot select the best checkpoint retrospectively—they must either use a held-out validation set (which requires its own compute and may not perfectly correlate with the evaluation benchmarks) or pick a fixed checkpoint (e.g., the last step), neither of which achieves the reported numbers.
What evidence exists in the paper. The paper does not report checkpoint-to-checkpoint performance curves, training variance within a single run, or performance at a fixed checkpoint (e.g., step 100). Table 2's ablations are also reported under the same within-budget selection protocol, so they inherit the same potential bias. The paper does not discuss whether the controller's stochasticity increases training variance or present evidence that ATESD's checkpoint-to-checkpoint stability is comparable to OPSD's. The error bars in Table 1 (±0.23 to ±1.04 Average points) reflect cross-seed variance of the maximum-within-window, which confounds seed-to-seed differences with checkpoint-to-checkpoint differences.
Mitigation status. The paper adopts the OPSD protocol as-is for comparability, which is a defensible choice—changing the evaluation protocol would make direct comparison to prior work impossible. However, the paper does not discuss the variance issue or provide supplemental evidence (e.g., learning curves, fixed-checkpoint results, or a held-out benchmark for checkpoint selection) that would address it. Section 6 ("Limitations and future directions") does not mention evaluation protocol bias. A practitioner should be aware that the reported gains are measured under a protocol that favors high-variance methods and may not fully translate to deployment settings where checkpoint selection is constrained.
The Teacher-Grounded Credit Score Requires Additional Forward Passes Not Accounted in the Training Budget
The assumption or constraint. The ATESD reward function includes a teacher-grounded credit term G_gt(t_0) (Equation 8), defined as the discounted average log-probability assigned by the exposure-modulated teacher to verified reference tokens over the lookahead window. Computing this term requires running the teacher model on reference solution tokens—a forward pass that is separate from the standard distillation forward pass (which scores student-generated rollouts). The paper describes g_t as "the average log-probability assigned by the exposure-modulated teacher to verified reference tokens" at step t (Section 4.3), implying additional computation at each student step during the lookahead window.
The consequence. The 100-step training budget counts only the student optimization steps (each involving one forward pass through student and teacher for distillation). The G_gt computation adds a second teacher forward pass per step during the lookahead window to compute log-probabilities on reference tokens. If the lookahead window is L = 20 steps and controller updates happen roughly every H + L steps, this adds up to 20 extra teacher forward passes per controller episode. Over the full 100-step training budget with ~4 controller updates, this is roughly 80 additional forward passes—an 80% increase over the nominal 100-step budget, ignoring the fact that teacher forward passes are typically cheaper than student forward+backward passes. The paper claims the controller is "lightweight" (Section 5.1) and that "ATESD keeps the OPSD student rollout, optimizer, LoRA training recipe, and problem-only prompting protocol unchanged" (Section 5.1), but this statement refers to the student side and does not account for the additional teacher computation required by the reward function.
For a practitioner, this overhead means that a fair compute-matched comparison would give OPSD additional training steps to compensate for ATESD's extra forward passes. If ATESD uses effectively 180 forward passes to OPSD's 100, the reported improvement of +0.95 to +2.33 Average points should be discounted by whatever improvement OPSD would achieve with 80% more steps. The paper does not perform this comparison.
What evidence exists in the paper. The paper does not report wall-clock time, FLOP counts, or forward pass counts for ATESD versus OPSD. The teacher-grounded score ablation in Table 2A shows that the full delayed reward (with G_gt) reaches 59.17 while discounted lookahead without G_gt reaches 58.06—a +1.11 gain from adding the teacher-grounded term. This is the incremental benefit of the additional computation; the 58.06 achieved without G_gt (and thus without the extra forward passes) represents a version of ATESD that more fairly matches OPSD's compute budget. The gap between ATESD-without-G_gt (58.06) and OPSD (57.20) is +0.86, roughly half the reported ATESD improvement (+1.97 on AIME 2024 at 1.7B). This suggests that a meaningful portion of ATESD's gain comes from the additional computation in the reward function, not from the exposure control mechanism alone.
Mitigation status. The paper does not acknowledge this as a limitation or a fairness concern. The teacher-grounded score is presented as a component of the reward design without discussion of its computational cost. Section 6 ("Limitations and future directions") does not mention compute overhead. A practitioner adopting ATESD should budget for the G_gt computation or consider the G_gt-free variant (the "Discounted lookahead" row in Table 2A) as a cheaper alternative with moderate performance. The paper provides no guidance on the compute–accuracy tradeoff between these variants.
Evaluation Is Restricted to a Single Task Family with Small Test Sets
The assumption or constraint. All experiments—diagnostic sweeps, controller training, ablation studies, and final evaluation—use competition-level mathematical reasoning benchmarks (AIME 2024, AIME 2025, HMMT 2025) with training on the OpenThoughts mathematical reasoning corpus. AIME has 30 questions per year; HMMT typically has 30–40 questions. The total evaluation set across all three benchmarks is approximately 100 questions. The paper uses the same benchmarks for checkpoint selection, meaning the effective sample size for model comparison is ~100 questions with no held-out data.
The consequence. The statistical power of the evaluation is limited. A 1–2 point difference in Average@12 on a 30-question benchmark corresponds to the model getting 0.3–0.6 more questions correct on average per sample—a difference of roughly one additional correct answer across the entire test set over 12 sampled completions. On such small test sets, benchmark-to-benchmark variance can be substantial (the paper's own results show ATESD improving OPSD by +1.97 on AIME 2024 but only +0.82 on AIME 2025 at 1.7B, a range of 1.15 points across two benchmarks of similar difficulty and size). This variability makes it difficult to determine whether the reported improvements are robust to benchmark choice or partially reflect idiosyncratic features of the specific test questions.
More fundamentally, the restriction to competition math leaves open the question of whether teacher-side exposure mismatch is a general phenomenon in self-distillation or a domain-specific artifact. Competition math problems have well-defined correctness criteria (the final answer is a number or expression), structured chain-of-thought reasoning with clear step-by-step progressions, and reference solutions that follow predictable patterns. These properties make exposure control via prefix truncation (Equation 2) semantically coherent—truncating a math solution at 50% of its reasoning tokens leaves a valid partial solution. In other domains—code generation, where reasoning and solution are interleaved; creative writing, where there is no ground-truth "answer"; or scientific reasoning, where chains of thought may involve domain-specific notation and non-linear structure—prefix truncation may produce incoherent or misleading teacher contexts, and the relationship between exposure and mismatch may differ qualitatively.
A practitioner using self-distillation for non-math reasoning tasks cannot determine from this paper whether ATESD would help, hurt, or have no effect. The exposure mismatch problem is plausibly general (any domain with a capability gap between student and privileged teacher should exhibit it), but the specific mechanism—prefix truncation of reasoning chains with preserved final answers—is tailored to the structure of math solutions and may not transfer.
What evidence exists in the paper. The paper explicitly acknowledges this limitation in Section 6:
"validating ATESD on larger model scales, code generation, and scientific reasoning remains important for testing whether exposure control extends beyond math contests and the benchmark suite studied here to broader reasoning domains"
However, the acknowledgment is in the context of future work, and the paper provides no preliminary evidence—even a small-scale experiment on a single non-math benchmark—to suggest that the phenomenon generalizes. The difficulty-bin analysis (Figure 2C) provides within-domain evidence that the optimal α varies with problem difficulty, but this is still within the math domain and does not address cross-domain transfer.
Mitigation status. The paper identifies domain generalization as future work. No cross-domain experiments are attempted. The 100-question effective test set size is not discussed as a statistical limitation, and no confidence intervals or significance tests are reported for benchmark-level comparisons. The error bars in Table 1 provide seed-level variance but do not account for benchmark-level variance or test-set size. A practitioner should treat the reported improvements as specific to the tested benchmarks and model family, and should expect to run their own validation before deploying ATESD in a new domain.
7. Implications and Future Directions
How This Work Changes the Landscape
This paper introduces a new axis—teacher exposure control—into the design space of on-policy self-distillation for LLM reasoning, and in doing so, it reframes a previously unquestioned default (full reference access) as an optimization problem in its own right. The magnitude of this shift is best understood as a conceptual reframing with immediate practical consequences, rather than a paradigm-level revolution. ATESD does not replace on-policy self-distillation; it extends it by identifying that the teacher's information level is a trainable variable orthogonal to the student-side adaptations that prior work focused on. The reframing matters because it converts an implicit, invisible assumption—"the teacher should see everything"—into an explicit design choice with demonstrated performance consequences.
The most significant landscape change is the identification of teacher-side exposure mismatch as a distinct failure mode that is the supervision-side analogue of the student-side rollout mismatch that OPSD was designed to eliminate. This diagnostic insight reconciles an implicit tension in the self-distillation literature: why does on-policy distillation (which fixes student-side mismatch) still leave performance on the table relative to what privileged supervision should theoretically enable? The paper's answer—because full teacher access creates targets the student cannot absorb—provides a unified explanation that was previously absent. It also explains why prior adaptive distillation methods that modulated only the student's view of a fixed teacher (temperature scaling, curriculum ordering) had limited headroom: they were optimizing the wrong thing. The teacher's information content, not just its presentation, is what constrains learning when the capability gap is large.
The fixed-exposure sweep in Figure 2A delivers a finding with potentially broad implications for the distillation community: full teacher exposure is not reliably optimal, and intermediate exposure can outperform it. At α = 0.5, the student achieves higher AIME 2024 accuracy than at α = 1.0 across all three seeds on Qwen3-1.7B. This single result challenges a default that every prior self-distillation method—OPSD, SDPO, GATES, and their follow-ups—inherited without justification. It means that practitioners running self-distillation pipelines are likely leaving accuracy on the table by giving the teacher complete reference solutions, and that a simple fixed-α sweep could recover some of those gains with zero change to training infrastructure. The paper's difficulty-bin analysis (Figure 2C) further shows that the optimal α varies by problem difficulty, ruling out the "just find the best fixed α" counterargument and motivating adaptive control specifically.
ATESD also shifts the conversation around meta-optimization in self-play training systems. The two-timescale controller with delayed credit assignment (Table 2A) demonstrates that when a meta-parameter affects the inner optimizer's targets, myopic one-step rewards are actively harmful—immediate feedback (52.22) underperforms even the best fixed α (57.44) on AIME 2024 at 1.7B, while delayed lookahead rewards (58.06–59.17) substantially outperform it. This is not an obvious result: a practitioner might reasonably expect that rewarding the controller based on the immediate loss change would work, and the 6.95-point gap between immediate and full delayed credit shows it definitively does not. The finding has implications beyond exposure control: any system that learns to modulate its own training signal (adaptive curricula, dynamic loss weighting, learned data selection) faces the same credit assignment problem, and this paper provides a template (hold-window + lookahead + discounted progress reward + grounding constraint) for addressing it.
The paper also redirects research attention from search algorithms to verifier/supervision quality within the distillation paradigm, analogous to how the RLHF community recognized reward hacking as the central bottleneck. ATESD's controller does not change what the student samples or how the student optimizes; it changes only the information content of the teacher's supervision. The fact that this single change—modulating α—produces +0.95 to +2.33 Average@12 gains across model scales (Table 1) suggests that supervision quality, not just supervision quantity, is the underexploited dimension in self-distillation. This makes research on better verifiers, better exposure policies, and better teacher-side adaptation more attractive than research on more sophisticated search or sampling strategies, at least for the distillation setting.
Finally, the paper normalizes the practice of treating training hyperparameters as learnable control variables rather than fixed constants. The Beta-policy controller demonstrates that a lightweight MLP (a few hundred parameters) can learn a training-state-conditioned policy for a continuous hyperparameter that previously required expensive grid search to tune. This is a methodological contribution that extends beyond ATESD: the hold-window + lookahead + REINFORCE architecture could be applied to other training-time variables (learning rate, loss weighting, data mixture ratios) in settings where the effect of a decision is only visible after multiple optimization steps. The paper makes this pattern concrete and reproducible.
Follow-Up Research This Work Enables
Per-sample exposure control conditioned on problem difficulty or student confidence. The most direct extension of ATESD addresses the gap between the paper's motivating evidence and its implemented method. Figure 2C shows that easy, medium, and hard problems prefer different α values (1.0, 0.5, and 0.0 respectively among the tested grid on AIME 2024 at 1.7B), but ATESD uses a single global α for all samples within a hold window. A natural follow-up would condition the exposure controller on per-sample features—estimated problem difficulty (e.g., from the base model's pass@1 rate, a lightweight difficulty classifier, or the student's own self-confidence on the current problem), the student's recent accuracy on similar problems, or even the reference solution's length and complexity. A strong experiment would compare three controllers: (a) global training-state-only (the current ATESD), (b) per-sample difficulty-conditioned using ground-truth difficulty labels (oracle upper bound), and (c) per-sample difficulty-conditioned using a predicted difficulty estimate (practical deployment). The research question is whether per-sample adaptation recovers additional gains beyond global adaptation, and how much of the oracle upper bound is achievable with predicted difficulty. The difficulty-bin analysis in Figure 2C suggests a 5–10 point spread between best and worst α per bin, so the potential headroom is significant.
Cross-domain validation of the exposure mismatch phenomenon. The paper's experiments are restricted to competition-level mathematical reasoning (AIME, HMMT) trained on the OpenThoughts corpus. Whether teacher-side exposure mismatch generalizes to other reasoning domains—code generation, scientific reasoning, multi-hop QA, logical deduction—is an open question with significant practical stakes given self-distillation's widespread use. A systematic follow-up would replicate the fixed-exposure sweep (Figure 2A) on 3–4 diverse benchmarks (e.g., HumanEval/MBPP for code, GPQA for science, ARC for logical reasoning, MMLU for broad knowledge) with a single model family (Qwen3 or similar) to establish: (a) whether full exposure is suboptimal in all domains or only in math, (b) whether the optimal fixed α is domain-specific or consistent, and (c) whether the monotonic mismatch growth pattern (Figure 2B) holds across domains. Negative results—domains where full exposure is optimal or where α has no effect—would be equally informative by establishing boundary conditions. The prefix-truncation mechanism (Equation 2) assumes a linear, scaffolded reasoning structure that may not hold in code (where reasoning and solution are interleaved) or creative writing (where there is no ground-truth answer), so alternative truncation strategies might be necessary.
Interaction between exposure control and training budget at scale. The paper's experiments use a fixed 100-step training budget, which is relatively short. The scale-dependent pattern in Table 1 shows ATESD's benefit growing from +0.95 (1.7B) to +2.05 (4B) to +2.33 (8B) Average points, but it's unknown whether this trend continues, plateaus, or reverses at larger scales and longer training horizons. A scaling study would train ATESD and OPSD at multiple model sizes (1B → 8B → 32B) and multiple training budgets (100, 500, 2000 steps), measuring: (a) the ATESD–OPSD gap as a function of scale, (b) whether ATESD's benefit is concentrated early in training (when the student is weak and exposure control matters most) or persists throughout, and (c) whether the learned α policy qualitatively changes with training duration (e.g., does α drift upward over long training as the student catches up?). This addresses a practical concern: if ATESD's benefit diminishes with training budget, practitioners running long distillation runs might prefer simpler fixed-α approaches, while those running short budgets would gain the most from adaptive control.
Stress-testing the credit assignment mechanism with counterfactual rewards. The delayed credit assignment in ATESD (Equation 8) uses a specific design: discounted learning progress with max(0,·) clipping, a teacher-grounded term, and an L = 20 lookahead window. Table 2A shows this combination works, but does not reveal which components are load-bearing or whether alternatives would work better. A systematic ablation-and-replacement study would test: (a) replacing the max(0,·) clipped progress with raw loss change (including negative changes), to measure whether the clipping is load-bearing or merely beneficial; (b) replacing the REINFORCE policy gradient with a learned value-function baseline (actor-critic) to test whether variance reduction helps; (c) replacing the hand-designed reward function with a learned reward model trained to predict future student accuracy from the controller's action and training state, testing whether end-to-end credit assignment outperforms the engineered signal; (d) varying L from 5 to 100 to characterize the sensitivity of the credit horizon. The teacher-grounded ablation in Table 2A already shows that removing G_gt drops performance from 59.17 to 58.06, suggesting it matters but is not dominant—a learned reward model might recover the loss or improve on it.
Exposure control combined with student-side adaptive distillation methods. ATESD modulates the teacher's information level; prior work modulated how the student receives that information (temperature scaling, example weighting, curriculum ordering). These are orthogonal interventions that could be composable. A combined experiment would test ATESD alongside a dynamic temperature schedule (e.g., Li et al., 2023) or a per-example difficulty-weighting scheme, measuring whether the combination is additive (the sum of individual gains), superadditive (the methods complement each other), or subadditive (diminishing returns because both address the same underlying mismatch). This would clarify whether exposure control is a substitute for or complement to existing adaptive distillation techniques. The paper's framework suggests complementarity: exposure controls what the teacher knows, while temperature/exampling controls how the student uses that knowledge, so they should operate on different parts of the learning dynamics.
Replacing the Beta controller with a simpler mechanism that achieves comparable gains. The paper's ablation in Table 2B shows that the learned Beta policy (59.17) substantially outperforms the best fixed α (57.44) and random exposure (54.94), establishing that feedback-driven adaptation matters. But it does not test whether a simpler adaptive mechanism—e.g., a hand-designed schedule that anneals α from 0.7 to 0.3 over training, or a heuristic rule that sets α proportional to the student's recent loss improvement rate—could achieve similar gains. This is a necessary stress test: if a simple heuristic matches ATESD, then the complexity of REINFORCE-trained Beta controller is unnecessary, and the paper's contribution reduces to "exposure should change during training" rather than "exposure should be learned from feedback." A strong experiment would compare ATESD to 3–4 heuristic adaptive policies (linear anneal, step-function anneal, loss-threshold-triggered adjustment, exponential moving average of the best recent fixed α) on the same benchmarks and budget. Negative results here—i.e., heuristics matching ATESD—would refine our understanding by showing that the value is in adaptivity per se, not in learned adaptivity, which would shift the research agenda toward simpler, cheaper adaptive mechanisms.
Practical Applications and Downstream Use Cases
Drop-in improvement for existing on-policy self-distillation pipelines with zero data changes. Any team currently running OPSD (or its variants like SDPO, GATES, or reasoning-compression self-distillation) can implement the exposure-modulated teacher (Equations 2–3, 5–6) as a minimal code change—truncate the reference solution before inserting it into the teacher prompt, controlled by a scalar α. Even without the learned controller, the fixed-exposure sweep (Figure 2A) provides actionable guidance: switching from full exposure (α = 1.0) to intermediate exposure (α ≈ 0.5) improved AIME 2024 accuracy by roughly 2 percentage points on Qwen3-1.7B across three seeds. For a team training a reasoning model on a budget, this is essentially free accuracy—no additional data, no model architecture changes, no increase in training steps. The paper's finding that the improvement grows with model scale (+0.95 → +2.05 → +2.33 Average from 1.7B to 8B with ATESD's learned policy) suggests the fixed-α benefit may also scale, making this a low-risk modification for larger models. The practical workflow is: run a small fixed-α sweep on a subset of your training data (testing α ∈ {0.0, 0.25, 0.5, 0.75, 1.0}), select the best value, and use it for the full training run. The sweep cost is at most 5× the cost of a short training run, amortized over the production training budget.
Self-improvement and iterative distillation loops where the student's capability evolves. In self-improvement pipelines where a model generates its own training data over multiple rounds (e.g., STaR, ReST^EM, or iterative rejection sampling), the student's capability changes substantially across rounds. The optimal teacher exposure for a weak student in round 1 (who needs minimal privileged context to avoid unabsorbable targets) is different from the optimal exposure for a strong student in round 3 (who can benefit from full reference reasoning). A static α cannot accommodate this, but ATESD's training-state-conditioned controller naturally adapts as the student improves. The controller's state includes loss EMAs and probe-NLL (Section 4.2), which track the student's evolving capability, so the learned policy should automatically shift α upward as the student becomes stronger. This makes ATESD a natural fit for multi-round self-improvement: rather than manually tuning α for each round, a single controller trained across rounds would learn to increase exposure as the student's competence grows. The paper's results at 1.7B, 4B, and 8B (Table 1) show the controller works across model capacities, which is a necessary precondition for cross-round adaptation to function.
Cheap difficulty estimation via the exposure-modulated teacher's behavior. The exposure-modulated teacher provides a side signal that could be repurposed for problem difficulty estimation without the expensive 2048-sample sweeps used in prior work. At inference time, running the teacher on a problem with α = 0 (answer only) and α = 1 (full reasoning) and measuring the KL between the two teacher distributions gives a per-problem signal of how much the privileged context changes the teacher's predictions. A large KL means the reference reasoning provides substantial additional information beyond the answer—likely indicating a hard problem where the reasoning steps are nontrivial. A small KL means the answer alone largely determines the teacher's predictions—an easier problem. This difficulty signal could be used to route problems: easy problems get a small test-time compute budget (few samples, no search), hard problems get a larger budget. The paper does not explore this, but the framework provides it for free: the exposure-modulated teacher already exists in an ATESD-trained pipeline, and computing the α = 0 vs. α = 1 KL requires only two forward passes per problem at inference time, a negligible cost compared to beam search or best-of-N sampling. This is a practical spin-off application that requires no new training, only a post-hoc analysis of the teacher's sensitivity to privileged context.
When to Prefer This Method
The paper explicitly positions ATESD as an extension of on-policy self-distillation rather than an alternative to it, and the method is designed to be a drop-in replacement for the full-exposure teacher in OPSD and its variants. The decision is therefore not "ATESD vs. RL" or "ATESD vs. SFT" but rather:
Prefer learned exposure control (ATESD) when:
- You are already using or planning to use on-policy self-distillation with a privileged teacher (the OPSD family of methods), since ATESD changes only the teacher context construction and adds a lightweight controller.
- Your training budget is fixed and you want to maximize accuracy within that budget without changing the student model, data, optimizer, or rollout protocol—the paper shows +0.95 to +2.33 Average@12 gains over OPSD's full-exposure default across Qwen3-{1.7B, 4B, 8B} under identical training-step budgets (Table 1).
- The training data contains a mix of problem difficulties where the student's capability varies relative to the reference solution difficulty, so a single fixed α would be a compromise—Figure 2C shows easy, medium, and hard problems prefer different α values.
- You can afford the one-time diagnostic cost of a small fixed-α sweep to validate that the phenomenon (full exposure suboptimality, Figure 2A) holds in your setting before committing to the full ATESD controller implementation.
Prefer a simpler fixed intermediate α when:
- Implementation simplicity is paramount and the ~1–2 point gain from learned control over the best fixed α (Table 2B: 59.17 vs. 57.44 on AIME 2024 at 1.7B) does not justify the controller infrastructure—a fixed α = 0.5 already improves over the OPSD default (57.20 → 57.44 on AIME 2024 at 1.7B, Figure 2A) with zero additional code beyond a teacher prompt truncation.
- Your training budget is long enough that the benefit of adaptive control may diminish as the student converges, and you lack the resources to run the controller ablations needed to determine whether the benefit persists—the paper's 100-step budget is relatively short and the interaction with training duration is untested.
Prefer the full-exposure OPSD default when:
- You cannot afford the fixed-α diagnostic sweep to verify that full exposure is suboptimal in your setting—the paper's evidence is from AIME 2024 at 1.7B only (Figure 2), and the phenomenon's generality across models, domains, and data distributions is not established.
- The student model is already near-ceiling performance on your target tasks, so the capability gap between student and privileged teacher is small and full exposure is likely close to optimal—the paper's scale-dependent gains suggest the benefit shrinks as models approach saturation, though this trend is inferred from cross-model comparisons rather than directly tested within a single training run.
