TCOD: Exploring Temporal Curriculum in On-Policy Distillation for Multi-turn Autonomous Agents

ArXiv: 2604.24005

🎯 Pitch

Vanilla on-policy distillation catastrophically fails in multi-turn agents because compounding errors drive the student outside the teacher's support, causing KL divergence to spike and success rates to collapse. TCOD rescues training by progressively expanding the student's trajectory exposure from short to long, improving performance by up to 18 points and even enabling the student to outperform the teacher on tasks where the teacher itself fails.

1. Executive Summary

This paper diagnoses and resolves a fundamental failure mode when applying on-policy distillation (OPD) to multi-turn autonomous agents—Trajectory-Level KL Instability, where compounding errors across turns cause KL divergence to escalate and success rates to collapse during training (e.g., the student's KL divergence starts around 1,000 and only converges to roughly 60 after severe instability). The authors propose TCOD (Temporal Curriculum On-Policy Distillation), a framework that controls the trajectory depth exposed to the student during training through two variants—Forward-to-Backward (F2B), which restricts the student to early-turn interactions and progressively extends the horizon, and Backward-to-Forward (B2F), which uses the teacher to navigate to near-terminal states and gradually expands the student's responsibility backward to the initial steps—evaluated across four student–teacher pairs on three multi-turn benchmarks (ALFWorld, WebShop, ScienceWorld). TCOD improves agent performance by up to 18 points over vanilla OPD, reduces training time by up to 32%, and enables students to surpass the teacher's success rate by up to 14 points on a hard split where the teacher itself fails, establishing that temporal curriculum pacing can stabilize multi-turn distillation and enable generalization beyond the teacher's capability boundary only when the student is progressively exposed to longer horizons instead of confronting full-length trajectories from the start.

2. Context and Motivation

The Specific Gap: On-Policy Distillation Breaks Down in Multi-Turn Settings

The paper addresses a specific, previously undocumented failure mode: on-policy distillation (OPD), which has been highly effective for transferring reasoning capabilities in single-turn tasks like math problem-solving, catastrophically destabilizes when naively applied to multi-turn autonomous agents. This is not a minor performance degradation—the paper presents evidence that small student models (e.g., Qwen3-1.7B, Qwen3-0.6B) trained with vanilla OPD in multi-turn environments experience a simultaneous escalation of KL divergence and collapse of task success rate to near-zero (Figure 2a, 2b). Even when the KL divergence eventually converges, it starts at values roughly two orders of magnitude higher than its final converged value (Figure 2c: initial KL ~1,000 vs. final KL ~60), indicating severe training instability that prior work on OPD had not observed or accounted for.

The core gap is conceptual and practical: existing OPD methods were designed and validated exclusively for static, single-turn tasks where each rollout is an independent reasoning trace. Multi-turn agents, by contrast, operate in environments where each action modifies the state and the next observation depends on the previous action, creating causal coupling across turns. The student's errors at turn $t$ propagate into the history $h_{t+1}$, shifting the state distribution progressively further from what the teacher model encounters during its own (correct) trajectories. This distribution shift renders the teacher's token-level probability predictions—which serve as the distillation supervision signal in Equation 2—increasingly unreliable as the trajectory lengthens.

Why This Problem Matters

The significance of this gap extends beyond an academic curiosity about training stability. It has direct implications for several practical and research trajectories:

1. The growing importance of agentic AI systems. Multi-turn autonomous agents represent one of the most promising application domains for LLMs, spanning embodied navigation (Shridhar et al., 2020), web interaction (Yao et al., 2022a), scientific reasoning (Wang et al., 2022), and general-purpose agentic frameworks (Wang et al., 2026; Contributors, 2026). As these systems move from research prototypes to production deployments, the ability to train smaller, more efficient models that can execute long-horizon tasks becomes critical—deploying frontier-scale models (e.g., 30B+ parameters) for every agent interaction is prohibitively expensive in terms of latency and compute cost. OPD represents a natural solution (distill a strong teacher into a small student), but if OPD itself breaks down precisely when trajectories become long and complex, it cannot serve this need.

2. Complementing reinforcement learning in agent training. Training multi-turn agents typically relies on reinforcement learning (RL) with sparse task-completion rewards. RL in these settings suffers from well-known challenges: long-horizon credit assignment (it's unclear which of many actions contributed to eventual success or failure), sample inefficiency in sparse-reward environments (Feng et al., 2025; Penaloza et al., 2026), and memory management difficulties (Shi et al., 2026). OPD was supposed to address these by replacing sparse scalar rewards with dense, token-level teacher supervision—effectively providing a rich learning signal at every step. The finding that OPD itself becomes unstable in multi-turn settings means that the practitioner is caught between two flawed options: sparse RL that is sample-inefficient, and dense distillation that is training-unstable. Resolving this instability would unlock OPD as a viable complement or alternative to RL for agent training.

3. The broader question of whether distillation techniques transfer across task structures. The paper's findings challenge an implicit assumption in the distillation literature: that methods developed and validated on single-turn reasoning tasks (mathematical problem solving, question answering) will generalize cleanly to multi-turn, interactive settings. The trajectory-level KL instability documented in Section 4.1 demonstrates that this assumption is false—multi-turn environments introduce qualitatively different failure dynamics that are invisible in single-turn benchmarks. Understanding why this transfer fails and how to fix it contributes to a more principled theory of when and how distillation works, rather than treating it as a black-box technique that can be applied uniformly across tasks.

Where Prior Approaches Fall Short

The paper systematically identifies the limitations of existing methods along several dimensions:

Single-Turn OPD Methods Don't Address Multi-Turn Compounding Errors

Recent work has improved OPD through several design choices, including objective design (Jang et al., 2026; Jin et al., 2026), optimization heuristics (Ko et al., 2026), and alternative supervision sources (Ye et al., 2026; Zhao et al., 2026). These methods focus on balancing forward and backward KL terms (Jang et al., 2026; Jin et al., 2026) and incorporating RL-style heuristics such as reward clipping (Ko et al., 2026) to improve training stability and convergence. However, as the paper explicitly notes in Section 2:

"these approaches are primarily designed for single-turn settings and do not directly address multi-turn agent environments."

The critical difference is that single-turn OPD operates on independent rollouts—the student generates one complete answer, the teacher provides token-level probabilities for that answer, and the KL divergence is computed over a fixed-length response. Errors in one rollout do not compound into the next because each rollout starts from the same initial state. In multi-turn settings, the student's action at turn $t$ becomes part of the input at turn $t+1$ (Equation 1: $h_t = (o_0, a_0, \ldots, a_{t-1}, o_t)$), so errors cascade through the trajectory. Prior OPD methods do not include any mechanism for controlling or mitigating this cascading effect—they treat all tokens in the trajectory identically, regardless of how far the student has drifted from the teacher's support.

The paper makes this distinction explicit in a key technical note (Section 4.1, Remark 1):

"Long-CoT increases the response length on the same environment state. However, multi-turn agents update the environment state at each interaction by incorporating new observations and actions, thereby amplifying compounding errors over the trajectory."

This is a crucial observation: the problem is not simply that trajectories are long (which would also apply to long chain-of-thought reasoning), but that the state itself changes based on student actions, creating a fundamentally different failure mode than what single-turn methods were designed to handle.

Curriculum Learning for RL Agents Imposes External Dependencies

Curriculum learning has been applied to agent training (Bengio et al., 2009; Shi et al., 2025; Wang & Ammanabrolu, 2025; Gong et al., 2026), but existing approaches typically rely on an external model to measure task difficulty or require additional data curation to construct the curriculum. For example, recent work on applying curriculum learning to GRPO (Guo et al., 2025) uses separate difficulty estimators to sort training examples from easy to hard. This adds complexity and breaks the self-contained nature of OPD, where the student learns purely from its own rollouts and the teacher's feedback without external signals.

The approach of Lauffer et al. (2025) is closer in spirit—it trains the student only on the expert's subsequent corrective actions—but it breaks the on-policy setting by relying on teacher-generated data rather than student-generated rollouts. This defeats one of the primary advantages of OPD: learning from the student's own distribution of errors rather than from pre-collected demonstrations, which can suffer from exposure bias.

The paper explicitly positions TCOD against these approaches in Section 2:

"Our approach avoids both [external difficulty measurement and breaking the on-policy setting] by defining difficulty through increasing trajectory depth, using only student-generated data, keeping training simple, on-policy, and more stable."

The key innovation here is that trajectory length itself serves as a natural difficulty metric—shorter trajectories are inherently easier because they reduce the opportunity for error accumulation—and this metric is built directly into the training process through a pacing schedule (Equation 4) rather than estimated externally.

Supervised Fine-Tuning Suffers from Exposure Bias

Standard supervised fine-tuning (SFT) on teacher-collected successful trajectories is the most straightforward alternative to OPD. However, as the paper notes in Section 2, SFT suffers from the well-known exposure bias in multi-turn settings: the model is trained on ground-truth teacher actions at every step, but at test time it must generate its own actions, and errors at early steps push the test distribution away from the training distribution. The experimental results in Table 2 confirm this: SFT on ALFWorld achieves only 32.14% success rate with Qwen2.5-3B, compared to 65.72% for vanilla OPD and 77.86–81.43% for TCOD variants. SFT provides a weak baseline that TCOD substantially outperforms, but the more relevant comparison is against vanilla OPD, which already addresses exposure bias (by training on student-generated rollouts) but introduces the trajectory-level KL instability that TCOD resolves.

How This Paper Positions Itself

The paper positions TCOD as a principled, minimal intervention that addresses a newly identified failure mode without requiring major architectural changes or external components. Several aspects of this positioning are worth examining:

1. The method is an extension, not a replacement, of OPD. TCOD does not modify the core OPD objective (Equation 2)—it still minimizes token-level KL divergence between student and teacher—nor does it change the teacher model, the student architecture, or the optimization procedure. The only modification is a constraint on which tokens contribute to the loss at each training step, controlled by the temporal curriculum. This is explicitly designed to be lightweight:

"This approach requires only minor code changes." (Section 4.2)

The simplicity of the intervention is itself a contribution: it demonstrates that the instability in multi-turn OPD is not a fundamental limitation of distillation as a technique, but rather a problem of how much of the trajectory the student is asked to learn from at once.

2. The paper presents TCOD as solving a diagnosis problem first, an intervention second. Section 4.1 provides a detailed empirical investigation of the failure mode before introducing the solution. This is important for two reasons. First, it establishes that the problem is real and systematic (not an artifact of a particular model or hyperparameter setting)—the observations span multiple student sizes (0.5B to 7B parameters), multiple teacher types (general-purpose 30B and domain-specific 7B), and multiple model families (Qwen3 and Qwen2.5). Second, the diagnosis directly motivates the intervention: if the root cause is compounding errors across turns that push the student beyond the teacher's effective support (as shown in Figure 2d, where per-turn KL divergence increases monotonically with turn index), then the natural solution is to limit the exposure to error-prone turns early in training and gradually expand it as the student becomes more competent.

3. TCOD is positioned as a framework with two instantiations that address complementary needs. The paper does not advocate for a single best variant. Instead, it presents TCOD-F2B and TCOD-B2F as two instantiations of the same core idea (temporal curriculum pacing), with different practical tradeoffs:

• TCOD-F2B requires no pre-collected teacher demonstrations and is simpler to implement—it simply truncates the student's rollout at $k$ steps and progressively increases $k$. This makes it a drop-in replacement for vanilla OPD with minimal overhead.
• TCOD-B2F leverages pre-collected successful teacher trajectories to give the student a "head start" from intermediate states, avoiding early-turn error accumulation entirely. It requires additional data (the teacher's successful trajectories) but may be more effective when the teacher is strong enough to provide useful starting points.

The paper presents both as valid options, with F2B recommended when demonstration data is unavailable or when minimal code changes are desired, and B2F recommended when teacher trajectories are accessible and early-turn errors are particularly severe.

4. The paper connects TCOD to curriculum learning while distinguishing it from prior curriculum approaches. The temporal curriculum in TCOD is fundamentally different from standard difficulty-based curriculum learning because difficulty is defined by the structure of the task itself (trajectory length) rather than by an external estimate. This is a key conceptual contribution: in multi-turn settings, trajectory length provides a natural, calibration-free difficulty signal that requires no additional model, no data preprocessing, and no human annotation. The paper makes this explicit:

"Our approach avoids both by defining difficulty through increasing trajectory depth, using only student-generated data, keeping training simple, on-policy, and more stable." (Section 2)

5. TCOD is positioned within a broader vision of making OPD viable for agent training. The paper's framing in Section 1 sets up the problem as a "critical open question" about whether OPD can "safely generalize to such dynamic, long-horizon environments." By demonstrating that a simple temporal curriculum resolves the instability, the paper positions TCOD not as a niche technique for specific benchmarks, but as a general principle that should be incorporated into any OPD pipeline for multi-turn tasks. The experiments across three diverse benchmarks (embodied navigation, web shopping, scientific reasoning) and multiple model scales (1.7B to 7B) support this generality claim.

The Underlying Mechanism: Why Compounding Errors Cause KL Instability

Understanding the paper's motivation requires a deeper look at the mechanism it identifies in Section 4.1. The standard OPD objective (Equation 2) computes the KL divergence between the teacher's and student's token-level distributions at each turn $t$, averaged over all turns and all trajectories generated by the student. This assumes that the teacher's probability distribution $\pi_\phi(a_t | h_t)$ provides a reliable supervision signal at every turn. However, as the student makes errors, the history $h_t$ includes states that the teacher rarely or never visits during its own correct trajectories. In these out-of-distribution states, the teacher's token probabilities become unreliable—the teacher may assign high probability to actions that are appropriate when the agent is on the correct path, but are nonsensical in the error state the student has actually reached.

This creates a vicious cycle visualized in Figure 2d:

1. The student makes an error at turn $t$, producing an action that deviates from what the teacher would have done.
2. This action produces observation $o_{t+1}$, which is incorporated into history $h_{t+1}$.
3. The history $h_{t+1}$ is now outside the teacher's effective support, so the teacher's probability distribution $\pi_\phi(\cdot | h_{t+1})$ is poorly calibrated.
4. The student trains on this unreliable signal (via KL divergence minimization), which can actually increase the student's subsequent errors.
5. The KL divergence at turn $t+1$ is higher than at turn $t$ because the student's distribution, conditioned on an out-of-distribution state, diverges further from the teacher's (poorly calibrated) distribution.
6. This cycle repeats, with KL divergence escalating monotonically across turns (Figure 2d) and success rate collapsing (Figure 2b, 2a).

Crucially, the paper shows that this is not just a problem for very small models. While small models (0.6B, 1.7B, 0.5B, 1.5B) experience catastrophic collapse to near-zero success rates (Figure 7 in Appendix B), larger models (3B, 7B) also exhibit prohibitively high initial KL divergence (~1,000) even though they eventually converge (Figure 2c). The instability is present across scales; only the severity varies. This means that even when vanilla OPD "works" for larger models (in the sense of eventually producing a trained agent), the training process is highly inefficient—the student spends many training steps recovering from early instability before making genuine progress.

The paper's diagnosis in Figure 2d is particularly revealing: the per-turn KL divergence is computed for two different teachers (a GRPO-trained Qwen2.5-7B and Qwen3-30B-A3B-Instruct), and both show the same escalating pattern. This confirms that the instability is not an artifact of a particular teacher model or training procedure, but rather a structural property of multi-turn distillation when the student is exposed to its own error-compounded states.

3. Technical Approach

3.1 Reader Orientation

TCOD is a training framework that modifies how a student language model learns from a teacher model during on-policy distillation for multi-turn agent tasks. Instead of asking the student to learn from full-length trajectories immediately—which causes training to destabilize because early mistakes compound into later turns—TCOD starts the student on short trajectory segments and gradually extends the horizon as training progresses, like teaching someone to navigate a building by first having them practice in a single room, then a floor, then the entire structure.

3.2 Big-Picture Architecture (Diagram in Words)

The TCOD system has five major components that operate together during training:

1. Student policy ($\pi_\theta$) — the language model being trained, which generates actions at each turn of the agent's interaction with the environment. This is the model that will ultimately be deployed.

2. Teacher policy ($\pi_\phi$) — a frozen, typically larger or domain-specialized model that provides token-level probability distributions as supervision signals. The teacher never updates its weights during TCOD training.

3. Environment ($E$) — the multi-turn interactive setting (e.g., ALFWorld, WebShop, ScienceWorld) that receives the agent's actions and returns observations. The environment maintains state that evolves based on agent actions.

4. Temporal Curriculum Controller — the core innovation of TCOD, which determines the trajectory depth $k$ that the student is allowed to explore at each training step. The controller implements a linear pacing schedule $k = k_{\text{start}} + \lfloor n / \eta \rfloor$, where $n$ is the current training step and $\eta$ is the growth rate, progressively increasing $k$ from an initial small value to the full task horizon $T$.

5. Asynchronous Training Infrastructure — a distributed system that decouples trajectory collection (actors) from model optimization (learner) using a shared replay buffer with staleness control, described in Section 4.3 but integral to the practical deployment of TCOD.

Information flows as follows: the temporal curriculum controller sets the current trajectory depth $k$ → the student generates a trajectory of length $k$ (in F2B) or takes over after the teacher executes a prefix (in B2F) → the environment returns observations at each step → the teacher provides token-level probability distributions for each student-generated action → the KL divergence between teacher and student distributions is computed at each step and aggregated into the loss → the student's parameters are updated via gradient descent → the process repeats with $k$ monotonically increasing according to the pacing schedule.

3.3 Roadmap for the Deep Dive

• First, the formal OPD objective for multi-turn agents (Equation 2), which establishes the baseline that TCOD modifies. Understanding what OPD computes and why it fails is prerequisite to understanding the TCOD intervention.

• Second, the empirical diagnosis of trajectory-level KL instability (Section 4.1), including the per-turn KL divergence analysis (Figure 2d) and the compounding error mechanism. This is why TCOD exists—the intervention makes no sense without the diagnosis.

• Third, the TCOD-F2B variant (Equation 3, Algorithm 1), which implements a forward curriculum by truncating student rollouts and progressively expanding the horizon. This is the simpler variant and establishes the core temporal curriculum concept.

• Fourth, the TCOD-B2F variant (Equation 5, Algorithm 2), which uses the teacher as a navigator to place the student at intermediate states and gradually reduces the teacher's prefix. This is the more sophisticated variant that addresses early-turn error accumulation directly.

• Fifth, the asynchronous training infrastructure (Section 4.3), including staleness-aware sub-trajectory replay and the distributed actor-learner architecture. These are practical design choices that significantly impact training stability and efficiency.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a method paper that diagnoses a previously undocumented failure mode in on-policy distillation for multi-turn agents and proposes a lightweight intervention—temporal curriculum pacing—that resolves the failure by controlling which portions of trajectories the student learns from at each training stage.

Formal Definition of Multi-Turn On-Policy Distillation

The paper builds on the standard formulation of on-policy distillation, adapting it to multi-turn agent environments where the agent interacts with an environment over a sequence of turns rather than producing a single static response.

Trajectory structure. A multi-turn interaction proceeds over turns indexed by $t \in \{0, \ldots, T-1\}$, where $T$ is the maximum number of steps before the task terminates (either by a termination action or by reaching the horizon limit). At each turn $t$:

• The agent receives an observation $o_t$ from the environment.
• The agent generates a response $a_t$, which consists of a chain-of-thought reasoning trace followed by an executable action (following the ReAct framework; Yao et al., 2022b).
• The environment processes the action and returns the next observation $o_{t+1}$.

The agent's state at turn $t$ is the full interaction history up to that point:

$h_t = (o_0, a_0, o_1, a_1, \ldots, o_{t-1}, a_{t-1}, o_t)$

where $h_t$ is the history state at turn $t$, $o_i$ is the observation at turn $i$, and $a_i$ is the agent's response (reasoning + action) at turn $i$.

What it computes: the history $h_t$ concatenates all observations and actions from the beginning of the episode up to the current turn. This history serves as the input context for the agent's policy at turn $t$—the agent sees everything that has happened and must decide what to do next based on this accumulating context.

Why this form: the agent operates in a partially observable environment where the current observation $o_t$ alone is insufficient to determine the optimal action. The history must include past actions and observations because the task requires reasoning about what has been attempted, what succeeded or failed, and what steps remain. This is a standard formulation in POMDPs (partially observable Markov decision processes) and is what distinguishes multi-turn agents from single-turn tasks: the state is not reset between turns, and errors propagate forward through the history.

A complete trajectory $\tau$ is then:

$\tau = (h_0, a_0, h_1, a_1, \ldots, h_{T-1}, a_{T-1})$

The trajectory terminates when either a termination action is taken (the agent signals task completion) or when the horizon $T$ is reached.

The OPD objective for multi-turn agents. Given a teacher policy $\pi_\phi$ (frozen, used only for inference) and a student policy $\pi_\theta$ (being trained), the standard on-policy distillation objective minimizes the expected KL divergence between the teacher's and student's token-level probability distributions, averaged over all turns in trajectories generated by the student:

$\mathcal{L}_{\text{OPD}}(\theta) = \mathbb{E}_{\tau \sim \pi_\theta} \left[ \sum_{t=0}^{T-1} D_{\text{KL}}\left( \pi_\phi(a_t | h_t) \parallel \pi_\theta(a_t | h_t) \right) \right]$

where $\tau \sim \pi_\theta$ denotes trajectories sampled by rolling out the student policy in the environment, $D_{\text{KL}}(\pi_\phi \parallel \pi_\theta) = \sum_{a_t} \pi_\phi(a_t | h_t) \log \frac{\pi_\phi(a_t | h_t)}{\pi_\theta(a_t | h_t)}$ is the KL divergence measuring how much the student's distribution diverges from the teacher's at turn $t$, $h_t$ is the history state at turn $t$ (which includes all previous student-generated actions and environment observations), and $T$ is the trajectory length (either the horizon or the turn where termination occurs).

What it computes: for each trajectory generated by the student, the teacher provides a probability distribution over all possible tokens at each turn $t$, conditioned on the history $h_t$ that the student has actually produced. The KL divergence measures the discrepancy between the teacher's distribution (what the teacher would have done given this history) and the student's distribution (what the student did do). The loss sums these divergences across all turns and averages over all trajectories. A lower KL divergence means the student's token probabilities more closely match the teacher's at each step.

Why this form: the key property of on-policy distillation is that the expectation is taken over trajectories generated by the student itself ($\tau \sim \pi_\theta$), not over teacher-generated demonstrations. This means the student learns from its own error distribution—the teacher provides supervision on states that the student actually visits, including error states, rather than only on correct trajectories. This is what distinguishes OPD from supervised fine-tuning (which trains only on teacher-collected successful trajectories and suffers from exposure bias at test time) and what makes OPD theoretically appealing: the student gets dense, token-level feedback even when it makes mistakes, allowing it to learn to recover from errors.

The critical assumption embedded in this objective—and the assumption that the paper demonstrates is violated in multi-turn settings—is that the teacher's probability distribution $\pi_\phi(a_t | h_t)$ remains a reliable supervision signal even when $h_t$ contains student-generated errors. As the paper shows in Section 4.1, this assumption breaks down as errors accumulate: the history $h_t$ drifts further from the teacher's training distribution, the teacher's token probabilities become poorly calibrated, and the KL divergence becomes an unreliable (and eventually harmful) learning signal.

Empirical Diagnosis: Trajectory-Level KL Instability

Before introducing TCOD, the paper conducts a systematic empirical investigation of what goes wrong when vanilla OPD is applied to multi-turn agents. This diagnosis is presented in Section 4.1 and is essential for understanding the design choices in TCOD.

Experimental setup for the diagnosis. The authors evaluate multiple student-teacher pairs on ALFWorld to characterize the failure mode:

• Qwen3 family: teacher is Qwen3-30B-A3B-Instruct; students are Qwen3-{0.6, 1.7, 4}B
• Qwen2.5 family: teacher is a GRPO-trained Qwen2.5-7B (domain-adapted on ALFWorld); students are Qwen2.5-{0.5, 1.5, 3, 7}B

The key metrics tracked during training are trajectory-level KL divergence (averaged over all tokens in all turns of sampled rollouts) and task success rate (fraction of rollouts that complete the task successfully).

Observation 1: KL escalation and success rate collapse co-occur for small models. As shown in Figure 2a and 2b, when training small student models (Qwen3-0.6B and Qwen3-1.7B) with vanilla OPD, the trajectory-level KL divergence increases sharply as training progresses, rather than decreasing as it does in single-turn settings. This escalation is accompanied by a simultaneous collapse of the task success rate to near-zero. This is fundamentally different from the behavior observed in single-turn OPD (e.g., for mathematical reasoning), where KL divergence typically decreases and stabilizes during training. The paper explicitly notes:

"Unlike prior work in single-turn settings such as mathematics or question answering, where the KL divergence consistently converges and decreases throughout training, we observe that the KL divergence escalates with the number of training steps in multi-turn agent scenarios."

The mechanism is destructive: as the student makes errors and receives unreliable teacher supervision on out-of-distribution states, its policy degrades, causing more errors in subsequent rollouts, further increasing KL divergence, and eventually collapsing performance entirely.

Observation 2: Even when KL converges, it starts prohibitively high. Larger models (Qwen3-3B, Qwen2.5-3B, Qwen2.5-7B) do not experience the catastrophic collapse seen in smaller models, but they still exhibit severe initial instability. Figure 2c shows that across multiple student-teacher pairs, the initial KL divergence at the start of training is approximately 1,000—roughly two orders of magnitude larger than the converged value of approximately 60. The paper states:

"across different student-teacher pairs... we consistently observe the initial KL divergence (~1000) is typically orders of magnitude larger than its converged value (~60), indicating severe instability during the training for multi-turn OPD."

This means that even when vanilla OPD eventually produces a functional agent (for larger models), the training process spends a large fraction of its steps recovering from early instability. The high initial KL is not just an inconvenience—it represents wasted computation and potential model damage before the student begins making genuine progress.

The underlying mechanism: per-turn KL divergence increases with turn index. To understand why KL escalates, the paper visualizes the per-turn KL divergence for Qwen2.5-3B distilled from two different teachers (GRPO-trained Qwen2.5-7B and Qwen3-30B-A3B-Instruct) in Figure 2d. The result is striking: KL divergence increases monotonically with the turn index for both teachers. At early turns (0–5), KL divergence is relatively low. By turns 20–25, KL divergence has grown substantially.

The paper explains the mechanism as compounding error amplification:

1. The student makes an error at turn $t$ (generates a token distribution that diverges from the teacher's).
2. This erroneous action produces an observation $o_{t+1}$ that the teacher would rarely or never see during its own correct trajectories.
3. The history $h_{t+1} = (\ldots, a_t^{\text{error}}, o_{t+1})$ is now outside the teacher's effective support—the teacher has never been trained on or evaluated on states that follow from this particular error.
4. The teacher's probability distribution $\pi_\phi(\cdot | h_{t+1})$ on this out-of-distribution state is poorly calibrated: the teacher may assign high probability to actions that would make sense if the agent were on a correct trajectory, but are nonsensical given the actual error state.
5. The student trains on this unreliable signal via KL divergence minimization, which can reinforce or amplify the student's errors rather than correcting them.
6. The KL divergence at turn $t+1$ is therefore higher than at turn $t$, because the student's distribution (conditioned on a confused state) diverges further from the teacher's (poorly calibrated) distribution.
7. This cycle repeats at each subsequent turn.

The paper makes a crucial distinction in Remark 1:

"Long-CoT increases the response length on the same environment state. However, multi-turn agents update the environment state at each interaction by incorporating new observations and actions, thereby amplifying compounding errors over the trajectory."

This is not simply a problem of long sequences. Long chain-of-thought reasoning produces many tokens, but all on the same input state. Multi-turn agents produce many tokens on evolving states, where each state depends on the correctness of all previous actions. The state-dependence is what creates compounding errors—a mistake at turn 5 reshapes turns 6 through $T$ in ways that single-turn reasoning does not experience.

Why this diagnosis matters for TCOD's design. The per-turn KL escalation pattern directly motivates the temporal curriculum approach: if errors compound because early-turn mistakes poison later-turn states, then the natural solution is to prevent the student from experiencing later-turn states until it has mastered early-turn behavior. This is exactly what TCOD does—it restricts the student to short trajectories (where error accumulation is limited) at the start of training, and only expands the horizon as the student demonstrates competence on the current trajectory depth.

TCOD-F2B: Forward-to-Backward Temporal Curriculum

TCOD-F2B implements a "shallow-to-deep" curriculum by restricting the maximum number of interaction steps the student can take during training, then progressively expanding this limit. The student begins by learning to complete tasks (or task prefixes) in only $k_{\text{start}}$ steps, and $k$ increases monotonically until it reaches the full task horizon $T$.

The TCOD-F2B objective. The modification to the OPD objective is minimal: instead of summing over all turns $0$ through $T-1$, the sum runs only over turns $0$ through $k-1$, where $k$ is the current curriculum-controlled trajectory depth:

$\mathcal{L}_{\text{TCOD-F2B}}(\theta) = \mathbb{E}_{\tau \sim \pi_\theta} \left[ \sum_{t=0}^{k-1} D_{\text{KL}}\left( \pi_\phi(a_t | h_t) \parallel \pi_\theta(a_t | h_t) \right) \right]$

where $\tau \sim \pi_\theta$ indicates trajectories sampled by the student policy, $k$ is the current maximum interaction steps (controlled by the curriculum schedule), $t$ indexes turns within the truncated trajectory, and all other symbols retain their definitions from the vanilla OPD objective (Equation 2).

What it computes: the student generates a trajectory by interacting with the environment for at most $k$ steps. At each of these $k$ steps, the teacher provides token-level probability distributions conditioned on the history $h_t$ that the student has produced. The KL divergence between teacher and student distributions is computed and summed over the $k$ steps. Crucially, the student does not see or learn from any turns beyond $k$—if the task would normally require 20 steps but $k = 5$, the student only trains on the first 5 steps of its interaction.

Why this form: the truncation breaks the compounding error cycle by preventing the student from reaching the later turns where errors have accumulated to the point of making teacher supervision unreliable. At $k = 1$, the student only needs to produce a correct first action—there is no history of previous errors to contaminate the state, so the teacher's supervision at turn 0 is reliable (the initial observation $o_0$ is always drawn from the environment's distribution, never from student errors). As $k$ increases, the student gradually learns to maintain correct behavior over longer horizons, but only after it has mastered the earlier turns that provide the foundation for later-turn performance.

The curriculum pacing schedule. The trajectory depth $k$ is controlled by a linear schedule:

$k = k_{\text{start}} + \lfloor n / \eta \rfloor$

where $n \in \{1, \ldots, N\}$ is the current training step, $N$ is the total number of training steps (set to 250 in all experiments), $k_{\text{start}}$ is the initial number of interaction steps (fixed to 1 in all experiments), and $\eta$ is the curriculum's growth rate (swept over $\{2, 4, 6\}$ in experiments).

What it computes: at each training step $n$, the maximum trajectory depth $k$ is determined by starting from $k_{\text{start}}$ and adding one additional step for every $\eta$ training steps. The floor function $\lfloor \cdot \rfloor$ ensures integer step counts. The final depth is capped at the environment's maximum horizon $T_{\text{max}}$ (30 for ALFWorld and ScienceWorld, 15 for WebShop). For example, with $k_{\text{start}} = 1$ and $\eta = 2$: at $n = 1$, $k = 1 + \lfloor 1/2 \rfloor = 1$; at $n = 3$, $k = 1 + \lfloor 3/2 \rfloor = 2$; at $n = 2T_{\text{max}}$, $k$ reaches $T_{\text{max}}$ and remains there.

Why this form: the linear schedule is chosen for simplicity and robustness. The paper demonstrates (Table 3) that performance varies by less than 2% across $\eta \in \{2, 4, 6\}$ on most benchmarks, indicating that the exact pacing rate is not critical—the key factor is that the curriculum exists and progresses monotonically, not the precise speed of progression. The linear form has the advantage of being parameter-free beyond $\eta$ (no learning rate decay, no adaptive thresholds) and requiring minimal code changes to implement. The paper explicitly notes: "This approach requires only minor code changes" (Section 4.2).

Algorithm 1: TCOD-F2B training procedure. The complete algorithm (reproduced from the paper) is:

Algorithm 1: Temporal Curriculum On-Policy Distillation: TCOD-F2B
1: Input: Student π_θ, Teacher π_φ, Environment E, total steps N,
          curriculum parameters k_start, η
2: Output: Trained student policy π_θ
3: for n = 1, 2, ..., N do
4:     k ← min(k_start + ⌊n/η⌋, T_max)
5:     Initialize s_0 ∼ E, history h_0 ← ∅
6:     for t = 0, 1, ..., k-1 do
7:         Sample a_t ∼ π_θ(· | h_t); execute a_t; update h_{t+1}
8:     end for
9:     L ← Σ_{t=0}^{k} D_KL(π_φ(a_t | h_t) ∥ π_θ(a_t | h_t))
10:    Update θ ← θ - ∇_θ L
11: end for
12: return π_θ

The procedure is straightforward: at each training step, determine the current trajectory depth $k$ from the pacing schedule, let the student interact with the environment for exactly $k$ steps, compute the KL divergence loss over those $k$ steps, and update the student's parameters. The student never sees turns beyond $k$, which prevents it from being exposed to states where error accumulation has made teacher supervision unreliable.

Design choice: why $k_{\text{start}} = 1$? The paper fixes the starting depth at 1 for all experiments. This means the student begins training by learning only the first action of each task—the simplest possible learning problem, with no compounding error possible because there are no previous turns to accumulate errors from. The paper does not ablate this choice, but it follows naturally from the diagnosis: if errors compound across turns, starting at $k = 1$ provides the cleanest possible initial learning signal (teacher supervision on the initial observation distribution, which is always in-distribution for the teacher).

Design choice: why linear pacing instead of exponential or adaptive? The paper explicitly acknowledges that "the optimal pace may vary with different environments or student–teacher pairs" (Appendix A) and suggests that "an adaptive mechanism that automatically adjusts the horizon based on the student's learning progress—such as through an exponential moving average of the KL divergence—could further improve generality." However, the linear schedule is chosen for the current work because it is robust (less than 2% performance variation across $\eta$ values), simple to implement, and sufficient for demonstrating the core contribution. The paper treats adaptive pacing as future work.

TCOD-B2F: Backward-to-Forward Temporal Curriculum

TCOD-B2F takes a complementary approach: instead of restricting the student to early turns and progressively extending forward, it uses the teacher to navigate the environment to near-terminal states (bypassing early-turn error accumulation entirely) and gradually expands the student's responsibility backward toward the initial state.

Core idea: the teacher as navigator. The key insight is that early-turn errors are the root cause of trajectory-level KL instability (since errors at turn $t$ poison all subsequent turns), so the most direct mitigation is to prevent the student from making early-turn errors at all during the early stages of training. TCOD-B2F achieves this by having the teacher execute the initial segment of a pre-collected successful trajectory, placing the environment in a state that the student can be confident is "on the right track," and then letting the student take over from that intermediate state to complete the task.

Pre-collection of successful teacher trajectories. Before training begins, the teacher policy $\pi_\phi$ is used to generate successful trajectories on the training tasks using a pass@10 sampling strategy: the teacher attempts each task up to 10 times, and trajectories that result in task completion are retained in a dataset $\mathcal{T}^* = \{\tau^*\}$. Only successful trajectories are used, ensuring that the states the student starts from are states that demonstrably lead to task completion under the teacher's policy.

The TCOD-B2F objective. The student does not generate the initial segment of the trajectory. Instead, the teacher replays the first $L - k$ steps of a pre-collected successful trajectory $\tau^*$ (where $L$ is the total length of that successful trajectory), placing the environment in state $h_{L-k}$. The student then takes over from step $L - k$ and generates the remaining $k$ steps. The loss is computed only over the student-generated portion:

$\mathcal{L}_{\text{TCOD-B2F}}(\theta) = \mathbb{E}_{\tau \sim (\pi_\phi, \pi_\theta)} \left[ \sum_{t=L-k+1}^{T-1} D_{\text{KL}}\left( \pi_\phi(a_t | h_t) \parallel \pi_\theta(a_t | h_t) \right) \right]$

where $\tau \sim (\pi_\phi, \pi_\theta)$ indicates that the trajectory is generated by first executing teacher actions (from the pre-collected successful trajectory) for $L-k$ steps, then switching to student-generated actions for the remaining steps, $L$ is the length of the sampled successful teacher trajectory $\tau^*$, $k$ is the current curriculum-controlled number of student steps, and $t$ indexes turns only over the student-generated portion (from $L-k$ to $T-1$).

What it computes: at each training step, a successful teacher trajectory $\tau^*$ of length $L$ is sampled from the pre-collected dataset. The teacher executes the first $L-k$ actions from $\tau^*$ in the environment (with gradient computation disabled—these are "stop gradient" steps that serve only to position the student). The environment state after these $L-k$ steps is $h_{L-k}$, which is a state known to be on a path to task completion. The student then generates actions for the remaining $k$ steps starting from this state. The KL divergence is computed only over the student-generated steps, and the loss updates the student's parameters.

Why this form: by having the teacher execute the early steps, TCOD-B2F completely avoids the early-turn error accumulation that causes KL instability in vanilla OPD. The student never makes errors at turns $0$ through $L-k-1$ because it never generates actions for those turns—the teacher handles them. This means the state $h_{L-k}$ that the student starts from is always an in-distribution state for the teacher (since it was reached by following a teacher trajectory), so the teacher's supervision on the student's first generated step is reliable. The student only needs to learn how to complete the task from progressively earlier starting points, which is a well-scaffolded learning problem.

The curriculum pacing in B2F. The number of student steps $k$ starts small (meaning the teacher executes most of the trajectory and the student only handles the final few steps) and increases according to the same linear schedule as F2B (Equation 4). When $k = 1$, the teacher executes $L-1$ steps and the student only has to produce the final action—the easiest possible learning problem, since the student is placed at the "doorstep of success" (as the paper describes it). As training progresses and $k$ increases, the teacher executes fewer steps and the student takes over from progressively earlier points in the trajectory. At the end of training, $k = L$ (the teacher prefix length is zero), and the student executes the full trajectory end-to-end from the initial state.

Algorithm 2: TCOD-B2F training procedure. The complete algorithm is:

Algorithm 2: Temporal Curriculum On-Policy Distillation: TCOD-B2F
1: Input: Student π_θ, Teacher π_φ, Environment E, total steps N,
          curriculum parameters k_start, η
2: Output: Trained student policy π_θ
3: Pre-collect teacher successful trajectories T* ← {τ*}
4: for n = 1, 2, ..., N do
5:     k ← min(k_start + ⌊n/η⌋, L)
6:     Sample τ* ∈ T* with length L; initialize s_0 ∼ E
7:     for t = 0, 1, ..., L-k-1 do
8:         Execute teacher action a*_t (stop gradient); update h_{t+1}
9:     end for
10:    for t = L-k, ..., L do
11:        Sample a_t ∼ π_θ(· | h_t); execute a_t; update h_{t+1}
12:    end for
13:    L ← Σ_{t=L-k}^{L} D_KL(π_φ(a_t | h_t) ∥ π_θ(a_t | h_t))
14:    Update θ ← θ - ∇_θ L
15: end for
16: return π_θ

The procedure has two phases within each training iteration. Phase 1 (lines 7–9): the teacher replays the first $L-k$ actions from the sampled successful trajectory, updating the environment state but contributing nothing to the gradient (stop gradient). Phase 2 (lines 10–12): the student takes over from the state reached by the teacher and generates $k$ actions, which are scored against the teacher's probabilities and used to compute the loss. The student's parameters are updated to minimize the KL divergence over its generated portion of the trajectory.

Addressing the train-test mismatch. A natural concern with TCOD-B2F is that during training, the student starts from teacher-navigated intermediate states, but at test time it must operate end-to-end from the initial state without any teacher assistance. The paper addresses this explicitly:

"we gradually reduce the teacher's prefix from $L-1$ steps down to zero, ensuring that by the end of training the student executes the full trajectory from the initial state with no teacher intervention, fully aligning the training and test distributions."

The curriculum schedule ensures a smooth transition: early in training ($k$ small), the student operates primarily from near-terminal states and learns how to finish tasks. Mid-training ($k$ medium), the student takes over from earlier states and learns how to navigate the middle portion of tasks. Late in training ($k = L$), the student executes full trajectories end-to-end, matching the test-time distribution exactly. The paper reports in Appendix D.5 that the end-to-end success rate on the test set "increases steadily with training steps, confirming that the smooth curriculum transition effectively prevents catastrophic distribution shift in practice."

Design choice: why pre-collect teacher trajectories instead of generating them on-the-fly? The paper pre-collects successful trajectories before training begins, which requires an initial data collection phase. This is a practical tradeoff: pre-collection decouples trajectory generation from training, avoiding the need to run the teacher during each training iteration (which would increase per-step latency). However, it also means the teacher trajectories are off-policy relative to the student—they were generated by the teacher alone, not by the student interacting with the teacher's prefix. The paper acknowledges this limitation in Appendix A: "TCOD-B2F relies on pre-collected successful teacher trajectories, which may require additional trajectory collection overhead. In such cases, the forward-to-backward variant (TCOD-F2B) provides a drop-in alternative that requires no demonstrations."

Design choice: why pass@10 for trajectory collection? The teacher attempts each task up to 10 times and retains only successful trajectories. This ensures that the pre-collected dataset contains only trajectories that demonstrably lead to task completion, which is necessary for B2F to work: the teacher must be able to navigate to states from which the student can realistically complete the task. Pass@10 provides a tradeoff between coverage (more attempts = more tasks with successful trajectories) and collection cost (more attempts = more computation).

Asynchronous Training Infrastructure

The paper describes several practical design choices for the training infrastructure in Section 4.3. While these are not the core contribution of TCOD, they significantly impact training stability and efficiency, and the paper's experimental results depend on them.

Distributed actor-learner architecture. Training is conducted on 8× NVIDIA H20 (96GB) GPUs, partitioned as follows:

• 4 GPUs for actor processes (trajectory collection)
• 2 GPUs for learner processes (model optimization)
• 2 GPUs for teacher models (inference only)

The actors and learner are decoupled into separate asynchronous processes. Actors continuously sample trajectories from the current student policy and deposit them into a shared buffer. The learner continuously samples from this buffer and performs gradient updates. The paper uses a "lock-free ring buffer to minimize synchronization overhead." This architecture maximizes GPU utilization by ensuring that the learner is never idle waiting for trajectories, and the actors are never idle waiting for the learner to finish an update.

Staleness-aware sub-trajectory experience replay. To maximize sample efficiency in multi-turn environments, each complete trajectory of length $n$ is decomposed into multiple prefix sub-trajectories, each of which is stored as an independent experience entry. Specifically, for a complete trajectory $\tau = (s_0, a_0, s_1, a_1, \ldots, s_n)$, the system stores each prefix $\tau_{1:t} = (s_0, a_0, \ldots, s_t)$ for $t \in \{1, \ldots, n\}$. This means a single 20-step trajectory produces 20 distinct training examples, each representing a sub-trajectory of different length. The paper states:

"To prevent the input context from exceeding the model's effective memory limit, leading to training instability, we encapsulate the interaction history within the prompt as a structured context. Consequently, the number of rollouts generated per batch is dynamic, depending on the varying lengths of collected trajectories."

The staleness control mechanism works as follows: each trajectory in the replay buffer is tagged with the version number of the policy that generated it ($\pi_{\theta_n}$). When the learner samples a trajectory, it checks whether $n_{\text{current}} - n_{\text{old}} > \Delta_{\text{max}}$. If the trajectory is too stale (generated by a policy more than $\Delta_{\text{max}}$ versions old), it is discarded. The paper sets $\Delta_{\text{max}} = 2$, stating that this "provides an optimal balance between sample efficiency and the strictness of the on-policy constraint."

Why staleness control matters for TCOD. The temporal curriculum in TCOD means that the student's policy is systematically changing not just due to parameter updates, but also due to the expanding trajectory horizon. A trajectory collected when $k = 3$ may behave very differently from what the current policy (now at $k = 7$) would produce, because the student has been trained on longer horizons and may have developed different strategies. The staleness filter ensures that the learner does not train on trajectories that are too far from the current policy's behavior, maintaining the on-policy property that is central to OPD's theoretical motivation.

Why sub-trajectory replay matters for efficiency. Multi-turn trajectories vary in length depending on when the task terminates (either by success, failure, or reaching the horizon). Without sub-trajectory decomposition, a batch would need to contain complete trajectories, leading to variable and potentially very large batch sizes (a batch of 16 trajectories of length 25 each would contain 400 total turns, while a batch of 16 trajectories of length 5 would contain only 80 turns). Sub-trajectory decomposition allows the system to construct batches with a consistent total number of turns by mixing prefixes of different lengths, improving GPU utilization and training throughput.

Summary of Design Choices and Their Justifications

This section synthesizes the key design decisions in TCOD and explains why each was made, connecting back to the empirical diagnosis in Section 4.1.

Temporal curriculum over full-trajectory training: motivated directly by Observation 1 and Observation 2—compounding errors across turns cause KL escalation and success rate collapse. Full-trajectory training exposes the student to all error-accumulated states from the start, which is inefficient at best (for larger models) and catastrophic at worst (for smaller models). The temporal curriculum prevents this by limiting the student's exposure to error-prone turns.

Trajectory length as difficulty metric over external difficulty estimators: prior curriculum learning approaches for agents (Shi et al., 2025; Wang & Ammanabrolu, 2025; Gong et al., 2026) use external models to measure task difficulty, adding complexity and breaking the self-contained nature of OPD. Trajectory length is a natural, calibration-free difficulty signal—longer trajectories provide more opportunities for error accumulation, making them inherently harder—and is built directly into the training process through the pacing schedule.

Linear pacing over exponential or adaptive: the paper demonstrates that a simple linear schedule works robustly across three benchmarks and multiple model scales (less than 2% performance variation across $\eta \in \{2, 4, 6\}$ in Table 3). The linear schedule is parameter-free beyond $\eta$, requires minimal code changes, and is sufficient to demonstrate the core contribution. Adaptive pacing is identified as future work (Appendix A).

Two complementary variants (F2B and B2F) over a single approach: F2B and B2F address the same problem (early-turn error accumulation) from opposite directions. F2B restricts the student to early turns and extends forward; B2F places the student at near-terminal states and extends backward. This dual approach accommodates different practical constraints—F2B requires no pre-collected demonstrations and is simpler to implement, while B2F provides stronger mitigation of early-turn errors (the student never makes early-turn mistakes because the teacher handles those steps) but requires pre-collected successful teacher trajectories.

$k_{\text{start}} = 1$ for both variants: starting from a single step provides the cleanest possible initial learning signal, with zero opportunity for compounding errors (there are no previous turns to accumulate errors from). This is the simplest way to ensure that the teacher's supervision is reliable at the start of training, when the student is least competent and most vulnerable to destabilization.

Staleness control ($\Delta_{\text{max}} = 2$) and sub-trajectory replay: these are practical choices motivated by the asynchronous training architecture and the need to maintain sample efficiency while respecting the on-policy constraint. Sub-trajectory replay maximizes the number of training examples extracted from each trajectory. Staleness filtering prevents the learner from training on data generated by significantly outdated policies, which is particularly important given the systematically changing policy induced by the temporal curriculum.

Fixed total training steps ($N = 250$): all experiments use 250 training steps. This is a relatively short training horizon, made feasible by the sample efficiency of OPD (dense token-level supervision) and further improved by TCOD's curriculum (which focuses the student's learning on manageable trajectory depths). The paper shows that TCOD reduces total training time by up to 32% compared to vanilla OPD (Figure 6), meaning that 250 steps under TCOD produces better results than 250 steps under vanilla OPD—the efficiency gain comes from completing training faster, not from training longer.

4. Key Insights and Innovations

Innovation 1: Diagnosing a Previously Invisible Failure Mode — Trajectory-Level KL Instability

The paper's most fundamental contribution is not a new method but the identification and empirical characterization of a failure mode that the field did not know existed: on-policy distillation destabilizes catastrophically in multi-turn agent settings due to compounding errors across turns, a phenomenon the authors name Trajectory-Level KL Instability. This is a genuinely new concept in the distillation literature, and its significance lies in exposing a hidden assumption that prior work had implicitly relied upon without articulating.

Before this paper, on-policy distillation had been validated exclusively on single-turn tasks—mathematical reasoning, question answering, code generation—where each rollout is an independent sample from a fixed initial state. The implicit assumption was that the teacher's token-level probability distribution provides a reliable supervision signal regardless of what the student generates, because the teacher is a more capable model and should be able to assess any state the student reaches. Prior work on improving OPD (Jang et al., 2026; Jin et al., 2026; Ko et al., 2026) focused on objective design and optimization heuristics—balancing forward and backward KL terms, reward clipping, entropy regularization—all aimed at improving convergence within the single-turn regime. None of this work questioned whether the fundamental premise of OPD (teacher supervision on student-generated states) might break down when the student's errors reshape the state distribution itself.

The paper's diagnosis reveals that this premise breaks down systematically in multi-turn settings. Figure 2d provides the smoking gun: per-turn KL divergence increases monotonically with turn index across multiple teacher-student pairs, demonstrating that the teacher's supervision becomes progressively less reliable the further the student progresses into a trajectory. This is not a problem of model capacity (it occurs with both 30B general-purpose teachers and 7B domain-specialized teachers) nor of model scale (it affects everything from 0.5B to 7B students, with severity varying but the pattern persisting). It is a structural property of multi-turn interaction: the student's action at turn t becomes part of the input at turn t+1 through the history ht+1, so errors causally propagate forward in a way that does not occur in single-turn reasoning.

What makes this diagnosis intellectually distinctive is that it reconceptualizes the difficulty of multi-turn distillation as a distribution shift problem, not a credit assignment problem. The standard intuition about why multi-turn agent training is hard invokes long-horizon credit assignment—it's unclear which of many actions caused success or failure, and sparse rewards provide weak learning signals. The RL for agents literature (Guo et al., 2025; Feng et al., 2025; Penaloza et al., 2026) has focused on this angle. TCOD's diagnosis suggests a fundamentally different bottleneck: even when a dense token-level supervision signal is available (from the teacher), that signal becomes actively harmful when the student reaches states the teacher cannot reliably evaluate. The problem is not that the signal is sparse; it's that the signal is dense but progressively corrupted.

This is a fundamental reframing, not an incremental observation. It implies that improving multi-turn agent training requires not better credit assignment or more sophisticated RL algorithms, but mechanisms for keeping the student within the teacher's effective support during training—exactly the design principle behind TCOD's temporal curriculum. The diagnosis also explains why SFT on teacher demonstrations (which avoids student-generated states entirely by training only on correct trajectories) outperforms vanilla OPD in some regimes (Table 2: SFT achieves 32% on Qwen2.5-3B versus 66% for OPD, but OPD presumably had to work through severe initial instability to reach that 66%). The field had no framework for understanding such tradeoffs before this paper's characterization of trajectory-level KL instability.

The diagnosis is supported by extensive evidence: Figure 2a and 2b show KL escalation co-occurring with success rate collapse for small models; Figure 2c shows prohibitively high initial KL (~1000 vs. converged ~60) even for larger models that eventually recover; Figure 2d shows per-turn KL increasing across multiple teacher types; and Appendix B, Figure 7 extends these observations across four additional student-teacher pairs with consistent patterns. The breadth of evidence across model families (Qwen3 and Qwen2.5), scales (0.5B to 7B), and teacher types (general-purpose 30B and domain-specific 7B) makes a strong case that this is a general phenomenon, not an artifact of specific hyperparameters or architectures.

Innovation 2: Trajectory Length as an Intrinsic, Calibration-Free Difficulty Metric for Curriculum Learning

The second conceptual contribution is the insight that trajectory length itself can serve as a difficulty metric for curriculum learning in multi-turn agent training, eliminating the need for external difficulty estimators, human annotations, or auxiliary models. This is a clean conceptual move that distinguishes TCOD from prior curriculum learning approaches and has implications beyond the specific OPD setting.

Prior work on curriculum learning for LLM agents has relied on external mechanisms to measure and rank task difficulty. Shi et al. (2025), Wang & Ammanabrolu (2025), and Gong et al. (2026) all use separate models or heuristics to sort training examples from easy to hard before or during training. Zhang et al. (2026) and Wang et al. (2025b) apply curriculum learning to pretraining and post-training respectively, but still require external criteria for what constitutes "easy" versus "hard" data. Lauffer et al. (2025) trains on teacher corrective actions, which implicitly defines difficulty by where the teacher intervenes, but breaks the on-policy setting. In all these approaches, the difficulty signal is extrinsic to the training process—it comes from a separate model, a preprocessing step, or human annotation.

TCOD's innovation is to define difficulty intrinsically through trajectory depth. The core reasoning, which follows directly from the diagnosis of KL instability, is elegant: shorter trajectories are inherently easier for distillation because they provide fewer opportunities for compounding errors to corrupt the teacher's supervision signal. At k = 1, the student learns from a single turn where the initial state o0 is always drawn from the environment's natural distribution (not from student errors), so the teacher's supervision is maximally reliable. As k increases, the student must maintain correct behavior over longer horizons, with each additional turn introducing the risk that an error at turn t will degrade the teacher's supervision at turn t+1 and beyond. The difficulty of the learning problem thus increases monotonically with k, and this increase is controlled entirely by the pacing schedule—no external difficulty estimator, no data preprocessing, no human annotation.

This is a conceptual reframing of what "difficulty" means in multi-turn agent training, not just a convenience. It shifts the definition of difficulty from "does this task require complex reasoning?" (which requires an oracle or a model to assess) to "how many steps of error-free behavior must the student sustain?" (which is directly measurable from the training process itself). The former is task-dependent and requires task-specific knowledge; the latter is task-independent and applies to any multi-turn environment regardless of domain. This explains why TCOD works across three fundamentally different benchmarks—embodied navigation (ALFWorld), web shopping (WebShop), and scientific reasoning (ScienceWorld)—without any domain-specific curriculum design.

The practical significance extends beyond OPD. The insight that trajectory length is a natural difficulty metric suggests a general principle for curriculum design in any multi-turn training setting (RL, imitation learning, or distillation): start by training on short horizons where error accumulation is limited, and progressively extend the horizon as the policy improves. This principle does not depend on the specific training algorithm (OPD vs. PPO vs. behavioral cloning) and could inform curriculum design across the broader agent training literature.

The paper's evidence for this innovation is primarily the robustness of TCOD to the curriculum growth rate η. Table 3 shows that across three benchmarks and multiple model sizes, varying η from 2 to 6 produces less than 2% variation in success rate. This insensitivity is strong evidence that the exact pacing schedule is not critical—what matters is the presence of the temporal curriculum itself, which is consistent with the interpretation that trajectory length is a naturally well-behaved difficulty metric rather than a brittle heuristic that requires careful tuning. The paper explicitly notes this in Section 5.4:

"Performance remains consistently stronger than vanilla OPD across settings, with less than 2% variation in success rate, demonstrating that TCOD-F2B/B2F is not sensitive to the specific choice of η."

Innovation 3: Two Complementary Curriculum Directions — Forward Truncation and Backward Navigation — That Address the Same Instability from Opposite Ends

The third conceptual contribution is the dual-variant design of TCOD, which demonstrates that the same underlying problem (compounding errors causing KL instability) can be addressed by restricting forward progress (F2B) or by providing a clean starting point (B2F). This is not merely a choice between two implementation options—it represents a deeper insight about the structure of the instability and the design space of interventions.

The paper's diagnosis reveals that the root cause of trajectory-level KL instability is early-turn errors: a mistake at turn 5 poisons all subsequent turns by pushing the history ht into states where the teacher's supervision is poorly calibrated. This suggests two natural, complementary intervention strategies:

1. F2B (Forward-to-Backward): Prevent the student from reaching error-prone later turns by truncating trajectories at k steps. Early in training, k is small, so the student never encounters the late-turn states where error accumulation would make teacher supervision unreliable. As k increases, the student gradually learns to sustain correct behavior over longer horizons. This addresses the problem by limiting how far errors can propagate.

2. B2F (Backward-to-Forward): Prevent the student from making early-turn errors at all by having the teacher execute the initial segment. The student first learns to complete tasks from near-terminal states (where the teacher has already navigated through the difficult early portion), then progressively takes over from earlier starting points. This addresses the problem by eliminating error accumulation in the most vulnerable portion of the trajectory.

What makes this dual design intellectually interesting is that both variants implement the same core principle—temporal curriculum pacing with k increasing monotonically—but through opposite mechanisms that have different practical properties and failure modes. F2B is simpler (no pre-collected demonstrations required) and more computationally efficient (Figure 6 shows F2B uses fewer action steps than B2F, which requires additional teacher-executed steps). B2F provides stronger mitigation of early-turn errors (the student literally cannot make mistakes at the beginning of training because it never generates actions for those turns) but requires pre-collected successful teacher trajectories and introduces a potential train-test mismatch that must be managed through the curriculum schedule.

This is an incremental but practically significant contribution to the design of curriculum learning for agents. Prior work on curriculum learning for RL agents has explored various difficulty metrics and pacing strategies, but the specific insight that trajectory truncation and state initialization are dual approaches to the same problem—and that both can be implemented within the same minimal framework—is novel. It also provides practical guidance: use F2B when simplicity and computational efficiency are priorities and the student is large enough to recover from some early errors; use B2F when teacher demonstrations are available and the student is small enough that early errors would be catastrophic (as in the Qwen3-1.7B experiments in Table 3, where vanilla OPD collapses to near-zero success rates and TCOD-B2F recovers to ~25%).

The evidence for this dual-variant design being effective comes from Table 2 and Table 3, where both F2B and B2F consistently outperform vanilla OPD, with the stronger variant depending on the specific setting. On ALFWorld with a domain-specialized teacher (Table 2), F2B achieves 81.43% for the 3B student while B2F achieves 77.86%—both substantial improvements over OPD's 65.72%, with F2B having a slight edge. On cross-benchmark evaluation with a general-purpose teacher (Table 3), both variants perform comparably (within ~1-2 points of each other on average), with neither consistently dominant. This supports the interpretation that both variants implement the same core principle effectively, and the choice between them depends on practical constraints (demonstration availability, computational budget) rather than fundamental superiority.

Innovation 4: Evidence That Temporal Curriculum Enables Students to Surpass the Teacher — Generalization Beyond the Teacher's Capability Boundary

The fourth intellectual contribution is the empirical demonstration that a properly scaffolded student can outperform its teacher on tasks where the teacher itself fails, establishing that temporal curriculum pacing does not merely stabilize imitation but enables genuine generalization. This finding challenges the common assumption that distillation necessarily produces a student that is, at best, as capable as its teacher.

The standard mental model for knowledge distillation is that the student learns to approximate the teacher's behavior—it recovers the teacher's capabilities, perhaps with some efficiency or compression benefits, but cannot exceed them. This assumption is implicit in the framing of most distillation work, where the teacher's performance serves as the theoretical upper bound. The paper's construction of a "Hard" split on ALFWorld—121 tasks where the teacher fails under pass@10 sampling—directly tests whether this assumption holds in multi-turn settings with temporal curriculum training.

The results in Table 2 are striking: on the Hard split, the teacher (GRPO-trained Qwen2.5-7B) achieves only 6.61% success rate, while TCOD-B2F with the same 7B backbone achieves 20.66%—a 14-point improvement over the teacher. TCOD-F2B achieves 18.18%, also substantially above the teacher. Both TCOD variants also surpass the teacher on the Unseen split (TCOD-F2B: 79.19% vs. Teacher: 76.87%; TCOD-B2F: 77.61% vs. Teacher: 76.87%). The student is not merely recovering the teacher's capability—it is exceeding it on tasks where the teacher is weak.

The mechanism behind this generalization is not fully explained in the paper, but the diagnosis of KL instability provides a plausible interpretation. The teacher's failures on the Hard split may stem from errors early in the trajectory that compound into failures later—the same compounding error phenomenon that causes KL instability during distillation. The student, trained with a temporal curriculum that systematically prevents early-error accumulation, may learn more robust early-turn behavior than the teacher exhibits. Because the student is never exposed to error-compounded states during early training (when k is small), its policy for early turns may be more consistent and less prone to the specific error patterns that cause the teacher to fail. In effect, the temporal curriculum acts as a form of error-correcting training that produces a policy more robust to compounding errors than the teacher's own policy.

This finding has significant implications for the broader field of agent training. It suggests that distillation with temporal curriculum is not merely a compression technique (making a smaller model behave like a larger one) but a potential improvement technique—a way to produce policies that are better than the best available teacher on specific tasks. This is reminiscent of the observation in AlphaGo Zero that training against a weaker opponent (the model's own previous version) can produce stronger play than training against a fixed strong opponent, because the curriculum of progressively harder opponents provides a better learning signal. Here, the curriculum of progressively longer trajectories provides a better learning signal than training on full trajectories from the start, even when the teacher is the same model.

The practical implication is that organizations with a domain-specialized teacher that performs well on most tasks but fails on a tail of hard cases might use TCOD (specifically B2F, since it requires teacher demonstrations for the easy cases) to train a student that matches the teacher on the easy cases while exceeding it on the hard ones. This transforms distillation from a "good enough" approximation technique into a potential tool for capability amplification.

The evidence for this innovation rests on Table 2's Hard and Unseen split results, which should be interpreted with the caveat that the 500-question ALFWorld test set provides relatively small per-split sample sizes (the Hard split contains 121 tasks). The consistency of the finding across both the Unseen and Hard splits, and across both TCOD variants, strengthens the claim, but replication on larger benchmarks would be valuable.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper evaluates on three text-based multi-turn agent benchmarks: ALFWorld (Shridhar et al., 2020), an embodied navigation environment with seen and unseen splits plus a custom Hard split (121 tasks where the teacher fails under pass@10 sampling); WebShop (Yao et al., 2022a), an e-commerce platform requiring multi-turn search and product selection; and ScienceWorld (Wang et al., 2022), a scientific reasoning environment covering 30 task types aligned with elementary science curricula. All benchmarks are described in Table 1, with maximum turns ranging from 15 (WebShop) to 30 (ALFWorld and ScienceWorld). The ALFWorld unseen split contains novel room layouts and object combinations not encountered during training, serving as an out-of-distribution (OOD) evaluation; the Hard split is constructed from the training set by selecting tasks where the teacher fails to complete the task in any of 10 sampling attempts (pass@10 = 0).

Base model(s). Two model families are used across experiments. For the main ALFWorld experiments (Table 2), students are Qwen2.5-3B and Qwen2.5-7B, with a GRPO-trained Qwen2.5-7B serving as the domain-specialized teacher. For the cross-benchmark evaluation spanning all three environments (Table 3), students are Qwen3-1.7B and Qwen3-4B, with Qwen3-30B-A3B-Instruct serving as the general-purpose teacher. This dual-family design tests TCOD across both domain-adapted teachers (trained specifically on ALFWorld via GRPO) and large general-purpose teachers (30B MoE architecture), spanning student scales from 1.7B to 7B parameters. The Qwen3-30B-A3B-Instruct teacher uses a mixture-of-experts architecture with 30B total parameters and 3B active parameters per token, testing whether TCOD works when the teacher and student have substantially different architectural properties.

Metrics. The primary metric is success rate (SR), measured as the percentage of test tasks completed successfully, where task completion is a binary outcome determined by the environment (e.g., the agent places the correct object in the correct location in ALFWorld, or purchases the specified product in WebShop). A secondary metric is average action rounds (Rounds), which measures the mean number of interaction steps the agent takes to complete tasks, with lower values indicating more efficient behavior. During training, additional monitoring metrics include trajectory-level KL divergence mean (averaged over all tokens in all turns of sampled rollouts), rollout success rate (fraction of training rollouts that succeed), critic advantages mean, maximum response length, and policy gradient loss (shown in Figures 4 and 5). These training metrics serve to diagnose the KL instability phenomenon and verify that TCOD stabilizes training dynamics.

Baselines. Four baselines establish performance boundaries and isolate the effect of the temporal curriculum:

• Teacher (Oracle / Upper Bound): The frozen teacher policy evaluated directly on the test environments. This represents the theoretical upper bound for standard distillation (the student is expected to at most recover the teacher's capability, though TCOD's Hard split results challenge this assumption).

• Zero-Shot Student (Lower Bound): The base student model evaluated directly on the interactive tasks without any task-specific fine-tuning or distillation. This establishes the starting point before any training, measuring the model's raw agent capability from pretraining alone.

• Supervised Fine-Tuning (SFT): The student is fine-tuned via standard negative log-likelihood loss on pre-collected successful teacher trajectories for 2 epochs. This is the fundamental imitation learning baseline that suffers from exposure bias in multi-turn settings—the model trains on ground-truth teacher actions at every step but must generate its own actions at test time, leading to a train-test distribution mismatch.

• Vanilla On-Policy Distillation (OPD): The standard multi-turn adaptation of OPD (Agarwal et al., 2024) where the student minimizes token-level KL divergence against the teacher's distribution over the student's entire generated trajectory, without any horizon constraints or temporal curriculum. This is the direct baseline that TCOD is designed to improve upon, and it exhibits the trajectory-level KL instability that Section 4.1 diagnoses.

Generation budget / compute accounting. The paper does not use a standardized "generation budget" metric for comparing methods at test time—instead, all methods are trained for a fixed total of 250 training steps (Table 4), and the evaluation is performed on the final checkpoint. Training time is reported in hours (Figure 6) for comparing the computational efficiency of TCOD versus vanilla OPD, with total wall-clock time measured on 8× NVIDIA H20 GPUs. The paper reports that TCOD-F2B and TCOD-B2F reduce total training time by up to 32% compared to vanilla OPD, attributed to shorter trajectory lengths during early curriculum stages enabling faster data collection. At test time, all methods are evaluated under the same conditions: temperature 0.4, top-p 1.0, maximum 4,096 generation tokens, and the environment-specific maximum steps (30 for ALFWorld and ScienceWorld, 15 for WebShop). The asynchronous training infrastructure allocates 4 GPUs for actors, 2 for learners, and 2 for teachers.

Statistical protocol. The paper uses a fixed random seed (42) across all experiments for reproducibility. Evaluation is performed every 5 training steps (Table 4), with the final checkpoint at step 250 used for all reported metrics. The paper does not report confidence intervals, standard deviations, or statistical significance tests for any of the reported success rates. Cross-validation is not used; instead, the ALFWorld benchmark provides separate seen and unseen splits that serve as in-distribution and out-of-distribution test sets, respectively, and the custom Hard split provides an additional OOD evaluation targeting tasks where the teacher fails. The lack of multiple random seeds or error bars is a limitation in assessing whether the reported improvements (e.g., +15.71 SR points on ALFWorld seen for Qwen2.5-3B, Table 2) are statistically reliable or within expected variance from training stochasticity.

Main Quantitative Results

TCOD vs. Vanilla OPD on ALFWorld with a Domain-Specialized Teacher

The headline results in Table 2 demonstrate that TCOD substantially outperforms vanilla OPD across both student sizes and both evaluation splits on ALFWorld, while simultaneously reducing the average number of action steps per task.

Qwen2.5-3B student (distilled from GRPO-trained Qwen2.5-7B):

• Zero-shot: 7.86% SR (seen), 2.24% (unseen), 0.83% (hard) — essentially untrained performance.
• SFT: 32.14% (seen), 25.37% (unseen), 4.96% (hard) — improves over zero-shot but substantially below OPD.
• Vanilla OPD: 65.72% (seen), 60.45% (unseen), 10.74% (hard) — strong gains over SFT, confirming OPD's advantage in addressing exposure bias.
• TCOD-B2F: 77.86% (seen, +12.14 over OPD), 70.90% (unseen, +10.45), 13.22% (hard, +2.48), with rounds reduced by 2.16 (seen), 1.65 (unseen), and 0.48 (hard).
• TCOD-F2B: 81.43% (seen, +15.71 over OPD), 79.19% (unseen, +18.74), 9.92% (hard, −0.82), with rounds reduced by 2.97 (seen), 3.74 (unseen), and 0.07 (hard).

The key pattern: both TCOD variants improve success rate by roughly 10–19 points over vanilla OPD on the seen and unseen splits, with the largest gains on the unseen split (TCOD-F2B: +18.74 points). On the Hard split, TCOD-B2F shows a small improvement (+2.48) while TCOD-F2B shows a marginal degradation (−0.82), suggesting that B2F's teacher-navigated initialization is particularly helpful for harder tasks where early-turn errors would otherwise be catastrophic.

Qwen2.5-7B student (distilled from GRPO-trained Qwen2.5-7B—same architecture as teacher):

• Zero-shot: 9.29% (seen), 8.96% (unseen), 1.65% (hard).
• SFT: 54.29% (seen), 48.73% (unseen), 8.26% (hard).
• Vanilla OPD: 75.37% (seen), 72.14% (unseen), 13.22% (hard).
• TCOD-B2F: 86.43% (seen, +11.06), 77.61% (unseen, +5.47), 20.66% (hard, +7.44), with rounds reduced by 2.12 (seen), 0.21 (unseen), and 0.82 (hard).
• TCOD-F2B: 82.14% (seen, +6.77), 76.12% (unseen, +3.98), 18.18% (hard, +4.96), with rounds reduced by −0.04 (seen, slight increase), 0.15 (unseen), and 0.52 (hard).

Notably, the 7B student with TCOD-B2F achieves 86.43% on the seen split, slightly exceeding the teacher's own performance of 85.71% (a +0.72 point gain). On the Hard split, TCOD-B2F with the 7B student reaches 20.66%, more than tripling the teacher's 6.61%—this is the clearest evidence that TCOD enables the student to surpass the teacher's capability boundary. TCOD-B2F's advantage is larger for the 7B student than for the 3B student (particularly on Hard: +7.44 vs. +2.48), suggesting that the combination of B2F's teacher-navigated initialization and larger student capacity provides stronger generalization.

The training dynamics in Figure 4a and 4b (for Qwen2.5-7B) show that TCOD maintains a higher success rate throughout training and achieves much more stable KL divergence than vanilla OPD. Figure 5a confirms that TCOD reduces average action rounds during training, with TCOD-F2B using fewer steps than TCOD-B2F (which requires additional teacher-executed steps that do not contribute to the loss).

TCOD vs. Vanilla OPD Across Three Benchmarks with a General-Purpose Teacher

Table 3 extends the evaluation to three diverse benchmarks (WebShop, ALFWorld, ScienceWorld) using Qwen3-30B-A3B-Instruct as the teacher and Qwen3-1.7B and Qwen3-4B as students. The teacher here is a general-purpose model, not domain-adapted, and its performance on the target domains is substantially lower than the GRPO-trained teacher in Table 2 (e.g., 39.57% on ALFWorld vs. 85.71%).

Qwen3-1.7B student:

• Teacher performance: 32.84% (WebShop), 39.57% (ALFWorld), 18.42% (ScienceWorld), average 30.28%.
• Vanilla OPD: 0.14% (WebShop), 0.32% (ALFWorld), 0.05% (ScienceWorld), average 0.17% — catastrophic collapse to near-zero success rates across all benchmarks, confirming Observation 1 (Section 4.1) that small models experience KL escalation and success rate collapse under vanilla OPD.
• TCOD variants across all η settings: performance recovers to 18–25% on WebShop, 23–25% on ALFWorld, and 9–11% on ScienceWorld, with averages ranging from 18.23% to 18.84%. The best configuration is TCOD-F2B with η=6 (average 18.84%).

The key finding here is recovery, not just improvement: vanilla OPD completely fails on the 1.7B student (near-zero success rates), while TCOD recovers performance to levels that are comparable to the teacher's own performance on the target domains (the gap to teacher is roughly 10–15 points on average). This is the most dramatic demonstration of TCOD's ability to mitigate trajectory-level KL instability—without the temporal curriculum, the small student learns essentially nothing; with it, the student reaches useful performance levels across all three benchmarks.

Figure 4c and 4d provide training dynamics for the 1.5B student (Qwen2.5 family) under TCOD-F2B with η=3 and η=6: TCOD maintains stable KL divergence and achieves an increasing success rate, while vanilla OPD (not shown explicitly in these subfigures but characterized in Section 4.1) would exhibit KL escalation and success rate collapse. Figure 5c and 5d show that TCOD prevents explosion in response length (max response length remains stable) and produces smoothly decreasing policy gradient loss, in contrast to the unstable dynamics of vanilla OPD.

Qwen3-4B student:

• Teacher performance: same as above (32.84% WebShop, 39.57% ALFWorld, 18.42% ScienceWorld, average 30.28%).
• Vanilla OPD: 30.12% (WebShop), 36.85% (ALFWorld), 15.95% (ScienceWorld), average 27.64% — the larger student does not collapse catastrophically, but still has a roughly 2.6-point gap to the teacher on average, consistent with Observation 2 that larger models exhibit high initial KL but eventually converge.
• TCOD variants: most configurations modestly outperform vanilla OPD, with the best configuration (TCOD-F2B, η=2) achieving an average of 29.54% (+1.90 over OPD). Individual benchmark improvements range from +0.42 to +2.10 points. Some configurations show slight degradations (TCOD-B2F with η=4: −0.91 on WebShop; TCOD-B2F with η=6: −1.07 on WebShop and −0.07 on ScienceWorld), but the best TCOD configuration consistently outperforms OPD.

The pattern for the 4B student is more nuanced: TCOD provides modest but consistent gains, not the dramatic recovery seen for the 1.7B student. This is consistent with the paper's diagnosis: larger models partially tolerate the KL instability (they don't collapse to zero), so the room for improvement through temporal curriculum is smaller. Nevertheless, TCOD still improves performance and—importantly—the improvements come with more stable training dynamics (Figure 4a, 4b) and reduced training time (Figure 6).

Generalization Beyond the Teacher's Capability

Table 2's results on the Unseen and Hard splits address the question of whether TCOD enables the student to surpass the teacher. The findings are definitive for the Hard split and suggestive for the Unseen split:

Unseen split: TCOD-F2B with Qwen2.5-3B achieves 79.19%, compared to the teacher's 76.87%—a +2.32 point improvement. TCOD-F2B with Qwen2.5-7B achieves 76.12%, slightly below the teacher's 76.87% (−0.75). TCOD-B2F with Qwen2.5-7B achieves 77.61%, a +0.74 point improvement. The pattern is mixed but leans toward TCOD matching or slightly exceeding the teacher on unseen environments.

Hard split (121 tasks where teacher fails under pass@10): The teacher achieves only 6.61%. TCOD-B2F with Qwen2.5-7B achieves 20.66%—a 14.05-point improvement, more than tripling the teacher's performance. TCOD-F2B with Qwen2.5-7B achieves 18.18% (+11.57). Even the 3B student with TCOD-B2F achieves 13.22% (+6.61), doubling the teacher's performance. This is the strongest evidence in the paper that TCOD enables generalization beyond the teacher's capability boundary—the student solves tasks that the teacher itself cannot, despite being trained via distillation from that same teacher.

The interpretation the paper offers (Section 5.3) is that TCOD produces a more robust policy by systematically preventing early-turn error accumulation during training. The teacher's failures on these hard tasks likely stem from compounding errors early in the trajectory (the same mechanism that causes KL instability), and the temporal curriculum trains the student to avoid these failure modes by learning early-turn behavior in a controlled, error-minimized setting before facing longer horizons.

Training Efficiency

Figure 6 compares total training time for TCOD versus vanilla OPD in two settings:

• On ScienceWorld with Qwen3-4B: OPD takes approximately 9.6 hours; TCOD-F2B takes approximately 6.6 hours (31% reduction); TCOD-B2F takes approximately 8.55 hours (11% reduction).
• On ALFWorld with Qwen2.5-7B: OPD takes approximately 7.18 hours; TCOD-F2B takes approximately 4.9 hours (32% reduction); TCOD-B2F takes approximately 5.5 hours (23% reduction).

The efficiency gain for TCOD-F2B is larger because it restricts the maximum number of interaction steps during early training, producing shorter trajectories and faster data collection. TCOD-B2F also shows gains but is less efficient because the student takes additional exploratory actions even from intermediate starting states, producing longer trajectories than F2B. Figure 5a confirms that TCOD-F2B uses fewer rollout action steps than both TCOD-B2F and vanilla OPD during training.

Ablation Studies and Robustness Checks

Curriculum growth rate η (Table 3, Figures 4c, 4d): The paper sweeps η ∈ {2, 4, 6} across all three benchmarks and both TCOD variants. Performance remains consistently stronger than vanilla OPD across all η values for both the 1.7B and 4B students, with less than 2% variation in success rate. For Qwen3-1.7B, the best average across benchmarks is 18.84% (F2B, η=6) and the worst among TCOD configurations is 18.23% (F2B, η=4)—a range of only 0.61 percentage points. For Qwen3-4B, the best is 29.54% (F2B, η=2) and the worst is 27.93% (B2F, η=4)—a range of 1.61 points. This insensitivity is a significant practical finding: TCOD does not require careful tuning of the pacing rate. However, the paper notes (Section 5.4) that larger η leads to more stable KL divergence during training (Figure 4d), as the student spends more iterations mastering the current trajectory depth before advancing. The practical recommendation is to start with a small η and increase it if KL instability is observed.

TCOD-F2B vs. TCOD-B2F (Tables 2, 3): Both variants consistently outperform vanilla OPD, but neither is uniformly dominant. On ALFWorld with the domain-specialized teacher (Table 2), F2B generally outperforms B2F for the 3B student (81.43% vs. 77.86% on seen), while B2F outperforms F2B for the 7B student (86.43% vs. 82.14% on seen). On the cross-benchmark evaluation with the general-purpose teacher (Table 3), the variants perform comparably, typically within 1–2 points of each other. The most notable difference is on the Hard split (Table 2), where B2F consistently outperforms F2B (20.66% vs. 18.18% for 7B; 13.22% vs. 9.92% for 3B), suggesting B2F's teacher-navigated initialization is particularly valuable for hard tasks.

Student model scale sensitivity (Table 2 vs. Table 3, Figure 7 vs. Figure 4): The benefit of TCOD is inversely related to student capacity—larger gains for smaller models. For the 1.7B student (Table 3), TCOD recovers performance from near-zero (vanilla OPD: 0.17% average) to useful levels (TCOD: ~18.5% average)—a recovery of approximately 18 points. For the 4B student (Table 3), TCOD provides ~1.9 points of improvement on average. For the 3B student (Table 2), TCOD provides 12–16 points of improvement, and for the 7B student (Table 2), 4–11 points. This pattern is consistent with the diagnosis: smaller models suffer more severely from KL instability because they make more errors at early turns, so the temporal curriculum provides a larger benefit by preventing those errors from compounding.

Teacher quality sensitivity (Table 2 vs. Table 3): The teacher's performance on the target domain strongly affects the upper bound of TCOD. In Table 2, the GRPO-trained Qwen2.5-7B teacher achieves 85.71% on ALFWorld seen, and TCOD-B2F with the 7B student reaches 86.43% (slightly above teacher). In Table 3, the Qwen3-30B-A3B-Instruct teacher achieves only 39.57% on ALFWorld, and the best TCOD student (Qwen3-4B) reaches 39.35%—still below the teacher. This suggests that TCOD's ability to surpass the teacher (as on the Hard split) is emergent rather than guaranteed: it requires that the teacher's failures stem from the specific compounding-error patterns that TCOD's curriculum addresses, not from fundamental capability gaps that affect all turns equally.

Number of training steps (implicit, from training dynamics): All experiments use a fixed 250 training steps. Figure 4a and 4b show that TCOD's success rate and KL divergence stabilize well before step 250 for the 7B student, suggesting that the training horizon is sufficient. For smaller models (Figure 4c, 4d with 1.5B), the success rate continues to increase through step 200, suggesting that additional training steps might yield further improvements. The paper does not ablate the total number of training steps to determine whether TCOD saturates earlier or later than vanilla OPD.

Sub-trajectory decomposition (implicit, from Section 4.3): The paper describes decomposing each trajectory into prefix sub-trajectories for replay buffer efficiency, but does not ablate this choice. It is unclear whether sub-trajectory decomposition itself contributes to training stability (by providing more diverse state coverage) or is purely a throughput optimization. The staleness filter (Δmax = 2) is also not ablated, leaving open the question of how sensitive TCOD is to the on-policy constraint strictness.

Prompt and history length (Table 4, Appendix E): The paper uses 10,240 maximum prompt tokens and a history length of 2 steps (meaning the most recent 2 observations and actions are included in the prompt), with maximum 512 response tokens. These values are fixed across all experiments and not ablated. The choice of history length is particularly interesting—keeping only the most recent 2 steps rather than the full history may partially mitigate compounding errors by limiting how far errors can propagate through the context, but this effect is not separated from the temporal curriculum's contribution.

Critical Assessment

Claim 1: "TCOD mitigates KL escalation and enhances KL stability, improving agent performance by up to 18 points over vanilla OPD."

Assessment: This claim is strongly supported by the evidence, but the "up to 18 points" figure requires careful contextualization. The largest single improvement is TCOD-F2B with Qwen2.5-3B on ALFWorld unseen: 79.19% vs. vanilla OPD's 60.45%, a difference of +18.74 points (Table 2). This is a genuine and substantial improvement, and it is backed by training dynamics (Figure 4) showing that TCOD maintains stable KL divergence while OPD oscillates. However, the "+18 points" is the maximum observed gain in a specific (favorable) configuration—the average improvement across all settings is smaller. For Qwen3-4B in Table 3, the improvements over OPD average only ~1.9 points. The claim would be more accurately stated as "up to 18 points, with gains inversely proportional to student capacity and teacher domain specialization."

The catastrophic-to-recovery pattern for the 1.7B student (Table 3: from 0.17% under OPD to ~18.5% under TCOD) is arguably more significant than the 18-point gain on the already-strong 3B student, because it demonstrates that TCOD can rescue training from complete failure—not just improve an already-working pipeline. However, saying "TCOD improves by 18 points" understates this: for the 1.7B student, TCOD improves by ~18 points over vanilla OPD but OPD itself is near zero, so TCOD essentially enables learning where none was possible before. The framing as "improvement" obscures the qualitative difference between recovery-from-collapse (1.7B) and incremental improvement (7B).

A weakness: the paper does not report whether any hyperparameter tuning was performed for vanilla OPD to mitigate the observed instability. Could the KL escalation be addressed by reducing the learning rate, adjusting the KL coefficient, or using gradient clipping? The paper uses a KL coefficient of 1.0 and learning rate of 1×10⁻⁶ (Table 4) for all methods—these were chosen for TCOD but applied identically to vanilla OPD. If OPD's instability could be resolved through standard hyperparameter tuning (e.g., a much lower learning rate to slow the KL escalation), then TCOD's advantage would be partially attributable to OPD being run with suboptimal hyperparameters.

Claim 2: "TCOD enables the student to surpass the teacher's performance on tasks where the teacher fails."

Assessment: This claim is supported by the Hard split results in Table 2, but with important caveats about what is being measured. The Hard split is constructed by selecting training tasks where the teacher fails under pass@10 sampling. The teacher achieves 6.61% on these tasks at evaluation time (which is slightly above 0% because pass@10 failure during data collection does not guarantee pass@1 failure at evaluation, or because the evaluation uses a different sampling configuration). TCOD-B2F with the 7B student achieves 20.66%, a 14-point improvement. This is genuine generalization beyond the teacher's capability on this specific subset of tasks.

However, two caveats apply. First, the Hard split contains 121 tasks—a relatively small sample for drawing strong conclusions about generalization. The consistency across both the Unseen and Hard splits (plus the mixed pattern on Unseen) suggests the finding is real but its statistical reliability on a per-split basis is uncertain without confidence intervals. Second, the teacher's failure under pass@10 means these are tasks where the teacher does not find any successful trajectory in 10 attempts, but the teacher may still have useful knowledge about these tasks (e.g., it knows what the correct first action is, but makes a mistake at a later turn that prevents recovery). TCOD-B2F's teacher-navigated initialization explicitly leverages this knowledge—the teacher's successful trajectories on other tasks are used to position the student at intermediate states even for hard tasks. This means the student's "surpassing" behavior is partially enabled by the teacher's knowledge of correct prefixes, even for tasks where the teacher cannot complete the full trajectory. The claim is accurate as stated but should be understood as "TCOD enables the student to complete tasks that the teacher cannot complete end-to-end, by leveraging the teacher's partial knowledge (successful prefixes) during training."

A missing experiment: does the student surpass the teacher on the Hard split when trained with TCOD-F2B (which does not use teacher prefixes)? Yes—Table 2 shows TCOD-F2B achieves 18.18% on Hard (vs. teacher's 6.61%), confirming that even without teacher-navigated initialization, the temporal curriculum alone enables surpassing the teacher. This strengthens the claim by ruling out the explanation that B2F's teacher prefixes directly provide the solution.

Claim 3: "TCOD is robust to the curriculum growth rate with less than 2% performance variation."

Assessment: Supported by Table 3, but "less than 2%" refers to success rate points (absolute difference), not relative percentage. For Qwen3-1.7B, the range across η values is 0.61 points (18.23% to 18.84%), which is indeed less than 2 points. For Qwen3-4B, the range is 1.61 points (27.93% to 29.54%), also less than 2 points. This robustness is practically important—it means practitioners can choose η without extensive tuning—but the sweep only covers η ∈ {2, 4, 6}, which is a narrow range. At η = 2 (fastest curriculum), the maximum depth increases by 1 every 2 training steps, reaching 30 turns by step 60. At η = 6 (slowest curriculum), it takes until step 180. Both are "reasonably paced" curricula; the paper does not test extreme values (η = 1, essentially no curriculum; η = 50, essentially always at kstart = 1 for most of training) that would establish the boundaries of robustness. The finding is more accurately characterized as "insensitive to moderate variations in pacing within the tested range" rather than "completely robust to any growth rate."

Claim 4: "TCOD reduces total training time by up to 32% compared to vanilla OPD."

Assessment: Supported by Figure 6, but the efficiency gain is measured in wall-clock time on a specific hardware configuration (8× H20 GPUs), which may not transfer to other setups. The paper attributes the gain to shorter trajectories during early curriculum stages enabling faster data collection. This mechanism is plausible—at k = 1, the student generates one action per task, while vanilla OPD generates full trajectories (potentially 30 steps). However, the absolute training times (4.9–9.6 hours) are relatively short, and the percentage reduction is less meaningful than the absolute time saved (~2.3 hours on ALFWorld for the 7B student). For practitioners training on larger datasets or for more steps, the 32% reduction would translate to more substantial absolute savings, but this extrapolation is not tested.

A missing detail: the paper does not report whether the total number of gradient updates is the same between TCOD and OPD (both use 250 training steps), so the efficiency gain comes entirely from faster data collection per step, not from requiring fewer steps to converge. An important ablation would be to compare TCOD at 250 steps against vanilla OPD at a larger number of steps (enough to match TCOD's training time), to determine whether TCOD is genuinely more sample-efficient or simply faster per step due to shorter trajectories.

Claim 5: "The underlying mechanism is compounding error amplification across turns."

Assessment: The mechanism is demonstrated convincingly in Section 4.1 (Figure 2d), which shows per-turn KL divergence increasing monotonically with turn index. This is a strong correlational finding—KL divergence is higher at later turns, consistent with the compounding error hypothesis. However, the paper does not provide direct causal evidence that errors at turn t cause increased KL at turn t+1. An experiment that would strengthen the mechanism claim: intervene by manually correcting the student's action at turn t (replacing it with the teacher's action) and measure whether this prevents the KL escalation at turn t+1. If the KL escalation disappears when errors are corrected, that would be strong causal evidence. Without such an intervention, alternative explanations are possible: for example, later turns might simply have higher intrinsic KL divergence (because the teacher's distribution is flatter in later-turn states, or because later turns have more possible actions), regardless of whether the student made errors at earlier turns. The paper's framing of the mechanism is plausible and well-motivated by the data, but would benefit from causal validation experiments.

General Experimental Weaknesses

Single training run. All results appear to be from a single training run with fixed random seed (42). Without multiple seeds, it is impossible to distinguish a genuine TCOD advantage from random variation in training dynamics. This is particularly concerning for the smaller splits (Hard: 121 tasks), where success rates could vary substantially across runs.

Fixed total steps (250). The paper does not investigate whether TCOD and OPD converge at different rates. If OPD eventually catches up to TCOD given enough steps, the advantage would be about training speed rather than asymptotic performance. The training curves in Figure 4a suggest that OPD's success rate for the 7B student is still slowly increasing at step 200, while TCOD appears closer to convergence, but this is not tested beyond 250 steps.

No combination of TCOD with prior OPD improvements. The paper compares TCOD only against vanilla OPD. Would TCOD combined with the objective design improvements from Jang et al. (2026) or Jin et al. (2026) (balancing forward and backward KL) yield further gains? The paper demonstrates that temporal curriculum addresses a different failure mode (compounding errors across turns) than these prior works (training instability within single turns), so the improvements are likely complementary, but this is not tested.

Limited teacher diversity. All teachers are from the Qwen family. It is unknown whether the KL instability pattern depends on teacher architecture (e.g., would a GPT-4 teacher exhibit the same per-turn KL escalation pattern with a Llama student?) or whether TCOD's effectiveness transfers across model families with different tokenization, training data, or behavioral characteristics.

No real-time adaptive curriculum. The paper explicitly mentions adaptive pacing as future work (Appendix A) but does not compare the fixed linear schedule against any adaptive alternative. A natural baseline would be to advance the curriculum when the student's success rate on the current trajectory depth exceeds a threshold—this would not require external difficulty estimators and would directly address the concern that the optimal pacing rate is environment-dependent. Its absence is a missed opportunity to strengthen the claim that curriculum pacing (rather than the specific linear schedule) is what matters.

6. Limitations and Trade-offs

6.1 Hard Problems Remain Essentially Unsolved — The Method Creates No New Capability

The assumption or constraint. TCOD operates entirely within the capability envelope defined by the student model's ability to produce correct actions at any turn, and the teacher's ability to provide useful supervision. The temporal curriculum stabilizes training and prevents compounding errors from destroying the learning signal, but it does not enable the student to learn correct behavior on turns where the teacher's supervision is fundamentally uninformative or where the student lacks the capacity to represent the correct policy. The paper is transparent about this boundary in the cross-benchmark experiments (Table 3): the teacher (Qwen3-30B-A3B-Instruct) achieves only 39.57% on ALFWorld, 32.84% on WebShop, and 18.42% on ScienceWorld, and TCOD students never substantially exceed these teacher baselines—the best TCOD student on ALFWorld reaches 39.35% (Qwen3-4B, B2F, η=6), actually slightly below the teacher.

The consequence. TCOD cannot amplify capability beyond what the teacher can demonstrate or what the student can represent. For genuinely hard tasks where neither the teacher nor the student has the foundational knowledge to produce correct actions (e.g., scientific reasoning that requires specialized domain knowledge absent from pretraining, or embodied tasks requiring spatial reasoning beyond the model's capacity), the temporal curriculum simply stabilizes training on an inadequate supervision signal—it does not make that signal more informative. The Hard split results in Table 2 (teacher success rate: 6.61%; TCOD-B2F with 7B student: 20.66%) appear to show capability amplification, but this is likely because the teacher has useful knowledge about these tasks (it can produce correct prefixes, as leveraged by B2F's teacher-navigated initialization) but fails to sustain it over full trajectories due to its own compounding errors. This is evidence that TCOD can produce a more robust policy than the teacher, not a more knowledgeable one. For tasks where the teacher fails because it genuinely does not know what actions to take (rather than because it makes compounding errors), TCOD provides no benefit—it can only teach what the teacher knows.

What evidence exists in the paper. The ScienceWorld results in Table 3 provide the clearest evidence: the teacher achieves 18.42%, and the best TCOD student (Qwen3-4B, F2B, η=2) reaches 17.85%—still below the teacher despite stable training. This is a benchmark where task difficulty likely stems from genuine knowledge gaps (elementary science concepts) rather than from error compounding in navigation or search, and TCOD cannot close the gap. Additionally, the Qwen3-1.7B student's performance across all benchmarks in Table 3 (average ~18.5% with TCOD) remains substantially below the teacher's average of 30.28%, indicating that for the smallest models, even stabilized distillation cannot overcome fundamental capacity limitations.

Mitigation status. The paper does not address this limitation explicitly as a limitation, but the results themselves demonstrate the boundary. The implication for practitioners is clear: TCOD is appropriate when the primary obstacle is training instability (the student could learn from the teacher if training were stable), not when the obstacle is teacher capability or student capacity. The paper does not suggest how to extend TCOD to address capability gaps—that would require fundamentally different techniques (e.g., better teachers, larger students, or integration with RL exploration).

6.2 TCOD-B2F Requires Pre-Collected Successful Teacher Trajectories — Creating an Upfront Data Collection Burden That Is Not Fully Accounted For

The assumption or constraint. TCOD-B2F depends on a dataset T∗ of successful teacher trajectories collected before training begins, using a pass@10 sampling strategy. The paper explicitly acknowledges this in Appendix A:

"TCOD-B2F relies on pre-collected successful teacher trajectories, which may require additional trajectory collection overhead. In such cases, the forward-to-backward variant (TCOD-F2B) provides a drop-in alternative that requires no demonstrations."

However, the cost of this pre-collection is not quantified, amortized into the reported training time comparisons (Figure 6), or analyzed for scaling behavior. For the ALFWorld experiments with the GRPO-trained Qwen2.5-7B teacher, the teacher achieves 85.71% success rate, meaning that pass@10 collection will successfully obtain trajectories for most training tasks with relatively few attempts. But for the cross-benchmark experiments with the Qwen3-30B-A3B-Instruct teacher (39.57% on ALFWorld, 32.84% on WebShop, 18.42% on ScienceWorld), pass@10 will fail to collect successful trajectories for a substantial fraction of tasks—those tasks are simply excluded from B2F training, creating a potential bias toward easier tasks that the teacher can already solve.

The consequence. Two practical problems arise. First, unaccounted compute cost: the pass@10 collection phase consumes teacher inference compute that is not included in Figure 6's training time comparison. For a teacher running on 2 GPUs (as described in Section 4.3's infrastructure), collecting up to 10 trajectories per training task across potentially hundreds of tasks represents a non-trivial overhead. Second, coverage bias: tasks where the teacher cannot produce any successful trajectory in 10 attempts are excluded from B2F's pre-collected dataset, potentially skewing the student's training distribution away from the hardest cases. The paper's Hard split is constructed precisely from such tasks, and the results show that B2F underperforms F2B on Hard for the 3B student (13.22% vs. 9.92% in Table 2? Wait—Table 2 shows B2F at 13.22% and F2B at 9.92% for the 3B student on Hard, so B2F actually outperforms F2B here, likely because the pre-collected trajectories come from the teacher's easy tasks but the curriculum's teacher-navigated initialization still helps on hard tasks). This complicates the coverage bias concern—B2F's teacher prefixes from other tasks may still provide useful starting points for hard tasks—but the fundamental dependence on teacher demonstrations remains a deployment constraint that F2B avoids entirely.

What evidence exists in the paper. The paper does not measure or report the cost of pass@10 trajectory collection. The training time comparison in Figure 6 reports only the distillation training phase, not the pre-collection phase. The acknowledgment in Appendix A is brief and does not quantify the overhead or analyze when it becomes prohibitive. Additionally, the paper does not report what fraction of training tasks lacked successful teacher trajectories under pass@10 for each benchmark and teacher pair—this would directly measure the coverage bias in B2F's training data.

Mitigation status. The paper partially mitigates this by providing TCOD-F2B as an alternative that requires no pre-collected demonstrations. However, F2B is not always the stronger variant (B2F outperforms it on the Hard split in Table 2 and on some configurations in Table 3), so practitioners who want the best performance may need to accept the pre-collection cost. The paper does not provide guidance on when the B2F overhead is justified relative to the performance gain over F2B, nor does it explore lighter-weight alternatives (e.g., collecting only a few demonstrations per task rather than pass@10, or generating demonstrations on-the-fly during training rather than pre-collecting).

6.3 The Fixed Linear Curriculum Schedule Is Robust Within a Narrow Tested Range but Would Benefit from Adaptive Pacing — Leaving Practical Tuning Partially Unresolved

The assumption or constraint. TCOD uses a fixed linear pacing schedule (Equation 4) controlled by a single parameter η (the growth rate), with k_start fixed at 1 for all experiments. The paper sweeps η ∈ {2, 4, 6} and demonstrates that performance varies by less than 2% across these values (Table 3, Section 5.4). However, the tested range is narrow: at η = 2, the maximum trajectory depth reaches 30 turns by training step 60 (out of 250 total steps), meaning the student spends only ~24% of training in the shallow-curriculum regime. At η = 6, depth 30 is reached at step 180, meaning the student spends ~72% of training at sub-maximal depths. All three tested values represent "reasonably paced" curricula—none approaches the extreme where the curriculum is effectively absent (η = 1, depth reaches 30 by step 30, essentially vanilla OPD after the first few steps) or where the curriculum stalls (η >> 250, the student never reaches full depth). The paper explicitly acknowledges this limitation in Appendix A:

"the optimal pace may vary with different environments or student–teacher pairs. An adaptive mechanism that automatically adjusts the horizon based on the student's learning progress—such as through an exponential moving average of the KL divergence—could further improve generality; we consider this a promising direction for future investigation."

The consequence. The reported robustness to η may not generalize to substantially different settings where the optimal pacing rate falls outside the tested range. For example, an environment with 100 maximum turns might require a much slower curriculum (larger η) to prevent KL instability at intermediate depths, or a student-teacher pair with very high initial alignment might benefit from a faster curriculum (smaller η) to avoid wasting training steps on unnecessarily shallow trajectories. A practitioner deploying TCOD on a new environment or with a new model family cannot simply adopt η ∈ {2, 4, 6} and expect robustness—they would need to tune η for their specific setting, and the paper provides no principled method for doing so beyond "start with a small η... and increase η if KL divergence instability is observed" (Section 5.4). This heuristic requires monitoring KL divergence during training, which adds a monitoring burden and may not be feasible in all deployment contexts.

What evidence exists in the paper. The η sweep in Table 3 demonstrates robustness across three benchmarks and two student sizes within the tested range, which is strong evidence that TCOD does not require precise tuning. However, the paper does not test boundary values (η = 1, η = 20, η = 250) to establish the limits of robustness. Figure 4d shows that larger η leads to more stable KL divergence, confirming that the curriculum pacing does affect training dynamics even if final success rates are similar—this suggests that the KL stability benefit of larger η may be practically important even if success rate differences are small. The paper does not correlate η with training time (Figure 6 reports only the best or default configuration, not the training time variation across η values).

Mitigation status. Partial. The explicit suggestion of adaptive pacing in Appendix A acknowledges the limitation and proposes a concrete direction (EMA of KL divergence as the trigger for curriculum advancement). However, no adaptive variant is implemented or evaluated, so the current TCOD leaves practitioners with a hyperparameter (η) that is robust within a tested range but not universally calibrated. The practical guidance in Section 5.4 ("start with a small η and increase η if KL divergence instability is observed") is reasonable but manual and reactive—it requires training runs to fail before the parameter is adjusted.

6.4 Results Are Based on Single Training Runs Without Statistical Replication — Limiting Confidence in the Reported Gains

The assumption or constraint. The paper uses a fixed random seed (42) for all experiments and reports results from what appear to be single training runs. No confidence intervals, standard deviations, or error bars are reported for any success rate, KL divergence measurement, or training time comparison. The evaluation benchmark sizes are moderate (500 test questions for ALFWorld, with the Hard split comprising only 121 tasks), making individual success rate measurements potentially sensitive to run-to-run variance.

The consequence. The reported improvements—particularly the headline +18.74 point gain on ALFWorld unseen for Qwen2.5-3B (Table 2)—cannot be assessed for statistical significance. Training runs for multi-turn agents are inherently stochastic: random initialization, random environment states, random action sampling (temperature 1.0 during training), and asynchronous actor-learner interaction all introduce variance. A +18.74 point improvement is large enough that it is unlikely to be purely noise, but smaller improvements (e.g., the +1.90 average gain for Qwen3-4B in Table 3) could plausibly fall within run-to-run variance. Without multiple seeds, practitioners cannot distinguish robust gains from favorable single-run outcomes, particularly for the smaller student-teacher configurations where the absolute success rates are lower and variance might be higher.

What evidence exists in the paper. None. The paper does not report multiple runs, confidence intervals, or any statistical analysis. This is a significant methodological weakness, especially given the growing expectation in the ML community for replication across random seeds when reporting benchmark comparisons. The paper's use of separate test splits (seen vs. unseen vs. hard for ALFWorld) provides some signal about generalization consistency, but this tests generalization across task distributions, not stability across training runs.

Mitigation status. Not addressed. The paper does not mention this limitation, and the fixed seed (42) is reported in Table 4 without comment on the number of runs. The asynchronous training infrastructure (Section 4.3) introduces additional sources of non-determinism (staleness-based filtering, dynamic batch sizes from sub-trajectory replay) that could amplify run-to-run variance, making replication particularly important.

6.5 All Experiments Use Qwen Family Models Exclusively — Generalization to Other Model Architectures, Tokenizers, and Training Regimes Is Untested

The assumption or constraint. Every experiment in the paper—both students and teachers—uses models from the Qwen family (Qwen2.5-{0.5, 1.5, 3, 7}B, Qwen3-{0.6, 1.7, 4}B as students; Qwen2.5-7B-GRPO and Qwen3-30B-A3B-Instruct as teachers). The distillation dynamics studied in Section 4.1, the KL instability diagnosis, and the TCOD solution are all characterized within a single model ecosystem sharing pretraining data, tokenization, architectural decisions, and training procedures. The paper makes no claim that TCOD would work with other model families (e.g., Llama, Gemma, DeepSeek, GPT), but also provides no evidence about whether the trajectory-level KL instability is specific to Qwen models or is a universal property of multi-turn distillation.

The consequence. There are at least three ways in which model-family dependence could matter. First, tokenizer mismatch: if the teacher and student use different tokenizers (e.g., a GPT-4 teacher and a Llama student), the token-level KL divergence in Equation 2 is not directly computable—the probability distributions are defined over different token vocabularies. The Qwen family guarantees tokenizer compatibility (Qwen2.5 and Qwen3 presumably share tokenization), but this is not universal. Second, behavioral similarity: Qwen models within the same family share pretraining data and training objectives, which may make their policies more naturally aligned than cross-family pairs. The high initial KL divergence (~1000 in Figure 2c) might be even higher—and the instability even more severe—when distilling across model families with different behavioral tendencies. Conversely, it might be lower if the families have converged on similar behaviors from different training. Third, architectural properties: Qwen3-30B-A3B-Instruct uses a mixture-of-experts architecture with 3B active parameters, which is structurally different from the dense student models. The paper's results with this teacher (Table 3) provide some evidence of cross-architecture generalization, but within the same model family.

What evidence exists in the paper. The paper uses two Qwen generations (2.5 and 3), two teacher types (GRPO-domain-adapted and general-purpose instruct), and both dense and MoE architectures—providing some diversity within the Qwen ecosystem. However, all experiments are within a single model family, and the paper does not discuss tokenizer compatibility, behavioral alignment across families, or potential failure modes when the teacher and student have substantially different output distributions due to different pretraining. The observation in Appendix B (Figure 8) that "teacher–student matching matters; stronger teachers are not always better" hints that distillation dynamics depend on teacher-student similarity, which could vary substantially across model families.

Mitigation status. Not addressed. The paper does not acknowledge this as a limitation or suggest cross-family experiments as future work. The benchmark diversity (three different environments) and model scale diversity (1.7B to 7B students, 7B to 30B teachers) partially mitigate concerns about narrow experimental scope, but do not address the model family monoculture directly. A practitioner using non-Qwen models should treat TCOD's effectiveness as plausible but unvalidated for their setting.

6.6 Training Time and Sample Efficiency Comparisons Are Measured at a Fixed 250 Steps — Leaving Convergence Behavior and Asymptotic Comparisons Unresolved

The assumption or constraint. All experiments in the paper train for exactly 250 steps (Table 4), and all comparisons between TCOD and baselines are made at this fixed training horizon. The training time reduction reported in Figure 6 (up to 32% faster than vanilla OPD) and the performance improvements in Tables 2 and 3 are therefore comparisons at equal training steps, not at equal convergence. The paper does not investigate whether vanilla OPD would catch up to TCOD given additional training steps, nor whether TCOD's advantage would grow, shrink, or plateau with extended training.

The consequence. The reported gains conflate two potentially distinct effects: (1) TCOD may learn faster (higher sample efficiency per training step) because the curriculum focuses learning on manageable trajectory depths, and (2) TCOD may achieve a higher asymptotic performance ceiling because it avoids the permanent policy damage that KL instability might cause early in training. The paper's evidence supports effect (1) strongly (training dynamics in Figures 4 and 5 show TCOD achieving higher success rates earlier), but effect (2) is not tested—we do not know whether vanilla OPD with, say, 1,000 training steps would eventually reach the same performance as TCOD at 250 steps. If it would, then TCOD is primarily a training acceleration technique, not an asymptotic improvement technique. If it would not (because early KL instability causes irreversible damage to the policy), then TCOD provides a more fundamental advantage.

What evidence exists in the paper. The training curves in Figure 4a and 4c provide partial evidence. For the Qwen2.5-7B student (Figure 4a), vanilla OPD's success rate appears to still be slowly increasing at step 200, while TCOD appears to have plateaued or is increasing more slowly. This suggests that OPD might continue to improve with more steps, potentially narrowing the gap. For the Qwen2.5-1.5B student (Figure 4c), vanilla OPD's success rate is not directly shown (the figure shows only TCOD variants), but Section 4.1's diagnosis indicates that small models under vanilla OPD experience success rate collapse—in this case, more steps would not help because training has already been destabilized. However, for the larger 4B and 7B models where OPD does not collapse, the convergence behavior at extended training horizons is unmeasured.

Mitigation status. Not addressed. The paper does not discuss the choice of 250 steps, ablate the total training horizon, or compare TCOD and OPD at equal wall-clock time rather than equal steps (which would give OPD more steps to compensate for its longer per-step data collection). The 32% training time reduction reported in Figure 6 is a genuine efficiency gain, but it does not answer the question of whether that efficiency translates to a higher performance ceiling or merely a faster path to the same ceiling.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper reshapes how the field should think about on-policy distillation for interactive agents by exposing a previously invisible failure mode—trajectory-level KL instability—and demonstrating that it can be resolved with a minimal structural intervention rather than with objective function redesign or architectural modification. The shift is not a paradigm overthrow but a diagnostic reframing: the bottleneck in multi-turn distillation is not the sparsity of learning signals (the traditional concern that motivated OPD as an alternative to RL in the first place) nor the design of the distillation objective itself (the focus of prior OPD work by Jang et al., 2026; Jin et al., 2026; Ko et al., 2026), but rather a distribution-shift problem where the student's own errors progressively corrupt the teacher's supervision quality as the trajectory lengthens. This is a genuinely new lens on why agent training is hard, and it redirects research attention from algorithmic complexity (better RL, more sophisticated distillation objectives) toward structural constraints on the training process (controlling which states the student learns from at each training stage).

The field-level implications are threefold. First, the paper recasts multi-turn distillation as a curriculum design problem rather than an optimization problem. Prior work on OPD for single-turn tasks treated the training dynamics as essentially solved—KL divergence decreases monotonically, the student converges reliably, and the research frontier was about squeezing out additional efficiency or performance through objective refinements. This paper shows that multi-turn settings break this clean picture entirely: KL divergence can increase during training (Figure 2a), success rates can collapse (Figure 2b), and even when convergence occurs, the initial phase is dominated by severe instability (Figure 2c: KL ~1000 decaying to ~60). The implication is that any multi-turn distillation pipeline must now account for trajectory-level instability as a first-class design concern, not as an afterthought that can be addressed through hyperparameter tuning. The paper's solution—temporal curriculum pacing—is notably simple, requiring "only minor code changes" (Section 4.2), which itself carries a methodological message: the failure was not in the optimization algorithm but in the assumption that the teacher's supervision remains reliable throughout the trajectory. Fix that assumption, and the optimization works.

Second, TCOD provides a unified explanation for the conflicting intuition that distillation should help with agent training (dense signals, no credit assignment problem) and the empirical observation that it often fails or underperforms. The field has been caught between two contradictory positions: on one hand, OPD is theoretically appealing because it replaces sparse environment rewards with token-level teacher guidance, addressing the credit assignment and sample efficiency problems that plague RL for agents; on the other hand, practitioners have observed that training agents with distillation can be frustratingly unstable, with models sometimes learning nothing or even degrading during training. This paper's diagnosis resolves the tension: OPD does provide dense, useful signals, but those signals become actively harmful when the student reaches states outside the teacher's effective support—and in multi-turn settings, the student inevitably reaches such states because errors compound across turns. The problem is not that distillation is the wrong paradigm; it's that vanilla distillation ignores the temporal structure of the learning problem. This reframing makes OPD for agents viable in a way that it wasn't before, because practitioners now have a concrete failure mode to diagnose and a concrete intervention (temporal curriculum) to apply.

Third, the paper demonstrates that distillation with an appropriate curriculum can produce policies that are more robust than the teacher itself on hard cases (Table 2, Hard split: TCOD-B2F achieves 20.66% vs. teacher's 6.61%), challenging the assumption that distillation is inherently bounded by teacher capability. This is not a claim of knowledge amplification—the student does not acquire new facts or reasoning skills that the teacher lacks—but it is a claim of robustness amplification: the student trained with a temporal curriculum learns early-turn behavior in a controlled, error-minimized setting, making it less susceptible to the compounding-error failures that cause the teacher to fail on hard tasks even though the teacher possesses the necessary knowledge. This opens up a new use case for distillation: not just compressing a large teacher into a smaller student for deployment efficiency, but using the distillation process itself to produce a policy that is more consistent than the teacher, particularly on tasks where the teacher's primary weakness is its own compounding errors. The field's mental model of distillation as a "good enough" approximation to a superior teacher now needs to account for the possibility that the approximation can be better on specific axes.

The research directions that become more attractive after this work include curriculum design for interactive learning (not just agent training but any setting with state-dependent error propagation), robust verifier training (the over-optimization analog in the distillation setting), and self-improvement pipelines where a model iteratively generates its own curriculum by training on progressively longer horizons. The directions that become less attractive—or at least need to be re-evaluated—include efforts to make vanilla OPD work for agents solely through better objective functions or optimization heuristics without addressing the temporal structure of the problem. The paper's diagnosis makes a strong case that trajectory-level KL instability is a structural property of multi-turn interaction, not an optimization artifact that can be tuned away with better learning rates or KL coefficients. Research that treats multi-turn distillation as a single-turn problem with longer sequences is now on notice: the state-dependence matters, and methods that ignore it are fighting the wrong battle.

Follow-Up Research This Work Enables

Adaptive pacing triggered by KL divergence stabilization rather than fixed linear schedules. The paper uses a fixed linear schedule $k = k_{\text{start}} + \lfloor n / \eta \rfloor$ and shows robustness to $\eta \in \{2, 4, 6\}$ (Table 3), but explicitly identifies adaptive pacing based on "an exponential moving average of the KL divergence" as a promising direction (Appendix A). A concrete experiment: implement curriculum advancement that increments $k$ only when the EMA of per-turn KL divergence within the current depth falls below a threshold (e.g., the converged value of ~60 observed in Figure 2c), and compare against the fixed linear schedule across the three benchmarks. The key measurement is whether adaptive pacing achieves equivalent or better final success rates while automatically handling environment-student-teacher combinations where the optimal $\eta$ falls outside the tested range, such as environments with 100-turn horizons or student-teacher pairs with severe initial misalignment. A negative result—adaptive pacing performing worse than fixed—would suggest that the curriculum's benefit comes primarily from monotonic progression itself (which adaptive pacing might slow too much) rather than from matching the pace to the student's actual readiness, refining our understanding of why the curriculum works.

Combining TCOD with prior OPD objective improvements to test whether the two classes of interventions address independent failure modes. The paper positions TCOD as orthogonal to prior OPD improvements like forward-backward KL balancing (Jang et al., 2026; Jin et al., 2026) and reward clipping (Ko et al., 2026)—those methods address instability within single-turn distillation dynamics, while TCOD addresses cross-turn compounding errors. A direct test: run TCOD-F2B with the entropy-aware OPD objective from Jin et al. (2026) on ALFWorld with the Qwen3-1.7B student (where vanilla OPD collapses to 0.17% average success rate, Table 3). If the combined method outperforms either alone, it confirms that the two failure modes are genuinely independent and that a complete multi-turn distillation pipeline should include both temporal curriculum and objective-level stabilization. If there is no improvement (or degradation), it would suggest that the temporal curriculum already addresses the instability that the objective modifications target, potentially simplifying the design space. The experiment is practical: the Jin et al. objective modifies the loss computation, which is orthogonal to TCOD's trajectory truncation, and both can be implemented within the asynchronous training infrastructure described in Section 4.3.

Cross-model-family replication of trajectory-level KL instability to establish whether it is a universal property or specific to Qwen-distribution similarities. The paper's diagnosis and solution are validated exclusively on Qwen-family models (Qwen2.5 and Qwen3 for both students and teachers). A cross-family experiment using a non-Qwen teacher (e.g., Llama-3-70B-Instruct) and a non-Qwen student (e.g., Gemma-2-9B) would test two things simultaneously: whether the per-turn KL escalation pattern (Figure 2d) generalizes when the teacher and student have different pretraining distributions, tokenization, and behavioral tendencies, and whether TCOD's temporal curriculum remains effective when the token-level KL divergence must be computed across potentially different token vocabularies (requiring a token-alignment step not needed in the Qwen-only setting). The key measurement is whether the initial KL divergence is substantially higher for cross-family pairs (suggesting that Qwen's shared pretraining provides a favorable starting alignment that makes TCOD's job easier) and whether TCOD still recovers performance or whether the cross-family distribution shift is too severe for the temporal curriculum to overcome. A negative result would bound the generality claims and motivate research into tokenizer-agnostic distillation objectives for multi-turn settings.

TCOD combined with process reward model (PRM) tree search for the revision setting—testing whether temporal curriculum stabilizes verifier-guided exploration. The paper studies temporal curriculum for on-policy distillation where the teacher provides token-level supervision, but an alternative paradigm for test-time computation is PRM-guided beam search over solution trajectories (as studied in the PRM search literature). A natural extension: replace the frozen teacher in TCOD with a process reward model that scores partial trajectories, and use the temporal curriculum to control the search depth rather than the distillation horizon. Early in training, the student performs shallow search (low beam width, short lookahead) while the PRM is reliable; as training progresses, search depth expands. The key measurement is whether temporal curriculum prevents the PRM over-optimization phenomenon (where search at high budgets finds solutions that score well under the PRM but are incorrect) in the same way it prevents KL escalation in distillation—by keeping the student within the PRM's reliable support region during early training. The experiment would use the ALFWorld environment with the same student models, replacing the teacher with a PRM trained via Monte Carlo rollouts on the base model's outputs, following the recipe from the PRM search literature. A strong result would show that temporal curriculum + PRM search outperforms either approach alone on medium-difficulty tasks.

TCOD applied to the self-improvement loop: using TCOD-trained students to generate training data for the next iteration. The paper's Introduction and Section 8 suggest "distilling the outputs of applying additional test-time compute back into the base LLM, enabling an iterative self-improvement loop" as a natural extension, but no experiments explore this. A concrete self-improvement experiment: start with the Qwen3-30B-A3B-Instruct teacher from Table 3, train a Qwen3-4B student with TCOD-F2B on ALFWorld, then use that student as the teacher for a Qwen3-1.7B student in a second distillation round. The key measurements are whether the 1.7B student in the second round achieves higher success rates than it would by distilling directly from the original 30B teacher (testing whether the TCOD-trained intermediate student is a "better teacher" than the original, despite being smaller and less capable overall), and whether performance degrades across iterations (the ReST$^{EM}$ failure mode from Appendix K of the prior paper, where iterative training amplifies spurious correlations). The experiment would distinguish between two hypotheses about why TCOD students sometimes surpass their teachers (Table 2, Hard split): Hypothesis 1 is that TCOD produces a more robust policy that generalizes better, which would make the student a genuinely better teacher for subsequent iterations. Hypothesis 2 is that TCOD's benefit is specific to the training dynamics of distillation and does not transfer to the student's ability to serve as a teacher, in which case iterative improvement would stall or reverse.

Stress-testing TCOD on environments with substantially longer horizons (50–100 turns) to test whether the linear curriculum schedule scales. The paper evaluates on environments with maximum 30 turns (Table 1: ALFWorld and ScienceWorld at 30, WebShop at 15). The trajectory-level KL instability is diagnosed as a function of turn count—per-turn KL increases monotonically with turn index (Figure 2d)—so environments with longer horizons should exhibit more severe instability and require correspondingly different curriculum pacing. An experiment on a 100-turn environment (e.g., a long-horizon web navigation task or a multi-stage scientific reasoning benchmark) would test whether the linear schedule $k = k_{\text{start}} + \lfloor n / \eta \rfloor$ with the same $\eta \in \{2, 4, 6\}$ generalizes or whether the optimal $\eta$ needs to scale with the maximum horizon. If $\eta = 6$ is optimal for 30-turn environments, does $\eta = 20$ become optimal for 100-turn environments? The key measurement is whether the performance curve (success rate vs. $\eta$) shifts systematically with horizon length—if it does, practitioners would need horizon-aware $\eta$ selection, and the robustness claim in Section 5.4 would need to be qualified by horizon range. If it does not (i.e., the same $\eta$ works across all horizons because what matters is the fraction of training spent at each depth, not the absolute depth), that would strengthen TCOD's generality claim substantially. A negative result would motivate the adaptive pacing approach described in Appendix A.

Practical Applications and Downstream Use Cases

Deploying small on-device agents for embodied or interactive tasks where a large teacher model is available for training but too expensive for inference. The most directly actionable finding from Table 3 is that TCOD can recover a small model (Qwen3-1.7B) from near-zero success rates under vanilla OPD (0.17% average across three benchmarks) to useful performance levels (~18.5% average with TCOD) when distilling from a strong teacher (Qwen3-30B-A3B-Instruct, 30.28% average). This is a deployment-relevant regime: a 30B teacher is too large for edge deployment on phones or embedded devices, but a 1.7B student is tractable. Without TCOD, distillation would be essentially non-functional (the student learns nothing due to KL collapse); with TCOD, the student becomes usable. The specific benefit: a practitioner with access to a large teacher model (API-based or self-hosted during training) and a target deployment requiring a small on-device model can now train the small model with TCOD-F2B (no pre-collected demonstrations needed, just the teacher for inference during training) and expect a functional agent rather than a collapsed one. The 32% training time reduction (Figure 6, F2B on ALFWorld: 4.9 hours vs. 7.18 for vanilla OPD) adds practical appeal by reducing the cloud GPU cost of the distillation phase.

Improving domain-specific agent performance through teacher-guided curriculum distillation, where the teacher is a fine-tuned specialist but still makes errors on a tail of hard cases. The Hard split results in Table 2—where the teacher (GRPO-trained Qwen2.5-7B, domain-specialized on ALFWorld) achieves only 6.61% success rate but TCOD-B2F with the same architecture reaches 20.66%—demonstrate that TCOD can amplify a specialist teacher's robustness on its own weak cases. This is a specific deployment scenario: an organization has invested in fine-tuning a model (via GRPO, SFT, or RL) on a domain-specific agent task, achieving strong average performance (85.71% on ALFWorld seen) but with a long tail of failure cases (6.61% on Hard). Rather than collecting more data or further fine-tuning the teacher (which may overfit or hit diminishing returns), the organization can distill the teacher into a student of the same size using TCOD-B2F, leveraging the teacher's own successful trajectories (already collected during training or evaluation) as the pre-collected dataset. The student then matches the teacher on easy cases while exceeding it on hard ones, effectively "rounding off" the performance tail. The concrete benefit is a ~14-point improvement on the hardest tasks without requiring new data, new architectures, or RL exploration—just a temporal restructuring of the distillation process.

Cost-efficient training of agents for multi-benchmark or multi-domain deployment, where a single general-purpose teacher must be distilled into smaller models for diverse environments. The cross-benchmark evaluation in Table 3 (WebShop, ALFWorld, ScienceWorld with a single Qwen3-30B-A3B-Instruct teacher) shows that TCOD works across fundamentally different environment types (embodied navigation, e-commerce, scientific reasoning) without domain-specific curriculum design. This matters for organizations building general-purpose agent platforms that must operate across multiple domains: rather than fine-tuning separate teachers for each domain (as in the GRPO-trained specialist in Table 2), they can use a single large general-purpose teacher and distill it into smaller domain-deployed models using TCOD with the same $\eta$ settings. The robustness of TCOD to $\eta \in \{2, 4, 6\}$ (less than 2% variation in Table 3) means that a single curriculum configuration can be deployed across domains without per-domain tuning, reducing the engineering overhead of multi-domain agent training. The concrete benefit is operational: one distillation pipeline, one set of hyperparameters, one teacher model, producing functional student agents across diverse environments—something that would be infeasible with vanilla OPD (which would collapse on the 1.7B student) or SFT (which suffers from exposure bias and achieves lower performance, as shown in Table 2, where SFT achieves 32.14% vs. TCOD-F2B's 81.43% on the same 3B student).

When to Prefer This Method

The paper itself articulates clear tradeoffs between TCOD variants and against vanilla OPD, grounded in both the empirical results (Tables 2 and 3) and the practical considerations discussed in Section 4.2 and Appendix A. These tradeoffs can be framed as a decision guide:

Prefer TCOD-F2B over TCOD-B2F when: (1) no pre-collected teacher demonstrations are available, or the cost of collecting them is prohibitive—F2B requires only the teacher for online inference during training, not a pre-built dataset of successful trajectories; (2) minimal code changes are desired—F2B simply truncates student rollouts at $k$ steps and can be implemented by modifying the trajectory collection loop, while B2F requires a warmup phase where the teacher executes prefix steps; (3) training time efficiency is the primary concern—F2B reduces total training time by 32% vs. vanilla OPD on ALFWorld (Figure 6), compared to 23% for B2F, because F2B's truncated trajectories during early curriculum stages are shorter than B2F's student-executed segments plus teacher-prefix steps; (4) the student model is large enough (3B+) to partially tolerate early-turn errors—the results in Table 2 show F2B outperforming B2F for the 3B student on seen and unseen splits (81.43% vs. 77.86% on seen; 79.19% vs. 70.90% on unseen), suggesting that larger students benefit more from F2B's forward curriculum than from B2F's teacher-navigated initialization.

Prefer TCOD-B2F over TCOD-F2B when: (1) pre-collected successful teacher trajectories are available (e.g., from prior teacher evaluations or training runs, collected through pass@10 sampling as described in Section 5.1); (2) the student is small (1.7B or below) and early-turn errors would be catastrophic under vanilla OPD—Table 3 shows both B2F and F2B recovering the 1.7B student from near-zero performance, but B2F's teacher-navigated initialization provides stronger protection against early-turn KL escalation by ensuring the student never generates actions for the most error-prone early turns; (3) the target tasks include a "hard tail" where the teacher's own policy fails due to compounding errors—the Hard split results in Table 2 show B2F achieving 20.66% vs. F2B's 18.18% for the 7B student, and 13.22% vs. 9.92% for the 3B student, indicating that B2F's teacher-prefix initialization is particularly valuable for tasks where early-turn errors are the primary failure mode; (4) the teacher is strong enough on the target domain that its successful trajectories provide useful starting points for most tasks—if the teacher fails on too many tasks (as with the Qwen3-30B teacher in Table 3, where success rates are 18–40%), B2F's pre-collected dataset will have coverage gaps, and F2B becomes the more reliable option.

Prefer either TCOD variant over vanilla OPD when: training a multi-turn agent of any scale, on any benchmark, with any teacher—the paper provides no evidence of any configuration where vanilla OPD outperforms both TCOD variants in final success rate, and the trajectory-level KL instability diagnosis in Section 4.1 establishes that vanilla OPD introduces unnecessary training instability even when it eventually converges (Figure 2c: initial KL ~1000 vs. converged ~60). The only scenario where vanilla OPD might be preferred is if the overhead of implementing the temporal curriculum (trivial for F2B, moderate for B2F) cannot be justified relative to the performance gain—but for the Qwen3-4B student in Table 3, where the gain is ~1.9 points on average, a practitioner might reasonably choose vanilla OPD if they have already tuned it and are satisfied with its stability. The paper does not address this "engineering overhead vs. gain" tradeoff explicitly, leaving it to practitioner judgment.