Distribution-Aligned Sequence Distillation for Superior Long-CoT Reasoning

ArXiv: 2601.09088

🎯 Pitch

A 4B reasoning model trained on just 448K examples—an order of magnitude fewer than competitors—surpasses 32B-scale counterparts by replacing standard 'data filtering' with a distillation process that actively aligns teacher and student probability distributions. Three simple mechanisms—temperature scheduling, divergence-aware sampling, and teacher-completed student prefixes—attack the core fallacy that student models simply need more high-quality teacher outputs, rather than outputs chosen to match their learning capacity.

---

1. Executive Summary

This paper introduces DASD-4B-Thinking, a lightweight 4B-parameter reasoning model built on Qwen3-4B-Instruct-2507 and trained via a Distribution-Aligned Sequence Distillation pipeline that systematically re-examines the prevailing paradigm of supervised fine-tuning on teacher-generated long chain-of-thought responses. The pipeline addresses three identified limitations of existing sequence-level distillation through three named mechanisms: temperature-scheduled learning (cold-starting on low-temperature samples before expanding to high-temperature diversity), divergence-aware sampling (prioritizing training examples where the teacher assigns high probability but the student assigns low probability, thereby aligning the teacher's output distribution with the student's learning capacity), and mixed-policy distillation (having the teacher complete truncated student-generated prefixes to mitigate exposure bias from teacher-forced training). Using only 448K training samples from the gpt-oss-120b teacher — an order of magnitude fewer than most existing open-source efforts — DASD-4B-Thinking achieves 83.3 on AIME25 and 88.5 on AIME24, 69.3 on LiveCodeBench v5, and 68.4 on GPQA-Diamond, outperforming all models of comparable scale and surpassing several 32B-scale counterparts, establishing that precise teacher-student distribution alignment during distillation can substitute for massive data scale only when the training signal is curated to match the student's learning dynamics rather than filtered by heuristic quality rules alone.

2. Context and Motivation

The Core Problem: Sequence-Level Distillation Is Widely Practiced But Poorly Understood

The fundamental question this paper tackles is deceptively straightforward: when distilling a large reasoning model into a smaller one by fine-tuning on the larger model's outputs, are we actually doing distillation, or are we just doing supervised fine-tuning? This distinction matters because it determines the entire design philosophy of the training pipeline. The paper argues that most of the community has been operating under the SFT perspective — treating teacher-generated responses as high-quality training examples to be filtered and curated with heuristic rules — while largely ignoring the core principle of knowledge distillation: enabling the student to learn the teacher's full output distribution so as to inherit its generalization capability.

This gap is significant for several practical reasons the authors highlight throughout Section 1:

• The open-source reasoning race. Following DeepSeek-R1's demonstration that distillation from powerful teachers can substantially empower smaller models (Guo et al., 2025), an explosion of open-source projects — OpenR1, OpenThoughts, NVIDIA AceReason, LIMO, s1, DeepMath, and many others — have emerged, collectively releasing millions of training examples and hundreds of model variants. Yet the methodological basis for these efforts remains largely unchanged from the original SFT formulation: collect questions, sample teacher responses, filter for quality, train. If the underlying distillation mechanism is not well-understood, improvements are limited to data scaling — adding more examples, more domains, more filtering rules — rather than more principled efficiency gains.

• The data scale barrier. Most existing open-source distilled reasoning models are trained on 1–2 million examples or more. For instance, AM-thinking-v1 uses 2.9M examples (Ji et al., 2025), while NVIDIA-OpenReasoning-Nemotron-7B uses an extraordinary 30M examples (Guha et al., 2025). This scale imposes substantial computational cost and creates a barrier to entry for researchers without access to massive inference budgets for data generation. If the distillation process itself could be made more efficient — extracting more learning signal per example — the barrier would be substantially lowered.

• A training-inference mismatch that degrades real-world performance. Models distilled via teacher-forced SFT are trained on the teacher's clean, coherent outputs but must generate autoregressively at inference time, relying on their own potentially flawed intermediate predictions. This exposure bias (Ranzato et al., 2016) causes the student to drift into regions of the output space it never encountered during training, compounding errors and producing degenerate behaviors like truncated or repetitive outputs. The paper provides empirical evidence for this phenomenon in Figure 7, showing that student-generated responses are cut off at an increasing rate as the teacher's reference length grows — a clear signature of reliance on teacher context that disappears at inference time.

Three Structural Limitations in Current Practice

The paper identifies three specific limitations that collectively represent a systemic absence of teacher-student interaction throughout the distillation process. These are not merely implementation details but conceptual gaps that follow directly from framing distillation as a data-filtering problem rather than a distribution-matching problem.

Limitation 1: Inadequate representation of the teacher's sequence-level distribution. Current approaches typically sample responses from the teacher at a fixed temperature and apply quality-based filtering rules (Yan et al., 2025; Lei et al., 2025). While such filtering removes obviously incorrect or degenerate outputs, it does not ensure adequate coverage of the teacher's output distribution. The teacher may have multiple valid reasoning strategies for the same problem — different modes of its distribution — and random sampling under restrictive filtering may capture only a narrow subset, particularly the high-probability modes. For smaller or less capable student models, this narrow coverage can be insufficient: the student never sees the breadth of reasoning approaches the teacher could produce, limiting its ability to generalize to novel problems.

A natural remedy — increasing the sampling temperature to flatten the distribution and cover rarer modes (Holtzman et al., 2020) — introduces its own problem: high-temperature samples are more diverse but also noisier, making learning more difficult, especially for a student model with limited capacity. The paper shows this directly in Figure 3(b): training loss on high-temperature (T=1.0) data converges more slowly and to a higher final loss than on low-temperature (T=0.6) data. So there is a fundamental tension between distributional coverage and learnability that a single fixed-temperature sampling strategy cannot resolve.

Limitation 2: Misalignment between the teacher's output distribution and the student's learning capacity. Standard sequence-level distillation is formalized in Section 2: the objective is to minimize the KL divergence between the teacher's and student's sequence-level distributions, but since computing the full expectation is intractable over the exponentially large output space $\mathcal{Y}$, the practical approximation replaces the teacher's distribution with a point mass at a sampled response $\hat{\boldsymbol{y}}$, reducing the loss to standard SFT cross-entropy on $\hat{\boldsymbol{y}}$. What this analysis reveals — and what the paper argues is overlooked in current practice — is that SFT on teacher-generated data only approximates the true distillation objective if the sampled response is representative of the teacher's distribution. More critically, SFT increases the student's probability on all tokens in the teacher-generated response, regardless of whether the student already assigns them high probability. This creates misleading gradients: for tokens where the teacher assigns low probability but the student assigns high probability (because of its own inductive biases or pretraining), SFT pushes those probabilities even higher, actively driving the student away from the teacher's distribution rather than toward it. This is in sharp contrast to classical logit-based distillation (Hinton et al., 2015; Gu et al., 2024), where the full logit distribution provides per-token calibration signals — increasing some probabilities and decreasing others — to precisely align the student with the teacher.

The paper frames this as a search for a better target sequence-level distribution: not just any teacher-generated responses, but a specific subset whose distribution is better aligned with the student's learning capacity. The challenge is that identifying such a subset requires understanding how the teacher's and student's probability assignments differ on the same responses — information that is not available when only looking at the teacher's outputs in isolation.

Limitation 3: Exposure bias from teacher-forced training versus autoregressive inference. This is a well-known problem in sequence generation (Ranzato et al., 2016), but the paper argues it is particularly severe in long-CoT distillation because reasoning traces can extend to tens of thousands of tokens. During SFT, the student is conditioned at every step on the ground-truth teacher prefix — this is teacher forcing. At inference time, the student must condition on its own previously generated tokens, which may contain errors, inconsistencies, or drift from the teacher's reasoning style. The longer the sequence, the more opportunity for divergence to accumulate. The paper's empirical investigation in Section 5 (Figure 7) confirms this: even on the training data itself, the student's autoregressive outputs are cut off at substantially higher rates than the teacher's, and this rate increases with reference response length. The student has learned to rely on the teacher's contextual guidance and cannot maintain coherence when generating independently.

Importantly, this is not a problem that better data filtering can solve — it is a structural mismatch between training and inference conditions that is inherent to off-policy distillation (training on teacher-generated data that the student never encounters during its own generation). Addressing it requires incorporating the student's own generation behavior into the training process, which is the motivation for mixed-policy distillation.

Where Prior Approaches Fall Short

The paper situates its contribution within a taxonomy of distillation approaches for reasoning:

Sequence-level distillation (the dominant paradigm). The vast majority of open-source reasoning models — DeepSeek-R1 distilled variants, OpenR1, OpenThoughts, NVIDIA AceReason, s1, LIMO, and others — follow this approach: collect challenging questions, use a powerful teacher (typically DeepSeek-R1 or a variant) to generate responses, apply quality filtering (correctness verification, length-based selection, preference filtering), and fine-tune a smaller model on the resulting dataset. While remarkably effective — these models achieve state-of-the-art or highly competitive performance — the paper argues that this line of work is viewed primarily through an SFT lens: the focus is on which data to keep, not how the student interacts with the teacher's distribution. Data filtering heuristics (correctness, reasoning difficulty, diversity) are designed to improve SFT data quality but do not explicitly optimize for the distribution-matching objective that defines true distillation. This leaves substantial efficiency on the table: if the goal is simply to filter the cleanest teacher outputs, scaling data volume becomes the primary lever for improvement, driving the massive dataset sizes observed in practice.

Logit-based distillation (the classical alternative). The original knowledge distillation formulation (Hinton et al., 2015) aligns the full logit distributions of teacher and student, providing rich per-token supervision that captures "dark knowledge" — the relative probabilities the teacher assigns to incorrect but plausible tokens. Recent large-scale models like Qwen3 (Yang et al., 2025) and Gemma (Kamath et al., 2025) adopt on-policy variants of this approach: the student generates responses, and the student's logits are aligned with the teacher's via KL divergence minimization. Thinking Machines Lab (Lu and Lab, 2025) released an open-source implementation. However, this approach has two significant drawbacks that the paper identifies: (1) it requires access to the teacher's full token-level logits, which are often unavailable for proprietary models and computationally expensive to request for open models; and (2) it becomes infeasible when the teacher and student use different tokenizers, as the output spaces are misaligned and direct probability mapping is not possible. The DASD paper explicitly targets the sequence-level paradigm because it imposes no arbitrary constraints on model architectures and does not require token-level access — a practical consideration for a pipeline that uses a proprietary teacher (gpt-oss-120b) and an open-weight student (Qwen3-4B-Instruct-2507).

The missing middle. Between these two paradigms lies a conceptual gap that the paper aims to fill: how can sequence-level distillation (which only requires sampled teacher outputs, not full logits) incorporate more of the distribution-matching logic that makes logit-based distillation powerful? The paper's three innovations are each answers to this question at different stages of the pipeline. Temperature-scheduled learning addresses distribution coverage without sacrificing learnability. Divergence-aware sampling identifies a teacher-derived target distribution aligned with the student's capacity — achieving per-example selection that serves an analogous role to the per-token calibration of logit distillation, but operating at the sequence level. Mixed-policy distillation addresses the training-inference mismatch by incorporating the student's own generation behavior.

How This Paper Positions Itself

The paper explicitly frames its contribution not as a new model or a new dataset, but as a methodological re-examination of the sequence-level distillation paradigm. The position is that the community's SFT-centric view of distillation — designing heuristic data filtering rules — has left the core principle of distillation underexploited. The paper does not reject the SFT-based approach; rather, it argues that SFT on teacher-generated data is a valid form of distillation (as shown by the derivation in Section 2, where the SFT loss is recovered as a sampled approximation of the KL divergence objective), but that treating it as distillation — with explicit attention to teacher-student distribution alignment — yields insights that the SFT-as-data-filtering perspective misses.

This positioning is supported by the theoretical grounding in Section 2. The KL divergence formulation $\min D_{KL}(p_T(\boldsymbol{y} \in \mathcal{Y}|\boldsymbol{x}) || p_S(\boldsymbol{y} \in \mathcal{Y}|\boldsymbol{x}))$ is the starting point, and the paper traces how the practical SFT approximation — replacing the teacher's distribution with a point mass at a sampled response — discards distributional information that could be leveraged for more effective learning. Each of the three innovations can be understood as reintroducing distributional awareness at a different point in the pipeline:

• Temperature-scheduled learning reintroduces awareness of the teacher's distribution's shape (concentrated at low temperature, broad at high temperature) and the student's learning dynamics (stable gradients early, diverse exploration later).
• Divergence-aware sampling reintroduces awareness of the discrepancy between teacher and student probabilities, selecting training examples where the gap is informative for learning rather than potentially misleading.
• Mixed-policy distillation reintroduces awareness of the distributional shift between training and inference conditions, using the teacher to provide corrective guidance on the student's own generation distribution.

The paper is careful not to claim that these innovations are individually revolutionary — temperature scheduling, distribution analysis, and on-policy data all have precedents in the literature. Rather, the contribution is the synthesis into a coherent distillation pipeline that treats teacher-student distribution alignment as the central design principle, and the demonstration that this principled approach can match or exceed the performance of brute-force data scaling approaches while using an order of magnitude fewer training examples. The 448K training samples used for DASD-4B-Thinking are explicitly compared against the 2.9M used by AM-thinking-v1 (Ji et al., 2025) and the 30M used by NVIDIA-OpenReasoning-Nemotron-7B (Guha et al., 2025), framing the paper's contribution as data efficiency through better distillation methodology rather than through better data filtering.

The paper also positions itself within the broader landscape of distillation for reasoning by deliberately choosing a teacher-student pair with substantial architectural differences — gpt-oss-120b (teacher) and Qwen3-4B-Instruct-2507 (student) — that differ in scale, architecture, vocabulary, tokenizer, and pretraining corpora. This choice is strategic: it demonstrates that the proposed innovations work across model families and are not dependent on the teacher and student sharing similar representations (as logit-based distillation would require), making the approach more broadly applicable to the common practical scenario of distilling from proprietary API-accessible teachers into open-weight students.

` tags for student-model compatibility.

• Repetitive content filtering: gpt-oss-120b "tends to generate repetitive content, particularly at lower temperatures" — repeated paragraphs, sentences, or phrases within a single response. These patterns can cause the trained student to produce "endlessly repetitive and excessively verbose outputs during inference." Regular expressions and n-gram matching are used to detect and remove such samples.

The final dataset distribution across domains is visualized in Figure 8. The total is 448K training samples (105K T=0.6 + 330K T=1.0 + 12.7K mixed-policy).

Stage 3: Temperature-Scheduled SFT.

• Sub-stage 3a (Cold Start): Fine-tune Qwen3-4B-Instruct-2507 on the 105K T=0.6 DAS data.

  • Hyperparameters: initial learning rate 5e-5 decaying to 1e-5 via cosine scheduler, cutoff length 64K tokens, greedy sequence packing to accelerate training, ZeRO-3 optimization with Liger kernels to reduce GPU memory consumption, global batch size 64, 6 epochs.
  • The authors note they "observe consistent performance improvements across epochs" — suggesting that the 6-epoch setting is chosen empirically rather than being a fixed prior.

• Sub-stage 3b (High-Temperature Expansion): Resume training from the Sub-stage 3a checkpoint using the 330K T=1.0 DAS data.

  • Training hyperparameters are kept identical to Sub-stage 3a. This is a deliberate design choice: by holding all optimization settings constant, any performance differences between stages can be attributed to the data temperature shift rather than learning rate schedules, batch sizes, or other confounds.
  • The model now sees the full 435K sample dataset, but with the ordering controlled: consistent, high-confidence patterns first, diverse rarer modes second.

Stage 4: Mixed-Policy Distillation.

• 50K questions are sampled from the DAS-curated training set.
• The student model from Stage 3 generates on-policy responses, capped at 1.5× the teacher's reference length.
• 15K truncated student responses are identified.
• For each truncated response, a random position beyond the halfway point is selected; the prefix is kept and the suffix is discarded.
• The teacher (gpt-oss-120b) generates completions from the truncation points.
• After quality filtering, 12.7K mixed-policy examples are retained.
• These are combined with 20K off-policy samples for balanced training, and the student is fine-tuned for one epoch.
• This yields the final DASD-4B-Thinking model.

Design rationale for the overall architecture. The three innovations are not merely stacked sequentially but are designed to address complementary aspects of the distillation gap:

• Temperature scheduling improves distribution coverage (making the point-mass approximation less lossy) while managing learnability (ensuring the student can absorb the covered modes).
• Divergence-aware sampling improves distribution alignment (selecting which samples within the covered modes best support the student's learning) while mitigating misleading gradients (prioritizing samples where SFT's direction matches the desired transfer direction).
• Mixed-policy distillation improves distribution robustness (reducing the mismatch between training and inference conditions) by introducing on-policy context.

Additionally, the paper inherits established practices from the broader distillation literature: quality verification through correctness checks, structural consistency enforcement (ensuring think/answer separation), and diversity curation through multi-domain question collection. These standard practices are presented as necessary but not sufficient — they ensure baseline data quality, while the three innovations provide the methodological advances that drive the reported performance gains.

---

Evaluation Setup and Baselines (Section 7)

The training pipeline produces the model; the evaluation (Section 7) measures its effectiveness. The evaluation configuration and baselines are part of the technical approach because they define the success criteria the pipeline was designed to optimize.

Benchmarks. Five complementary reasoning benchmarks are used:

• AIME24 and AIME25: Each comprising 30 problems from the American Invitational Mathematics Examination. These are the most challenging math benchmarks used, requiring multi-step competition-level reasoning with exact numeric answers.
• GPQA Diamond: 198 graduate-level multiple-choice questions in physics, chemistry, and biology, designed to be "Google-proof" — requiring deep academic reasoning rather than factual retrieval.
• LiveCodeBench v5 and v6: A continuously updated coding benchmark with temporal partitioning to prevent data contamination. v5 covers problems from October 2024–February 2025; v6 covers February–May 2025. Beyond code generation, it evaluates self-repair, executable correctness, and test-output prediction.

Evaluation protocol. All evaluations use a unified setup: temperature 1.0, top-p 1.0. For every benchmark, 64 responses are sampled per question, and the average accuracy is reported. This 64-sample averaging provides reliable, stable evaluation results by reducing variance from individual sampling runs. For AIME24 and AIME25 (extreme difficulty), the maximum generation length is set to 102,400 tokens; for LiveCodeBench and GPQA-D, it is 81,920 tokens. These long context limits accommodate the extended chain-of-thought reasoning traces the model produces.

Why pass@64 rather than pass@1? The paper reports average accuracy across 64 samples, which is essentially pass@64. This is consistent with the evaluation methodology used by most comparable models in the reasoning literature — reasoning models are typically evaluated with multiple samples and majority voting or best-of-N selection because single-sample evaluation has high variance on difficult benchmarks. The 64-sample protocol ensures the reported numbers are comparable to the baselines.

Baseline selection. The paper compares against two categories of models:

• Open-Weights Only: Models that release weights publicly but keep training data proprietary. These represent the strongest available performance from the broader community but lack full reproducibility. Included: Qwen3 series (4B-Thinking-2507, 8B, 14B, 32B), DeepSeek-R1-0528-Qwen3-8B, GLM-Z1 series (32B-0414, 9B-0414), Mistral 3 series (3B, 8B).

• Open-Weights & Open-Data: Models that release both weights and training data, enabling full reproducibility. Included: AM-thinking-v1 (2.9M data), OpenThoughts3-7B (1.2M data), Pai-DistillQwen-ThoughtY (4B, 8B, 365K data), POLARIS-4B-Preview, NVIDIA-OpenReasoning-Nemotron-7B (30M data), NVIDIA-Nemotron-Ultra-253B.

This dual categorization enables two types of comparison: (a) how DASD-4B-Thinking performs against the strongest available models regardless of reproducibility, and (b) how it performs against models where the full training recipe is publicly documented, enabling direct methodological comparison. The paper explicitly notes DASD-4B-Thinking's 448K training samples against AM-thinking-v1's 2.9M and NVIDIA-Nemotron's 30M, framing its contribution around data efficiency.

MoE evaluation (Section 7.7). To test scalability, the pipeline is extended to a Mixture-of-Experts architecture: Qwen3-30B-A3B-Instruct-2507 is used as the student, but crucially, no new data is collected. The exact same 105K T=0.6 DAS dataset curated for the 4B student is reused directly, testing cross-architecture transfer. Only the first stage (low-temperature training) is applied, yielding DASD-30B-A3B-Thinking-Preview. This achieves 86.7 on AIME25, 72.8 on LCB v6, 72.3 on GPQA-D, with an average of 77.3 — competitive with much larger MoE models trained on orders of magnitude more data. This result strongly supports the paper's claim that DAS data captures teacher-student distributional alignment that generalizes across student architectures.

---

Summary of Design Choices and Their Justifications

• Sentence-level probability decomposition over token-level: avoids the tokenizer-compatibility requirement of logit-based distillation while still capturing fine-grained distributional patterns; the geometric mean normalizes for sentence length.
• Two-stage temperature curriculum (cold-then-hot) over single-temperature or hot-then-cold: stable early gradients establish foundational reasoning patterns; high-temperature data then expands mode coverage without destabilizing already-learned behaviors. The ordering matters empirically — reversing it would expose the untrained student to noisy, diverse samples before it has learned the basic patterns.
• Divergence-aware selection (teacher-high, student-low) over random sampling or teacher-confidence-based selection: the Teacher Sentence pattern correlates with downstream correctness; selecting for high-divergence examples ensures the student focuses its limited capacity on the distributional gaps that matter most for performance.
• Geometric mean for sentence probability over arithmetic mean or product: the product of per-token probabilities would penalize long sentences (more multiplications of numbers <1), while the arithmetic mean would not appropriately weight outliers. The geometric mean provides a length-normalized average that treats each token equally in log-space.
• Mixed-policy with student prefix retained (no masking) over masked variant or pure off-policy: retaining the student prefix in the training loss exposes the student to its own generation context, which is the mechanism that mitigates exposure bias. Masking removes this benefit, as shown in Table 5.
• Teacher completion with quality filtering over student self-completion or teacher-only generation: the teacher provides high-quality corrective supervision from the student's actual generation context, combining the benefits of on-policy exposure with off-policy quality.
• 64-sample evaluation averaging over single-sample evaluation: reduces variance on difficult benchmarks where pass@1 can be noisy; consistent with community practice for reasoning model evaluation.
• Identical hyperparameters across temperature stages over stage-specific tuning: ensures that performance differences are attributable to the data temperature curriculum rather than optimization hyperparameter changes, providing cleaner scientific evidence for the temperature-scheduling hypothesis.
• Reusing 4B-curated DAS data for 30B MoE student over re-curating data: tests the cross-architecture generalizability of the divergence-aware selection criterion and demonstrates that DAS captures properties of the teacher-student distributional relationship that are not specific to a particular student architecture.

4. Key Insights and Innovations

Innovation 1: Reframing Sequence-Level Distillation as a Distribution-Matching Problem, Not a Data-Filtering Problem

The paper's most fundamental contribution is not any single algorithmic technique but a diagnostic reframing of what sequence-level distillation is. The dominant paradigm in the reasoning community — exemplified by DeepSeek-R1 distillation (Guo et al., 2025) and its many open-source replications (OpenR1, OpenThoughts, NVIDIA AceReason, s1, LIMO, and others) — treats distillation from the SFT perspective: the teacher generates responses, researchers design heuristic filters to select the "best" ones (by correctness, reasoning difficulty, output length, diversity), and the student is fine-tuned on these curated examples. The methodological question in this paradigm is always which data to keep.

The DASD paper argues — through the explicit derivation in Section 2 tracing from the KL-divergence objective $\min D_{KL}(p_T \| p_S)$ to the sampled SFT approximation — that this framing discards the core insight of knowledge distillation: the goal is to match distributions, not to maximize likelihood on a filtered subset of the teacher's outputs. The point-mass approximation $p_T(\boldsymbol{y} \mid \boldsymbol{x}) \approx \mathbb{1}\{\boldsymbol{y} = \hat{\boldsymbol{y}}\}$ that reduces the KL objective to standard SFT loss is a necessary practical simplification, but the community has largely forgotten that it is an approximation and has consequently stopped asking whether the chosen $\hat{\boldsymbol{y}}$ faithfully represents $p_T$. The three innovations in the paper are all answers to the question: what would the pipeline look like if we designed it to better recover the distribution-matching objective that the SFT loss approximates?

This reframing is significant because it changes the axes of optimization. In the SFT-as-filtering view, the primary levers are data volume (more examples), data coverage (more domains), and filtering sophistication (better heuristics). This explains why the field has gravitated toward massive datasets — 2.9M examples for AM-thinking-v1 (Ji et al., 2025), 30M for NVIDIA-OpenReasoning-Nemotron-7B (Guha et al., 2025) — and increasingly elaborate filtering pipelines. By contrast, in the distribution-matching view, the primary levers are distributional coverage (does the sampled data span the teacher's modes?), distributional alignment (does the training signal steer the student toward the teacher's distribution rather than away from it?), and distributional consistency (do training and inference conditions produce matched distributions?). These are different questions that lead to different design interventions — temperature scheduling rather than more data, divergence-aware selection rather than better quality heuristics, mixed-policy correction rather than longer training. The paper's result that 448K examples with distributionally-aware curation outperform 2.9M examples with SFT-style filtering (83.3 vs. 74.4 on AIME25 for AM-thinking-v1, Table 6) is not merely an efficiency gain — it is evidence that the distribution-matching framing identifies optimization axes that data scaling alone cannot reach.

This is best understood as a fundamental reframing rather than an incremental method improvement. The paper does not propose a new loss function or a new model architecture — it proposes a different way of thinking about what the training pipeline is trying to accomplish, and shows that this conceptual shift yields concrete design decisions that were invisible from the SFT perspective. The three limitations identified in the Introduction (inadequate distribution coverage, misalignment with student capacity, exposure bias) only become visible as problems to solve once the distribution-matching goal is taken seriously. From the SFT perspective, inadequate coverage is solved by collecting more data (which these projects do), misalignment is invisible (it's not a concept the SFT framing has), and exposure bias is a known issue but treated as a generic sequence-generation problem rather than a specific consequence of distributional drift in long-CoT distillation.

Innovation 2: Diagnosing Misleading Gradients via Sentence-Level Distribution Decomposition

The paper's second conceptual contribution is the distribution decomposition framework introduced in Section 4, which identifies four canonical sentence types (Teacher, Student, Shared, Boosted) based on probability discrepancies between the teacher ($p_T$), the pre-distillation student ($p_S$), and the distilled student ($p_D$). This framework is distinctive not as a training technique — divergence-aware sampling is the technique — but as a diagnostic apparatus that makes visible what is happening inside the distillation process at a granularity that prior work could not access.

The key insight is that standard SFT on teacher-generated data produces asymmetric gradient signals: for tokens where the student already assigns high probability (but the teacher assigns low probability), SFT pushes those probabilities even higher, driving the student away from the teacher's distribution. This is a direct consequence of the point-mass approximation — the SFT loss only sees the teacher's sampled token as "correct" and has no mechanism to suppress tokens where the student is overconfident. Classical logit-based distillation avoids this through full-vocabulary calibration (Hinton et al., 2015; Gu et al., 2024), but that requires token-level access to the teacher's logits, which is often unavailable. The paper's innovation is showing that the same diagnostic information — which parts of a teacher-generated response produce misleading gradients — can be recovered from sequence-level probabilities alone, without needing the teacher's full logit distribution.

The four-category decomposition is itself a conceptual contribution. By comparing $p_T$, $p_S$, and $p_D$ on the same sentences, the framework reveals provenance: which parts of a distilled model's reasoning trace originate from the teacher, which were already present in the pre-distillation student, which are shared by both, and which were amplified by the distillation process. The empirical finding in Figure 6 — that Teacher Sentences (where $p_T \gg p_S$) consistently correlate with correct answers, while Boosted Sentences may correlate negatively — provides an explanatory mechanism for why some distillation data is more effective than others that goes beyond surface-level quality metrics.

The framework's significance extends beyond the specific divergence-aware sampling technique it enables. It provides a language for analyzing distillation dynamics that was previously absent from the sequence-level distillation literature. Prior work could observe that some training examples were "better" than others based on downstream evaluation, but had no principled way to characterize what made them better. The Teacher/Student/Shared/Boosted taxonomy provides such a characterization, grounded in the distributional relationship between teacher and student rather than in heuristic properties of the data. This makes it a fundamental diagnostic tool rather than an incremental improvement — it changes how researchers can think about and analyze distillation, opening the door to future work that goes beyond the specific DAS implementation in this paper. The fact that the same decomposition patterns are replicated on the independently trained DeepSeek-Distill-Qwen3-8B (Figure 6, right panels) suggests the taxonomy captures general properties of reasoning distillation rather than artifacts of the DASD training setup.

Innovation 3: Exposing Exposure Bias as a Distributional Drift Problem in Long-CoT Distillation, and Addressing It Through Constructive Mixed-Policy Correction

Exposure bias — the mismatch between teacher-forced training and autoregressive inference — is a well-known problem in sequence generation (Ranzato et al., 2016), and on-policy distillation has been proposed as a remedy in general language model training (Agarwal et al., 2024; Chen et al., 2025a). The DASD paper's contribution is not the general concept but rather demonstrating that exposure bias manifests in a specific and severe form in long-CoT distillation, and proposing a targeted, lightweight intervention that differs from standard on-policy approaches.

The specific manifestation the paper identifies (Section 5, Figure 7) is that the student's autoregressive outputs are truncated at substantially higher rates than the teacher's, and this truncation rate increases with the teacher's reference response length. This is not merely "the student makes more errors" — it is a specific failure mode where the student, deprived of the teacher's contextual guidance, cannot maintain coherent reasoning over long horizons and simply stops generating before reaching a conclusion. The mechanism is distributional drift: the student was trained to reason in the context of correct teacher prefixes, and when it must reason in the context of its own (potentially flawed) prefixes, the distribution of intermediate states it encounters differs from the training distribution, causing compounding deviations that eventually lead to degenerate outputs.

The proposed solution — mixed-policy distillation — is distinctive in its construction method. Rather than the standard on-policy approach of having the student generate complete responses and filtering for correctness (Agarwal et al., 2024), or the logit distillation approach of aligning per-token distributions (Hinton et al., 2015), the paper's method constructs hybrid trajectories: the student generates a prefix, the prefix is randomly truncated, and the teacher completes the sequence. This is a "constructive" approach because it creates training data where the context is on-policy (from the student's actual generation) but the completion is off-policy (high-quality, from the teacher). The student thus learns to recover from its own generation context — the specific skill that exposure bias degrades.

The key finding from the ablation in Table 5 — that masking the student-generated prefix (training only on the teacher completion) worsens performance — confirms the mechanism. If the benefit came purely from additional high-quality teacher data, masking should not hurt. The fact that masking does hurt demonstrates that the on-policy context (the student's own prefix) is the active ingredient — the student needs to see its own generation behavior in the training signal to learn robustness to distributional drift. This is a small but precise empirical finding that validates the distributional interpretation of exposure bias.

The significance of this innovation is that it provides a lightweight, architecture-agnostic method for addressing exposure bias in sequence-level distillation. Unlike full on-policy distillation, which requires the student to generate complete responses for all training examples (expensive), or logit-based methods, which require token-level access to the teacher (often unavailable), mixed-policy distillation requires only a small set of targeted interventions — 12.7K examples in the final pipeline — and works across heterogeneous model pairs. The paper frames this as a modular correction rather than a fundamental overhaul of the distillation process, which makes it practical to integrate into existing pipelines. The consistent gains across all benchmarks in Table 7 (+0.3% to +0.9%) from this lightweight stage, on top of an already strong model, provide evidence that exposure bias is a real limiting factor even in well-constructed distillation pipelines, and that constructive mixed-policy correction is an effective targeted intervention.

Innovation 4: Demonstrating That Distribution-Aware Curation Generalizes Across Student Architectures, Establishing the DAS Criterion as a Property of the Teacher-Student Relationship

The MoE evaluation in Section 7.7 reveals a finding that is easy to overlook but has significant implications: the divergence-aware sampling data curated for the Qwen3-4B student transfers effectively to a different student architecture — Qwen3-30B-A3B-Instruct-2507 — with no re-curation, no re-sampling, and no architectural adaptation. The DASD-30B-A3B-Thinking-Preview, trained on only the first-stage 105K T=0.6 DAS dataset curated for the 4B model, achieves competitive or superior performance against powerful MoE baselines trained on orders of magnitude more data (Table 8: 86.7 AIME25, 72.8 LCB v6, 77.3 average, versus NVIDIA-Nemotron-3-Nano-30B-A3B trained on 18M SFT examples + RL).

This finding is conceptually significant because it implies that the divergence-aware sampling criterion — which selects samples where $p_T$ (from gpt-oss-120b) is high and $p_S$ (from the original 4B student) is low — captures something about the teacher's distribution relative to a class of student models, not just relative to the specific 4B student used for selection. The teacher-high/student-low divergence pattern is correlated with downstream correctness regardless of which student architecture is used at inference time, suggesting that it identifies genuinely informative reasoning patterns rather than artifacts of a particular student's inductive biases.

This is a transferability result that was not obvious a priori. One might have expected DAS to be student-specific — that the divergence patterns it identifies would be idiosyncratic to the 4B student's particular probability assignments and would not generalize to a 30B MoE student with different pretraining, different inductive biases, and a different internal representation of reasoning. The fact that it does generalize suggests that the DAS criterion is capturing something more fundamental: perhaps the teacher's high-confidence regions that are "distant" from any smaller model's pretraining distribution, or perhaps reasoning patterns that are genuinely difficult for smaller models to represent but that the teacher can reliably produce. If this interpretation holds, DAS is not just a data selection method but a measure of reasoning difficulty relative to model capacity — a conceptual bridge between the long-CoT distillation literature and the broader question of what makes reasoning examples effective for transfer.

The practical implication is substantial: it means that divergence-aware data can be curated once (using a single reference student) and reused across multiple student architectures, dramatically reducing the computational cost of applying the method at scale. The paper does not fully exploit this finding — the MoE experiment is a preview, and the reported model uses only the first training stage — but the result opens the door to a more efficient distillation workflow where DAS curation is a one-time cost amortized across an entire family of student models. This is a fundamental insight about the nature of the DAS criterion rather than an incremental performance improvement, and it positions DAS as a general-purpose tool for reasoning distillation rather than a model-specific optimization.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper evaluates on five benchmarks: AIME24 and AIME25 (30 competition-level math problems each, requiring exact numeric answers), GPQA Diamond (198 graduate-level multiple-choice questions in physics, chemistry, biology), and LiveCodeBench v5 and v6 (continuously updated coding benchmarks with temporal partitioning to prevent data contamination; v5 covers October 2024–February 2025, v6 covers February–May 2025). These are all standard benchmarks in the reasoning literature, not custom datasets. No dataset split is used — all are treated as test sets only, since training data comes entirely from teacher-generated responses on separate question collections described in Section 6.1.

• Base model(s). The primary student model is Qwen3-4B-Instruct-2507 (Yang et al., 2025), a 4B-parameter dense model from the Qwen3 family. It is chosen because it represents a widely available, compact model at a scale where distillation gains are both practically valuable and measurable — the base model's performance is non-trivial but far from saturation (AIME25 passes from 47.4% at baseline to 83.3% after full training, Table 7). The teacher is gpt-oss-120b (Agarwal et al., 2025), a proprietary 120B model, chosen deliberately to test cross-architecture distillation: the teacher and student differ in scale, architecture, vocabulary, tokenizer, and pretraining corpora. For the MoE experiment (Section 7.7), the student is Qwen3-30B-A3B-Instruct-2507, a 30B-parameter Mixture-of-Experts model with 3B active parameters.

• Metrics. The primary metric is benchmark accuracy (%), computed as the fraction of questions for which the model's selected final answer matches the ground truth. For all benchmarks, 64 responses are sampled per question (temperature 1.0, top-p 1.0), and the average accuracy across these 64 samples is reported. This is pass@64 with no majority voting or verifier-based selection — it measures the model's raw generation capability rather than its ability to self-verify. For AIME24 and AIME25, the maximum generation length is 102,400 tokens; for LiveCodeBench and GPQA-D, it is 81,920 tokens. These long limits accommodate the extended chain-of-thought reasoning traces the model produces.

• Baselines. The paper compares against two categories, as enumerated in Section 7.2:

  • Open-Weights Only (weights public, training data proprietary): Qwen3-4B-Thinking-2507, Qwen3-14B, Qwen3-32B (Yang et al., 2025); DeepSeek-R1-0528-Qwen3-8B (Guo et al., 2025); GLM-Z1-32B-0414, GLM-Z1-9B-0414 (Zeng et al., 2024); Mistral3-3B, Mistral3-8B (Mistral AI Team, 2025).
  • Open-Weights & Open-Data (both public): AM-thinking-v1 (Ji et al., 2025, 2.9M training examples); OpenThoughts3-7B (Guha et al., 2025, 1.2M training examples); Pai-DistillQwen-ThoughtY-4B and -8B (Cai et al., 2025, 365K training examples); POLARIS-4B-Preview (An et al., 2025); NVIDIA-OpenReasoning-Nemotron-7B (Guha et al., 2025, 30M training examples); NVIDIA-Nemotron-Ultra-253B. All baseline metrics are taken from the corresponding official reports — the paper does not re-evaluate any baseline. This is standard practice but means that minor evaluation protocol differences (e.g., exact sampling temperature, number of samples for averaging) could slightly affect comparability.

• Generation budget / compute accounting. The paper does not report a FLOPs-based compute accounting. The implicit unit of compute is number of training examples (448K total for the final DASD-4B-Thinking pipeline, versus the baseline datasets ranging from 365K to 30M). At inference time, the evaluation protocol fixes the number of samples per question at 64 across all benchmarks, so the inference compute per evaluation is held constant. The paper's efficiency claims are based on training data volume, not inference FLOPs — a choice that reflects the community's focus on data efficiency in distillation but does not account for the computational cost of the DAS curation process itself (computing student-side probabilities on teacher-generated responses, which requires forward passes through the student model).

• Cross-validation / statistical protocol. The paper does not employ cross-validation or report confidence intervals. The main results (Table 6) are single-run evaluations with 64-sample averaging per question, which provides low-variance point estimates but no formal statistical significance testing. The ablations in Sections 3, 4, and 5 use controlled experimental setups (fixed random seeds implied, identical hyperparameters across compared conditions) but do not report variance across runs. The staged training ablation (Table 7) reports monotonic improvements across stages, but without error bars it is impossible to assess whether the smaller gains from mixed-policy distillation (+0.3% to +0.9%) are statistically reliable or within the noise floor of the 64-sample evaluation.

---

Main Quantitative Results

The experimental results are organized across multiple sections. Sections 3, 4, and 5 present controlled ablations validating individual innovations. Section 7 presents the full pipeline evaluation against external baselines. I present these in logical order: first the component-level ablations that establish each innovation's contribution, then the composite results that establish state-of-the-art status.

Temperature-Scheduled Learning (Section 3)

The headline finding: cold-starting on low-temperature (T=0.6) data followed by training on high-temperature (T=1.0) data substantially outperforms training on either temperature alone, and high-temperature data alone outperforms low-temperature data alone (Table 1). With gpt-oss-120b as teacher and Qwen3-4B-Instruct-2507 as student, training on 50K math responses:

• T=0.6 only: 81.7 AIME24, 71.9 AIME25
• T=1.0 only: 83.1 AIME24, 76.1 AIME25
• T=1.0 with T=0.6 cold start: 85.2 AIME24, 81.3 AIME25

The cold-start + high-temperature combination outperforms T=1.0-only by +2.1 on AIME24 and +5.2 on AIME25 — substantial gains given the modest total data volume. Scaling the T=1.0-only data to 100K (doubling) yields only marginal improvement over 50K (83.1→83.1 on AIME24, 76.1→78.9 on AIME25), suggesting diminishing returns from data volume that the temperature schedule circumvents.

The same pattern replicates with a different teacher (Qwen3-Next-80B-A3B-Thinking) and smaller data scale (25K math responses): T=0.6 cold start + T=1.0 achieves 83.1 AIME24 and 73.1 AIME25, versus 82.9/70.2 for T=1.0-only and 79.0/71.3 for T=0.6-only (Table 1, lower section). The performance ordering is consistent across teacher-student pairs.

Multi-domain validation (Table 2). Training on a mixture of 45K samples (25K math + 10K code + 10K science) with cold-start + T=1.0 achieves:

• AIME25: 77.5 (vs. 74.6 T=0.6-only, 75.2 T=1.0-only)
• LiveCodeBench v6: 51.0 (vs. 44.1 T=0.6-only, 47.3 T=1.0-only)
• GPQA-D: 66.4 (vs. 65.5 T=0.6-only, 65.4 T=1.0-only)

Doubling the T=1.0-only data to 90K samples (50K math + 20K code + 20K science) achieves only 75.8 AIME25, 51.3 LCB v6, 65.4 GPQA-D — the cold-start approach with half the data matches or exceeds these scores on AIME25 and GPQA-D while matching LCB v6. This is the paper's first demonstration that data efficiency gains generalize across domains: the cold-start temperature curriculum achieves with 45K samples what static-temperature training requires 90K samples to approximate. The LCB v6 result (51.0 vs. 51.3) is the only case where doubling the data volume roughly equals the temperature schedule; the paper attributes this to the relatively small proportion of code data in the mixture, suggesting further scaling could still yield gains there.

Divergence-Aware Sampling (Section 4)

The headline finding: DAS consistently outperforms random sampling at matched data volumes, and in some cases surpasses the performance of RS with 2× the data (Tables 3 and 4). With gpt-oss-120b teacher and Qwen3-4B-Instruct-2507 student:

At T=0.6 with 50K math responses:

• RS: 81.7 AIME24, 71.9 AIME25
• DAS: 83.3 AIME24, 74.2 AIME25

At T=1.0 with 50K math responses:

• RS: 83.1 AIME24, 76.1 AIME25
• DAS: 85.0 AIME24, 79.2 AIME25
• RS with 100K (doubled data): 83.1 AIME24, 78.9 AIME25

The critical finding here is that DAS at 50K (79.2 AIME25) slightly exceeds RS at 100K (78.9 AIME25) while matching RS at 100K on AIME24 (85.0 vs. 83.1). This means DAS achieves with half the data volume what RS requires the full volume to approximate, directly validating the claim that distributionally-aware sampling is more data-efficient than random sampling. The gain is larger at T=1.0 (+1.9 AIME24, +3.1 AIME25 DAS vs. RS at 50K) than at T=0.6 (+1.6 AIME24, +2.3 AIME25), suggesting DAS is particularly valuable when the teacher's distribution is broader and more modes are represented (the high-temperature regime).

Cross-teacher validation (Table 3, lower section). With Qwen3-Next-80B-A3B-Thinking as teacher and only 25K math responses:

• RS: 79.0 AIME24, 71.3 AIME25
• DAS: 82.5 AIME24, 71.9 AIME25

The AIME24 gain is substantial (+3.5), but the AIME25 gain is negligible (+0.6). The paper does not comment on this asymmetry, but it suggests DAS's benefit may be partly benchmark-dependent — AIME25 (more recent, less likely to be in the teacher's training data) may benefit less from distributional alignment when the teacher's own advantage over the student is smaller.

Multi-domain validation (Table 4). On the 45K mixed-domain dataset (25K math + 10K code + 10K science):

• RS: 74.6 AIME25, 44.1 LCB v6, 65.5 GPQA-D
• DAS: 75.6 AIME25, 47.3 LCB v6, 65.7 GPQA-D

Gains on math (+1.0) and code (+3.2) are clear; the science gain (+0.2) is within noise range. The large code improvement (+3.2 on LCB v6) is noteworthy because code reasoning traces may contain particularly distinctive teacher-student divergence patterns that DAS captures. The paper does not analyze per-domain DAS effectiveness in detail.

Mixed-Policy Distillation (Section 5)

The headline finding: a lightweight mixed-policy stage (only 7.7K non-masked examples, one epoch) yields measurable gains, and masking the student prefix eliminates these gains (Table 5). Starting from the model trained on 50K DAS data at T=0.6:

• Baseline (no mixed-policy): 83.3 AIME24, 74.2 AIME25
• Mixed-policy (7.7K, no mask): 83.3 AIME24, 74.8 AIME25
• Mixed-policy (7.7K, with mask): 80.8 AIME24, 72.3 AIME25

Two findings matter here. First, the unmasked variant improves AIME25 (+0.6) while preserving AIME24 — a directional but small improvement. Second and more importantly, the masked variant substantially degrades performance (AIME24 drops 2.5 points, AIME25 drops 1.9 points below the baseline). The masking variant removes the student-generated prefix from the training loss, training only on the teacher's completion. The fact that this hurts performance — rather than simply providing no benefit — confirms that exposure to the student's own generation context is the active ingredient, not the additional teacher data. The paper interprets this as evidence for the exposure bias mechanism: when the student sees its own prefixes during training with teacher completions, it learns to recover from its typical generation patterns; when those prefixes are masked, the training becomes pure off-policy and provides no robustness benefit.

The small absolute gain from the unmasked variant (+0.6 AIME25) might seem modest, but the paper frames this as expected given the lightweight nature of the intervention: only 7.7K examples and one epoch of training. The more important result is the negative ablation: the masking condition demonstrates that the method's design (retaining student context) is essential, not incidental.

Full Pipeline Results: DASD-4B-Thinking vs. Baselines (Section 7.4)

The headline result is DASD-4B-Thinking's state-of-the-art performance at the 4B scale, summarized in Table 6. I excerpt the key comparisons:

On AIME24 and AIME25:

• DASD-4B-Thinking: 88.5 AIME24, 83.3 AIME25
• Qwen3-4B-Thinking-2507 (same base model family, proprietary training): —, 81.3 AIME25
• Qwen3-32B (8× larger): 81.4 AIME24, 72.9 AIME25
• DeepSeek-R1-0528-Qwen3-8B (2× larger, specialized reasoning model): 86.0 AIME24, 76.3 AIME25
• GLM-Z1-32B-0414 (8× larger): 80.8 AIME24, 63.6 AIME25
• AM-thinking-v1 (32B, 2.9M training examples): 85.3 AIME24, 74.4 AIME25
• NVIDIA-OpenReasoning-Nemotron-7B (7B, 30M training examples): 84.7 AIME24, 78.2 AIME25
• NVIDIA-Nemotron-Ultra-253B (63× larger): 80.8 AIME24, 72.5 AIME25

The 4B DASD model outperforms the 32B Qwen3 on both AIME benchmarks (+7.1 AIME24, +10.4 AIME25) and the 253B Nemotron Ultra (+7.7 AIME24, +10.8 AIME25). Against the most comparable open-data model (AM-thinking-v1, 32B, 2.9M data), DASD-4B-Thinking leads by 3.2 on AIME24 and 8.9 on AIME25 while using 6× fewer training examples (448K vs. 2.9M). These numbers are visualized in Figure 9 (left panel for AIME25), where DASD-4B-Thinking appears in the extreme top-left — highest performance at smallest scale.

On LiveCodeBench:

• DASD-4B-Thinking: 69.3 LCB v5, 67.5 LCB v6
• Qwen3-4B-Thinking-2507: —, 55.2 LCB v6
• Qwen3-32B: 65.7 LCB v5, — LCB v6
• DeepSeek-R1-0528-Qwen3-8B: 60.5 LCB v5
• NVIDIA-OpenReasoning-Nemotron-7B: 63.9 LCB v5
• NVIDIA-Nemotron-Ultra-253B: 68.1 LCB v5

DASD-4B-Thinking (69.3 LCB v5) surpasses the 253B Nemotron Ultra (68.1) and the 32B Qwen3 (65.7) on code generation. The gap over Qwen3-4B-Thinking-2507 on LCB v6 is striking: 67.5 vs. 55.2 (+12.3). This is the largest relative improvement across all benchmarks, suggesting that the distillation pipeline is particularly effective at transferring code reasoning capabilities that the base 4B model lacks.

On GPQA-Diamond:

• DASD-4B-Thinking: 68.4
• Qwen3-4B-Thinking-2507: 65.8
• Qwen3-32B: 68.4
• NVIDIA-Nemotron-Ultra-253B: 76.0

DASD-4B-Thinking matches Qwen3-32B exactly (68.4) and closes to within 7.6 points of the 253B Nemotron Ultra. The paper notes that GPQA is "notoriously challenging for compact models due to its heavy reliance on parametric knowledge" — the 4B student cannot match the 120B teacher's factual knowledge, so closing the gap to this extent through reasoning capability alone is notable. However, the remaining gap to the 253B model (7.6 points) is larger than the gaps on math and code, consistent with the interpretation that parametric knowledge advantages are harder to transfer through distillation than reasoning patterns.

Staged Training Ablation (Section 7.5)

The pipeline's cumulative contribution is traced in Table 7, starting from the Qwen3-4B-Instruct-2507 base model:

The low-temperature stage provides the largest single gain: AIME25 jumps from 47.4 to 74.0 (+26.6), LCB v6 from 35.1 to 50.6 (+15.5). The high-temperature stage adds substantial further gains, particularly on coding: LCB v5 improves from 56.6 to 68.4 (+11.8), LCB v6 from 50.6 to 67.2 (+16.6). The mixed-policy stage adds small but consistent gains across all five benchmarks (range: +0.3 to +0.9). The GPQA-D trajectory is unusual: low-temp training provides a large gain (62.5→67.7, +5.2), but high-temp training provides essentially no additional benefit (67.7→67.6, −0.1), and mixed-policy provides a modest recovery (67.6→68.4, +0.8). This suggests that scientific reasoning at the 4B scale benefits primarily from the stable, high-confidence reasoning patterns in low-temperature data, with high-temperature diversity providing negligible additional benefit — a domain-specific saturation effect that the paper does not analyze in detail.

MoE Transfer Results (Section 7.7)

The headline finding: DAS data curated for the 4B student transfers effectively to a 30B MoE student with no re-curation (Table 8). Using only the first-stage 105K T=0.6 DAS data:

• DASD-30B-A3B-Thinking-Preview: 86.7 AIME25, 72.8 LCB v6, 72.3 GPQA-D, 77.3 average
• Qwen3-30B-A3B-Thinking-2507: 85.0 AIME25, 66.0 LCB v6, 73.4 GPQA-D, 74.8 average
• gpt-oss-20b (teacher variant): 91.7 AIME25, 61.0 LCB v6, 71.5 GPQA-D, 74.7 average
• NVIDIA-Nemotron-3-Nano-30B-A3B (18M SFT + RL): 89.1 AIME25, 68.3 LCB v6, 73.0 GPQA-D, 76.8 average

DASD-30B-A3B-Thinking-Preview achieves the highest average (77.3) despite using only 105K training examples versus NVIDIA-Nemotron-3-Nano's 18M (a 171× data efficiency ratio) and no RL. The LCB v6 lead is particularly large: 72.8 vs. 68.3 for NVIDIA (+4.5) and 72.8 vs. 66.0 for Qwen3-30B-A3B-Thinking-2507 (+6.8). On AIME25, the 86.7 score trails NVIDIA (89.1) by 2.4 points, which the paper attributes to using only the first training stage — full temperature scheduling and mixed-policy distillation, as applied to the 4B model, would likely close this gap. This is a preview result, not a complete evaluation, but it provides strong evidence for the cross-architecture transferability of DAS-curated data.

---

Ablation Studies and Robustness Checks

Temperature comparison at matched data volume (Table 1). Training on 50K T=1.0 responses outperforms 50K T=0.6 responses across all configurations: +1.4 AIME24 and +4.2 AIME25 for gpt-oss-120b teacher; +3.9 AIME24 but −1.1 AIME25 for Qwen3-Next-80B-A3B-Thinking teacher. The AIME25 reversal with the Qwen teacher (−1.1) is the only case where T=0.6 outperforms T=1.0, suggesting teacher-specific distributional properties affect the optimal temperature choice.

Data scaling at T=1.0 (Table 1). Doubling T=1.0 data from 50K to 100K yields diminishing returns: AIME24 stays flat at 83.1; AIME25 improves only from 76.1 to 78.9 (+2.8). This is substantially smaller than the gain from adding T=0.6 cold-start to 50K T=1.0 data (AIME25: 76.1→81.3, +5.2), demonstrating that the temperature curriculum provides benefits that data scaling alone cannot replicate.

Multi-domain scaling (Table 2). Doubling the T=1.0 mixture from 45K to 90K: AIME25 improves 75.2→75.8 (+0.6), LCB v6 improves 47.3→51.3 (+4.0), GPQA-D stays flat at 65.4. The cold-start approach with 45K data achieves 77.5 AIME25 (+1.7 over doubled T=1.0), 51.0 LCB v6 (−0.3), 66.4 GPQA-D (+1.0). These results show that the temperature schedule provides complementary benefits to data scaling, with the specific benefit varying by domain.

DAS vs. RS at matched and doubled data (Table 3). DAS at 50K T=1.0 (85.0/79.2) vs. RS at 100K T=1.0 (83.1/78.9): DAS matches or exceeds RS with 2× data. At T=0.6: DAS at 50K (83.3/74.2) vs. RS at 50K (81.7/71.9): consistent 1.6–2.3 point gains. The DAS benefit is robust across temperatures but relatively larger at T=1.0, where the teacher's distribution is broader and random sampling is less efficient at covering informative modes.

Masked vs. unmasked mixed-policy (Table 5). Unmasked (7.7K examples): 83.3 AIME24, 74.8 AIME25. Masked: 80.8 AIME24, 72.3 AIME25. The masked variant degrades below the baseline (83.3/74.2), confirming that the student-generated context is the active mechanism and that teacher completions alone — without the student's prefix in the training loss — actively harm performance through distributional shift.

Effect of DAS on data distribution shape (Figure 10, Section 7.6). The response probability distributions with and without DAS are "nearly identical" — DAS induces "negligible perturbation to the underlying response probability distribution." This is an important robustness check: it demonstrates that DAS is not simply selecting for higher-probability or lower-probability responses, which would shift the distribution and potentially interact with the temperature schedule. The orthogonality between DAS and temperature scheduling is what enables their combination to produce cumulative gains.

Cross-teacher validation of temperature-scheduled learning (Table 1, lower section). The pattern replicates with Qwen3-Next-80B-A3B-Thinking: cold-start + T=1.0 (83.1 AIME24, 73.1 AIME25) outperforms T=1.0-only (82.9/70.2) and T=0.6-only (79.0/71.3). The probability distribution shift between temperatures for this teacher is visually smaller (Figure 4 vs. Figure 3), yet the curriculum still helps, suggesting the mechanism is not simply about bridging a large distribution gap but about ordering samples from easier-to-harder learnability.

Sentence-type correlation with correctness (Figure 6). Across two independently trained models (internal DASD model and DeepSeek-Distill-Qwen3-8B, a completely different distillation pipeline), Teacher Sentences consistently show higher probability in correct answers (solid lines above dashed lines). The pattern generalizes across models and training procedures, supporting the claim that the Teacher/Student/Shared/Boosted taxonomy captures general properties of reasoning distillation rather than artifacts of the DASD setup.

Negative result: Data scaling saturates quickly at T=1.0. The paper shows repeatedly (Tables 1, 2) that doubling high-temperature data volume produces marginal gains compared to the temperature curriculum. This is a negative result for the "just collect more data" approach that dominates the field, and it provides the empirical motivation for shifting focus from data quantity to distributional coverage quality.

---

Critical Assessment

Does the paper demonstrate that DASD-4B-Thinking achieves state-of-the-art performance at its scale?

Yes, with strong evidence. Table 6 and Figure 9 provide comprehensive comparisons against an extensive set of baselines at multiple scales (from 3B to 253B parameters), and DASD-4B-Thinking leads all models of comparable size while surpassing several much larger models. The 83.3 on AIME25 exceeds the 32B Qwen3 (72.9), the 32B GLM-Z1 (63.6), and the 253B Nemotron Ultra (72.5) — these are not cherry-picked weak baselines but leading models from major industrial labs. The coding benchmark results are similarly strong (69.3 LCB v5 exceeds the 253B Nemotron Ultra at 68.1). However, all baseline numbers are taken from official reports, not re-evaluated under the DASD paper's exact evaluation protocol. If baselines used different sampling temperatures, different numbers of samples for averaging, or different maximum generation lengths, the comparisons are not perfectly controlled. The paper's 64-sample, temperature-1.0, 102K-token-limit protocol is explicit, but the baseline papers may have used different settings. This is standard practice in the field but introduces an uncontrolled variable that could affect the precision of the comparisons.

Does the paper demonstrate that data efficiency (448K vs. millions of examples) is achieved through methodological innovations rather than through particularly clean data sources or fortunate teacher-student pairing?

Partially, with some gaps. The paper demonstrates that its innovations improve over random-sampling baselines in controlled experiments (Tables 1–5), which isolates the methodological contribution from data source quality. However, the full-pipeline comparison (448K achieving 83.3 AIME25) relies on the specific combination of teacher (gpt-oss-120b), student (Qwen3-4B), question sources (AceReason, OpenCodeReasoning, OpenScience, AM-1.4M), and the three innovations. The paper does not ablate how much of the final performance comes from the question sources alone (i.e., training with random sampling on the same 435K data without DAS or temperature scheduling). The closest comparison is Table 2's multi-domain experiment, where temperature-scheduled + DAS achieves 77.5 AIME25 on 45K data vs. T=1.0-only random sampling at 75.2 (same data, +2.3 gain). Extrapolating this 2.3-point gain to the full 435K dataset is reasonable but not directly verified. Additionally, gpt-oss-120b is a very strong teacher — the paper does not compare against a baseline that uses the same teacher, same questions, and static-temperature random sampling at the full 435K data scale. Such a baseline would cleanly isolate the contribution of the three innovations from the teacher and question quality. The paper provides component-level evidence that each innovation helps (Sections 3, 4, 5), but the magnitude of their combined contribution at the final 448K scale is not directly measured against a matched-data random-sampling baseline.

Does the paper demonstrate that the three innovations address the three identified limitations?

For Limitation 1 (inadequate coverage of teacher's distribution): Temperature-scheduled learning demonstrably improves over single-temperature training (Tables 1, 2), and the mechanism is plausible — low-temperature data covers high-probability modes, high-temperature data adds rarer modes. However, the paper does not directly measure distribution coverage. Figure 3(a) shows that T=1.0 sampling broadens the response probability range, and Table 1 shows that this broader coverage improves downstream performance when combined with cold-start curriculum. But the paper does not quantify mode coverage (e.g., how many distinct reasoning strategies are captured at each temperature), so the claim that temperature scheduling "broadens coverage of the teacher's full mode structure" is supported by proxy evidence (better downstream performance) rather than direct measurement.

For Limitation 2 (misalignment with student's learning capacity): DAS demonstrably improves over random sampling (Tables 3, 4), and the sentence-type analysis (Figure 6) provides a plausible mechanism — Teacher Sentences correlate with correctness, and DAS explicitly selects for high $p_T$, low $p_S$ examples. However, the paper does not directly demonstrate that DAS mitigates the "misleading gradients" problem it identifies. The claim is that SFT on DAS-selected data "naturally mitigates misleading gradients" because Teacher Sentences have $p_T \gg p_S$, meaning the student's probability is low relative to the teacher's. But this is a selection criterion, not a gradient modification. DAS selects examples, but once selected, they are still trained with standard cross-entropy loss, which still pushes probabilities up on all tokens. DAS does not produce per-token gradient calibration — it selects examples where the gradient direction is more likely to align with the teacher's distribution. This is an important distinction: DAS is a data selection method that probabilistically avoids misleading gradients, not a gradient modification method that eliminates them. The positive results support DAS's effectiveness, but the paper has not isolated whether the benefit comes from avoiding misleading gradients or from some other property of high-divergence examples (e.g., they may simply be more informative or challenging). A direct test would involve measuring gradient alignment with the teacher's distribution for DAS vs. random sampling, but this is not done.

For Limitation 3 (exposure bias): The mixed-policy distillation evidence is the most direct. Figure 7 shows that the student's autoregressive outputs diverge from the teacher's (demonstrating the problem), and Table 5 shows that mixed-policy training with student prefixes retained helps while masking them hurts (demonstrating the solution). The masked-vs-unmasked ablation is a clean, well-designed experiment that isolates the mechanism. However, the absolute gains from mixed-policy distillation are small — +0.3 to +0.9 across benchmarks in Table 7 — and without error bars it's unclear if they exceed evaluation noise. The paper frames these as meaningful because they are applied on top of an already strong model and are consistent across all five benchmarks. This is a reasonable interpretation but would benefit from either multiple runs to establish variance or a larger mixed-policy data scale to see if gains continue to increase.

Are there genuine weaknesses in the experimental design?

1. No matched-data random-sampling baseline at the full 448K scale. The strongest evidence for the pipeline's data efficiency would be a direct comparison: same teacher, same questions, same total data volume (448K), but with random sampling instead of DAS, single-temperature instead of scheduled, and no mixed-policy distillation. The paper provides component-level ablations that strongly suggest such a baseline would underperform, but the aggregate gap is not directly measured. This gap matters because some of the pipeline's efficiency may come from the teacher quality or question curation rather than the innovations.

2. No statistical significance reporting. All results are single-run point estimates with 64-sample averaging. The 64-sample averaging reduces variance compared to single-sample evaluation, but it does not eliminate run-to-run variance from training (different random seeds for data ordering, dropout, etc.). The mixed-policy gains (+0.3 to +0.9) and some DAS gains (+0.2 on GPQA-D in Table 4) are small enough that they could plausibly fall within run-to-run variance. Re-running the key experiments with multiple seeds and reporting standard deviations would substantially strengthen the evidence.

3. DAS computation cost is not accounted for in efficiency claims. Computing DAS requires forward passes through the student model on teacher-generated responses to obtain student-side token probabilities. The paper does not report this computational cost or factor it into the data efficiency comparison. If DAS requires scoring (e.g.) 500K teacher responses to select 50K, then the total computation (teacher generation + student scoring + training) may be comparable to or greater than training on 100K randomly sampled responses (teacher generation + training). The "half the data" efficiency claim would then apply to training data volume but not necessarily to total pipeline computation. The paper is transparent that DAS requires student-side probabilities (Section 4) but does not quantify the overhead.

4. The cross-architecture transfer experiment (Section 7.7) is a preview, not a complete result. DASD-30B-A3B-Thinking-Preview uses only the first training stage (T=0.6 DAS). It demonstrates that the data transfers, but it does not demonstrate that the full pipeline (temperature scheduling, mixed-policy) would produce proportional gains on the MoE architecture. The strong LCB v6 result (72.8, +4.5 over NVIDIA Nemotron-3-Nano) is encouraging but preliminary.

5. The "misleading gradients" claim is theoretical, not directly tested. The paper identifies misleading gradients as a key problem in Section 4 but does not design an experiment to directly measure gradient alignment between SFT on random samples vs. DAS-selected samples. The evidence for DAS addressing misleading gradients is the downstream performance improvement, which is consistent with the hypothesis but does not rule out alternative explanations (e.g., DAS simply selects more informative or higher-quality examples for reasons unrelated to gradient alignment).

6. No comparison against logit-based distillation baselines. The paper explicitly chooses to improve sequence-level distillation rather than adopt logit-based methods, and it provides practical reasons for this choice (tokenizer mismatch, proprietary teacher access). However, for the specific case where both models are available locally (e.g., Qwen3-Next-80B-A3B-Thinking as teacher, Qwen3-4B as student, same tokenizer family), a logit-distillation baseline would help quantify how much of the "distribution-matching gap" the proposed methods recover. Without this, it's unclear how close DASD gets to the performance ceiling that full logit supervision would provide.

7. The evaluation setup uses pass@64, which is different from the standard pass@1 metric used by some baselines. The paper reports 64-sample average accuracy, which is functionally pass@64. If some baselines report pass@1 (single-sample accuracy), the numbers are not directly comparable — pass@64 is always higher than pass@1 because it benefits from sampling variance. The paper uses the "official report" numbers for all baselines and does not specify whether those baselines also used multi-sample averaging. If Qwen3-32B reports pass@1 numbers and DASD-4B-Thinking reports pass@64, the comparison overstates DASD's advantage. The consistency of the 64-sample protocol across the DASD evaluations is good, but the baseline numbers may use different protocols.

What experiments would strengthen the paper but are absent?

• Full-pipeline baseline: Random sampling + static temperature + no mixed-policy on the exact same 448K data sources, to quantify the aggregate gain from all three innovations.
• Multiple training runs with error bars: For the key ablations (Tables 1, 3, 5, 7) and especially the final pipeline results (Tables 6, 7), running 3–5 seeds and reporting mean ± std would establish whether the small gains from mixed-policy and some DAS configurations are reliable.
• Direct gradient alignment measurement: Compute whether DAS-selected examples produce gradients whose direction is better aligned with the teacher's distribution (measured via KL divergence decrease on a held-out set) compared to randomly selected examples, to directly test the "misleading gradients" hypothesis.
• Logit-distillation comparison: On the Qwen3-Next-80B-A3B-Thinking → Qwen3-4B pair (same tokenizer family), train a logit-distillation baseline to quantify the ceiling that full distributional supervision provides, and measure how much of that ceiling DASD recovers.
• Pass@1 evaluation: Report pass@1 in addition to pass@64 to enable cleaner comparison with baselines that may use different sampling protocols, and to assess whether the improvements hold at lower inference budgets.
• DAS cost analysis: Quantify the compute overhead of divergence-aware sampling (how many teacher responses must be scored by the student to select the final training set) and compare total pipeline FLOPs against a random-sampling pipeline with equivalent total compute, not just equivalent training data volume.
• Difficulty-stratified analysis: Break down benchmark performance by problem difficulty (similar to the MATH difficulty quintile analysis in the reference paper) to understand whether DASD's gains are concentrated in particular difficulty regimes — this would provide richer insight into what capabilities distillation is actually transferring.

6. Limitations and Trade-offs

6.1 The Difficulty Estimation Cost Is Not Accounted For in Efficiency Claims

The assumption or constraint. Divergence-aware sampling requires computing student-side token probabilities on teacher-generated responses to identify Teacher Sentences where p_T ≫ p_S. The paper makes this requirement explicit in Section 4:

"our method only requires, for each token in the teacher-generated response, its predicted probability by both the teacher and the student. The teacher-side probabilities are naturally obtained during sampling... while the student-side probabilities are readily computed from the local model."

The paper also acknowledges in the same paragraph that "the teacher-side probabilities for the student's outputs are typically unavailable for proprietary models" — but this is framed as an advantage of DAS over logit-based methods, not as a cost. What is not addressed is the computational overhead of this per-token scoring. To select (e.g.) 50K DAS examples from a larger pool of teacher-generated responses, the student model must perform forward passes on potentially hundreds of thousands of complete reasoning traces — each potentially tens of thousands of tokens long — to obtain the per-token probabilities needed for divergence computation.

The consequence. The paper's headline efficiency claims compare training data volume (448K examples for DASD vs. 2.9M for AM-thinking-v1, Table 6; vs. 30M for NVIDIA-OpenReasoning-Nemotron-7B) but do not include the computation required for DAS curation itself. If DAS requires (hypothetically) scoring 500K teacher responses to select 50K training examples, the total pipeline FLOPs may be comparable to — or exceed — training on 100K randomly sampled responses without DAS scoring. The "half the data" efficiency claim (e.g., DAS at 50K T=1.0 matches RS at 100K T=1.0, Table 3) would then apply only to training data volume, not to total pipeline computation. A practitioner deciding whether to implement DAS needs to weigh the training data reduction against the scoring overhead, and the paper provides no quantification to inform this tradeoff. Additionally, for very long CoT traces (the paper's cutoff length is 64K tokens, Section 6.4.1), the student-side forward pass for probability computation may itself be a non-trivial fraction of the teacher's generation cost — particularly if many candidate responses must be scored per selected example.

What evidence exists in the paper. None. The paper does not report (a) how many teacher-generated responses were scored to produce the final DAS training sets, (b) the total FLOPs consumed by student-side probability computation during data curation, or (c) any comparison against a random-sampling pipeline matched on total FLOPs rather than total training examples. The efficiency comparisons throughout Sections 3, 4, and 7 are all based on training data volume, not total pipeline computation.

Mitigation status. Not addressed. The paper does not mention the scoring cost in its discussion of DAS efficiency, nor does it discuss strategies for reducing this cost (e.g., approximate scoring via a lightweight proxy model, scoring only a subset of tokens, or caching student probabilities across related responses). The MoE transfer result (Section 7.7) indirectly mitigates the amortized cost — one DAS curation serves multiple student architectures — but the per-pipeline cost for the initial curation remains unquantified. The "Future Work" section does not mention DAS cost reduction as a direction.

---

6.2 The Method Is Validated on a Single Cross-Architecture Teacher-Student Pair Under a Narrow Task Distribution

The assumption or constraint. The full pipeline results (Table 6, Section 7.4) are demonstrated on a single teacher-student pair: gpt-oss-120b as teacher and Qwen3-4B-Instruct-2507 as student. The paper chose this pair deliberately to demonstrate cross-architecture compatibility (different tokenizers, vocabularies, pretraining corpora — Section 1), and the component-level ablations validate individual innovations with a second teacher (Qwen3-Next-80B-A3B-Thinking, Tables 1 and 3) and a second student (Qwen3-30B-A3B-Instruct-2507, Table 8). However, the full three-stage pipeline is only validated on the single gpt-oss-120b → Qwen3-4B pair. Furthermore, all benchmarks are drawn from a narrow set of reasoning-focused tasks: competition mathematics (AIME), competitive programming (LiveCodeBench), and graduate-level science QA (GPQA Diamond). The paper does not evaluate on open-ended generation, dialogue, factual knowledge recall, instruction following beyond the training data's instruction-following component (which is used for training but not evaluated), or non-English benchmarks.

The consequence. Several aspects of the method's effectiveness may be specific to the tested configuration:

• Teacher specificity. gpt-oss-120b is described as a 120B model (Section 1). The optimal temperature schedule (T=0.6 cold-start + T=1.0 expansion) may depend on this particular teacher's output distribution — a teacher with different calibration properties, different reasoning style diversity, or different temperature-sensitivity might require different scheduling parameters. The paper implicitly acknowledges this by noting that "optimal temperature combinations can be selected based on the model's evaluation performance and the response probability distributions observed" (Section 3), but does not provide guidance for this selection.

• Student scale. The 4B student has substantial room for improvement (base Qwen3-4B: 47.4 AIME25, 35.1 LCB v6, Table 7), enabling the pipeline to demonstrate large gains. Whether the same innovations would produce meaningful improvements on a student that is already closer to the teacher's capability (e.g., a 70B model distilling from a 120B teacher) is unknown — the distributional gaps that DAS exploits may be narrower, reducing the selection criterion's informativeness.

• Task domain. The reasoning benchmarks emphasize verifiable, closed-form answers. The DAS criterion (selecting samples where p_T ≫ p_S) may behave differently for open-ended tasks — creative writing, summarization, dialogue — where correctness is subjective and probability divergence may correlate more with stylistic differences than reasoning quality. The paper provides no domain-diversity analysis of DAS's effectiveness; Table 4's multi-domain validation shows DAS gains across math (+1.0), code (+3.2), and science (+0.2), but the science gain is negligible.

• Training task distribution. The training data spans four domains (math, code, science, instruction-following — Section 6.1), but evaluation only covers the first three. Instruction-following performance is not reported, making it impossible to assess whether the pipeline transfers to the full training distribution.

What evidence exists in the paper. The component-level cross-teacher validation (Tables 1 and 3, Qwen3-Next-80B-A3B-Thinking) and cross-student validation (Table 8, Qwen3-30B-A3B) provide partial evidence for generalizability, but neither replicates the full three-stage pipeline. The Qwen3-Next teacher only validates temperature scheduling and DAS in isolation (math-only, small scale); the MoE student only validates the first training stage (T=0.6 DAS, 105K examples). The full pipeline's composite behavior — the interaction between DAS, temperature scheduling, and mixed-policy distillation across multiple domains — is validated only on the single gpt-oss-120b → Qwen3-4B configuration.

Mitigation status. Partially acknowledged. The paper frames the cross-architecture choice as demonstrating "broad compatibility" (Section 1) and provides the MoE transfer result as evidence for scalability (Section 7.7). However, it does not discuss the narrow task distribution as a limitation, nor does it suggest multi-domain evaluation as future work. The "Future Work" section mentions integrating "agentic capabilities — such as knowledge retrieval and tool use" but not broader task evaluation. The absence of instruction-following evaluation despite including it in training data is not explained.

---

6.3 Hard Problems Near the Student's Capability Ceiling Show Diminishing or Negligible Returns from High-Temperature Diversity

The assumption or constraint. Temperature-scheduled learning assumes that high-temperature data — which covers rarer teacher modes — provides complementary benefits to low-temperature data across the difficulty spectrum. However, the staged ablation (Table 7) reveals that high-temperature training provides essentially zero additional benefit on GPQA-Diamond (67.7 → 67.6, a −0.1 change after high-temperature training), while providing large gains on math (+9.0 AIME25) and code (+11.8 LCB v5, +16.6 LCB v6). This is not an isolated anomaly — it also appears in Table 2's multi-domain experiment, where high-temperature training over T=0.6-only provides no GPQA-D gain (65.5 → 65.4 at T=1.0-only vs. 65.5 at T=0.6-only).

The consequence. This pattern suggests a domain-specific capability ceiling that temperature-scheduled learning cannot breach. GPQA-Diamond is described by the paper as "notoriously challenging for compact models due to its heavy reliance on parametric knowledge" (Section 7.4). The implication is that on tasks where the base 4B student fundamentally lacks the factual knowledge required to reason correctly — regardless of reasoning quality — broader coverage of the teacher's reasoning modes provides no benefit because the bottleneck is knowledge, not reasoning strategy. Temperature scheduling assumes that coverage of rarer teacher modes transfers generalizable reasoning skills; this assumption fails when the missing capability is factual rather than procedural.

A practitioner evaluating DASD for a domain where the student has limited parametric knowledge (e.g., specialized scientific reasoning beyond the GPQA distribution, legal reasoning requiring extensive case law knowledge, medical diagnosis requiring domain-specific facts) cannot assume that temperature scheduling will provide benefits proportional to those observed on math and code. The paper provides no principled way to predict which domains will benefit and which will hit the knowledge ceiling. The diminishing returns from scaling T=1.0 data alone (Table 1: AIME25 improves only 76.1 → 78.9 when doubling from 50K to 100K) further suggests that there are hard limits to what temperature-based diversity can achieve even within a single domain.

What evidence exists in the paper. The GPQA-Diamond saturation is visible in Table 7 (67.7 → 67.6 after high-temperature training) and corroborated in Table 2 (GPQA-D gains from high-temperature are minimal: 65.5 T=0.6 → 65.4 T=1.0 → 65.4 T=1.0 doubled). The paper notes GPQA's knowledge-dependence in Section 7.4 but does not connect this observation to a limitation of the temperature-scheduling methodology. The data scaling saturation at T=1.0 (Tables 1 and 2) provides additional evidence that high-temperature diversity has bounded value independent of the temperature scheduling ordering.

Mitigation status. Not addressed. The paper does not discuss domain-specific capability ceilings, does not analyze why high-temperature training fails to improve GPQA-D, and does not provide guidance on when the temperature schedule should be abridged (e.g., skipping high-temperature training for knowledge-intensive domains). The "Future Work" section mentions integrating "knowledge retrieval" as a direction for developing "more powerful, domain-adapted reasoning models," which indirectly acknowledges the knowledge bottleneck, but does not frame it as a limitation of the current distillation methodology. The paper's conclusion that "diverse exploration under higher temperature effectively expands the policy's solution coverage once a stable baseline has been established" (Section 7.5) is presented as a universal finding, despite the GPQA-D counter-evidence in the very same ablation table.

---

6.4 Mixed-Policy Distillation Provides Only Marginal Gains at the Current Scale, and Its Scaling Behavior Is Unknown

The assumption or constraint. Mixed-policy distillation is introduced as a solution to exposure bias (Section 5) and is included as a distinct stage in the final pipeline, contributing to the released DASD-4B-Thinking model. However, the gains from this stage are small and consistent but narrow: +0.3 to +0.9 across all five benchmarks in the staged ablation (Table 7). These gains are obtained from only 12.7K mixed-policy examples (Section 6.4.2) and one epoch of training, with 20K additional off-policy samples added for balance. The paper itself characterizes this as a "lightweight" stage with "minimal training overhead" (Section 7.5). The small absolute magnitude of the gains, combined with the absence of any statistical significance testing or multiple-run variance estimation, raises uncertainty about whether these improvements exceed the noise floor of the 64-sample evaluation protocol.

The consequence. A practitioner deciding whether to implement mixed-policy distillation faces an unclear cost-benefit assessment:

• If the gains are real but small, the stage adds complexity to the pipeline (requiring student generation, truncation, teacher completion, and quality filtering for an additional dataset) for at most a 0.9% absolute improvement. Whether this is worthwhile depends on the deployment context — for a model that is already state-of-the-art at its scale, an additional 0.9% on AIME24 (87.7 → 88.5) may be valuable; for a rapid prototyping setting, the pipeline complexity may not justify the marginal gain.

• If the gains are within noise, the stage adds unjustified complexity. The paper does not report variance, so this possibility cannot be ruled out. The fact that the masked variant degrades performance substantially (Table 5: 83.3 → 80.8 AIME24) — providing a clear negative result — suggests the effects are real, but the magnitude of the positive direction remains uncertain.

• Scaling behavior is unknown. The paper uses 12.7K mixed-policy examples because that is what survived quality filtering after generating from 50K training questions (Section 6.4.2). Would 50K, 100K, or 500K mixed-policy examples produce larger gains? Would the gains saturate? Is there an optimal ratio of off-policy to mixed-policy data? None of these questions are explored. The paper states that "these results motivate continued exploration of this promising direction in the future" (Section 5), explicitly delegating the scaling investigation to future work.

What evidence exists in the paper. Table 5 provides the ablation (unmasked vs. masked, 7.7K examples, T=0.6-only base model), showing +0.6 AIME25 with unmasked and −1.9 AIME25 with masked relative to baseline. Table 7 shows the full-pipeline contribution: +0.8 AIME24, +0.3 AIME25, +0.9 LCB v5, +0.3 LCB v6, +0.8 GPQA-D. All are single-run point estimates without error bars. The paper does not report any experiment varying the amount of mixed-policy data, the truncation position distribution, the teacher completion temperature, or the off-policy/mixed-policy ratio.

Mitigation status. Partially acknowledged. The paper describes mixed-policy distillation as a "lightweight" intervention (Sections 5 and 7.5) and explicitly states that it "motivates continued exploration of this promising direction" (Section 5), implying that the current implementation is preliminary. However, the paper does not discuss the uncertainty around the small absolute gains, does not report variance, and does not identify the scaling of mixed-policy data as a limitation. A practitioner reading only the abstract and main results would see mixed-policy distillation listed as one of three core innovations without being informed that its quantitative contribution is an order of magnitude smaller than the contributions from temperature scheduling (+9.0 AIME25) and divergence-aware sampling (implicitly included in the temperature-staged gains).

---

6.5 The Distribution Decomposition Framework Requires Access to a Pre-Distillation Student Model and Is Not Validated as a Real-Time Selection Mechanism

The assumption or constraint. The divergence-aware sampling criterion introduced in Section 4 — selecting training examples where the teacher's probability is high and the student's probability is low — depends on computing student-side probabilities on teacher-generated responses. This requires having access to (a) the teacher model for generation and probability computation, and (b) the pre-distillation student model (the base model before SFT) for probability computation on those same responses. The sentence-type analysis in Figure 6 that motivates DAS further requires access to a post-distillation student model (the distilled model after training) to categorize sentences into Teacher/Student/Shared/Boosted — but this third model is only needed for the diagnostic analysis, not for the DAS selection itself.

For the DASD-4B-Thinking pipeline, the pre-distillation student is Qwen3-4B-Instruct-2507, and the post-distillation analysis uses the trained DASD models. This works because both are openly available. However, this requirement creates constraints:

• For a new student architecture, the DAS data must be curated with that specific student's probability computations. The MoE transfer result (Section 7.7) shows that DAS data curated for the 4B student transfers to the 30B student, which is encouraging but not guaranteed — the paper does not provide a theoretical justification for this transfer, and there may exist student architectures where the divergence patterns differ enough that re-curation is necessary.

• For rapid iteration, computing student probabilities on every candidate teacher response before training adds a preprocessing step that scales with the number of candidate responses considered. If a practitioner wants to experiment with DAS on a new teacher, a new student, or a new domain, they must first perform this scoring step.

The consequence. DAS is not a real-time or universal selection mechanism — it requires instance-level probability computation involving both the teacher and the specific student that will be trained. This contrasts with data filtering heuristics (correctness verification, length-based selection, repetition filtering) that can be applied to teacher-generated data independently of the student model. A practitioner who wants to pre-curate a general-purpose distillation dataset for use with multiple student models (as the open-source community often does — e.g., OpenThoughts, OpenR1, AM-1.4M) cannot apply DAS without committing to a specific student model for probability computation. The MoE transfer result partially mitigates this concern (showing that one curation can serve multiple architectures), but (a) the result is preliminary (only one transfer, only first training stage), and (b) both the 4B and 30B models share the Qwen3 architecture family and pretraining — transfer to a completely different architecture (e.g., a LLaMA-based or Mistral-based student) is untested.

What evidence exists in the paper. The MoE transfer experiment (Table 8) provides the only cross-student evidence. The paper explicitly notes that the 4B-curated data is reused "to test the cross-architecture transferability of our data" and that "we do not re-collect or re-curate training samples" (Section 7.7). This is a positive result but limited to a single transfer pair within the same model family. No experiment tests transfer to a different architecture family. The paper does not analyze whether the DAS criterion's effectiveness depends on the student model's scale, architecture, pretraining data, or tokenizer compatibility with the teacher.

Mitigation status. Partially addressed. The MoE result is presented as evidence for cross-architecture transferability. The paper states that DAS "does not require re-sampling data for every new student model" (Section 4), citing the MoE result in Section 7.7. However, this claim is based on a single positive example and is not accompanied by any theoretical argument about why DAS scores should generalize across students. A practitioner adopting DAS for a different student architecture would be extrapolating from a sample size of one transfer pair within the same model family. The "Future Work" section does not mention investigating the generalizability of DAS across student architectures.

---

6.6 Exposure Bias Is Documented But Not Quantitatively Connected to Downstream Failure Modes

The assumption or constraint. Section 5 presents the core evidence for exposure bias: Figure 7 shows that student-generated responses are "cut off" (hit the maximum generation length without producing a complete answer) at increasingly high rates as the teacher's reference response length grows, and this cut-off rate is substantially higher than what would be expected if the student faithfully reproduced the teacher's behavior. The paper then presents mixed-policy distillation as a solution, validating it through downstream benchmark improvements (Tables 5 and 7). However, the paper does not directly connect the exposure bias it measures (truncation rate as a function of teacher response length) to downstream error patterns — it does not analyze whether the benchmarks where mixed-policy distillation provides gains are the same benchmarks where exposure bias is most severe, or whether the specific failures that mixed-policy distillation corrects are the long-context reasoning failures that the truncation analysis predicts.

The consequence. The causal chain from exposure bias → downstream errors → mixed-policy correction is plausible but not directly verified. Several alternative explanations for mixed-policy distillation's small gains are compatible with the evidence:

• Additional training data: The mixed-policy stage adds 12.7K new training examples (plus 20K off-policy balancers). The gains could simply be from additional training data rather than from the on-policy context specifically.
• Domain-specific effects: The truncation analysis is domain-agnostic, but the mixed-policy gains vary by benchmark (Table 7: +0.9 LCB v5, +0.8 GPQA-D, +0.3 LCB v6). If exposure bias is the mechanism, why does LCB v6 benefit less than LCB v5?
• Teacher quality improvement: The teacher completions in the mixed-policy data may be higher quality (because they are generated from the student's actual context) than the original off-policy data, providing a better training signal independent of the exposure bias mechanism.

The masked-vs-unmasked ablation (Table 5) partially rules out the "additional data" explanation — the masked variant provides the same number of additional training examples but degrades performance. However, it does not distinguish between the "exposure bias correction" and "teacher quality improvement" explanations. The student's prefix context may help the teacher generate more targeted, context-appropriate completions that are simply better training examples than generic teacher-generated responses, regardless of whether the student learns to handle its own generation context.

What evidence exists in the paper. The truncation analysis (Figure 7) quantifies the exposure bias phenomenon. The masked-vs-unmasked ablation (Table 5) provides strong evidence that the student's prefix context is mechanistically important (masking hurts). The staged ablation (Table 7) shows small but consistent gains. However, no experiment directly links the truncation rates measured in Figure 7 to the benchmarks where mixed-policy distillation provides gains. The paper does not report, for example, whether AIME25 problems with longer teacher reference responses benefit more from mixed-policy distillation than problems with shorter responses, which would directly test the exposure bias mechanism. The paper does not analyze whether the student's errors on specific benchmarks are concentrated in long-context reasoning steps (where distributional drift would be most severe) or in short-context factual errors (unrelated to exposure bias).

Mitigation status. Not addressed. The paper presents the truncation analysis and the mixed-policy gains as related through the exposure bias hypothesis, but does not test the hypothesized causal link. The "Future Work" section mentions "further refining the mixed-policy distillation approach to enhance training efficiency and stability" but does not mention investigating the specific mechanism by which mixed-policy data improves performance, or validating that it operates through exposure bias reduction rather than through other channels. A practitioner deciding how to allocate resources between off-policy data scaling and mixed-policy data generation currently has no basis for predicting which error types mixed-policy distillation will correct.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a diagnostic reframing, not a new architecture or loss function. The core shift is treating sequence-level distillation for long-CoT reasoning as a distribution-matching problem rather than a data-filtering problem, and showing that this reframing surfaces optimization axes that the dominant SFT-centric view renders invisible. The magnitude of the change is moderate — the paper does not overturn the SFT-on-teacher-data paradigm but rather re-interprets it through the lens of knowledge distillation, recovering insights that the community has largely ignored since Kim and Rush (2016) first formalized the sequence-level KL objective.

The most concrete landscape-altering finding is the data efficiency ceiling under pure SFT filtering: the paper demonstrates repeatedly (Tables 1, 2) that doubling high-temperature data volume produces marginal gains — AIME25 improves only 76.1→78.9 (+2.8) when scaling from 50K to 100K T=1.0 math samples with gpt-oss-120b — while the temperature schedule alone (adding a T=0.6 cold start to 50K T=1.0 data) improves AIME25 to 81.3 (+5.2). This quantifies a diminishing-returns curve that the "just collect more data" approach is already hitting, and it provides an empirical foundation for shifting community effort from data volume scaling toward distributional alignment. If this result replicates across other model families and domains, it implies that the current race to release ever-larger distillation datasets (1M, 2M, 30M examples) is solving the wrong optimization problem — what matters is not how many teacher responses you collect, but how well the collected set represents the teacher's distribution in ways the student can effectively learn from.

This work also reconciles a latent tension in the distillation literature. On one side, logit-based distillation (Hinton et al., 2015) provides rich per-token supervision but requires tokenizer compatibility and teacher logit access, making it impractical for the heterogeneous teacher-student pairs common in open-source reasoning distillation (proprietary API teachers, open-weight students). On the other side, sequence-level distillation (SFT on teacher outputs) is simple and architecture-agnostic but discards almost all distributional information. The DASD paper argues that this tension is false — there exists a middle ground where sequence-level methods can recover substantial distributional information without requiring token-level logits, through sentence-level probability discrepancy analysis (Section 4) and temperature-based distribution coverage (Section 3). The paper does not fully close the gap to logit-based supervision — an explicit logit-distillation baseline is absent — but it demonstrates that the sequence-level paradigm is not inherently limited to point-mass approximations of the teacher's distribution. This opens a research program into distribution-aware sequence-level methods that achieve more of logit distillation's benefits while retaining sequence-level distillation's architectural flexibility.

Finally, the paper addresses a specific contradiction that has puzzled practitioners: why do some distillation efforts succeed with modest data while others require millions of examples? The DASD framework provides an explanatory mechanism: data that aligns the teacher's output distribution with the student's learning capacity (the DAS criterion) transfers reasoning capabilities far more efficiently than data selected by surface quality heuristics. The empirical consequence — 448K DASD examples outperforming 2.9M AM-thinking-v1 examples on AIME25 (83.3 vs. 74.4, Table 6) — is not merely an efficiency gain but evidence that the choice of which teacher responses to train on matters more than the total volume, and that this choice should be guided by teacher-student distributional alignment rather than by human-designed filtering rules. This finding, if it generalizes, fundamentally changes how practitioners should allocate their distillation compute budgets: invest in distributional curation (temperature scheduling, divergence analysis) before investing in data volume scaling.

Follow-Up Research This Work Enables

1. Directly measure gradient alignment between DAS-selected and randomly-selected training examples to validate the "misleading gradients" hypothesis. The paper identifies misleading gradients as a key limitation of standard SFT on teacher data — SFT pushes student probabilities up on all tokens regardless of the teacher's distribution — and proposes DAS as a selection mechanism that probabilistically avoids this problem by prioritizing examples where the student's probability is already low relative to the teacher's. However, this claim is tested only through downstream performance, not through direct gradient analysis. A direct experiment would: (a) compute the gradient of the cross-entropy loss on a DAS-selected batch versus a randomly-selected batch, (b) project both gradients onto the direction that minimizes KL divergence from the teacher's distribution (measured on a held-out set of teacher responses), and (c) test whether DAS-selected batches produce gradients with larger positive projection onto the KL-descent direction. A positive result would provide mechanistic validation for DAS beyond the current correlational evidence. A negative result — DAS improves performance without better gradient alignment — would suggest that DAS's benefit comes from some other property (e.g., selecting more informative or challenging examples) and would redirect future work toward identifying that property.

2. Characterize the domain-dependence of temperature-scheduled learning by testing the GPQA-D saturation pattern on additional knowledge-intensive benchmarks. The staged ablation (Table 7) reveals that high-temperature training provides no benefit on GPQA-Diamond (67.7→67.6, −0.1 after high-temperature) while providing large gains on math (+9.0 AIME25) and code (+11.8–16.6 LCB). The paper attributes this to GPQA's "heavy reliance on parametric knowledge," but tests this interpretation on only one benchmark. A systematic follow-up would evaluate temperature-scheduled distillation on a suite of benchmarks that vary along a knowledge-vs-reasoning spectrum: knowledge-intensive tasks (MMLU, TriviaQA, MedQA), reasoning-intensive tasks (MATH, GSM-Symbolic, ARC), and mixed tasks (BIG-Bench Hard, HumanEval). If high-temperature training consistently fails to improve knowledge-intensive benchmarks regardless of domain, this would establish a principled boundary condition for temperature scheduling — practitioners could skip the high-temperature stage for knowledge-centric applications. If high-temperature training fails on some knowledge tasks but not others, the interaction between temperature scheduling and parametric knowledge transfer would be more nuanced and merit deeper investigation.

3. Scale mixed-policy distillation to determine whether its gains saturate or compound at larger data volumes, and whether the optimal truncation strategy is length-dependent. The paper applies mixed-policy distillation with only 12.7K examples and one epoch, achieving small but consistent gains (+0.3 to +0.9, Table 7). The ablation establishes that the on-policy context is mechanistically important (masking hurts, Table 5), but does not explore scaling behavior. Would 50K, 100K, or 500K mixed-policy examples produce proportionally larger gains, or does the benefit saturate quickly? Is the random truncation position (anywhere beyond half the response length, Section 6.4.2) optimal, or would a difficulty-aware truncation strategy — truncating earlier for problems where the student makes errors early in reasoning, later for problems where errors accumulate — produce larger gains? A direct experiment would sweep mixed-policy data volume (e.g., 10K, 25K, 50K, 100K examples) and truncation strategies (random vs. error-position-targeted vs. length-proportional) on a fixed base model, measuring gains on benchmarks stratified by reference response length. If gains from more mixed-policy data saturate quickly, practitioners can use the minimal effective dose; if they compound, mixed-policy distillation would warrant a larger compute allocation in the distillation budget than the paper currently allocates.

4. Test DAS cross-architecture generalization across fundamentally different model families to determine whether the MoE transfer result (Section 7.7) reflects a general property or a Qwen-family artifact. The paper shows that DAS data curated for Qwen3-4B transfers to Qwen3-30B-A3B (both Qwen3 architecture, shared pretraining, Section 7.7). This is encouraging but insufficient to claim general cross-architecture transferability. A direct follow-up would curate DAS data once (e.g., using gpt-oss-120b teacher and Qwen3-4B student), then train students from different architecture families on the same DAS data: a LLaMA-based student (e.g., LLaMA-3.2-3B), a Mistral-based student (e.g., Mistral-7B), and a Gemma-based student (e.g., Gemma-2-9B). If DAS transfers (i.e., DAS-trained models outperform random-sampling-trained models of the same architecture at matched data volumes), it establishes DAS as a general-purpose curation method where data is collected once and reused across the broader community. If it fails to transfer for some architectures, the characteristics that predict transfer success (shared tokenizer family? similar pretraining data? comparable scale?) become important to identify for practitioners selecting teacher-student pairs.

5. Develop a lightweight difficulty predictor that replaces per-token DAS scoring with a single forward-pass approximation, reducing the curation overhead. The paper does not quantify the computational cost of DAS curation — computing student-side token probabilities on all teacher-generated candidate responses before selection. For a practitioner working with a 4B student and long CoT traces (up to 64K tokens), this scoring cost could rival or exceed the cost of training on the selected data. A natural extension is to approximate the DAS criterion without full token-level probability computation: for example, train a lightweight binary classifier on (prompt, teacher response) pairs to predict whether a response would be DAS-selected, using DAS scores on a subset of data as training labels; or use only the student's sequence-level probability (the geometric mean of per-token probabilities, which the paper already computes for distribution visualization in Figure 3) as a proxy for the full per-token divergence. A successful approximation that recovers (e.g.) 90% of DAS's performance gain at 10% of the scoring cost would make DAS practical at much larger scales and for rapid iteration. The paper's own result that DAS induces "negligible perturbation" to the response probability distribution (Figure 10, Section 7.6) suggests that sequence-level signals may be sufficient — if DAS is not substantially shifting the overall probability distribution, perhaps a cheaper sequence-level proxy can approximate the per-token selection criterion.

6. Compare DASD against a logit-distillation baseline on a same-tokenizer teacher-student pair to quantify how much of the distribution-matching gap is recovered. The paper deliberately avoids logit-based methods due to tokenizer incompatibility with gpt-oss-120b, but the component-level experiments use Qwen3-Next-80B-A3B-Thinking as a second teacher (Tables 1, 3), which shares the Qwen tokenizer family with the Qwen3-4B student. This pair enables a clean comparison: train one model with DASD (temperature-scheduled + DAS + mixed-policy, sequence-level only), and another with a state-of-the-art on-policy logit distillation method (e.g., the Thinking Machines Lab approach from Lu and Lab, 2025, which minimizes KL divergence between student and teacher logits on student-generated sequences), both using the same teacher, same student, and same total training data budget. The gap between the two would quantify the distribution-matching ceiling that DASD leaves on the table — how much additional performance could be extracted if full logit supervision were available. If the gap is small, DASD effectively approximates logit distillation without requiring logit access; if large, it identifies a target for further sequence-level innovations (e.g., distribution-aware reweighting during SFT, which the paper mentions as future work in Section 8) and establishes an upper bound that practitioners can weigh against the practical costs of logit-based methods.

Practical Applications and Downstream Use Cases

1. Cost-efficient distillation for resource-constrained research groups producing open-source reasoning models. The paper's central result — 448K training examples producing a 4B model that outperforms a 32B model trained on 2.9M examples on AIME25 (83.3 vs. 74.4, Table 6) — directly enables groups with limited inference budgets to produce competitive reasoning models. A research lab with access to a strong teacher API (e.g., gpt-oss-120b or equivalent) but limited GPU hours for data generation can apply the DASD pipeline: curate ~450K examples with temperature scheduling and divergence-aware sampling, train with standard SFT hyperparameters (the paper's 6-epoch, 64K-context, ZeRO-3 configuration is fully specified in Section 6.4.1), and achieve state-of-the-art performance at the 4B scale without needing to generate millions of teacher responses. The specific resource multiplier is ~6× fewer training examples than the most comparable open-data model (AM-thinking-v1 at 2.9M) and ~67× fewer than the most data-intensive baseline (NVIDIA-OpenReasoning-Nemotron-7B at 30M). For groups working with open-weight teachers where the entire pipeline can run locally, the paper provides a complete recipe: question sources (Section 6.1), temperature settings (T=0.6 cold start + T=1.0 expansion), DAS implementation details (per-token probability comparison between teacher and pre-distillation student), and mixed-policy construction (student prefix retention, Section 6.4.2). The MoE transfer result (Section 7.7) further enables amortization: one DAS curation can serve multiple student architectures within the same model family.

2. Targeted deployment of compact reasoning models in domains where knowledge depth matters less than reasoning quality. The GPQA-Diamond saturation pattern (Table 7: high-temperature training provides no gain on GPQA-D) establishes a clear boundary condition: DASD's efficiency gains are concentrated in reasoning-procedural domains (math, code) rather than knowledge-intensive domains (graduate-level science). This directly informs deployment decisions. For applications where the reasoning task draws on general logical deduction rather than specialized factual knowledge — automated math tutoring, code review and debugging, formal verification of logical arguments, competitive programming assistance — a compact DASD-distilled model at 4B scale can approach or exceed the performance of much larger general-purpose models (Table 6: DASD-4B-Thinking surpasses Qwen3-32B on AIME25 by 10.4 points and on LCB v5 by 3.6 points). This enables on-device or low-latency deployment of high-quality reasoning without the hardware requirements or inference cost of 32B+ models. Conversely, for applications requiring deep domain knowledge — medical diagnosis, legal analysis, specialized scientific research — the paper's results suggest that distillation alone cannot close the gap to larger models (DASD-4B-Thinking at 68.4 on GPQA-D trails NVIDIA-Nemotron-Ultra-253B at 76.0 by 7.6 points), and practitioners should either use larger base models or integrate retrieval augmentation (which the paper mentions as future work in Section 8).

3. Data curation-as-a-service for the open-source reasoning ecosystem, enabled by DAS's cross-architecture transfer property. The MoE experiment (Table 8) demonstrates that DAS-curated data transfers from a 4B student to a 30B student within the same architecture family, achieving competitive performance (86.7 AIME25, 77.3 average) at the first training stage. If the cross-architecture generalizability is validated across model families (see Follow-Up Research #4), this enables a new workflow: a well-resourced organization curates a single DAS-optimized dataset once (using a strong teacher and a reference student), and releases it publicly for use by the broader community with arbitrary student architectures. This is analogous to how ImageNet pretraining enabled transfer learning across vision architectures, but at the level of reasoning data curation rather than model weights. The efficiency multiplier is substantial: instead of hundreds of research groups independently spending inference compute to generate and filter teacher responses for their specific student models, one high-quality DAS curation serves the entire ecosystem. The paper's release of the DASD training dataset alongside the model weights (Section 1) is a concrete step toward this model, but the full realization requires establishing that the DAS criterion generalizes beyond the Qwen family — a question the paper raises but does not answer conclusively.

When to Prefer This Method

The paper explicitly positions DASD as an improvement over the standard SFT-on-teacher-data distillation paradigm (Section 1) and as an alternative to logit-based distillation (Section 2). The decision boundaries that emerge from the paper's experimental results and stated assumptions are:

Prefer DASD over standard random-sampling SFT distillation when:

• You are distilling from a proprietary or API-accessible teacher that does not expose token-level logits, but does expose sequence-level output probabilities (which many APIs provide). DASD requires only per-token probabilities for the generated sequence — not full-vocabulary logits — making it compatible with teachers that logit-based methods cannot use.
• Your teacher and student use different tokenizers or vocabularies — a scenario where logit-based distillation is infeasible. The paper's primary teacher-student pair (gpt-oss-120b and Qwen3-4B-Instruct-2507) deliberately tests this condition.
• Your student model has substantial room for improvement relative to the teacher (e.g., 4B student distilling from 120B teacher). The DAS criterion exploits large teacher-student probability discrepancies; if the student is already close to the teacher (e.g., 70B distilling from 120B), the divergence signal may be too weak to provide informative selection.
• Your domain is reasoning-procedural rather than knowledge-intensive. The temperature scheduling benefits are concentrated on math and code; GPQA-Diamond shows saturation (Table 7). For knowledge-heavy domains, standard SFT on quality-filtered teacher data at the maximum affordable volume may be equally effective, since high-temperature diversity adds little (Tables 2 and 7).
• You are operating under a constrained data generation budget (e.g., limited API credits for teacher queries). DASD achieves with 448K examples what comparable open-data models require 2.9M–30M examples to match or exceed (Table 6).

Prefer logit-based on-policy distillation over DASD when:

• Your teacher and student share the same tokenizer and vocabulary, and you have access to the teacher's full token-level logits. Under these conditions, logit distillation provides richer per-token calibration that DASD only approximates through sequence-level selection. The paper does not directly compare against logit distillation, so the performance gap is unknown, but the theoretical argument (Section 2) implies logit distillation is an upper bound on the distribution-matching objective that DASD approximates.
• You can afford the computational cost of on-policy generation — having the student generate responses, computing teacher logits on those student-generated sequences, and aligning distributions at the token level. This cost scales with the number of training iterations rather than being a one-time pre-training curation cost.

Prefer standard SFT with large-scale random sampling over DASD when:

• You are curating a general-purpose distillation dataset for release to the community, intended for use with arbitrary student architectures that you cannot anticipate in advance. DASD requires selecting a reference student model for divergence computation; if that reference student is not representative of downstream users' architectures, the DAS criterion may not transfer (the MoE result is encouraging but limited to one family). Random sampling, despite being less efficient, makes no assumptions about the student.
• Your compute bottleneck is training FLOPs rather than data generation FLOPs, and the DAS curation overhead (scoring all teacher responses with the student model before training) is significant relative to your training budget. The paper does not quantify this overhead, but for very large candidate pools, it could dominate the pipeline cost.