Pre-training Distillation for Large Language Models: A Design Space Exploration

🎯 Pitch

Distilling knowledge from a 9B teacher into a 1.9B student during pre-training yields an 8% accuracy gain over standard training, yet using a larger 145B teacher actually hurts until the student reaches a critical size threshold—roughly 10% of the teacher's parameters—revealing a counterintuitive capacity gap. This systematic exploration across logits processing, loss functions, scaling laws, and offline vs. online settings uncovers practical recipes that challenge the assumption bigger teachers always teach better.

1. Executive Summary

This paper systematically explores the design space of pre-training distillation (PD) — extending knowledge distillation from the post-training phase into the pre-training phase of large language models, where a student LLM learns from the teacher model's logits on the pre-training corpus rather than from instruction-response pairs. Using GLM-4-9B as the teacher to distill student LLMs from 330M to 6.8B parameters on up to 500 billion tokens, the authors investigate four design dimensions: logits processing (top-p-k truncation and temperature normalization), loss selection (KLD vs. NLL vs. MSE, and static vs. scheduled combinations of distillation and LM loss), scaling law (varying student and teacher sizes and corpus volume), and offline vs. online logits. The exploration yields a better PD configuration that achieves an 8.0% relative improvement over standard LM-loss pre-training for the 1.9B student model, while establishing that larger student LLMs generally benefit more from pre-training distillation and that a larger teacher LLM does not necessarily guarantee better results — a capacity-gap dynamic where effectiveness emerges only once the student reaches roughly 10% of the teacher's parameter count.

2. Context and Motivation

The Core Problem: We Know Post-Training Distillation Works, but Pre-Training Distillation Is Unexplored

Knowledge distillation has become a standard practice in the large language model pipeline, but almost exclusively during the post-training phase. In post-training KD, a student LLM learns from a set of instruction-response pairs generated by a more capable teacher model (typically GPT-4 or similar) using standard language modeling loss. This approach underpins many widely-used open-source LLMs — Alpaca, Vicuna, WizardLM, and others — and has proven remarkably effective for aligning smaller models with instruction-following behavior at relatively low computational cost.

The fundamental question this paper raises is: if distillation works so well after pre-training, why not apply it during pre-training itself? The intuition is straightforward. A teacher model's logits contain richer information than one-hot ground-truth tokens. As Gou et al. (2021) established, these soft targets serve as a form of label smoothing — they encode the teacher's uncertainty, its knowledge of token similarity, and its implicit ranking of plausible next tokens. In principle, exposing a student model to this signal during its foundational pre-training phase should accelerate learning and improve the final model quality.

But despite this clear potential advantage, pre-training distillation remains largely unexplored. The paper identifies this as the central gap:

"there is limited exploration on how to better apply PD. Therefore, in this paper, we take an initial step in exploring the design space of pre-training distillation."

This gap is not just an academic oversight — it represents a missing piece in the LLM development toolkit. If pre-training distillation can reliably improve student model quality, it could reshape how organizations allocate their pre-training compute budgets.

Why This Problem Matters: Three Practical and Theoretical Stakes

1. The economics of pre-training at scale. Pre-training large language models from scratch is extraordinarily expensive. The largest models require trillions of tokens processed across thousands of GPUs for months. If distillation during pre-training can produce a higher-quality model from the same number of tokens — or equivalently, reach a target quality with fewer tokens — the cost savings are substantial. The paper's finding that a better PD configuration yields an 8% relative improvement over standard LM-loss pre-training (Table 5, WSD-α+WSD-LR configuration) translates directly to better return on pre-training investment. Even a 1–2% improvement at the scale of modern pre-training budgets (which can run into the tens of millions of dollars) justifies serious engineering attention.

2. The compression ratio problem for small models. The paper's scaling law experiments (Figure 4) reveal a nuanced dynamic: larger student LLMs generally benefit more from pre-training distillation, and a larger teacher does not guarantee better student results. This connects to a well-documented phenomenon in knowledge distillation — the capacity gap (Mirzadeh et al., 2020), where a student model that is too small relative to its teacher cannot effectively absorb the teacher's knowledge, because the teacher's representations encode distinctions the student simply cannot represent. Understanding exactly when pre-training distillation is effective (the paper finds the threshold is roughly when the student reaches 10% or more of the teacher's parameter count) has direct implications for organizations deciding whether to distill or independently pre-train smaller models.

3. A new axis in the scaling laws conversation. The broader LLM scaling literature — from Kaplan et al. to Chinchilla (Hoffmann et al., 2022) — has focused almost exclusively on the relationship between model size, data volume, and compute during pre-training, treating the training signal (one-hot token prediction) as a fixed given. The post-training distillation literature has demonstrated that teacher-provided signals can dramatically improve sample efficiency during alignment. This paper connects these two threads, asking whether the training signal itself can be upgraded during pre-training. If so, the standard scaling laws (which relate compute, parameters, and tokens) may need to be revised when the loss function includes a distillation term — the "value per token" of pre-training data is not constant when a teacher model enriches it.

Where Prior Approaches Fall Short

The paper identifies two categories of prior work that touch on pre-training distillation but leave the design space unexplored.

Category 1: Pre-ChatGPT small model distillation. Works like DistilBERT (Sanh et al., 2019), TinyBERT (Jiao et al., 2020), MiniLM (Wang et al., 2020a), and MobileBERT (Sun et al., 2020b) demonstrated that distillation could produce compact versions of BERT-scale models (typically hundreds of millions of parameters at most). These approaches proved that pre-training distillation is conceptually feasible. However, the paper argues their findings are not directly transferable to the billion-parameter LLM regime for several reasons:

Scale difference: Distilling a 110M-parameter BERT into a 66M-parameter student is a fundamentally different problem from distilling a 9B-parameter LLM into a 1.9B-parameter student. The capacity gap dynamics, the storage requirements for teacher logits, and the optimization challenges all scale non-linearly.
Training configuration mismatch: Pre-ChatGPT models used different architectures (encoder-only BERT), different training objectives (masked language modeling vs. autoregressive), different vocabulary sizes (typically 30K vs. 150K tokens), and different data scales. The design choices that worked for BERT distillation may not be optimal for GPT-style autoregressive LLM pre-training.
Infrastructure constraints: The paper highlights a practical problem that did not exist at BERT scale: storing teacher logits. For a 9B teacher model with a 150K vocabulary and 100 billion training tokens, storing full logits in float32 requires approximately 58.6 PB of disk space. This is not just expensive — it is infeasible at commodity infrastructure scales. The paper's top-p-k truncation strategy is itself a contribution motivated by this practical constraint.

Category 2: Recent LLM distillation with limited transparency. Several recent LLM releases mention using distillation during pre-training, but provide minimal detail on how it was actually implemented. The paper cites:

Gemma 2 (Team et al., 2024): mentions distillation but provides limited specifics on the distillation configuration.
AFM / Apple Intelligence Foundation Models (Gunter et al., 2024): uses a form of distillation (the paper notes AFM uses top-1 truncation, equivalent to using teacher-generated labels with standard LM loss) but does not systematically explore alternatives.
LokiLM (Kiefel and Shah, 2024): employs pre-training distillation without detailed ablation of design choices.
Minitron (Turuvekere Sreenivas et al., 2024): focuses primarily on pruning combined with distillation rather than standalone distillation design.

The paper explicitly positions itself against this opacity:

"Gemma 2, AFM, LokiLM, and Minitron employ pre-training distillation but provide limited details on the distillation process."

"While Muralidharan et al. (2024) explores the best practices for pruning and distillation of LLMs, it mainly focuses on pruning and does not systematically explore pre-training distillation."

The closest prior work, Muralidharan et al. (2024), studies distillation in the context of pruning — starting from a larger pre-trained model, pruning it, and then distilling. This is a different setting from training a student from scratch, which is the focus of this paper. The design decisions that matter when recovering pruned model quality may differ from those that matter when guiding a freshly initialized student through its entire pre-training trajectory.

How This Paper Positions Itself

The paper frames itself as a systematic design space exploration, not a method proposal. It does not claim to invent pre-training distillation — the preliminary experiment in Section 3.1 shows that vanilla PD (NLL loss, static α = 1, top-0.95-100 truncation, τ = 1.0) already provides a 1.6% average improvement over LM-loss pre-training, validating that the basic concept works. The contribution is in mapping the space of design choices and identifying which factors actually matter and which do not.

Specifically, the paper structures its investigation around four dimensions that it identifies as the key degrees of freedom in Equation 1:

Logits processing (F): How to truncate and normalize the teacher's logits to make storage feasible without destroying useful signal. This is a practical necessity (storing full logits requires 58.6 PB for 100B tokens with a 150K vocabulary) that becomes a design opportunity — different truncation strategies may improve or degrade the distillation signal.
Loss selection (L and α): Whether to use negative log-likelihood, Kullback-Leibler divergence, or mean squared error as the distillation loss, and how to combine distillation loss with standard LM loss — as a static weighted sum or a dynamically scheduled mixture. Prior work in image classification (Kim et al., 2021) found MSE to be superior, while LLM-specific findings are absent.
Scaling law: How the effectiveness of PD changes with student model size, teacher model size, and total pre-training corpus volume. This is the dimension with the most direct practical implications for organizations deciding whether and how to invest in pre-training distillation.
Offline vs. online logits: Whether to use logits from a fully pre-trained teacher (offline, requiring an additional forward pass over the entire corpus) or logits captured during the teacher's own pre-training (online, adding no inference cost but using a partially converged teacher). This is a cost-quality tradeoff with no prior characterization.

The paper's positioning is explicitly exploratory rather than conclusive:

"We hope our exploration of the design space will inform future practices in pre-training distillation."

This framing is appropriate given the scale of the experiments (up to 500B tokens, models up to 6.8B parameters, teacher models at 9B and 32B) and the combinatorial explosion of possible configurations. The paper does not claim to find the globally optimal PD configuration — it maps the terrain and identifies which directions are promising.

The Practical Constraint That Shapes the Entire Investigation

A crucial piece of context that the reader must understand is the storage problem that motivates the entire logits processing investigation (Section 3.1, Section 3.2). The paper computes:

GLM-4-9B has a vocabulary of approximately 150,000 tokens.
For 100 billion training tokens, storing one float32 logit per vocabulary entry per token requires: 100B × 150K × 4 bytes ≈ 58.6 PB (petabytes).
This is unaffordable at any reasonable infrastructure scale.

The offline distillation paradigm — pre-compute teacher logits, store them to disk, then train the student by reading logits during training — is what the paper uses throughout, and it is only feasible because of the top-p-k truncation strategy. The paper achieves a 4,000× reduction in storage (from 58.6 PB to approximately 15 TB for 100B tokens) by keeping only the highest-probability tokens.

This storage constraint is not merely an implementation detail — it fundamentally shapes the design space. Every design choice about truncation (what values of p and k) trades off storage cost against information retention. The paper's finding that "smaller p and k can be used to further reduce storage space" without significantly harming distillation quality (Section 3.2) is practically valuable precisely because storage is the binding constraint.

Moreover, the online logits approach (Section 3.5) can be understood as an alternative solution to the storage problem: rather than storing logits at all, capture them during the teacher's pre-training and immediately use them to train the student in a pipelined fashion. The 180 TB storage cost reported for the 400B-token online experiment is still substantial but eliminates the separate inference pass. The paper's recommendation — "if one aims to pre-train a series of LLMs of varying sizes, one can first pre-train the largest LLM while storing online logits, and then pre-train smaller LLMs with online logits" — reflects a practical deployment strategy shaped by storage economics.

The Unstated Premise: Why Logits Rather Than Text?

The paper focuses exclusively on logits-based KD — the student learns from the teacher's output probability distribution, not from the teacher's generated text. This is an important design commitment that the paper does not extensively justify, so it is worth understanding the implicit reasoning.

Text-based distillation (sequence-level KD; Kim and Rush, 2016) would involve having the teacher generate the next token, and the student learning to predict that token — essentially, treating the teacher's outputs as ground truth. This is what post-training KD typically does (learn from teacher-generated responses). Logits-based KD instead gives the student the teacher's full probability distribution over the vocabulary, which encodes much more information: which tokens the teacher considers plausible alternatives, how confident it is in its top prediction, and the shape of the probability mass across the vocabulary.

The paper's finding that top-1 truncation (k = 1 in Figure 3) — which is equivalent to text-based distillation where the student learns from the teacher's top prediction — still yields improvement over standard LM loss supports the value of even minimal teacher signal. But the finding that larger k values (up to k = 50) yield somewhat better results suggests that the richer probability distribution provides useful information beyond just the teacher's top-1 prediction.

Summary: What the Reader Should Take Forward

Before diving into Section 3's experiments, the reader should understand:

Pre-training distillation is intuitively promising but systematically unexplored. The post-training KD literature proves that teacher signal helps; this paper asks whether the same holds during the far more expensive pre-training phase, where even small improvements justify significant engineering investment.
The prior literature either operates at a different scale (pre-ChatGPT small models) or provides insufficient detail (recent LLM releases). No existing work systematically maps the design space: how to process logits, select losses, schedule the KD-LM loss mixture, scale across model sizes, or choose between offline and online teacher logits.
Storage constraints are not an implementation detail — they are the central practical challenge. The 58.6 PB storage requirement for naïve logits-based PD makes truncation strategies essential, and the entire logits processing dimension is motivated by the need to reduce storage while retaining useful signal.
The paper is an exploration, not a prescription. It aims to map the design space so that future practitioners can make informed decisions, not to declare a single best PD configuration. The 8.0% improvement from the "better configuration" (WSD-α+WSD-LR in Table 5) should be understood as evidence that the design space matters, not as a final answer.

3. Technical Approach

3.1 Reader Orientation

The system is a pre-training pipeline for large language models where, instead of learning to predict the next token from the ground-truth text alone, the model additionally learns to match the probability distribution produced by a larger, already-trained teacher model on every token of the pre-training corpus. The problem it solves is that standard pre-training with one-hot language modeling loss throws away the rich uncertainty information contained in a stronger model's predictions — pre-training distillation captures this signal to accelerate student learning and improve final model quality, with the central design challenge being how to process, weight, and schedule this teacher signal across billions of tokens and model sizes spanning an order of magnitude.

3.2 Big-Picture Architecture (Diagram in Words)

The pre-training distillation system has five major components that operate in a sequential pipeline:

Teacher LLM (GLM-4-9B or GLM-4-32B): A pre-trained large language model that processes the entire pre-training corpus token-by-token and produces, for each token position, a logit vector over its full vocabulary (~150K tokens). This model is frozen — no gradients flow through it during student training.
Logits processing module (F): A truncation and normalization stage that takes the teacher's raw logits, applies top-p-k truncation to retain only the highest-probability tokens (reducing storage by ~4,000×), re-normalizes the truncated logits into a proper probability distribution, and optionally applies temperature scaling. This module exists purely to make storage feasible — storing full float32 logits for 100B tokens with a 150K vocabulary requires ~58.6 PB, which is unaffordable.
Student LLM (330M to 6.8B parameters): The model being trained from scratch on the pre-training corpus. It produces its own next-token probability distribution at each token position, which is compared against both the one-hot ground-truth token (the LM loss) and the teacher's processed probability distribution (the KD loss).
Loss computation module: Takes the student's predicted distribution, the ground-truth next token, and the teacher's processed distribution, and computes a weighted combination of language modeling loss and distillation loss according to a mixing coefficient α. The choice of distillation loss function (KLD, NLL, MSE) and the schedule for α (static, linear, WSD) are the central design degrees of freedom.
Disk storage for offline logits: A persistent storage layer that holds the teacher's truncated logits for the entire pre-training corpus. In the offline setting, logits are pre-computed in a separate pass and read during student training. In the online setting, logits are captured during the teacher's own pre-training, streamed to storage, and later read for student training — eliminating the need for a separate inference pass but using a partially-converged teacher.

Information flows as follows: corpus text → teacher LLM forward pass → raw logits per token → top-p-k truncation + temperature normalization → disk storage → student LLM reads text + stored logits → student produces its own distribution → loss computation mixes LM loss and KD loss → gradient update to student parameters.

3.3 Roadmap for the Deep Dive

First, the formal distillation pre-training objective (Equations 1–4), which defines the mathematical framework that unifies all design dimensions — every subsequent section explores variations within this single objective.
Second, the preliminary experiment configuration, since it establishes the baseline pipeline (two-stage paradigm, storage constraints, evaluation protocol) that all subsequent controlled experiments inherit and modify.
Third, logits processing — the F function in Equation 1 — because truncation and temperature are the first operations applied to teacher logits and determine what information is available to all downstream components.
Fourth, loss selection — L and α — because once the teacher signal is processed, the choice of how to compare student and teacher distributions and how to weight that comparison against standard LM loss determines what the student actually learns.
Fifth, scaling law experiments — because model size and corpus size modulate the effectiveness of all upstream design choices and determine when pre-training distillation is worth the engineering investment.
Sixth, offline vs. online logits — because this choice determines how teacher logits are sourced and affects both the cost and the quality of the teacher signal, representing the highest-level architectural decision.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical design space exploration paper whose core idea is that pre-training distillation is feasible and effective for billion-parameter LLMs, but the specific configuration choices — how to truncate teacher logits, which distillation loss to use, how to combine distillation with standard LM loss, when scaling makes PD effective, and whether to use offline or online teacher logits — significantly impact the outcome, and the paper systematically maps these choices through controlled experiments.

The Formal Pre-Training Distillation Objective

The paper formalizes the pre-training distillation problem as a single unified optimization objective that encompasses all design dimensions. The notation operates at the token level: given a text sequence $x = \{x_t\}_{t=1}^T$ of $T$ tokens, a student LLM parameterized by $\theta_S$ , and a teacher LLM parameterized by $\theta_T$ , the training objective is:

$\theta_S^* = \arg\min_{\theta_S} \mathcal{L} = \arg\min_{\theta_S} \left[(1 - \alpha) \mathcal{L}_{\text{lm}} + \alpha \mathcal{L}_{\text{kd}}\right]$

where $\theta_S^*$ is the optimal student parameter vector that minimizes the composite loss, $\mathcal{L}_{\text{lm}}$ is the standard language modeling loss, $\mathcal{L}_{\text{kd}}$ is the knowledge distillation loss, and $\alpha \in [0, 1]$ is a mixing coefficient that controls the relative weight of the two loss terms.

What it computes: this equation defines the overall training objective as a convex combination of two loss functions. When $\alpha = 0$ , the student learns purely from the one-hot ground-truth tokens — this is standard pre-training. When $\alpha = 1$ , the student learns purely from the teacher's probability distribution — this is pure distillation. Intermediate values blend the two signals.

Why this form: the convex combination is the simplest way to mix two loss signals while preserving the property that the total loss is bounded between the two pure extremes. It also makes $\alpha$ directly interpretable: $\alpha = 0.9$ means the student receives 90% of its training signal from the teacher and 10% from the ground truth. The paper's exploration of dynamic scheduling (Section 3.3) keeps this same convex combination form but makes $\alpha$ a function of training step rather than a constant, allowing the mixture to evolve during training.

The standard language modeling loss is defined as:

$\mathcal{L}_{\text{lm}} = \frac{1}{T} \sum_{t=1}^{T} -\log P_{\theta_S}(x_t | x_{<t})$

where $P_{\theta_S}(x_t | x_{<t})$ is the probability the student model assigns to the true next token $x_t$ given the preceding context $x_{<t}$ , and $T$ is the sequence length (4,096 tokens in all experiments).

What it computes: the average negative log-probability of the correct next token across all positions in the sequence. This is the standard autoregressive language modeling objective: the model is penalized when it assigns low probability to the token that actually appears next in the training corpus.

Why this form: negative log-likelihood is the maximum-likelihood objective for categorical distributions, which is the natural choice when the target is a single discrete token. The $\frac{1}{T}$ averaging across sequence positions ensures the loss is comparable across sequences of different lengths and prevents the loss magnitude from scaling with $T$ .

The knowledge distillation loss is defined as:

$\mathcal{L}_{\text{kd}} = \frac{1}{T} \sum_{t=1}^{T} \mathcal{L}\left(P_{\theta_S}(x_t | x_{<t}),\; \mathcal{F}(P_{\theta_T}(x_t | x_{<t}))\right)$

where $\mathcal{L}$ is the distillation loss function (KLD, NLL, or MSE), $P_{\theta_T}(x_t | x_{<t})$ is the teacher model's raw probability distribution over the vocabulary at position $t$ , and $\mathcal{F}$ is a processing function that truncates and normalizes the teacher's logits before they are used as the distillation target.

What it computes: the average discrepancy between the student's predicted distribution and the teacher's processed distribution, taken across all token positions. The teacher processes the same context $x_{<t}$ as the student and produces its own next-token distribution; the student is trained to make its distribution similar to the teacher's under the chosen loss function $\mathcal{L}$ .

Why this form: the key architectural decision is that the teacher's distribution is processed by $\mathcal{F}$ before comparison. This is not merely an implementation detail — it is a necessary step for storage feasibility and a design degree of freedom. The teacher's raw logits span the full 150K vocabulary, creating an impossible storage requirement (58.6 PB for 100B tokens at float32). The $\mathcal{F}$ function is where truncation, re-normalization, and temperature scaling happen, and the paper's Section 3.2 explores exactly how to design this function.

The logits processing function is formalized as:

$\mathcal{F}(z) = \text{softmax}\left(\frac{\text{Truncate}(z)}{\tau}\right)$

where $z$ is the teacher's raw logit vector (one value per vocabulary token), $\text{Truncate}(z)$ applies the top-p-k truncation operation that retains only a subset of the logits, $\tau$ is the temperature parameter that controls the sharpness of the resulting distribution, and $\text{softmax}$ converts the truncated, temperature-scaled logits into a proper probability distribution that sums to 1.

What it computes: starting from the teacher's raw logits $z$ , the function first discards all but the highest-probability tokens according to the top-p-k criterion, then divides the remaining logits by temperature $\tau$ , and finally applies softmax to produce a normalized probability distribution. The output is a sparse probability vector where most vocabulary entries have exactly zero probability (those eliminated by truncation) and the retained entries form a valid distribution.

Why this form: the truncation-before-normalization order is critical. If normalization happened first (softmax then truncation), the resulting probabilities would not sum to 1. The temperature $\tau$ is applied after truncation because scaling the logits of only the retained tokens changes the shape of the distribution differently than scaling all logits would. When $\tau < 1$ , the distribution sharpens (higher peaks, lower entropy), making the teacher appear more confident. When $\tau > 1$ , the distribution flattens (approaching uniform), making the teacher appear less certain. The paper's experiments with different $\tau$ values in Table 2 explore how this sharpness affects student learning.

Preliminary Experiment Configuration

Before exploring the design space, the paper establishes a baseline PD pipeline and validates that pre-training distillation works at all. This configuration serves as the reference point from which all subsequent controlled experiments vary exactly one factor.

Two-stage paradigm. The paper uses an offline distillation paradigm with two distinct phases:

Logits generation and storage: The pre-trained teacher LLM (GLM-4-9B, frozen) processes the entire 100-billion-token pre-training corpus with a context window of 4,096 tokens per chunk. For each token position in each chunk, the teacher produces a logit vector over its ~150K vocabulary. These logits are truncated using top-p-k (described below) and stored to disk.
Student training: The student LLM (1.9B parameters, randomly initialized) is trained from scratch on the same 100B tokens, reading both the text and the stored teacher logits for each token. The student never interacts with the teacher directly — all teacher signal comes from the pre-computed logits on disk.

Logits truncation for storage feasibility. The paper computes that storing full float32 logits for 100B tokens with a 150K vocabulary would require approximately 58.6 PB of disk space, which is unaffordable. The baseline truncation strategy is a two-stage top-p-k method:

Stage 1 — top-p truncation with p = 0.95: For each token position, the teacher's logits are sorted by probability. Tokens are retained in descending probability order until their cumulative probability mass reaches 0.95. All remaining tokens (the long tail of the distribution) are discarded. The intuition is that tokens outside the 95% probability mass are individually so unlikely that their precise probabilities carry negligible information for the student.
Stage 2 — top-k truncation with k = 100: After top-p truncation, at most 100 tokens are retained, keeping the highest-probability ones. This acts as a safety cap: if the teacher's distribution is unusually flat (e.g., at the start of a sequence or in highly ambiguous contexts), top-p alone might retain thousands of tokens, and top-k truncation ensures storage cost is bounded.

This two-stage method reduces storage by approximately 4,000× — from 58.6 PB (hypothetical full logits) to approximately 15 TB for 100B tokens. The retained logits are re-normalized with softmax at temperature $\tau = 1.0$ (no sharpening or flattening) before being stored.

Baseline loss configuration. The preliminary experiment uses pure distillation with no LM loss component: $\alpha = 1$ in Equation 1. The distillation loss function $\mathcal{L}$ is negative log-likelihood (NLL): the student is trained to maximize the log-probability it assigns to each token, weighted by the teacher's probability for that token. In practice, this means:

$\mathcal{L}_{\text{kd}} = -\mathcal{F}(P_{\theta_T}(x_t | x_{<t})) \cdot \log P_{\theta_S}(x_t | x_{<t})$

This is equivalent to treating the teacher's processed distribution as a soft target distribution and training the student with standard cross-entropy where the target is the teacher's distribution rather than a one-hot vector.

Student model architecture. The 1.9B student LLM uses the architecture specified in Table 7: 2,048 hidden size, 6,912 FFN hidden size, 24 layers, 16 attention heads, 2 query groups (grouped-query attention; Ainslie et al., 2023), and no weight tying between embeddings and output. All models are trained in BFLOAT16 format (Kalamkar et al., 2019).

Training hyperparameters. Both the baseline LLM-LM (standard pre-training) and LLM-KD (distillation pre-training) use identical optimization settings: Adam optimizer (Kingma, 2014) with a batch size of 2,048 sequences, maximum sequence length of 4,096 tokens, a cosine learning rate scheduler with maximum learning rate $6 \times 10^{-4}$ and minimum learning rate $6 \times 10^{-5}$ , and 1% warmup rate (the learning rate linearly increases from 0 to the maximum over the first 1% of training steps, then follows a cosine decay to the minimum).

Supervised fine-tuning after pre-training. The paper notes a critical practical issue: at 1.9B parameters, the student LLM's zero-shot and few-shot performance on challenging benchmarks like MMLU and C-Eval is close to random guessing, making direct comparison difficult. To make evaluation meaningful, after pre-training distillation completes, all models undergo supervised fine-tuning (SFT; Ouyang et al., 2022) on 10 billion tokens of high-quality instruction-tuning data plus an additional 10 billion tokens of pre-training text corpus. During SFT, only the language modeling loss is used ( $\alpha = 0$ ), and for instruction-tuning data, the loss is computed only on the response portion of each example. SFT uses a smaller batch size of 256, a cosine learning rate scheduler with maximum $4 \times 10^{-5}$ and minimum $4 \times 10^{-6}$ , and 1% warmup rate.

Why SFT is necessary: the paper's goal is to measure whether pre-training distillation improves the fundamental quality of the student model, not just its perplexity on the pre-training corpus. SFT serves as a probe: after both LLM-LM and LLM-KD undergo identical fine-tuning, any performance difference on downstream benchmarks must be attributable to differences in the pre-trained representations, not to differences in the fine-tuning process. Without SFT, the benchmark numbers would be dominated by noise (near-random performance), making comparisons unreliable.

Evaluation protocol. The paper evaluates on eight datasets spanning English understanding, Chinese understanding, and math: HellaSwag (zero-shot), WinoGrande (zero-shot), PIQA (zero-shot), MMLU (6-shot), KBQA (zero-shot), C3 (5-shot), C-Eval (5-shot), and GSM8k (8-shot). All evaluations use sampling temperature 0 (greedy decoding). The choice of zero-shot for some datasets and few-shot for others reflects standard practice in the LLM evaluation literature, balancing the need for meaningful scores against evaluation cost.

Preliminary results and their interpretation. Table 1 shows LLM-KD achieves an average 1.6% relative improvement over LLM-LM across the eight benchmarks, with notable gains on GSM8k (+24.6% relative) and C-Eval (+3.2%). The paper interprets this as validation that pre-training distillation is feasible:

"we can observe that generally LLM-KD performs better than LLM-LM, though the improvement is marginal, indicating that pre-training distillation is feasible, but the current distillation configurations may not be optimal."

The phrasing "marginal" is important: the baseline configuration works, but the paper's premise is that a 1.6% average improvement is the floor, not the ceiling. The entire design space exploration that follows is motivated by the hypothesis that better configurations can extract substantially more value from the teacher signal.

Logits Processing (Design Dimension #1): The $\mathcal{F}$ Function

The logits processing function $\mathcal{F}$ is the first operation applied to teacher logits and determines what information the student receives. The paper explores two sub-dimensions: truncation strategy (how many logits to retain) and temperature (how to scale logits before softmax normalization). All experiments in this section use the same setup as the preliminary experiment except for the specific processing parameter being varied.

Top-p-k truncation: the two-stage method. The paper's truncation strategy combines top-p and top-k in sequence. Top-p truncation (Holtzman et al., 2019) was originally proposed for nucleus sampling during text generation — it retains only the smallest set of tokens whose cumulative probability mass reaches a threshold $p$ , discarding the long tail of unlikely tokens. Top-k truncation retains exactly the $k$ highest-probability tokens, providing a hard cap on storage.

The paper identifies a complementary dynamic between the two methods:

"When the logits distribution is sharp, top-p truncation is enough; when the distribution is more uniform with long-tailed non-trivial values, top-k truncation works as a secondary truncation."

In sharp distributions (teacher is confident), top-p with $p = 0.95$ may retain only a handful of tokens, and top-k provides no additional constraint. In flat distributions (teacher is uncertain, e.g., at the beginning of a sentence or in highly ambiguous contexts), top-p might retain hundreds or thousands of tokens because probability mass is spread thinly; top-k with $k = 100$ caps the storage at 100 entries per token position regardless of distribution shape.

Experiment 1: varying p in top-p-100 truncation. The paper fixes $k = 100$ and sweeps $p$ across values: 0.5, 0.6, 0.7, 0.8, 0.85, 0.9, 0.95. The results in Figure 2 and Table 8 show that all values of $p$ produce similar relative improvements over LLM-LM (clustered around 1–5% with no clear trend). The logits size per token drops substantially as $p$ decreases: the paper estimates that top-0.5-100 retains far fewer logits on average than top-0.95-100, and the "p-inf" curve in Figure 2 (top-p with no top-k cap) shows that without the $k$ cap, storage cost grows rapidly with $p$ .

"different p leads to similar improvements. A possible explanation is that in distillation pre-training, student LLM primarily captures the mass of the logits. This suggests that a smaller p can be used to further reduce storage space."

The operational implication: practitioners can use more aggressive truncation (smaller $p$ ) to reduce storage costs without measurably degrading student performance. The student model appears to learn primarily from the overall shape and mode of the teacher's distribution, not from the precise probabilities of long-tail tokens.

Experiment 2: varying k in top-0.95-k truncation. The paper fixes $p = 0.95$ and sweeps $k$ across values: 1, 3, 5, 10, 20, 50, 100. The results in Figure 3 and Table 8 show:

All values of $k$ yield improvements over LLM-LM.
$k = 50$ achieves the best average performance (39.6% vs. 37.7% for LM baseline), though differences are modest.
Even $k = 1$ (retaining only the teacher's top-1 prediction per token) provides improvement. The paper notes this is "equivalent to using the LM loss but with labels generated from the teacher LLM" — the student learns to predict the teacher's most likely token rather than the ground-truth token.

The $k = 1$ result is theoretically interesting because it demonstrates that teacher signal helps even in its most degenerate form: simply replacing the ground-truth next token with the teacher's top prediction. The paper speculates:

"This may be due to the teacher LLM conducting implicit noise filtering in pre-training corpora."

The teacher model, having been trained on a larger and potentially cleaner corpus, may assign high probability to tokens that are genuinely appropriate for the context even if they differ from the exact token in the training data. In cases where the training data contains noise, errors, or suboptimal phrasings, the teacher's top prediction may be a "corrected" version that provides a cleaner training signal than the raw text.

Why truncation order matters. The top-p-k method applies top-p first, then top-k, not the reverse. If top-k were applied first (keeping the top-k tokens by probability), top-p would then discard tokens from this already-truncated set, potentially eliminating tokens that are within the desired probability mass but outside the top-k. The paper's ordering ensures that all tokens within the desired probability mass are candidates for retention, and top-k only intervenes to cap storage when the distribution is unusually flat. Figure 2 confirms this logic: the "p-inf" curve (top-p without top-k) shows rapidly growing token counts as $p$ increases, while the "p-100" curve (top-p with top-100 after) is essentially flat — the top-k cap is binding and prevents storage blowup.

Temperature $\tau$ in logits normalization. After truncation, the retained logits are scaled by temperature $\tau$ before softmax normalization. The paper sweeps static $\tau$ values: 0.05, 0.1, 0.2, 0.5, 1.0, 2.0, 5.0, 10.0 (Table 2 and Table 9).

The results show:

Lower temperatures ( $\tau \leq 2.0$ ) produce similar improvements, clustered around 1.6–2.7% relative improvement.
At $\tau = 5.0$ , improvement drops to near zero (-0.1%).
At $\tau = 10.0$ , improvement returns to 1.0% — but the paper interprets the $\tau = 5.0$ result as evidence that "learning from a more uniform distribution may be not efficient for student LLM."

The mechanism: temperature controls the entropy of the teacher's distribution after truncation. At low $\tau$ , the distribution is sharp — the top token receives near-total probability mass, and the student receives a strong, unambiguous signal about which token the teacher considers most likely. At high $\tau$ , the distribution approaches uniform — the teacher's signal becomes weak because all retained tokens receive similar probability, providing little information beyond "these tokens are all plausible." The student cannot efficiently extract useful gradient information from a near-uniform target because the loss surface becomes flat: almost any distribution the student produces will be similarly close to the uniform target.

Adaptive temperature methods. The paper explores three methods that compute a per-token temperature based on properties of the teacher's logits:

NormKD (Chi et al., 2023): Applies adaptive temperature normalization to both teacher and student logits before computing the distillation loss. The paper uses the hyperparameters $T_{\text{norm}} = 1.0$ and $\alpha = 0.5$ (note: this is a different $\alpha$ from the one in Equation 1 — it is internal to the NormKD loss calculation).
WTTM (Zheng and YANG, 2024): Applies adaptive temperature only to the teacher logits (Weighted Teacher Temperature Matching). Hyperparameters: $\gamma = 0.1$ , $\beta = 1.0$ .
AdaKD (proposed in this paper): A simplified adaptive method that computes temperature based on the sharpness of the teacher's logits. Two variants are tested:
- AdaKDSD: Uses the standard deviation of the logits directly as the temperature $\tau$ . When the teacher is confident (sharp distribution, high standard deviation), $\tau$ is high, flattening the distribution and preventing over-confident targets. When the teacher is uncertain (flat distribution, low standard deviation), $\tau$ is low, preserving what little structure exists.
- AdaKDH: Computes temperature based on the entropy $H$ of the logits distribution:
$\tau_H = \tau_{\text{max}} - (\tau_{\text{max}} - \tau_{\text{min}}) \times \frac{H}{H_{\text{max}}}$

where $H$ is the entropy of the teacher's logits at the current token, $H_{\text{max}}$ is the maximum entropy observed across an estimation set of 10 million tokens (set to 4.8 in the experiments), $\tau_{\text{max}} = 2.0$ , and $\tau_{\text{min}} = 0.1$ . When entropy is high (flat distribution, uncertain teacher), the fraction $H / H_{\text{max}}$ is large, so $\tau_H$ approaches $\tau_{\text{min}}$ , sharpening the distribution. When entropy is low (peaked distribution, confident teacher), $\tau_H$ approaches $\tau_{\text{max}}$ , flattening the distribution.

Why this form for AdaKDH: the formula linearly interpolates between $\tau_{\text{max}}$ and $\tau_{\text{min}}$ based on normalized entropy. The subtraction from $\tau_{\text{max}}$ means the mapping is inverted: low entropy (confident teacher) maps to high temperature, while high entropy (uncertain teacher) maps to low temperature. This has the effect of compressing the dynamic range of the teacher's confidence — confident predictions are tempered down, uncertain predictions are sharpened up — which the paper hypothesizes helps the student focus on the "most important parts" while not being overwhelmed by the teacher's overconfidence.

Results of adaptive temperature (Table 3). NormKD performs worse than static temperature baselines (37.2% average, -1.3% relative to LLM-LM, compared to 38.7% for $\tau = 0.5$ static). WTTM shows essentially no improvement over LLM-LM (37.8%, +0.2%). AdaKDSD achieves 38.2% (+1.2%), comparable to static temperature baselines. AdaKDH achieves 38.8% (+2.8%), the best among adaptive methods but not significantly better than the best static temperature ( $\tau = 0.5$ achieved 38.7%).

"compared to static temperature (τ = 0.5), adaptive temperature does not show significant additional improvement."

The practical takeaway: adaptive temperature adds implementation complexity without clear benefit. A well-chosen static temperature ( $\tau \leq 2.0$ ) achieves comparable or better performance with simpler implementation. The paper does not recommend adaptive temperature for PD.

Loss Selection (Design Dimension #2): $\mathcal{L}$ and $\alpha$

Loss selection has two sub-dimensions: the choice of distillation loss function $\mathcal{L}$ that measures discrepancy between student and teacher distributions, and the mixing coefficient $\alpha$ that balances distillation against standard LM loss. For all experiments in this section, the logits processing uses the preliminary experiment's configuration (top-0.95-100, $\tau = 1.0$ ).

Distillation loss function $\mathcal{L}$ . The paper tests three loss functions, with $\alpha = 1$ (pure distillation, no LM loss) to isolate the effect of $\mathcal{L}$ :

Negative log-likelihood (NLL): As used in the preliminary experiment. The student is trained to assign high probability to each token $x_t$ , weighted by the teacher's probability for that token:

$\mathcal{L}_{\text{NLL}} = -\sum_{v \in \mathcal{V}} \mathcal{F}(P_{\theta_T}(v | x_{<t})) \cdot \log P_{\theta_S}(v | x_{<t})$

where $\mathcal{V}$ is the vocabulary and the sum is over all retained (non-truncated) vocabulary entries. This is equivalent to cross-entropy with the teacher's processed distribution as the target. The student is penalized most heavily when it assigns low probability to tokens the teacher considers highly likely, and penalized less for tokens the teacher considers unlikely.
Kullback-Leibler divergence (KLD): Measures the information lost when the student's distribution is used to approximate the teacher's distribution:

$\mathcal{L}_{\text{KLD}} = \sum_{v \in \mathcal{V}} \mathcal{F}(P_{\theta_T}(v | x_{<t})) \cdot \log \frac{\mathcal{F}(P_{\theta_T}(v | x_{<t}))}{P_{\theta_S}(v | x_{<t})}$

KLD is equal to cross-entropy minus the entropy of the teacher's distribution. Since the teacher's entropy is constant with respect to student parameters, KLD produces identical gradients to NLL — the optimization dynamics are the same. The practical difference (if any) comes from the absolute magnitude of the loss values, which can affect learning rate scheduling and early stopping.
Mean squared error (MSE): Measures the squared difference between the student's logits and the teacher's processed logits:

$\mathcal{L}_{\text{MSE}} = \frac{1}{|\mathcal{V}_{\text{retained}}|} \sum_{v \in \mathcal{V}_{\text{retained}}} \left(\text{logit}_S(v) - \mathcal{F}(\text{logit}_T(v))\right)^2$

where $\mathcal{V}_{\text{retained}}$ is the set of vocabulary entries retained after truncation, and $\text{logit}_S(v)$ and $\text{logit}_T(v)$ are the pre-softmax logit values from the student and teacher, respectively. MSE operates on logits rather than probabilities, treating the distillation problem as regression rather than distribution matching.

Results (Table 5). The clear finding:

NLL achieves 38.3% average (+1.6% relative to LLM-LM). Solid improvement.
KLD achieves 38.7% average (+2.6% relative). Slightly better than NLL in aggregate, though NLL performs better on some challenging datasets (MMLU: 27.8% for NLL vs. 26.7% for KLD; C-Eval: 26.7% for NLL vs. 25.4% for KLD).
MSE achieves 34.9% average (-7.6% relative). A significant and consistent degradation. MSE underperforms on every benchmark except WinoGrande, with particularly catastrophic drops on GSM8k (3.9% vs. 8.6% for LM baseline) and HellaSwag (44.6% vs. 53.3%).

Why MSE fails so badly. The paper notes this finding "contrasts with prior research in image classification (Kim et al., 2021), which finds MSE loss is the most superior choice in knowledge distillation." The discrepancy reveals a fundamental difference between image classification KD and LLM pre-training KD:

In image classification KD, the teacher produces a single distribution over a fixed set of classes (typically 1,000 for ImageNet). MSE on logits is well-behaved because the logit space is relatively low-dimensional and the teacher's logits are calibrated to produce meaningful probability differences.
In LLM pre-training KD, the vocabulary is large (150K tokens), and for each token position, the vast majority of vocabulary entries have near-zero probability. MSE on logits treats a prediction error of 0.001 vs. 0.0001 as equally important as an error of 0.9 vs. 0.1 — the scale of the logits can vary enormously across the vocabulary, and MSE is sensitive to these variations in ways that are uninformative for the student. The cross-entropy family (NLL, KLD) is naturally robust to this because it operates on probabilities, which are always in [0, 1] and sum to 1, providing an implicit normalization that MSE lacks.

"the pre-training distillation of LLMs involves new training dynamics and requires further investigation."

Combination of $\mathcal{L}_{\text{lm}}$ and $\mathcal{L}_{\text{kd}}$ : static α. The paper first explores fixed combinations of LM and KD loss by sweeping $\alpha$ across values: 0.0 (LM-only baseline), 0.1, 0.5, 0.6, 0.7, 0.8, 0.9, 0.95, 1.0 (KD-only). The KD loss is NLL. Results are in Table 4 and Table 10.

The pattern is an inverted-U shape:

At $\alpha = 0.1$ : +0.1% — essentially no improvement, the teacher signal is too weak.
As $\alpha$ increases from 0.5 to 0.9, improvement grows: +1.5% → +1.4% → +2.9% → +2.0% → +3.6%.
At $\alpha = 0.9$ : the best performance at +3.6% relative improvement (39.1% average).
At $\alpha = 0.95$ : improvement drops to +2.5%.
At $\alpha = 1.0$ : improvement is +1.6% (the preliminary experiment baseline).

The optimal $\alpha = 0.9$ reveals an important dynamic: pure distillation ( $\alpha = 1.0$ ) is not optimal. A small proportion (10%) of standard LM loss improves results. The paper hypothesizes:

"while a higher proportion of distillation loss can boost the distillation performance, an appropriate ratio (about 10%) of LM loss can further enhance pre-training distillation performance."

Why 10% LM loss helps: the LM loss provides a grounding signal — it ensures the student can still predict the actual next token in the training data, not just match the teacher's distribution. The teacher's distribution, while richer than one-hot targets, is not perfect: it reflects the teacher's own biases, uncertainties, and potential errors. The LM loss acts as a regularizer that prevents the student from overfitting to imperfections in the teacher signal. With 10% LM loss, the student must balance two objectives: match the teacher (KD loss) and get the right answer (LM loss). This multi-objective training likely produces more robust representations.

Dynamic scheduling of α. The paper explores five strategies for varying α during training, motivated by the hypothesis that different phases of training may benefit from different loss mixtures:

Linear increase (Linear Inc): α linearly increases from 0 to 1 during training. The intuition: early training with pure LM loss helps mitigate the capacity gap by allowing the student to learn basic language patterns from clean one-hot targets before attempting to match the teacher's more complex distribution. Late training with high α lets the student refine its representations toward the teacher's.
Linear decrease (Linear Dec): α linearly decreases from 1 to 0 during training. The intuition: using KD loss early provides better optimization initialization (Yim et al., 2017), giving the student a head start by learning from the teacher's rich signal before switching to fine-tuning on ground-truth data.
Periodic: α alternates between 0.9 and 0 every four batches — three batches with α = 0, one batch with α = 0.9, repeating. This is adapted from Kiefel and Shah (2024) and represents a form of interleaved training where the student switches rapidly between distillation and standard pre-training.
WSD-α with cosine learning rate (WSD-α+Cos-LR): α follows a warmup-stable-decay (WSD; Hu et al., 2024) schedule with a cosine learning rate scheduler. Specifically: α linearly increases from 0 to 1.0 during the warmup stage, stays at 1.0 during the stable stage, and then follows cosine decay from 1.0 to 0 during the decay stage. The warmup ratio is 10% of total training steps, and the decay ratio is 1%. The learning rate follows a standard cosine schedule (as in the preliminary experiment).
WSD-α with WSD learning rate (WSD-α+WSD-LR): The same α schedule as above, but the learning rate also follows a WSD schedule: warmup to maximum, stay at maximum, then cosine decay to minimum. The warmup and decay ratios for the learning rate are the same as for α (10% warmup, 1% decay). The intuition is critical: when the learning rate is at its maximum (the "stable" phase), the model is making its largest updates and most of the learning happens. Using a high proportion of KD loss during this phase may maximize the benefit of the teacher signal.
WSD-β with WSD-LR: An ablation where β (defined as $1 - \alpha$ , the proportion of LM loss) follows the WSD schedule instead of α. This means the LM loss proportion is high during the stable phase and decreases during decay. This tests whether the benefit comes from the WSD schedule itself or specifically from scheduling α (the KD proportion) with WSD.

Results (Table 5). The findings are striking:

Linear Inc: 38.1% (+1.1%) — worse than static $\alpha = 0.9$ . Using LM loss early and KD loss late is not effective.
Linear Dec: 39.2% (+4.1%) — substantially better than static $\alpha = 0.9$ . Starting with KD loss and transitioning to LM loss works well. The paper interprets: "involving more KD loss in the early pre-training stage is more beneficial."
Periodic: 38.0% (+0.9%) — no meaningful advantage. Interleaving distillation sporadically does not help.
WSD-α+Cos-LR: 38.6% (+2.3%) — better than static α but not as good as Linear Dec.
WSD-α+WSD-LR: 40.7% (+8.0%) — the best configuration in the entire paper, achieving a substantial jump over the LM baseline (37.7%) and over the preliminary PD configuration (38.3%).
WSD-β+WSD-LR: 38.1% (+1.1%) — essentially no improvement, confirming that the benefit is not from the WSD schedule alone but specifically from scheduling KD loss with WSD.

The comparison between WSD-α+WSD-LR and WSD-β+WSD-LR is the key ablation. Both use the WSD learning rate schedule. The only difference is whether α (KD proportion) or β (LM proportion) is high during the stable phase. The dramatic performance difference (40.7% vs. 38.1%) confirms the paper's central hypothesis about loss scheduling:

"using KD loss when maintaining a high learning rate effectively enhances model performance."

Why WSD-α+WSD-LR works so well: the WSD learning rate schedule divides training into three phases. During the stable phase (80% of training), the learning rate is at its maximum — this is when the model makes its largest parameter updates, and the quality of the gradient signal matters most. By setting $\alpha = 1.0$ (pure KD) during this phase, the student receives the richest possible training signal from the teacher during its period of maximum plasticity. During the decay phase (1% of training), α decreases to 0, introducing LM loss for final fine-tuning and preventing the student from over-specializing to the teacher. This phased approach — learn from the teacher when learning rate is high, fine-tune on ground truth when learning rate is low — maximizes the benefit of both signals.

The comparison with 0-α+WSD-LR (LM-only training with WSD learning rate, achieving 38.2% or +1.2%) further isolates the effect: WSD-LR alone provides a small benefit even without distillation (the 0.7 percentage point gap between WSD-LR and baseline cosine LR, 38.2% vs. 37.7%). But the majority of the 8.0% improvement comes from combining WSD-LR with the WSD-α distillation schedule.

The comparison between WSD-α+WSD-LR and 1-α+WSD-LR (pure KD with WSD-LR, achieving 39.6% or +5.0%) shows that including the decay phase with some LM loss (+40.7% vs. +39.6%) provides an additional boost — consistent with the static α finding that ~10% LM loss is beneficial.

Scaling Law (Design Dimension #3): Model Size and Corpus Size

The scaling law experiments ask: does pre-training distillation become more or less effective as we scale up the student model, the teacher model, or the corpus size? These experiments use the preliminary experiment's configuration (NLL loss, α = 1, top-0.95-100, τ = 1.0) unless otherwise noted, varying only the model sizes and corpus volume.

Varying student and teacher model sizes (Figure 4, Table 11). The paper tests students at 330M, 670M, 1.9B, 3.8B, and 6.8B parameters, distilled from teachers at 9B (GLM-4-9B) and 32B (GLM-4-32B). The architectures are specified in Table 7, with design choices that scale across sizes: the number of query groups in grouped-query attention (GQA) increases from 2 to 8, and weight tying (tying the embedding and output projection matrices) is used only for the 330M model.

The key finding from Figure 4:

"Larger student LLMs generally benefit more from pre-training distillation."

The relative improvement over LM-only baselines grows monotonically with student size for the 9B teacher: 330M shows essentially no improvement (32.1% KD vs. 32.6% LM, actually slightly worse), 670M shows modest improvement (33.8% vs. 33.6%), 1.9B shows clear improvement (38.3% vs. 37.7%), 3.8B shows strong improvement (44.9% vs. 42.0%), and 6.8B shows the largest gain (48.0% vs. 44.9%).

For the 32B teacher, the pattern is inverted: the 32B teacher underperforms the 9B teacher for several student sizes. Specifically:

330M student: 9B teacher achieves 32.1%, 32B teacher achieves 31.6%. Both are worse than LM-only (32.6%).
670M student: 9B teacher achieves 33.8%, 32B teacher achieves 34.2%. Both are similar.
1.9B student: 9B teacher achieves 38.3%, 32B teacher achieves 37.9%. 9B teacher is better.
3.8B student: 9B teacher achieves 44.9%, 32B teacher achieves 42.9%. 9B teacher is substantially better.
6.8B student: 9B teacher achieves 48.0%, 32B teacher achieves 47.4%. 9B teacher remains better.

The paper interprets this through the lens of the capacity gap (Mirzadeh et al., 2020):

"Distilling from a larger teacher LLM does not necessarily yield better performance. This may be due to the capacity gap between teacher and student LLMs."

The capacity gap arises because the teacher's internal representations encode information at a level of granularity and abstraction that the student's smaller architecture simply cannot represent. The 32B teacher's knowledge is more fine-grained and specialized; when a 1.9B student tries to match its distribution, the student lacks the representational capacity to capture the nuances, and the training signal becomes noisy rather than informative. The 9B teacher produces a distribution that is coarser and more learnable by the smaller student.

"From a compression perspective, larger LLMs compress information more effectively and achieve better compression rates, potentially making it harder for smaller LLMs to learn."

The paper identifies a rough threshold:

"pre-training distillation is effective when the size of the student LLM reaches about 10% or more of the teacher LLM size, and as the proportion increases, the benefits of pre-training distillation grow until reaches the turning point."

For the 9B teacher, this threshold is approximately 900M parameters — the 670M student (7.4% of teacher) shows marginal improvement, while the 1.9B student (21%) shows clear improvement. For the 32B teacher, even the 6.8B student (21%) shows only modest benefit, suggesting the effective threshold may be higher for larger teachers. The paper acknowledges:

"Due to computational constraints, we do not explore the turning point of performance gain to the proportion, which we leave as future work."

The paper also flags the weak-to-strong generalization direction (Burns et al., 2024) — using a small teacher to help train a large student — as a promising unexplored regime.

Varying pre-training corpus size (Figure 5, Table 12). The paper trains 1.9B and 3.8B students on 500 billion tokens (5× more than the baseline 100B) using GLM-4-9B as teacher, saving checkpoints every 10,000 optimization steps (~83B tokens) plus a final checkpoint at the end of training. For each checkpoint, the model is fine-tuned with SFT before evaluation (same protocol as the preliminary experiment).

The results in Figure 5 show:

Pre-training distillation consistently outperforms LM-only pre-training at every checkpoint throughout the 500B-token training run, for both model sizes.
The gap widens during early-to-mid training: at 83B tokens, the 1.9B KD model achieves ~38.2% vs. ~37.7% for LM (small gap). At 419B tokens, the 1.9B KD model achieves ~45.5% vs. ~44.2% for LM (larger gap).
The gap converges slightly toward the end but remains significant: at the final checkpoint (59,604 steps, ~500B tokens), the 1.9B KD model achieves ~45.4% vs. ~44.2% for LM. The 3.8B KD model achieves ~53.7% vs. ~50.2% for LM.
The 3.8B student benefits more from PD than the 1.9B student throughout training, consistent with the model size scaling findings.

The paper interprets:

"The gains from pre-training distillation increase initially during pre-training and then converge with a slight decrease, and are still significant are the end of pre-training. This suggests that pre-training distillation not only enhances training efficiency but also improves the performance upper bound of student LLMs."

The "upper bound" claim is significant: it means PD does not merely accelerate convergence to the same ceiling — it raises the ceiling. The student trained with PD achieves a higher final performance than the LM-only student even with unlimited training. This has direct implications for large-scale pre-training: even if you plan to train on trillions of tokens, incorporating PD during pre-training may yield a strictly better final model.

The paper acknowledges the computational limitation:

"Due to computational limitations, we do not reach trillion-level tokens for pre-training which are used by most advanced LLMs. We believe that pre-training distillation is also effective using several trillion tokens and encourage future LLM development to incorporate pre-training distillation."

This is a reasoned extrapolation: the gap at 500B tokens is still substantial (especially for the 3.8B model, where it is 3.5 percentage points), and there is no sign of the gap disappearing at scale. However, the slight convergence at the end of training suggests the gap might narrow further at multi-trillion-token scales, which remains an open empirical question.

Offline vs. Online Logits (Design Dimension #4): Sourcing Teacher Signal

The final design dimension addresses how teacher logits are obtained. The paper considers two paradigms:

Offline: Logits are generated from a fully pre-trained, converged teacher model. This is what all previous experiments use. It requires a separate forward pass of the teacher over the entire pre-training corpus, which is computationally expensive — effectively, the cost of running inference with a 9B or 32B model on 100B+ tokens.
Online: Logits are generated simultaneously during the teacher model's own pre-training, stored on disk, and then later used to train the student. This eliminates the separate inference pass but uses logits from a teacher that may be far from convergence, potentially containing substantial noise.

Experimental setup. Due to the high cost of pre-training GLM-4-9B from scratch, the paper pre-trains a new 9B LLM from scratch using 400 billion tokens (4× the baseline corpus) while simultaneously storing the logits for each token. The teacher training uses a batch size of 1,728, sequence length of 4,096, a cosine learning rate scheduler with maximum $6 \times 10^{-4}$ and minimum $6 \times 10^{-5}$ , and 1% warmup. Storing the logits during teacher pre-training consumes about 180 TB of disk space — still substantial but eliminating the separate inference cost.

Three student configurations are tested:

LLM-Online-100B-L: Distilled using the first 100 billion tokens of the teacher's pre-training — the logits from the very beginning when the teacher is essentially random.
LLM-Online-100B: Distilled using the last 100 billion tokens of the teacher's pre-training — the logits from the most converged portion of the teacher.
LLM-Online-100B:* Same as LLM-Online-100B but with modified configuration: $\alpha = 0.1$ (mostly LM loss with light KD) and top-0.95-50 truncation (more aggressive than the baseline top-0.95-100), to test whether different processing helps when the teacher signal is noisier.

All students use the preliminary experiment's configuration otherwise (1.9B parameters, NLL loss, $\tau = 1.0$ ), with the noted modifications for LLM-Online-100B*.

Results (Table 6). The findings reveal a sharp quality gradient:

LLM-Online-100B-L: 29.8% average (-20.9% relative to LLM-LM). Catastrophic degradation. The teacher at the beginning of pre-training produces essentially random logits, and the student learning from this noise is actively harmed.
LLM-Online-100B: 36.3% average (-3.9% relative to LLM-LM). Still worse than LM-only pre-training but much closer. The teacher's later-stage logits are informative enough to avoid catastrophic degradation but not good enough to beat the baseline.
LLM-Online-100B:* 37.9% average (+0.5% relative to LLM-LM). With lighter KD weighting ( $\alpha = 0.1$ ) and more aggressive truncation, online logits from the later stages of teacher pre-training achieve a small but positive improvement over LM-only pre-training.

The paper draws several conclusions from these results:

"The reason may be that the teacher LLM is far from convergence, and hence the logits contain substantial noise."

The teacher at 400B tokens is still training — it has not reached the quality of the fully pre-trained GLM-4-9B used in offline experiments. The quality gap between offline and online logits explains why offline PD achieves 38.3% while online PD achieves at best 37.9%.

"even logits generated by a non-converged teacher LLM can help pre-training student LLM, suggesting that using online logits is also effective and better practice is to utilize the logits from the later stages of the teacher LLM's pre-training."

The LLM-Online-100B* result (positive improvement, albeit small) shows that online logits are viable when configured appropriately — specifically, when the distillation signal is deemphasized ( $\alpha = 0.1$ ) to account for its lower quality, and when truncation is more aggressive (top-50 vs. top-100) to reduce noise.

Practical deployment strategy. The paper recommends a scenario-dependent approach:

"if one aims to pre-train only an LLM, using offline logits of a pre-trained teacher LLM is better; if one aims to pre-train a series of LLMs of varying sizes, one can first pre-train the largest LLM while storing online logits, and then pre-train smaller LLMs with online logits."

The logic: for training a single model, the inference cost of generating offline logits from a pre-trained teacher is a one-time expense that yields higher-quality student models. For training a family of models (e.g., 1.9B, 3.8B, 6.8B), the offline approach would require running the teacher on the corpus for each student separately, multiplying the inference cost. The online approach generates logits once during the largest model's pre-training and then amortizes that cost across all smaller students — a more economical pipeline for model families.

The paper does not explore the obvious hybrid: use the final checkpoint of the online teacher (which is a fully pre-trained model) as an offline teacher for subsequent students. This would combine the highest-quality logits with the amortized cost structure, and represents a natural direction for future work.

The Better PD Configuration (PD∗)

Based on the design space exploration, the paper synthesizes findings into a single recommended configuration (Section 3.1, Appendix A.6):

Logits processing: top-0.95-50 truncation (slightly more aggressive than baseline, justified by the finding that smaller k works well), temperature $\tau = 2.0$ (in the effective range from Table 2).
Loss function: KLD as the distillation loss (slightly better than NLL in aggregate from Table 5).
Loss combination: WSD-α with WSD-LR, where α follows warmup to 0.9 (not 1.0), stable at 0.9, decay to 0. The maximum α of 0.9 rather than 1.0 incorporates the finding that ~10% LM loss improves results. The WSD warmup ratio is 10% and decay ratio is 1%.
Teacher: GLM-4-9B (the 32B teacher provides no benefit and may be worse due to capacity gap).
Logits source: Offline (pre-computed from the pre-trained teacher).

Results for this configuration on 1.9B, 3.8B, and 6.8B students are in Table 13 and Figure 1:

1.9B: 41.2% average — a significant jump from the LLM-LM baseline (37.7%) and the preliminary PD configuration (38.3%).
3.8B: 45.7% average — up from 42.0% (LM) and 44.9% (vanilla PD).
6.8B: 49.8% average — up from 44.9% (LM) and 48.0% (vanilla PD).

The improvements grow with student size (2.3 percentage points for 3.8B, 4.9 points for 6.8B), consistent with the scaling law findings that larger students benefit more from well-configured PD.

Summary of Design Choices and Their Justifications

Top-p-k truncation with p = 0.95, k = 50 over full logits: Reduces storage by ~4,000× (from unaffordable 58.6 PB to manageable ~15 TB for 100B tokens) without significantly degrading distillation quality. Even more aggressive truncation (smaller p, smaller k) can be used for additional storage savings.
Static temperature τ = 2.0 over adaptive temperature: Adaptive methods add complexity without clear benefit. Static temperatures in the range 0.1–2.0 perform similarly, and τ = 2.0 provides a slight edge in the better PD configuration.
KLD over NLL or MSE: KLD and NLL perform similarly (both in the cross-entropy family), but KLD shows marginally better aggregate performance. MSE fails catastrophically because operating on raw logits in a 150K-dimensional space without probability normalization creates poorly calibrated training signals.
WSD-α+WSD-LR over static α or simpler schedules: Placing high KD proportion during the high-learning-rate stable phase maximizes the teacher signal's impact during the model's most plastic period. Including a small amount of LM loss (α = 0.9 rather than 1.0) prevents overfitting to teacher imperfections.
Teacher smaller than 32B (9B sufficient): Larger teachers suffer from capacity gap — their more fine-grained representations are not learnable by smaller students. The student-to-teacher ratio should be at least ~10% for PD to be effective.
Offline logits over online logits when training a single model: Pre-computed logits from a fully converged teacher are higher quality, and the one-time inference cost is justified by better student performance. Online logits are viable for model families where amortizing logit generation across multiple students is economical.
SFT after PD for meaningful evaluation: At small model scales (≤1.9B), zero-shot and few-shot performance on challenging benchmarks is near random. SFT provides a standardized probe that isolates pre-training quality differences from evaluation noise.

4. Key Insights and Innovations

Innovation 1: Pre-Training Distillation as a Design Space, Not a Binary Choice

The dominant framing in prior work — both in the pre-ChatGPT distillation literature (Sanh et al., 2019; Jiao et al., 2020) and in recent LLM releases that mention distillation (Gemma 2, AFM, LokiLM, Minitron) — treats pre-training distillation as a single technique: you either do it or you don't. The implicit assumption is that if distillation helps, it helps in roughly the same way regardless of configuration, and the practitioner's job is simply to decide whether to turn it on.

This paper makes a fundamentally different conceptual move: it reframes pre-training distillation as a multi-dimensional design space where the configuration choices — how you truncate teacher logits, which loss function you use, how you schedule the mixing between distillation and standard LM loss, which teacher you distill from, and whether you use offline or online logits — are not minor implementation details but first-order determinants of effectiveness. The evidence for this is not a single table but the cumulative structure of the paper: every section reveals that different choices along each dimension produce meaningfully different outcomes, and the gap between a naive configuration (the preliminary experiment's +1.6%) and a well-tuned one (WSD-α+WSD-LR's +8.0%) spans nearly an order of magnitude in relative improvement.

This reframing is significant because it changes what it means to "do pre-training distillation." In the binary view, the relevant question is "should we distill?" In the design-space view, the relevant questions become "how should we configure distillation for our specific student-teacher size ratio, corpus budget, and storage constraints?" The paper does not answer all of these questions — it explicitly leaves the trillion-token regime and the interaction effects between dimensions unexplored — but it establishes the intellectual framework in which those questions become meaningful.

The paper's own language supports this reading. The title and abstract emphasize "design space exploration," not "method proposal." The conclusion states "we hope our exploration will inform future practices in pre-training distillation" — a framing that positions the contribution as mapping the terrain rather than staking a single claim.

Innovation 2: The WSD Schedule as a General Principle for Distillation Signal Allocation

The paper's strongest single empirical result — the WSD-α+WSD-LR configuration achieving +8.0% relative improvement (Table 5) — is not merely a better hyperparameter setting. It encodes a conceptual principle about when distillation signal is most valuable during training: the teacher's distribution provides the richest learning signal during the phase when the learning rate is highest and the model is making its largest parameter updates.

This principle is non-obvious because it contradicts two intuitive alternatives that the paper explicitly tests and rejects. The first alternative (Linear Inc, +1.1%) assumes that early training should use pure LM loss to build basic language competence before introducing the more complex teacher signal — essentially, "crawl before you walk." The second alternative (static α, best at +3.6%) assumes a constant mixture throughout training — "the teacher signal is equally valuable at all stages." The WSD-α+WSD-LR result shows both are suboptimal: the teacher signal is most valuable precisely when the model is learning fastest, and the small amount of LM loss (α decaying from 0.9 to 0 in the final 1% of training) serves as a regularizer that prevents overfitting to teacher imperfections.

The paper strengthens this argument through its ablation: WSD-β+WSD-LR (+1.1%) schedules the LM loss proportion to be high during the stable phase — the opposite of the optimal configuration — and achieves essentially no improvement. This confirms that the benefit is not from the WSD learning rate schedule alone (0-α+WSD-LR achieves only +1.2%) but specifically from the interaction between high KD proportion and high learning rate. The paper does not develop this into a full theoretical framework (it remains an empirical finding), but the principle — "align the richest training signal with the period of maximum plasticity" — is a conceptual contribution that generalizes beyond the specific α scheduling curves tested here.

Where this sits on the incremental-to-fundamental spectrum: it is a mid-level innovation. The WSD learning rate scheduler itself was proposed by Hu et al. (2024), and the idea of scheduling the loss mixture is not entirely novel (curriculum learning has explored related ideas; Soviany et al., 2022). But the specific finding that KD proportion should be maximized during the stable high-learning-rate phase — and the demonstration that getting this wrong (WSD-β+WSD-LR) eliminates essentially all benefit — is a non-trivial empirical discovery with direct practical implications for anyone implementing PD.

Innovation 3: The Capacity Gap as the Central Scaling Dynamic in Pre-Training Distillation

The paper's scaling law experiments (Figure 4) establish a finding that is both empirically clear and theoretically significant: larger teachers do not guarantee better students. The 32B teacher underperforms the 9B teacher for 1.9B and 3.8B students, and even at 6.8B (21% of the 32B teacher's size), the 9B teacher remains better. This is not a marginal difference — for the 3.8B student, the 9B teacher achieves 44.9% vs. 42.9% for the 32B teacher, a gap of 2 percentage points.

This finding is conceptually important because it challenges the default intuition that "more teacher knowledge is always better." The capacity gap concept (Mirzadeh et al., 2020), which the paper invokes, explains why: a teacher's internal representations encode distinctions and nuances that require a certain minimum model capacity to express. When the student falls below this threshold, trying to match the teacher's distribution becomes counterproductive — the student cannot represent what the teacher knows, and the training signal becomes noisy rather than informative.

What makes this a genuine innovation rather than a simple replication of the capacity gap concept is the paper's operationalization of it: the identification of a rough 10% threshold (student parameters / teacher parameters) below which PD provides minimal benefit, and the observation that this threshold may shift upward for larger teachers. The 330M student (3.7% of 9B teacher) actually performs slightly worse with PD than with LM-only pre-training (32.1% vs. 32.6%). The 670M student (7.4%) shows marginal improvement. Clear benefits emerge only at 1.9B (21%). For the 32B teacher, even 6.8B (21%) shows only modest gains, suggesting the effective threshold may be higher for larger teachers.

This has direct implications for how organizations should think about pre-training distillation investments: if you have a 32B teacher and want to distill a 1.9B student (6% ratio), the paper's results suggest you would be better off using a 9B teacher instead, or not distilling at all. The teacher-selection problem becomes an optimization in its own right — you want the smallest teacher that is still large enough to provide useful signal for your target student size, because a teacher that is too large creates a capacity gap that degrades rather than enhances learning.

The paper explicitly connects this to a compression perspective: "larger LLMs compress information more effectively and achieve better compression rates, potentially making it harder for smaller LLMs to learn." This reframes the capacity gap not as a failure of the distillation algorithm but as a fundamental information-theoretic constraint: a more compressed representation (from a larger model) requires more capacity to decompress and learn from. This is a speculative but intriguing theoretical framing that the paper does not fully develop but that points toward a deeper understanding of why distillation works (and fails) at scale.

Innovation 4: Online Logits as a Viable (If Weaker) Alternative to Offline Pre-Computation

The comparison between offline and online logits (Section 3.5, Table 6) establishes a finding with significant practical implications: logits from a partially-trained, still-converging teacher can provide useful distillation signal, provided the configuration is adjusted appropriately. The LLM-Online-100B* result (+0.5% over LM-only baseline) demonstrates that even a teacher that has not completed pre-training can improve student quality when the distillation signal is properly calibrated (α = 0.1, more aggressive truncation).

This finding addresses a real economic constraint. The offline paradigm used throughout most of the paper requires a fully pre-trained teacher and a separate inference pass over the entire corpus — for a 9B teacher on 100B tokens, this is a non-trivial computational expense. The online paradigm eliminates this cost: logits are captured as a byproduct of the teacher's own pre-training, costing only additional storage (180 TB for 400B tokens in the paper's experiment) rather than additional computation.

The innovation here is not the technical mechanism (online logit capture is conceptually straightforward) but the characterization of the cost-quality tradeoff. The paper maps out the quality gradient: logits from the very beginning of teacher training are harmful (-20.9%), logits from later stages are slightly sub-baseline (-3.9%), and logits from later stages with adjusted configuration become slightly above baseline (+0.5%). This gradient is actionable: if you are pre-training a family of models, you should capture logits from the later stages of the largest model's training (not the beginning) and use a lower α value (0.1 rather than 0.9–1.0) to account for the teacher's incomplete convergence.

The paper's deployment recommendation — offline for single models, online for model families — is a practical heuristic derived from this characterization. It recognizes that the offline approach yields strictly better student models (as expected, since the teacher is fully converged) but that the online approach's elimination of a separate inference pass makes it economically preferable when the logit generation cost can be amortized across multiple students. This is an engineering contribution rather than a scientific one, but it addresses a real bottleneck (the 58.6 PB storage problem and the inference cost problem) that stands between the paper's experimental results and practical deployment.

The finding also opens a direction that the paper does not explore: using the final checkpoint of the online teacher (which, after full pre-training, is essentially an offline teacher) as the logit source for subsequent students. This hybrid approach would combine the highest-quality logits with the amortized cost structure of the online paradigm, and represents the natural endpoint of the cost-quality tradeoff characterization that this innovation provides.

5. Experimental Analysis

Evaluation Methodology

Dataset. All experiments use a randomly sampled subset of 100 billion tokens from a pre-training corpus (Section 3.1). For the scaling law experiments on corpus size (Section 3.4), this is extended to 500 billion tokens. For the online logits experiment (Section 3.5), the teacher is pre-trained on 400 billion tokens. The paper does not specify the exact source or composition of the pre-training corpus beyond describing it as a general text corpus used for autoregressive language model pre-training. For downstream evaluation, eight benchmark datasets are used: HellaSwag (Zellers et al., 2019; zero-shot), WinoGrande (Sakaguchi et al., 2020; zero-shot), PIQA (Bisk et al., 2020; zero-shot), MMLU (Hendrycks et al., 2021; 6-shot), KBQA (Duan, 2016; Duan and Tang, 2018; zero-shot), C3 (Sun et al., 2020a; 5-shot), C-Eval (Huang et al., 2024; 5-shot), and GSM8k (Cobbe et al., 2021; 8-shot). All evaluations use sampling temperature 0 for deterministic greedy decoding (Appendix A.1).
Base model(s). The primary teacher model is GLM-4-9B (GLM et al., 2024), a 9-billion-parameter autoregressive LLM with a vocabulary of approximately 150,000 tokens. For scaling law experiments (Section 3.4), GLM-4-32B (a 32-billion-parameter model) serves as an additional teacher. Student LLMs range from 330M to 6.8B parameters, with architectures specified in Table 7: the 330M model uses 1,024 hidden size, 4,096 FFN hidden size, 12 layers, 16 attention heads, 2 query groups (grouped-query attention; Ainslie et al., 2023), and tied embeddings; the 670M model uses the same hidden dimensions but 24 layers and no weight tying; the 1.9B model uses 2,048 hidden size, 6,912 FFN, 24 layers, 16 heads, 2 query groups; the 3.8B model uses 3,072 hidden size, 8,192 FFN, 28 layers, 24 heads, 8 query groups; the 6.8B model uses 4,096 hidden size, 12,800 FFN, 28 layers, 32 heads, 8 query groups. All models are trained in BFLOAT16 format (Kalamkar et al., 2019). The paper argues the 9B teacher is "representative of the capabilities of many contemporary LLMs" (implied throughout, though not stated for the teacher specifically), and the student sizes were chosen to span the regime where PD might be expected to show scaling trends.
Metrics. The primary metric throughout is average accuracy (%) across the eight downstream evaluation benchmarks described above, computed after supervised fine-tuning (SFT) on 10B instruction-tuning tokens plus 10B pre-training corpus tokens. Accuracy is computed per-benchmark using standard evaluation protocols (exact match or equivalent for each dataset), and then averaged across all eight benchmarks to produce the single "Average" column reported in every results table. The paper reports this as an unweighted mean — each benchmark contributes equally regardless of its absolute score scale. For experiments reporting Δ values (relative improvement), this is computed as (Accuracy_PD - Accuracy_LM) / Accuracy_LM × 100% where Accuracy_LM is the baseline LLM-LM performance on the same average metric. Individual dataset scores are reported in the appendix tables (Tables 8–13) but all headline claims use the average.
Baselines. The central baseline is LLM-LM: the same student model architecture trained from scratch on the identical pre-training corpus using only standard language modeling loss (α = 0 in Equation 1), with all other training hyperparameters (learning rate schedule, batch size, sequence length, optimizer, SFT protocol) held identical to the PD experiments. This baseline is re-trained for each experimental condition (different student sizes, different corpus sizes) to ensure fair comparison — for example, Table 11 reports separate LM baselines for each student size at each teacher size. For the online logits experiment (Section 3.5), additional baselines include LLM-Online-100B-L (distilled from the first 100B tokens of teacher pre-training) and LLM-Online-100B (distilled from the last 100B tokens) against the same LLM-LM baseline. The paper also includes a 0-α+WSD-LR baseline in Table 5: LM-only training but using the WSD learning rate scheduler rather than cosine, to isolate whether WSD-LR benefits come from the schedule itself or from its interaction with distillation.
Generation budget / compute accounting. The paper does not use "generations" as a compute unit (unlike inference-time scaling work). Instead, all experiments are token-matched: the student LLM processes exactly the same number of pre-training tokens regardless of whether distillation is used. The main experiments use 100 billion tokens; scaling law experiments extend to 500 billion tokens; online logits experiments use 400 billion tokens for teacher pre-training. The paper reports disk storage requirements for logits as the relevant resource constraint: storing full logits for 100B tokens with a 150K vocabulary in float32 requires approximately 58.6 PB, which the top-p-k truncation reduces to approximately 15 TB (a ~4,000× reduction). For the online logits experiment, storing logits during teacher pre-training on 400B tokens consumes approximately 180 TB. Training is conducted on Nvidia H800 GPUs (Appendix A), but GPU-hours are not reported. The paper does not account for the computational cost of the teacher's forward pass to generate offline logits in its efficiency claims — this is a one-time pre-computation cost that is amortized if the logits are reused for multiple students.
Cross-validation / statistical protocol. The paper does not employ cross-validation or report confidence intervals. Each experimental configuration is trained once (a single pre-training run from random initialization followed by a single SFT run), and the reported numbers are point estimates from that single run. This is standard practice in LLM pre-training research due to computational constraints (training a 1.9B model on 100B tokens is expensive enough that multiple seeds would be prohibitive), but it means the paper cannot distinguish genuine configuration effects from run-to-run variance. The paper also does not report statistical significance tests for any comparisons. The two-fold cross-validation used in the inference-time compute paper (the reference example) is notably absent here — the paper simply reports the raw accuracy of each configuration and interprets differences qualitatively. For the 500B-token scaling law experiments, checkpoints are saved every 10,000 steps (~83B tokens), and each checkpoint is independently fine-tuned with SFT before evaluation, providing a longitudinal view of training dynamics, but still from a single training run.

Main Quantitative Results

Preliminary Validation: Vanilla PD Provides a Modest but Real Improvement

Table 1 establishes the feasibility baseline. The LLM-KD model (vanilla PD: NLL loss, α = 1, top-0.95-100 truncation, τ = 1.0, 9B teacher, 1.9B student, 100B tokens) achieves an average accuracy of 38.3% across the eight benchmarks, compared to 37.7% for LLM-LM — a +1.6% relative improvement. The gains are uneven across benchmarks: GSM8k shows the largest relative improvement (+24.6%, from 8.6% to 10.8%), while several benchmarks show small degradations (PIQA: -0.5%, MMLU: -0.5%, KBQA: -1.3%). The paper notes this improvement is "marginal" and uses it to motivate the design space exploration.

Logits Processing: Truncation Is Robust, Temperature Has an Upper Bound

Top-p-k truncation (Figures 2–3, Tables 8–9). Sweeping p from 0.5 to 0.95 with k = 100 (Figure 2) shows that all values produce similar relative improvements (clustered around 1–5%, no clear monotonic trend), while the number of stored logits per token drops substantially with smaller p. The best configuration is p = 0.5 (38.5% average, Table 8) but differences are within ~0.4 percentage points across all p values. Sweeping k from 1 to 100 with p = 0.95 (Figure 3) shows that k = 50 achieves the best performance (39.6% average), with k = 1 (equivalent to using teacher-generated labels with LM loss) still achieving 38.4%. The paper's key operational finding: "different p and k values in top-p-k truncation shows improvements with limited differences, and one can adopt smaller p and k in logits truncation to save storage disk space."

Static temperature (Table 2, Table 9). For τ values in {0.05, 0.1, 0.2, 0.5, 1.0, 2.0, 5.0, 10.0}, temperatures τ ≤ 2.0 produce similar relative improvements (clustered around +1.6% to +2.7%), with the best at τ = 0.2 and τ = 0.5 (both 38.7% average). At τ = 5.0, improvement drops to -0.1% (37.7% average, essentially identical to LLM-LM). At τ = 10.0, improvement returns to +1.0% (38.1%). The paper interprets the τ = 5.0 result as evidence that overly uniform teacher distributions are inefficient for student learning, though the non-monotonic pattern (improvement returns at τ = 10.0) complicates this interpretation and is not discussed.

Adaptive temperature (Table 3). NormKD achieves 37.2% (-1.3%, worse than LLM-LM). WTTM achieves 37.8% (+0.2%, essentially no improvement). AdaKDSD achieves 38.2% (+1.2%). AdaKDH achieves 38.8% (+2.8%), the best among adaptive methods. However, the best static temperature (τ = 0.5, 38.7%) matches or exceeds AdaKDH, so the paper concludes "adaptive temperature does not show significant additional improvement."

Loss Selection: KLD and NLL Are Similar; MSE Fails; WSD Scheduling Matters Critically

Distillation loss function (Table 5, top rows). With α = 1 (pure distillation):

LLM-NLL: 38.3% (+1.6%), the preliminary experiment baseline.
LLM-KLD: 38.7% (+2.6%), marginally better than NLL in aggregate but worse on MMLU (26.7% vs. 27.8%) and C-Eval (25.4% vs. 26.7%).
LLM-MSE: 34.9% (-7.6%), a catastrophic degradation. Every benchmark degrades except WinoGrande (55.0% vs. 54.8% for LM baseline). GSM8k collapses to 3.9% (vs. 8.6% for LM baseline).

Static α (Table 4, Table 10). Sweeping α in {0.0, 0.1, 0.5, 0.6, 0.7, 0.8, 0.9, 0.95, 1.0} with NLL loss reveals an inverted-U pattern: improvement grows from +0.1% at α = 0.1 to a peak of +3.6% at α = 0.9 (39.1% average), then declines to +2.5% at α = 0.95 and +1.6% at α = 1.0. The optimal α = 0.9 outperforms pure distillation (α = 1.0) by 0.8 percentage points, demonstrating that a small proportion (~10%) of standard LM loss is beneficial.

Dynamic α scheduling (Table 5, bottom rows).

Linear Inc (α: 0 → 1): 38.1% (+1.1%) — worse than static α = 0.9.
Linear Dec (α: 1 → 0): 39.2% (+4.1%) — better than any static α configuration, showing that starting with KD and transitioning to LM is effective.
Periodic: 38.0% (+0.9%) — no advantage over static α.
WSD-α+Cos-LR: 38.6% (+2.3%) — better than static α but worse than Linear Dec.
WSD-α+WSD-LR: 40.7% (+8.0%) — the single best result in the paper. This is 2.4 percentage points above the LM baseline (37.7%) and 2.4 points above the preliminary PD configuration (38.3%). On individual benchmarks, it achieves the highest scores on HellaSwag (56.4%), WinoGrande (57.7%), MMLU (31.8%), C-Eval (33.8%), and GSM8k (12.5%).
WSD-β+WSD-LR: 38.1% (+1.1%) — essentially no improvement, confirming that the benefit comes from scheduling KD proportion (α) with WSD, not from WSD-LR alone or from scheduling LM proportion (β).

The comparison between 1-α+WSD-LR (pure KD with WSD-LR, 39.6% or +5.0%) and WSD-α+WSD-LR (WSD-scheduled α with WSD-LR, 40.7% or +8.0%) isolates the effect of the α schedule: adding the WSD α schedule on top of WSD-LR provides an additional +3.0 percentage points relative improvement beyond what WSD-LR alone provides with pure KD. The comparison between 0-α+WSD-LR (LM-only with WSD-LR, 38.2% or +1.2%) and WSD-α+WSD-LR shows the full effect of distillation with optimal scheduling: a +6.8 percentage point relative gain attributable to distillation, on top of the +1.2% gain from WSD-LR alone.

Scaling Law: Larger Students Benefit More; Larger Teachers Do Not

Model size scaling (Figure 4, Table 11). Using the 9B teacher, relative improvements over LM baselines are: 330M: -1.5% (32.1% KD vs. 32.6% LM — PD actually hurts), 670M: +0.6% (33.8% vs. 33.6%), 1.9B: +1.6% (38.3% vs. 37.7%), 3.8B: +6.9% (44.9% vs. 42.0%), 6.8B: +6.9% (48.0% vs. 44.9%). The improvement grows monotonically with student size (except the jump from 3.8B to 6.8B where the relative improvement is identical, though absolute gains are larger: +3.1 points for 6.8B vs. +2.9 points for 3.8B).

Using the 32B teacher, relative improvements are: 330M: -3.1% (31.6% vs. 32.6%), 670M: +1.8% (34.2% vs. 33.6%), 1.9B: +0.5% (37.9% vs. 37.7%), 3.8B: +2.1% (42.9% vs. 42.0%), 6.8B: +5.6% (47.4% vs. 44.9%). The 32B teacher underperforms the 9B teacher for every student size except 670M (34.2% vs. 33.8%, a marginal difference). For the 3.8B student, the 9B teacher achieves 44.9% vs. 42.9% for the 32B teacher — a gap of 2.0 percentage points.

Corpus size scaling (Figure 5, Table 12). Training 1.9B and 3.8B students on 500B tokens with the 9B teacher:

1.9B, final checkpoint (59,604 steps): KD achieves 45.4% vs. 44.2% for LM — a +2.7% relative improvement, sustained from the earliest checkpoint (38.2% vs. 37.7% at 10k steps) through the final one. The gap widens during training (0.5 points at 10k steps, 2.4 points at 50k steps) and narrows slightly at the end (1.2 points at the final checkpoint).
3.8B, final checkpoint: KD achieves 53.7% vs. 50.2% for LM — a +7.0% relative improvement. The gap is larger than for the 1.9B model at every checkpoint, consistent with the model size scaling findings. At 40k steps, the gap peaks at 3.5 percentage points (52.4% vs. 48.9%) before settling to 3.5 points at the final checkpoint.

Offline vs. Online Logits: Online Works but Is Weaker

Online logits (Table 6). All using 1.9B students, NLL loss, and the preliminary experiment's configuration unless noted:

LLM-Online-100B-L (first 100B tokens of teacher pre-training): 29.8% average, -20.9% relative to LLM-LM. Individual benchmarks collapse: HellaSwag at 30.1% (vs. 53.3% for LM), GSM8k at 2.4% (vs. 8.6%).
LLM-Online-100B (last 100B tokens): 36.3%, -3.9% relative. Still below LLM-LM but dramatically better than the early-logits variant. GSM8k recovers to 8.0% (close to LM baseline 8.6%).
LLM-Online-100B* (last 100B tokens, α = 0.1, top-0.95-50): 37.9%, +0.5% relative. Achieving a small positive improvement over LM-only pre-training, with notable gains on C3 (57.0% vs. 54.7% for LM) and GSM8k (10.0% vs. 8.6%).

The gradient from catastrophic (-20.9%) to slightly positive (+0.5%) demonstrates that online logit quality depends critically on teacher training stage and on adjusting α and truncation to account for incomplete teacher convergence.

The Better PD Configuration (PD∗)

PD∗ results (Figure 1, Table 13). Synthesizing findings: top-0.95-50 truncation, τ = 2.0, KLD loss, WSD-α+WSD-LR with max α = 0.9, 9B teacher, offline logits:

1.9B: 41.2% average — a gain of 3.5 percentage points over LLM-LM (37.7%) and 2.9 points over the preliminary PD configuration (38.3%).
3.8B: 45.7% — a gain of 3.7 points over LLM-LM (42.0%) and 0.8 points over the vanilla PD with 9B teacher (44.9%).
6.8B: 49.8% — a gain of 4.9 points over LLM-LM (44.9%) and 1.8 points over vanilla PD (48.0%).

The improvements grow with model size in absolute terms (3.5 → 3.7 → 4.9 points), consistent with the scaling law findings. The 1.9B student under PD∗ (41.2%) nearly matches the 3.8B student under LM-only pre-training (42.0%), demonstrating that well-configured distillation can effectively close the gap to a model roughly 2× larger trained with standard pre-training — though the paper does not make this comparison explicitly.

Ablation Studies and Robustness Checks

Top-p sweep with fixed k = 100 (Table 8): Varying p from 0.5 to 0.95 produces average accuracies between 38.2% (p = 0.85 and p = 0.9) and 38.5% (p = 0.5 and p = 0.7). The range is only 0.3 percentage points, confirming that the exact p value is not critical as long as it is ≥0.5. The storage savings from smaller p are substantial (Figure 2: top-0.5-100 retains far fewer logits per token than top-0.95-100), making this a "free" efficiency gain.
Top-k sweep with fixed p = 0.95 (Table 8): Varying k from 1 to 100 produces averages from 38.1% (k = 3) to 39.6% (k = 50). The performance difference between k = 1 (38.4%) and the best k = 50 (39.6%) is 1.2 percentage points — modest but consistent. The k = 1 result (teacher's top-1 token used as label) demonstrates that even minimal teacher signal helps.
Temperature sweep (Table 9): Low temperatures (0.05–2.0) cluster in 38.3–38.7% range. The drop at τ = 5.0 (37.7%) and recovery at τ = 10.0 (38.1%) is non-monotonic and unexplained. The paper does not discuss possible causes (e.g., interaction between temperature and the truncated logits' distribution shape, or sensitivity to the specific one-shot training run).
Adaptive temperature comparisons (Table 3): NormKD underperforms LM baseline (-1.3%) despite being designed for KD. The paper does not ablate NormKD hyperparameters, so it is unclear whether the negative result reflects the method itself or suboptimal hyperparameter choices for the PD setting. AdaKDH outperforms AdaKDSD (38.8% vs. 38.2%), showing entropy-based sharpness measurement is preferable to standard-deviation-based, but both underperform or match the best static temperature.
Distillation loss function comparison (Table 5): The MSE failure is dramatic (-7.6%) and consistent across all benchmarks. The paper does not attempt any variants of MSE (e.g., MSE on probabilities rather than logits, or MSE with different normalization), so it is unclear whether the failure is fundamental to MSE loss or specific to this implementation.
Static α sweep (Table 10): The inverted-U shape (peak at α = 0.9) with a clear drop at α = 1.0 confirms that pure distillation is suboptimal and that a small LM loss component is beneficial. The degradation at α = 0.5 (+1.5%) vs. α = 0.9 (+3.6%) shows that too little KD signal also underperforms — the optimal mixture is strongly KD-weighted but not exclusively KD.
α schedule comparisons (Table 5): Linear Dec (39.2%) vs. Linear Inc (38.1%) shows that the order of loss mixture matters: KD-first-then-LM outperforms LM-first-then-KD by 1.1 percentage points. The WSD-β+WSD-LR ablation (38.1%) confirms that WSD-LR alone does not explain the WSD-α+WSD-LR gains — the specific interaction of KD proportion with high learning rate is necessary.
Model size scaling (Table 11): For the 9B teacher, the student size sweep from 330M to 6.8B (a 20.6× range) shows monotonic growth in PD benefit, from negative (-1.5%) to strongly positive (+6.9%). For the 32B teacher, the 1.9B student achieves only +0.5% while the 3.8B student achieves +2.1%, and the 6.8B achieves +5.6% — but all three underperform their 9B-teacher counterparts. The 330M and 670M students show essentially no benefit or slight harm from any teacher, establishing a practical lower bound on student size for PD effectiveness.
Corpus size checkpoints (Table 12): For the 1.9B student on 500B tokens, the LM baseline improves from 37.7% (10k steps) to 44.2% (final) — a gain of 6.5 points from additional pre-training. The KD model improves from 38.2% (10k steps) to 45.4% (final) — a gain of 7.2 points. The KD advantage is present at every checkpoint but narrows from +0.5 points (10k) to +1.2 points (final), suggesting diminishing returns. For the 3.8B student, the LM baseline improves from 42.6% to 50.2% (+7.6 points), while KD improves from 43.9% to 53.7% (+9.8 points), with the KD advantage growing from +1.3 to +3.5 points. The 3.8B model shows no sign of diminishing PD returns at 500B tokens.
Online logits configurations (Table 6): The α = 0.1 adjustment for LLM-Online-100B* substantially improves over the α = 1.0 LLM-Online-100B (37.9% vs. 36.3%), confirming that when teacher signal is noisier (online), a lower KD proportion preserves more of the student's ability to learn from ground truth. The top-0.95-50 truncation (vs. top-0.95-100 in the baseline) also contributes, though the paper does not ablate truncation and α separately for online logits.

Critical Assessment

Claim 1: Pre-training distillation is feasible and effective for billion-parameter LLMs. The preliminary experiment (Table 1) establishes feasibility: the vanilla PD configuration achieves +1.6% average improvement over LM-only pre-training. However, this improvement is uneven — several benchmarks show slight degradation — and the absolute magnitude is modest. The "better PD configuration" (Table 13) substantially strengthens this claim: PD∗ achieves +9.3% relative improvement for 1.9B (41.2% vs. 37.7%), +8.8% for 3.8B, and +10.9% for 6.8B. These are meaningful gains that would justify the engineering investment in PD for organizations pre-training models at this scale. The claim is supported with qualifications: it holds for student sizes at 1.9B and above, using a 9B teacher, with careful configuration choices (KLD, WSD-α+WSD-LR, proper truncation). It does not hold for very small students (330M, where PD slightly hurts) or when the teacher is substantially larger than the student (32B teacher, 1.9B student yields only +0.5%). The paper tests only one model family (GLM) and one teacher architecture — whether these results transfer to other model families (LLaMA, Qwen, etc.) is unverified.

Claim 2: The WSD-α+WSD-LR configuration achieves +8.0% relative improvement and represents a substantially better configuration than naive PD. This claim is the paper's strongest single result and is well-supported by Table 5. The ablation structure (WSD-β+WSD-LR, 1-α+WSD-LR, 0-α+WSD-LR) cleanly isolates the effect. However, the claim comes with an important caveat: this result is from a single training run. The paper does not report multiple seeds, and the difference between WSD-α+WSD-LR (40.7%) and Linear Dec (39.2%) is 1.5 percentage points — potentially within the range of run-to-run variance for a 1.9B model trained on 100B tokens. The paper provides no way to assess whether this difference is statistically reliable. Additionally, the WSD-α+WSD-LR result is reported only for the 1.9B student — the paper does not verify that the same configuration is optimal for the 3.8B or 6.8B students (the PD∗ configuration in Table 13 uses WSD-α+WSD-LR for those sizes as well, but does not ablate whether a different schedule would be better for larger students).

Claim 3: Larger student LLMs generally benefit more from pre-training distillation. This claim is strongly supported by Figure 4 and Table 11. The monotonic trend from 330M (negative) to 6.8B (+6.9%) is clear and substantial. The 20.6× range in student size provides good coverage of the regime where PD might be considered. However, the paper does not establish whether the trend continues to even larger students (e.g., distilling a 12B student from a 32B or 70B teacher), which is the regime where many practitioners would be most interested. The 500B-token results (Figure 5) provide additional support: the 3.8B student benefits more from PD than the 1.9B student at every checkpoint. The finding is consistent with theoretical expectations (larger models can better represent the teacher's knowledge), but the absence of data points beyond 6.8B limits the practical guidance for larger-scale distillation.

Claim 4: A larger teacher LLM does not necessarily guarantee better results. This claim is supported by Figure 4 and Table 11, but with an important nuance. The 32B teacher underperforms the 9B teacher for the 1.9B and 3.8B students (gaps of 0.4 and 2.0 percentage points respectively), but for the 6.8B student, the gap narrows to 0.6 points (47.4% vs. 48.0%). This suggests the capacity gap may close as the student grows, but the paper does not test students large enough (e.g., 12B+) to verify whether the 32B teacher eventually overtakes the 9B teacher. The paper's framing ("does not necessarily guarantee better results") is appropriately hedged, but the evidence actually supports a stronger claim: for student sizes up to at least 6.8B (21% of the 32B teacher), the 9B teacher is strictly better. The threshold at which the 32B teacher becomes preferable is unknown and represents a significant gap in the scaling analysis.

Claim 5: Online logits are a viable (if weaker) alternative to offline logits. This claim is weakly supported. The LLM-Online-100B* result (+0.5%) is positive but small — well within the range that could be explained by a single favorable training run. The paper does not train multiple online configurations with the same adjusted α and truncation to verify consistency, nor does it train the online variant for multiple student sizes. The 400B-token teacher pre-training is only 4× the student's training budget, and the teacher may still be substantially undertrained — using a fully pre-trained teacher (offline) but capturing logits during its final training stage (online from the last checkpoint) would be a stronger test. The paper also does not report the online variant with the better PD configuration (WSD-α+WSD-LR, KLD loss), which might close more of the gap to offline PD.

Genuine weaknesses in the experimental design:

Single training runs with no error bars. Every result in the paper is a point estimate from one pre-training run. For a 1.9B model on 100B tokens, run-to-run variance from random initialization, data order, and hardware non-determinism could plausibly explain differences of 0.5–1.5 percentage points — the range within which many of the paper's comparative claims fall (e.g., the difference between KLD and NLL, between different p values in top-p truncation, between static τ = 0.5 and AdaKDH). Without multiple seeds or confidence intervals, the reader cannot distinguish genuine configuration effects from noise.
The SFT probe introduces an uncharacterized confound. All downstream evaluations are performed after SFT on 10B instruction-tuning tokens plus 10B pre-training text. The paper assumes that SFT is a neutral probe — that post-SFT performance differences reflect pre-training quality differences. But SFT itself involves a separate training process with its own hyperparameters (learning rate, batch size, data mixture), and the interaction between pre-training quality and SFT efficiency is uncharacterized. It is possible that PD improves pre-training in ways that make the model more (or less) amenable to SFT, and the reported improvements conflate pre-training quality with SFT amenability. The paper does not report pre-SFT evaluation metrics (e.g., perplexity on held-out text) that would cleanly separate these effects.
Missing baselines. The paper does not compare against several natural alternatives: (a) pre-training the student on the teacher's generated text (sequence-level KD) rather than logits — this would be a cheaper baseline that avoids the storage problem entirely; (b) using an ensemble of teachers; (c) using a teacher assistant (Mirzadeh et al., 2020) — an intermediate-sized model between teacher and student — which is the standard approach for addressing the capacity gap that the paper identifies as a key limitation; (d) using different teacher architectures (e.g., a 9B non-GLM model) to test whether findings are architecture-specific.
The difficulty estimation for online logits is incomplete. The paper shows that online logits from the first 100B tokens of teacher pre-training are harmful (-20.9%) and from the last 100B tokens are slightly harmful (-3.9%) or slightly helpful (+0.5% with adjustments). But the teacher's pre-training trajectory is 400B tokens — the paper does not report results for logits from intermediate stages (e.g., 100B–200B, 200B–300B), which would provide a finer-grained characterization of when online logits become "good enough." The recommendation to use "later stages" is vague without this characterization.
No interaction effects between design dimensions are explored. The paper studies each of the four design dimensions independently, varying one factor while holding others at the preliminary experiment's defaults. It does not investigate whether the optimal truncation strategy depends on the temperature, whether the optimal α schedule depends on the student-teacher size ratio, or whether the online-vs-offline choice changes the optimal loss function. The PD∗ configuration in Table 13 selects the best setting from each dimension independently and combines them — but there is no guarantee that these independently-optimal choices are jointly optimal. The paper acknowledges this limitation explicitly: "The main limitation of this work is that we do not explore the interactions between different factors in pre-training distillation... this is unaffordable, as these experiments are too resource-intensive given the complexity of factor combinations." This is a fair acknowledgment, but it means the PD∗ configuration is a heuristic combination rather than a verified optimum, and the reported +8.0% improvement (from WSD-α+WSD-LR alone) may understate or overstate what a jointly-optimized configuration could achieve.
The evaluation benchmarks mix zero-shot and few-shot settings. HellaSwag, WinoGrande, PIQA, and KBQA use zero-shot evaluation; C3, C-Eval, MMLU, and GSM8k use 5-shot, 5-shot, 6-shot, and 8-shot respectively. The paper computes a simple unweighted average across these benchmarks. This means a 1-point improvement on GSM8k (a math reasoning benchmark where absolute scores are low, 8.6% for LM) contributes equally to the average as a 1-point improvement on PIQA (a commonsense benchmark where scores are high, 72.9% for LM). The relative importance of these benchmarks is not discussed, and the averaging procedure gives equal weight to benchmarks with very different score scales and evaluation protocols. A more standard approach would normalize scores or report per-benchmark trends rather than relying heavily on the unweighted average.
Storage cost is discussed but not accounted for in comparisons. The paper emphasizes the 58.6 PB → 15 TB storage reduction as a key practical contribution, but the 15 TB figure is for 100B tokens — scaling this to trillion-token regimes would require ~150 TB, which is manageable but non-trivial. The paper does not discuss the I/O bottleneck of reading logits from disk during student training (the student must read both text tokens and logit vectors for each token position, substantially increasing data loading requirements compared to LM-only pre-training), nor does it report training throughput comparisons between LM and KD training. These practical considerations would matter for any organization evaluating whether to adopt PD.

Missing experiments that would have strengthened the paper:

Multiple training runs for the key comparisons (at minimum, 2–3 seeds for LLM-LM, the preliminary PD configuration, and the PD∗ configuration) to establish whether the reported differences exceed run-to-run variance.
Pre-SFT evaluation (perplexity on a held-out validation set) to verify that PD improves pre-training quality directly, not just SFT amenability.
Scaling law experiments at larger student sizes (e.g., 12B, 20B) to determine whether the "larger students benefit more" trend continues and whether the 32B teacher eventually overtakes the 9B teacher.
Interaction experiments — at minimum, testing whether the optimal α schedule changes with student size (since the capacity gap changes) and whether the optimal truncation changes with temperature.
Sequence-level KD baseline — training the student on text generated by the teacher (teacher-forcing on teacher outputs) — which would be substantially cheaper than logits-based KD and would establish the marginal value of the full logits signal over just the teacher's top-1 predictions.
A teacher assistant baseline for the 32B → 1.9B distillation path, where the capacity gap is known to be problematic, to determine whether the standard mitigation from the KD literature helps.
Compute-matched comparisons — training the LM baseline for more tokens to match the total FLOPs of the KD training (which includes the teacher inference cost for offline logits), to establish whether PD is more efficient than simply training the baseline for longer. The paper's token-matched comparisons are fair for isolating the distillation effect but do not address whether PD is compute-efficient in a total-cost sense.

In summary, the paper's central claims about the effectiveness and design sensitivity of pre-training distillation are generally supported by the reported experiments, but the evidence is weakened by the absence of multiple seeds, the reliance on SFT as a proxy evaluation, the lack of interaction experiments, and the limited scale range for some of the scaling law conclusions. The WSD-α+WSD-LR result is the most compelling single finding, while the online logits and capacity gap findings are suggestive but would benefit from additional validation. The paper successfully establishes that PD is a design space worth exploring — the specific configuration recommendations are likely robust in direction (WSD scheduling helps, larger students benefit more) but the precise magnitudes (+8.0% vs. maybe +6% or +10% with a different seed) should be treated as indicative rather than definitive.

6. Limitations and Trade-offs

No Statistical Reliability Assessment — Single Training Runs Throughout

The assumption or constraint. Every result in the paper is a point estimate from exactly one pre-training run followed by one SFT run. The paper does not train multiple seeds for any configuration, does not report confidence intervals, and does not conduct statistical significance tests for any comparison. The paper never explicitly states this as a limitation in the main text, though the Limitations section acknowledges that "the complexity of factor combinations" made interaction experiments "unaffordable" — the same reasoning implicitly applies to multi-seed experiments.

The consequence. For a 1.9B-parameter model trained on 100B tokens — a stochastic process involving random initialization, data shuffling, and hardware-induced non-determinism — run-to-run variance could plausibly produce differences of 0.5–1.5 percentage points in downstream accuracy. This means many of the paper's comparative claims fall within the range where noise and genuine configuration effects are indistinguishable. The central result — WSD-α+WSD-LR achieving 40.7% vs. 38.3% for the preliminary PD configuration — represents a 2.4 percentage point gap that is likely robust. But finer comparisons throughout the paper are unverifiable: the difference between KLD (38.7%) and NLL (38.3%) distillation loss, the difference between p = 0.5 (38.5%) and p = 0.85 (38.2%) in top-p truncation, the difference between static τ = 0.5 (38.7%) and AdaKDH (38.8%), and the difference between online logits with adjusted configuration (+0.5%) and the LM baseline all fall within 0.1–0.6 percentage points. The paper interprets these differences qualitatively as evidence for or against particular design choices, but without variance estimates, the reader cannot know whether re-running the same experiment would reproduce the same ordering.

What evidence exists in the paper. The paper provides no evidence about variance. There are no multiple-seed experiments, no bootstrap confidence intervals, no report of training loss variance across runs or across batches. The 500B-token checkpoint results (Table 12, Figure 5) provide a longitudinal view — multiple evaluations from a single training trajectory — which demonstrates smooth convergence but does not speak to cross-run variance. The closest thing to a replication is the independent re-training of different model sizes (330M through 6.8B) and the independent training of the online teacher, but these are different experimental conditions, not repetitions of the same condition.

Mitigation status. Not addressed. The paper does not mention this as a limitation. It is standard practice in LLM pre-training research to report single-run results due to computational cost, and the paper's scale (1.9B–6.8B models on 100B–500B tokens) makes multi-seed experiments genuinely expensive. However, the paper could have addressed this transparently by (a) acknowledging the issue explicitly, (b) running at least 2–3 seeds for the most critical comparison (LLM-LM vs. PD∗) at the 1.9B scale to bound the variance, or (c) reporting training metrics (validation perplexity during pre-training, pre-SFT) that are typically lower-variance than downstream benchmark accuracy after SFT. The absence of any of these makes the finer-grained design recommendations (e.g., "k = 50 yields the best results," "KLD outperforms NLL") impossible to distinguish from noise.

The SFT Probe Confounds Pre-Training Quality with Fine-Tuning Amenability

The assumption or constraint. All downstream evaluation in the paper is conducted after supervised fine-tuning (SFT) on 10B instruction-tuning tokens plus 10B pre-training text tokens. The paper presents this as a necessary methodological choice:

"Given the limited capacity of the student LLM, its performance on some evaluation datasets, such as MMLU and C-Eval, is close to random guessing, making the results incomparable. Therefore, we conduct supervised fine-tuning (SFT) with additional 10B high-quality instruct-tuning data after pre-training."

The implicit assumption is that SFT is a neutral, uniform probe: models that are better after SFT were better before SFT, and the SFT process does not interact with pre-training quality in ways that could reverse or distort pre-training comparisons.

The consequence. SFT is itself a substantial training process — 10B tokens of instruction data with a separate learning rate schedule, batch size, and data mixture — and its effectiveness may depend on properties of the pre-trained model that are not captured by downstream accuracy. A model pre-trained with distillation might develop representations that are more (or less) amenable to the specific SFT data mixture used, independent of its raw language modeling quality. If PD produces features that happen to align better with the instruction-tuning distribution — for example, because the teacher model was itself instruction-tuned or trained on data that overlaps with the SFT data — then post-SFT accuracy would overstate the pre-training improvement. Conversely, if PD produces features that are somehow less amenable to SFT (e.g., the teacher's distribution biases the student toward patterns that do not transfer well to instruction-following), post-SFT accuracy would understate the pre-training improvement. The paper provides no way to disentangle these possibilities because it never reports pre-SFT metrics (perplexity on held-out text, zero-shot accuracy without SFT, or intermediate checkpoints before SFT).

What evidence exists in the paper. The paper reports only post-SFT accuracy. There is no perplexity measurement on a held-out validation set during pre-training, no pre-SFT evaluation on any benchmark (even those where the 1.9B model's accuracy might be above random, such as HellaSwag or PIQA), and no ablation of the SFT process itself (varying SFT data, SFT duration, or SFT learning rate) to confirm that PD gains are robust to SFT configuration. The 500B-token checkpoint experiments (Table 12) evaluate each checkpoint after SFT — so they show the interaction of PD with pre-training duration, but still always through the lens of SFT. The paper does not discuss this confound.

Mitigation status. Not addressed. The paper treats SFT as an unproblematic evaluation protocol and does not acknowledge it as a potential confound. The standard approach for evaluating pre-training quality in the LLM literature includes reporting validation perplexity during pre-training (which directly measures the model's language modeling capability without any post-processing) alongside downstream task performance. The paper's omission of perplexity is unusual and makes it impossible to verify that PD improves pre-training quality itself, rather than just producing models that fine-tune more effectively under the specific SFT recipe used. A practitioner evaluating whether to adopt PD would want to know whether the gains are in the pre-training (which matters for all downstream uses) or in the SFT interaction (which might be specific to the instruction-tuning setup).

Design Space Exploration Is Factorial, Not Joint — No Interaction Effects Are Tested

The assumption or constraint. The paper studies four design dimensions (logits processing, loss selection, scaling law, offline vs. online) by varying one factor at a time while holding all others at the preliminary experiment's default configuration. The PD∗ configuration in Table 13 is constructed by selecting the best-independently-performing setting from each dimension and combining them. The paper explicitly acknowledges this limitation:

"The main limitation of this work is that we do not explore the interactions between different factors in pre-training distillation, that is, the different combinations of factors. This is unaffordable, as these experiments are too resource-intensive given the complexity of factor combinations."

The consequence. There is no guarantee that the independently-optimal choices are jointly optimal. Design choices in machine learning systems routinely interact: the optimal temperature for logits normalization might depend on the truncation strategy (since truncation changes the shape of the distribution that temperature scales), the optimal α schedule might depend on the student-teacher size ratio (since the capacity gap changes how much the student should rely on teacher signal at different training stages), and the optimal loss function might depend on whether logits are offline (clean) or online (noisy). The PD∗ configuration — applying WSD-α+WSD-LR scheduling with KLD loss, τ = 2.0, and top-0.95-50 truncation simultaneously — assumes these choices are independent. If they are not, PD∗ could be either better than reported (if the interactions are synergistic) or worse (if the independently-optimal choices conflict when combined). The paper provides no evidence either way.

The practical consequence for a practitioner is ambiguity: the paper's design recommendations ("use KLD loss," "use τ = 2.0," "use WSD-α+WSD-LR") are derived from experiments where only one factor was varied at a time. If the practitioner's setting differs from the preliminary experiment's defaults in any way — different student size, different teacher, different corpus, different vocabulary — the optimal choice for each factor might shift. The paper does not provide the information needed to predict how these recommendations transfer.

What evidence exists in the paper. The paper provides extensive evidence that each individual design choice matters (Tables 2–6, 8–11). But it provides zero evidence about interactions. The PD∗ results (Table 13, Figure 1) demonstrate that combining the independently-best choices yields improvements over the preliminary configuration (41.2% vs. 38.3% for 1.9B), but this does not test whether a different combination would be even better. There is no experiment that varies two factors simultaneously — for example, testing whether the optimal α schedule is different for the 3.8B student than for the 1.9B student, or whether the optimal temperature changes when using KLD vs. NLL loss. The paper's factorial design (one factor at a time) is appropriate for an initial exploration but cannot detect or quantify interactions.

Mitigation status. Acknowledged explicitly in the Limitations section. The paper frames this as a resource constraint rather than a methodological oversight:

"Our controlled variable experiments have already incurred significant computational costs... While searching the combinations of factors could identify best practices, we believe our experiments and explorations are sufficiently solid to inform future practices in pre-training distillation."

This is a reasonable position given the scale of the experiments. A full factorial exploration of even 2-way interactions (e.g., 4 temperature values × 3 loss functions × 5 α schedules × 5 student sizes = 300 configurations, each training a 1.9B+ model on 100B tokens) would be cost-prohibitive at current hardware prices. However, the paper does not attempt even a minimal interaction experiment — for instance, testing whether the best α schedule from Section 3.3 (WSD-α+WSD-LR) remains optimal for the 3.8B student (where the capacity gap is smaller and the optimal KD proportion might differ), or whether the optimal temperature from Section 3.2 changes when using KLD instead of NLL. A few well-chosen interaction experiments at the 1.9B scale would have substantially strengthened the PD∗ recommendation without exploding the experimental budget.

Model and Benchmark Specificity — No Evidence of Cross-Architecture or Cross-Domain Generality

The assumption or constraint. All experiments use a single model family (GLM; GLM et al., 2024) for both teacher and student, a single pre-training corpus (described only as a general text corpus, with no details on composition, language mix, or domain coverage), and downstream evaluation on eight benchmarks spanning English understanding, Chinese understanding, and math. The paper does not test with other model architectures (e.g., LLaMA, Qwen, or non-GLM architectures), other teacher-student architecture pairs (e.g., a LLaMA teacher distilling a GLM student, which would test whether architectural compatibility matters), or other pre-training data distributions. The paper's claims about the effectiveness of PD are implicitly claims about PD as a technique, but the evidence is conditional on the specific models and data used.

The consequence. Several of the paper's findings may be architecture-specific or data-specific in ways that practitioners cannot assess from the reported results:

The capacity gap threshold (~10% student-to-teacher ratio). This threshold depends on architectural compatibility between teacher and student, vocabulary overlap (all GLM models share the same ~150K vocabulary), and the specific representational capacity of the GLM architecture. A student using a different architecture with the same parameter count might have different effective capacity and a different threshold. The paper's finding that the 32B teacher underperforms the 9B teacher for 1.9B–6.8B students could be specific to how GLM models scale — a different architecture family might show different teacher-size scaling behavior.
The MSE loss failure. The paper interprets this as evidence that "the pre-training distillation of LLMs involves new training dynamics" compared to image classification KD, but it could also be specific to the GLM vocabulary size (150K), the GLM logit distribution properties, or the specific truncation strategy used. A model with a smaller vocabulary or different logit calibration might not show the same catastrophic MSE degradation.
The effectiveness of PD for English vs. Chinese tasks. The evaluation benchmarks include both English (HellaSwag, WinoGrande, PIQA, MMLU, GSM8k) and Chinese (KBQA, C3, C-Eval) datasets. The preliminary PD configuration shows gains on some English benchmarks (HellaSwag: +1.7%) and some Chinese benchmarks (C3: +1.9%, C-Eval: +3.2%), but the paper never breaks down results by language. If PD is more effective for Chinese (perhaps because the teacher was trained on more Chinese data, or because the 150K vocabulary provides richer signal for Chinese token distributions), this would be important for practitioners working primarily in English.

What evidence exists in the paper. The paper provides no cross-architecture experiments, no teacher-student architecture mismatch experiments, no details on training data composition, and no per-language breakdown of results. The better PD configuration (PD∗, Table 13) shows consistent improvements across all eight benchmarks for all three student sizes — which is reassuring — but this is still within a single model family on a single (undisclosed) data distribution. The paper does not discuss architecture or data specificity as a limitation.

Mitigation status. Not addressed. The paper does not claim its findings are architecture-agnostic, but it also does not caution readers that the results may not transfer. The framing of the paper as a "design space exploration" implies that the explored dimensions (logits processing, loss selection, scaling law, offline vs. online) are general properties of PD, not artifacts of the GLM architecture or training data. A practitioner using a non-GLM model family would need to assume — without evidence — that the paper's recommendations (WSD-α+WSD-LR, KLD loss, τ ≤ 2.0, 10% size threshold) apply to their setting. Given the well-documented sensitivity of distillation to architectural choices (e.g., the original capacity gap paper by Mirzadeh et al., 2020, used the same architecture for teacher and student), this assumption may not hold.

Online Logits Evaluation Is Insufficient to Support the Deployment Recommendation

The assumption or constraint. Based on a single online logits experiment — pre-training a new 9B teacher from scratch on 400B tokens, capturing logits, and distilling three 1.9B student variants — the paper recommends:

"if one aims to pre-train a series of LLMs of varying sizes, one can first pre-train the largest LLM while storing online logits, and then pre-train smaller LLMs with online logits."

This recommendation assumes that the online logits approach is sufficiently effective and well-characterized to guide deployment decisions, and that the +0.5% result for LLM-Online-100B* (Table 6) would scale to larger students and longer teacher pre-training.

The consequence. The online logits experiment has several properties that make the deployment recommendation premature:

The teacher pre-training length (400B tokens) is only 4× the student's training budget (100B tokens). Modern LLMs are often pre-trained on multiple trillions of tokens — the 400B-token teacher is still substantially undertrained, and its logits at 300B–400B tokens may be much noisier than offline logits from a fully-converged teacher. The paper has no evidence about how online logit quality evolves beyond 400B tokens or whether the +0.5% improvement would grow, shrink, or reverse if the teacher were trained to convergence (1T+ tokens).
Only one student size (1.9B) is tested. The deployment recommendation is for "a series of LLMs of varying sizes," but the paper has no evidence that online logits work for larger students (3.8B, 6.8B) or smaller ones (330M, 670M). Given the capacity gap findings — smaller students benefit less from PD in general — online PD (which provides a weaker teacher signal) may be ineffective or harmful for very small students.
The adjusted configuration (α = 0.1, top-0.95-50) was found by manual tuning for this specific setting. The paper does not sweep α, truncation, or loss function for online logits to establish whether these are robustly optimal choices or specific to the 1.9B/400B/100B setup.
The 180 TB storage cost for 400B tokens of online logits is mentioned but not factored into the economic analysis. At trillion-token scales, online logit storage would reach ~450 TB — manageable but requiring dedicated infrastructure. The paper's recommendation does not discuss whether this storage cost, plus the I/O overhead during student training, is preferable to the alternative of simply running the fully-trained teacher as an offline logit generator.

What evidence exists in the paper. Table 6 provides three data points (LLM-Online-100B-L at -20.9%, LLM-Online-100B at -3.9%, LLM-Online-100B* at +0.5%). The paper does not report additional online configurations (different α values, different loss functions, different student sizes), does not report results using online logits from intermediate teacher training stages (100B–200B, 200B–300B, 300B–400B) to characterize the quality trajectory, and does not compare online PD at 400B tokens to offline PD using the same teacher after it completes pre-training (which would be the natural hybrid baseline).

Mitigation status. The paper frames the online logits experiment as preliminary ("Due to the high cost of pre-training GLM-4-9B from scratch, we preliminarily pre-train...") and does not present the deployment recommendation as definitive. However, the recommendation appears in the main text (Section 3.5) and the abstract without caveats about its preliminary nature, which could lead practitioners to over-weight it relative to the strength of the supporting evidence. The paper does not explicitly call out the need for additional online logits experiments at larger scale or with different student sizes as future work.

Storage and I/O Overhead Not Accounted for in Practical Deployment Analysis

The assumption or constraint. The paper's entire offline PD paradigm depends on pre-computing and storing teacher logits for every token of the pre-training corpus. The paper reports the storage reduction from top-p-k truncation (~4,000×, from a hypothetical 58.6 PB to ~15 TB for 100B tokens with top-0.95-100) and mentions the 180 TB cost for storing online logits during 400B tokens of teacher pre-training. However, the paper never reports training throughput (tokens per second) for PD compared to standard LM pre-training, never discusses the I/O bottleneck of reading logit vectors from disk during student training, and never computes the total cost (GPU-hours + storage + I/O infrastructure) of PD relative to simply training the student for longer with standard LM loss.

The consequence. A practitioner evaluating PD needs to compare the total cost of PD (teacher inference + logit storage + student training with disk I/O) against the cost of achieving equivalent student quality through other means — most obviously, training the student on more tokens with standard LM loss, or training a slightly larger student with LM loss. The paper's token-matched comparisons (same number of student training tokens for LM and KD) are appropriate for isolating the pedagogical effect of distillation, but they do not answer the economic question: is PD a cost-effective way to improve student models compared to simply scaling up standard pre-training?

The unaccounted costs include:

Teacher inference cost. Generating offline logits requires running the frozen teacher LLM on every token of the pre-training corpus — for a 9B teacher on 100B tokens with 4,096 context length, this is approximately 100B forward passes (one per token, since autoregressive models produce one logit vector per token). This is a substantial computational expense.
Logit storage and I/O. During student training, each token requires reading the ground-truth token ID (a few bytes) plus the teacher's truncated logit vector (up to 100 float32 values = 400 bytes for top-100, or 200 bytes for top-50). This means the data loading volume per token is ~50–100× larger for PD than for standard LM pre-training. If the training cluster's storage I/O is the bottleneck (which it often is for large-scale pre-training), PD could significantly reduce training throughput.
The SFT stage. The paper's evaluation protocol adds 20B tokens of SFT after pre-training. If the PD gains are partly due to better SFT amenability (see Limitation 2), then the SFT cost is part of the PD deployment cost and should be factored into comparisons.

What evidence exists in the paper. The paper reports storage requirements for logits (15 TB for 100B tokens, 180 TB for 400B tokens of online logits) but nothing about training throughput, I/O overhead, teacher inference cost in GPU-hours, or total cost comparisons. The paper never compares PD to a compute-matched LM baseline (e.g., training the LM student on additional tokens to match the total FLOPs spent on teacher inference + PD student training). The evaluation is strictly token-matched, not FLOP-matched or dollar-matched.

Mitigation status. Not addressed. The paper treats the storage reduction from truncation as a practical contribution but does not consider the broader cost picture. For a practitioner at an organization with a fixed pre-training budget, the relevant question is not "does PD improve student quality at a given token count?" but "does PD improve student quality per dollar (or per GPU-hour) relative to the next best alternative?" The paper provides none of the information needed to answer this question. A FLOPs-matched comparison — or at minimum a back-of-the-envelope calculation of the teacher inference cost and its amortization across students — would substantially increase the practical value of the findings.

This limitation interacts with the online logits recommendation: online logits eliminate the separate teacher inference cost (since logits are captured during the teacher's own pre-training), but the paper's online results are too weak (+0.5% at best) and too preliminary (single student size, single teacher training length) to establish whether online PD is cost-effective. The economic case for PD — offline or online — remains unquantified.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not introduce a fundamentally new technique — knowledge distillation during pre-training was already known to be feasible from the DistilBERT and TinyBERT era — but it changes the field's relationship to pre-training distillation in three concrete ways.

First, it converts pre-training distillation from a binary on/off decision into a multi-dimensional engineering problem with meaningful stakes. Before this work, a practitioner deciding whether to use PD would have asked "should we use distillation or not?" — and likely would have defaulted to whatever configuration was used in a prior paper or internal codebase, assuming the configuration didn't matter much. After this work, that question becomes "what truncation strategy, loss function, mixing schedule, and teacher size should we use?" — and the gap between a naive configuration (+1.6%) and a well-tuned one (+8.0% for the 1.9B student, +10.9% for the 6.8B student) shows that the configuration choices are not minor optimizations but first-order determinants of whether PD is worth doing at all. The paper's central empirical contribution is not any single configuration finding but the demonstration that configuration variance dominates the naive PD-vs-LM effect size: a practitioner who naively adopts PD with suboptimal settings (say, MSE loss with static α = 1.0) would see a 7.6% degradation and conclude PD doesn't work, while one who tunes the design space would see an 8.0% improvement and conclude PD is transformative — both from the same underlying technique. This changes PD from a technique to a design space, and that reframing matters for how seriously practitioners take it and how they invest engineering effort.

Second, it establishes the WSD schedule as a general principle for when teacher signal is most valuable during training — not as a learning rate trick, but as a signal-allocation strategy. The WSD learning rate scheduler (Hu et al., 2024) was previously proposed as a training efficiency method — keep the learning rate high for most of training, then rapidly decay. This paper shows something different: the WSD schedule for the loss mixture (WSD-α), combined with WSD-LR, dramatically amplifies the benefit of distillation. The ablation structure (WSD-α+WSD-LR at +8.0% vs. WSD-β+WSD-LR at +1.1%, vs. 1-α+WSD-LR at +5.0%, vs. 0-α+WSD-LR at +1.2%) demonstrates that this is not simply WSD-LR providing a small generic benefit on top of distillation — it is the specific interaction of placing high KD proportion during the high-learning-rate stable phase that matters. The conceptual principle — "align the richest training signal with the period of maximum plasticity, then regularize with ground truth during decay" — generalizes beyond the specific α curves tested here and could inform curriculum design, multi-task training schedules, and other settings where training signal quality varies across sources. This is a mid-level contribution: it repurposes an existing technique (WSD) for a new purpose (distillation signal scheduling) and provides mechanistic evidence for why it works, not just that it works.

Third, it clarifies the scaling boundaries of PD — specifically, that the student-to-teacher ratio is a first-class hyperparameter with a rough 10% threshold, and that the economically intuitive "use the biggest available teacher" strategy is counterproductive below a certain student scale. The finding that a 32B teacher underperforms a 9B teacher for 1.9B and 3.8B students (Figure 4) directly contradicts the default assumption that more teacher knowledge is always better. The capacity gap (Mirzadeh et al., 2020) was previously known as a theoretical concern; this paper operationalizes it at the billion-parameter scale and quantifies the penalty (e.g., 2.0 percentage points lower for the 3.8B student with the 32B teacher vs. the 9B teacher). For organizations planning distillation pipelines, this reframes teacher selection from "use the largest model you have" to "use the smallest teacher that exceeds ~10× the student's parameter count" — a substantially different engineering guideline. The paper also provides the first evidence (to my knowledge) that PD improvements persist — and even grow — through 500B tokens of pre-training (Figure 5), suggesting PD raises the asymptotic performance ceiling, not just the convergence rate. This has implications for the trillion-token pre-training regimes used by frontier models: if PD gains are sustained at scale, the case for incorporating PD into the largest training runs becomes stronger, not weaker, even if the relative gain appears modest (a 2–3% relative improvement on a multi-million-dollar training run is substantial in absolute terms).

What this work does not change. The paper does not resolve the question of whether PD is compute-efficient in a total-cost sense — the token-matched comparisons leave unanswered whether PD's teacher inference cost and storage overhead make it preferable to simply training the student for longer with standard LM loss. It does not provide evidence that PD works across architectures or data distributions beyond the GLM family on its undisclosed corpus. And it does not provide a recipe for PD at the frontier scales (trillion+ tokens, 70B+ teachers, 10B+ students) where most industrial interest lies. These are not failures — the paper explicitly scopes itself as an initial design space exploration — but they bound the landscape change as one of reframing and enabling rather than solving: the paper tells the field that PD is worth tuning seriously, shows how to tune it at the billion-parameter scale, and leaves the frontier-scale validation to future work.

Follow-Up Research This Work Enables

Joint optimization of the PD design space with interaction effects. The paper's central acknowledged limitation is that it studies each design dimension independently and combines the independently-best choices into PD∗ without testing whether they interact. A natural follow-up would systematically explore 2-way interactions at the 1.9B/100B-token scale where the computational budget is manageable. Specific questions: Does the optimal α schedule change with student size? The capacity gap shrinks as the student grows — the 1.9B student (21% of teacher) faces a larger gap than the 6.8B student (76% of teacher) — so the optimal KD proportion during the stable phase might be lower for smaller students (which need more ground-truth grounding) and higher for larger students (which can absorb more teacher signal). Does the optimal temperature depend on the loss function? KLD operates on probabilities and is scale-invariant; NLL is equivalent to KLD in gradient but not in absolute magnitude; the optimal temperature for sharpening or flattening the teacher distribution might differ between them. A well-designed experiment with 2–3 values per dimension (loss function × temperature × α schedule × student size) — perhaps 36–54 configurations — would cost more than the current paper's budget but would provide a much stronger basis for configuration recommendations. The paper's PD∗ results (Table 13) already show that combining the independently-best choices yields gains, but without interaction data, practitioners cannot know whether they're leaving additional performance on the table or whether a different combination would work better for their specific student-teacher size ratio.

PD at trillion-token scale with frontier model sizes. The paper's 500B-token experiments (Figure 5) show PD gains persisting through the end of training, with the 3.8B student's advantage actually growing (+1.3 points at the first checkpoint to +3.5 points at the final checkpoint). But 500B tokens is well below the multi-trillion-token regimes used by models like LLaMA 3 (15T+ tokens), Qwen 2.5 (18T tokens), and DeepSeek-V2 (8.1T tokens). A critical open question: do PD gains asymptote, continue growing, or eventually reverse at trillion-token scales? The slight convergence in the 1.9B student's advantage at the final 500B-token checkpoint (gap narrows from +2.4 to +1.2 points) hints that diminishing returns may set in for smaller students, but the 3.8B student shows no such convergence. A follow-up experiment training a 3.8B or 6.8B student with PD∗ on 1T–2T tokens — and comparing against an LM-only baseline trained for the same token count — would directly address whether PD is worth the engineering investment at frontier scales. This experiment is expensive but highly leveraged: if PD provides even a 2% relative improvement at the 1T-token scale, the absolute performance gain on downstream benchmarks would likely exceed what could be achieved by many architectural tweaks or data filtering improvements that consume comparable engineering effort. The paper's recommendation that "we believe that pre-training distillation is also effective using several trillion tokens" is reasonable but unverified — this experiment would verify or refute it.

Cross-architecture PD: does architectural compatibility matter? All the paper's experiments use GLM-family models for both teacher and student, with identical vocabularies (~150K tokens) and similar architectural choices (grouped-query attention, same activation functions, same layer structure). This is the cleanest setting for studying PD, but real-world deployment often involves distilling across architectures — for example, distilling from a proprietary model (GPT-4, Claude) into an open-weight architecture (LLaMA, Mistral), where the teacher and student have different vocabularies, different attention mechanisms, and different training data distributions. A follow-up experiment would distill from a 9B GLM teacher into a 1.9B non-GLM student (e.g., a LLaMA-architecture model with a different tokenizer), using either vocabulary projection (mapping the teacher's logits to the student's vocabulary via token alignment) or text-based distillation (sequence-level KD) as baselines. This would test whether the paper's design recommendations (top-p-k truncation, WSD-α+WSD-LR, KLD loss) transfer across architectures, or whether architectural mismatch introduces new failure modes (e.g., the teacher's logits over tokens that don't exist in the student's vocabulary provide noisier signal, or the capacity gap interacts with architectural differences). A negative result — PD works poorly across architectures even with careful configuration — would be practically important, as it would mean PD is only viable within model families, limiting its applicability.

Sequence-level KD as a cheap baseline for PD. The paper focuses exclusively on logits-based KD, which requires storing and reading teacher logit vectors. A natural ablation is to compare against sequence-level KD: use the teacher to generate the most likely next token at each position (greedy teacher-forcing), and train the student to predict those teacher-generated tokens as if they were ground truth. This eliminates the storage problem entirely (the student only reads token IDs, same as standard pre-training) and the I/O overhead, at the cost of discarding the teacher's full distribution. The paper's top-1 truncation result (k = 1 in Figure 3, achieving 38.4% average) is effectively sequence-level KD, and it shows a 1.8% improvement over LM-only pre-training — competitive with the preliminary PD configuration (+1.6%) but far behind PD∗ (+9.3%). A proper sequence-level KD experiment would train a student on teacher-generated tokens with the same WSD-α+WSD-LR schedule and other PD∗ optimizations, testing whether the scheduling and loss function improvements transfer to the simpler signal. If sequence-level KD with optimal scheduling can close most of the gap to logits-based PD, the practical case for logits-based PD (with its storage and I/O costs) weakens substantially. If it cannot, the experiment quantifies the marginal value of the full logits distribution over just the teacher's top predictions — a number with direct economic implications.

The capacity gap threshold as a function of absolute scale, not just ratio. The paper identifies a rough 10% student-to-teacher ratio threshold for PD effectiveness and observes that the 32B teacher underperforms the 9B teacher for students up to 6.8B (21% of teacher). But the paper cannot distinguish whether the relevant variable is the ratio (student/teacher parameters) or the absolute gap (teacher minus student parameters). If ratio is what matters, a 1.9B student distilling from a 19B teacher would face the same capacity gap as a 3.8B student distilling from a 38B teacher — both at 10%. If absolute gap matters, the 38B teacher case would be harder (34.2B gap vs. 17.1B gap). A follow-up experiment would test students at multiple sizes (1.9B, 3.8B, 6.8B) with teachers spanning a wider range — e.g., 4B, 9B, 19B, 32B, 70B — to map the capacity gap surface as a function of both ratio and absolute scale. This would directly inform teacher selection for PD pipelines: if ratio is the dominant variable, a practitioner with a 70B teacher can straightforwardly distill a 7B student. If absolute gap matters, the same practitioner might need a teacher assistant (Mirzadeh et al., 2020) — distilling 70B → 20B → 7B — to keep the gap manageable at each step. The paper's open question about weak-to-strong generalization (Burns et al., 2024) — using a small teacher to improve a larger student — fits naturally into this experiment: test whether a 4B teacher can improve a 6.8B student, reversing the capacity gap direction.

Online PD with full teacher convergence and multi-student amortization. The paper's online logits experiment is its weakest link — the teacher is trained for only 400B tokens, the best online student achieves only +0.5%, and only one student size is tested. A definitive online PD experiment would: (a) pre-train a teacher to full convergence (1T+ tokens, ideally matching the training budget of frontier models), capturing logits from the final 100B–200B tokens; (b) use those late-stage logits to distill multiple student sizes (1.9B, 3.8B, 6.8B) with the PD∗ configuration (WSD-α+WSD-LR, KLD, appropriate truncation and temperature); and (c) compare against both an LM-only baseline and an offline PD baseline using the same converged teacher. This would answer: how much of the offline PD advantage comes from teacher convergence quality vs. from the offline paradigm itself? If online PD with a fully-converged teacher achieves, say, +5–6% improvement (vs. offline PD's +8–9%), the tradeoff becomes quantitative — 5–6% improvement with zero additional inference cost (since logits are captured during teacher training) vs. 8–9% improvement with one additional teacher inference pass over the corpus. For organizations training model families (e.g., a 70B flagship model plus 7B, 13B, and 30B variants), this tradeoff could favor online PD even with a modest performance gap. The paper's current recommendation — offline for single models, online for model families — is directionally correct but needs this quantitative backing to inform actual resource allocation decisions.

Practical Applications and Downstream Use Cases

Pre-training model families with a shared teacher. An organization pre-training a family of LLMs at different scales — for example, a 70B flagship model plus 7B, 13B, and 30B variants for different deployment tiers — would benefit directly from the paper's PD∗ configuration. The paper's scaling law results (Figure 4) show that larger students benefit more from PD (the 6.8B student gains +6.9% from vanilla PD with the 9B teacher, and PD∗ pushes this further), and the 500B-token experiments (Figure 5) show that PD gains are sustained and even grow through extended training. The recommended pipeline: pre-train the 70B teacher to convergence, generate offline logits (or capture online logits during the final training stage), and distill all smaller models using PD∗ with WSD-α+WSD-LR scheduling, KLD loss, and appropriate truncation. The paper's finding that the teacher should not be too large relative to the student — the 32B teacher underperforms the 9B teacher for students under ~7B — suggests that for a 7B student, a teacher in the 30B–70B range may be near the capacity-gap threshold where PD becomes less effective. The practical guideline: use the smallest teacher that is at least ~10× the student size, not the largest available teacher, and consider teacher assistants for very large teacher-to-student ratios.

Pre-training under fixed token budgets where quality is the binding constraint. For organizations with a fixed pre-training token budget (e.g., 500B tokens due to data availability or compute constraints) who need to maximize downstream model quality, PD∗ offers substantial gains without requiring additional tokens. The paper's 500B-token results (Table 12) show that the 3.8B student with PD reaches 53.7% average accuracy vs. 50.2% for LM-only — a 3.5 percentage point absolute gain at the same token count. For a 6.8B student (extrapolating from the 100B-token results where PD∗ provides a +4.9 point gain), the absolute improvement could be even larger. The storage and I/O cost of PD (15 TB for 100B tokens, scaling linearly with corpus size) is the main deployment barrier, but the paper's finding that aggressive truncation (smaller p and k in top-p-k) does not significantly degrade quality (Figures 2–3) means practitioners can tune storage cost downward without proportional quality loss. For a 500B-token corpus, ~75 TB of logit storage (using top-0.95-50 or more aggressive truncation) is well within the storage budget of any organization capable of pre-training a 6.8B model.

Distillation as an alternative to data filtering for noisy corpora. The paper's observation about top-1 truncation — "this may be due to the teacher LLM conducting implicit noise filtering in pre-training corpora" (Section 3.2) — suggests a use case that the paper does not explicitly develop. In settings where the pre-training corpus contains substantial noise (web text with typos, grammatical errors, factual inconsistencies, or toxic content), the teacher model's probability distribution naturally down-weights inappropriate tokens and up-weights plausible alternatives. Training a student on the teacher's distribution effectively applies a soft, token-level denoising filter that would be difficult to replicate with rule-based or classifier-based data filtering pipelines. The k = 1 result (38.4% vs. 37.7% LM baseline) shows that even the teacher's top-1 prediction — essentially, an autoregressive correction of the training data — provides a small improvement, suggesting the teacher is indeed "correcting" the training data in some cases. A practitioner working with a unavoidably noisy corpus (e.g., a new domain where clean data is scarce) could use PD as a form of learned data cleaning, with the teacher providing a distribution over corrected tokens rather than binary keep/discard decisions at the document level. The PD∗ configuration's +8–10% improvements on clean data suggest the denoising effect compounds with other PD benefits.

When to Prefer This Method

The paper does not articulate explicit tradeoffs against named alternative pre-training strategies (e.g., "prefer PD over curriculum learning when..."). The closest it comes is the implicit comparison between PD and standard LM-only pre-training at the same token count, and the offline-vs-online deployment recommendation. The decision framework is therefore:

Prefer PD over standard LM-only pre-training when the student model is at least ~10% the size of the teacher (1.9B+ for a 9B teacher), the teacher is available and architecturally compatible, and storage infrastructure can accommodate the truncated logits (~15 TB per 100B tokens of pre-training data). The benefits scale with student size (larger students benefit more), and even modest configuration tuning (WSD-α+WSD-LR, KLD loss) yields substantially larger gains than naive PD. The paper's evidence supports this for GLM-family models up to 6.8B students and 500B tokens.
Prefer offline over online PD when training a single student model, since offline logits from a fully converged teacher provide strictly higher-quality signal (38.3% offline vs. 37.9% online at best, with the offline configuration using α = 1.0 vs. online's α = 0.1). The one-time teacher inference cost is justified by better student quality. This recommendation is supported for 1.9B students with a 9B teacher trained on 400B tokens, but the gap may narrow or close if the online teacher is trained to full convergence.
Prefer online PD when training a family of models at different scales, since the logit generation cost is amortized across all students. The paper's evidence for online PD is preliminary (+0.5% at best vs. +1.6%+ for offline), so this recommendation should be treated as directional — online PD avoids the teacher inference cost but may require accepting a smaller quality improvement unless the teacher is trained to convergence before logit capture.
Prefer standard LM-only pre-training when the student is very small relative to the teacher (<10% of teacher size), since PD can actually degrade performance (the 330M student drops from 32.6% LM to 32.1% PD with the 9B teacher), or when storage and I/O infrastructure cannot support the logit volume. In these cases, the paper's results suggest that PD's benefits do not materialize and may be negative.

🎯 Pitch

1. Executive Summary

2. Context and Motivation

The Core Problem: We Know Post-Training Distillation Works, but Pre-Training Distillation Is Unexplored

Why This Problem Matters: Three Practical and Theoretical Stakes

Where Prior Approaches Fall Short

How This Paper Positions Itself

The Practical Constraint That Shapes the Entire Investigation

The Unstated Premise: Why Logits Rather Than Text?

Summary: What the Reader Should Take Forward

3. Technical Approach

3.1 Reader Orientation

3.2 Big-Picture Architecture (Diagram in Words)

3.3 Roadmap for the Deep Dive

3.4 Detailed, Sentence-Based Technical Breakdown

The Formal Pre-Training Distillation Objective

Preliminary Experiment Configuration

Logits Processing (Design Dimension #1): The $\mathcal{F}$ Function

Loss Selection (Design Dimension #2): $\mathcal{L}$ and $\alpha$

Scaling Law (Design Dimension #3): Model Size and Corpus Size

Offline vs. Online Logits (Design Dimension #4): Sourcing Teacher Signal

The Better PD Configuration (PD∗)

Summary of Design Choices and Their Justifications

4. Key Insights and Innovations

Innovation 1: Pre-Training Distillation as a Design Space, Not a Binary Choice

Innovation 2: The WSD Schedule as a General Principle for Distillation Signal Allocation

Innovation 3: The Capacity Gap as the Central Scaling Dynamic in Pre-Training Distillation

Innovation 4: Online Logits as a Viable (If Weaker) Alternative to Offline Pre-Computation

5. Experimental Analysis

Evaluation Methodology

Main Quantitative Results

Preliminary Validation: Vanilla PD Provides a Modest but Real Improvement

Logits Processing: Truncation Is Robust, Temperature Has an Upper Bound

Loss Selection: KLD and NLL Are Similar; MSE Fails; WSD Scheduling Matters Critically

Scaling Law: Larger Students Benefit More; Larger Teachers Do Not

Offline vs. Online Logits: Online Works but Is Weaker

The Better PD Configuration (PD∗)

Ablation Studies and Robustness Checks

Critical Assessment

6. Limitations and Trade-offs

No Statistical Reliability Assessment — Single Training Runs Throughout

The SFT Probe Confounds Pre-Training Quality with Fine-Tuning Amenability

Design Space Exploration Is Factorial, Not Joint — No Interaction Effects Are Tested

Model and Benchmark Specificity — No Evidence of Cross-Architecture or Cross-Domain Generality

Online Logits Evaluation Is Insufficient to Support the Deployment Recommendation

Storage and I/O Overhead Not Accounted for in Practical Deployment Analysis

7. Implications and Future Directions

How This Work Changes the Landscape

Follow-Up Research This Work Enables

Practical Applications and Downstream Use Cases

When to Prefer This Method

Adjacent in domain

A Survey of Small Language Models

RATIONALYST: Mining Implicit Rationales for Process Supervision of Reasoning