ArXiv: 2408.11796

🎯 Pitch

A short teacher correction phaseβ€”just 100B tokens of fine-tuningβ€”is all it takes to adapt a large model to an unseen dataset and enable compression via structured pruning and distillation, producing an 8B model from Mistral NeMo 12B that surpasses its teacher on all benchmarks using 40Γ— fewer tokens.

1. Executive Summary

This work introduces a practical adaptation of the Minitron compression recipe for producing small language models when the original pretraining dataset is unavailable, applying it to distill Mistral NeMo 12B and Llama 3.1 8B down to 8B and 4B parameters respectively. The core methodological contributions are a teacher correction phaseβ€”a lightweight fine-tuning (~100B tokens) of the unpruned teacher model on the distillation dataset before serving as supervisionβ€”and a downstream-task-based saliency criterion for depth pruning (using Winogrande accuracy rather than validation loss to select which contiguous layer blocks to drop). The resulting MN-Minitron-8B model outperforms the Llama 3.1 8B teacher on all common benchmarks while using 40Γ— fewer training tokens (380B vs. 15T), and the Llama-3.1-Minitron-4B-Width variant approaches the 8B teacher's accuracy using 150Γ— fewer tokens (94B vs. 15T), establishing that structured pruning with distillation can recover teacher-competitive performance on an unseen data distribution only when the teacher is first adapted to that distribution through correction.

2. Context and Motivation

The Problem: Compressing LLMs Without Access to Original Training Data

The fundamental problem this paper tackles is a practical but pervasive one: how do you compress a large language model into a smaller, more efficient variant when you do not have access to the dataset the large model was originally trained on?

This might seem like a niche constraint, but it is increasingly common for an important reason. Frontier LLM providers β€” including Meta (Llama 3.1), Mistral AI (Mistral NeMo), and many others β€” have shifted toward training their largest models on proprietary, private datasets that are never publicly released. The Llama 3.1 8B model, for example, was pretrained on 15 trillion tokens from a carefully curated mixture of public and internal data sources; the exact composition, preprocessing, and sampling ratios are undisclosed. Mistral NeMo 12B was trained on a similarly opaque data pipeline. When practitioners want to derive smaller models from these large open-weight checkpoints β€” for deployment on edge devices, in latency-sensitive applications, or under constrained compute budgets β€” they face a direct obstacle: the original data distribution that the large model internalized is unknown and inaccessible.

This creates a gap between what the model compression literature promises and what practitioners can actually achieve. Compression methods like structured pruning combined with knowledge distillation have been shown to produce strong small models at dramatically reduced training cost β€” but only under the assumption that distillation happens on the same data distribution the teacher was trained on. When that assumption breaks, the teacher provides suboptimal guidance, the student model underperforms, and the compression gains evaporate.

Why This Problem Matters

The significance of this problem extends in several directions:

Economic and environmental impact. Training large language models from scratch is extraordinarily expensive. The Llama 3.1 8B model required 15 trillion training tokens β€” a compute budget measured in millions of GPU-hours. Gemma 2 2B was trained on 2 trillion tokens. The Minitron approach demonstrated that smaller models in a family can be obtained using only ~380B distillation tokens, representing a 40Γ— reduction in training cost compared to training a same-sized model from scratch. But this cost saving is only realizable if the distillation process works effectively. If practitioners must retrain from scratch because they lack the original data, the economic argument for compression collapses β€” and every new model family developed from open-weight checkpoints would require massive, redundant training runs.

Democratization of model development. Open-weight models like Llama 3.1 have been transformative for the research community and smaller organizations, enabling fine-tuning, domain adaptation, and application development without the capital expenditure of pretraining. But the creation of model families at different scales remains concentrated among the few organizations that can afford multi-trillion-token pretraining runs. Effective compression methods that work without access to original training data would allow a much wider range of practitioners to produce efficient, deployment-ready models from available open-weight checkpoints, lowering the barrier to entry for creating customized model families.

Deployment constraints. The gap between a 12B-parameter model and an 8B-parameter model, or between 8B and 4B, has concrete operational consequences. As the paper demonstrates in its throughput analysis (Figure 10), moving from Llama 3.1 8B to a 4B depth-pruned variant yields a 2.7Γ— speedup in inference throughput on a single GPU. For applications requiring real-time responses, high query volumes, or resource-constrained hardware (mobile devices, edge servers, consumer GPUs), these throughput differences determine whether a model is deployable at all. If compression to smaller sizes requires retraining from scratch because the original data is unavailable, the deployment benefits are offset by prohibitive training costs.

Prior Approaches and Their Limitations

The paper builds on a lineage of work in structured pruning, knowledge distillation, and model compression that has evolved rapidly in the LLM era. Understanding the trajectory of this prior work clarifies which gaps remain.

The Original Minitron Recipe and Its Assumption

The most direct precursor is the Minitron approach described by Muralidharan et al. (2024). That work established a powerful pipeline: start with a large pretrained LLM, apply structured pruning (removing entire neurons, attention heads, layers, or embedding channels), and then retrain the pruned model using knowledge distillation from the original unpruned teacher. The key findings were that (1) structured pruning provides a dramatically better initialization than random weights for the smaller architecture, (2) distillation-based retraining recovers accuracy using far fewer tokens than training from scratch, and (3) the combination of pruning and distillation produces models competitive with same-sized models trained from scratch on the full pretraining corpus.

However, the original Minitron work β€” like much of the distillation literature β€” implicitly assumed that the distillation dataset matches the teacher's training distribution. The teacher model provides logit-level supervision that encodes its knowledge about token probabilities given context, and that knowledge is specific to the distribution it was trained on. If the distillation dataset differs substantially in sub-word token distribution, topic coverage, or data mixture, the teacher's logits become miscalibrated guidance β€” the teacher confidently predicts tokens that are appropriate for its original training distribution but suboptimal for the data the student actually sees.

This assumption is noted only tangentially in the original work because the experiments were conducted in settings where the original training data was available. The present paper confronts the case where it is not.

Distillation Under Distribution Shift

Knowledge distillation β€” originally proposed by Hinton et al. (2015) β€” transfers knowledge from a teacher model to a student by having the student mimic the teacher's output distribution, typically by minimizing the Kullback-Leibler divergence between teacher and student logits. In the standard formulation, the teacher and student see the same data, and the teacher's outputs serve as "soft targets" that encode richer information than hard labels alone (e.g., that a car is more similar to a truck than to a carrot, even if all three are distinct classes).

In the LLM domain, distillation has been widely used during pretraining (e.g., the original Minitron), during task-specific fine-tuning, and for domain adaptation. But across these settings, a common thread is that the teacher is assumed to be well-calibrated for the data being used as distillation input. When the teacher encounters out-of-distribution data, its logit distributions shift β€” often becoming overly confident on tokens that happened to co-occur in its training data but are inappropriate in the new context, or underconfident on tokens that are common in the new distribution but rare in the old one. This miscalibration propagates directly to the student through the distillation loss.

The paper does not cite specific prior work on distribution-shift-robust distillation for LLMs, which suggests this is a relatively underexplored area. Most distillation work assumes data availability from the teacher's training distribution, and the methods developed for domain adaptation through distillation typically assume the teacher is jointly fine-tuned or that the shift is intentional and structured. The gap the paper identifies is the case where the shift is unintentional β€” introduced by using a different dataset because the original one is unavailable β€” and where the practitioner needs a lightweight, general-purpose fix.

Structured Pruning for LLMs

The paper's pruning methodology draws on several lines of work in structured pruning:

Width pruning. Methods like SliceGPT (Ashkboos et al., 2023) and Sheared LLaMA (Xia et al., 2023) remove entire rows and columns from weight matrices, reducing hidden dimensions, MLP intermediate dimensions, and potentially attention heads. These methods compute importance scores β€” typically based on weight magnitudes, activation norms, or sensitivity to removal β€” and then trim the least important components. The original Minitron work refined this by using purely activation-based importance estimation (no gradient computation, only forward passes) and single-shot pruning (trimming all selected components at once rather than iteratively).

Depth pruning. Removing entire transformer layers is an orthogonal compression axis. Methods like ShortGPT (Men et al., 2024) and Shortened LLaMA (Kim et al., 2024) demonstrate that many layers in large LLMs are redundant and can be removed with minimal accuracy loss, particularly in deeper layers. The key design choice is the saliency criterion β€” how to decide which layers to drop. The dominant approaches have been Block Importance (BI), introduced by Men et al. (2024), which uses a linear approximation to estimate the impact of removing each layer on the loss, and direct validation loss evaluation, which removes each layer or layer block and measures the resulting perplexity increase. The original Minitron work found that validation loss-based importance outperformed BI, and that removing contiguous blocks of layers was more effective than removing individually-selected non-contiguous layers β€” a finding independently observed by Gromov et al. (2024) in "The Unreasonable Ineffectiveness of the Deeper Layers."

Where prior depth pruning falls short. A critical observation in the current paper is that validation loss is not a reliable proxy for downstream task performance when selecting layers to remove. Figure 8 shows that dropping 16 non-contiguous layers selected by validation loss minimization produces a better validation loss than dropping 16 contiguous layers (e.g., layers 16–31), but Figure 9 reveals the opposite pattern on downstream task accuracy: the contiguous removal dramatically outperforms the non-contiguous removal on Winogrande (59.5% vs. 50.0% accuracy). This disconnect β€” where a pruning decision that looks good by perplexity turns out to be poor by task accuracy β€” has not been systematically addressed in prior depth pruning work. The standard practice of using validation loss or its approximations (BI) as the saliency criterion implicitly assumes that perplexity improvements translate to task improvements, which this paper shows can fail.

Compressing Without Original Data: The Missing Piece

Across the prior work β€” the Minitron pipeline, knowledge distillation methods, and structured pruning techniques β€” there is a consistent gap: no established recipe for applying these techniques when the original pretraining data is unavailable. The individual components (pruning for initialization, distillation for recovery) are well-studied, but the practical problem of adapting the teacher to a new distillation dataset has not been systematically addressed. This is the gap the paper fills.

How This Paper Positions Itself

The paper positions itself as a practical extension of the Minitron compression paradigm, not a fundamental rethinking of it. The core pipeline β€” importance estimation, structured pruning, distillation-based retraining β€” is inherited directly from Muralidharan et al. (2024). The contributions are specifically about making that pipeline work in the realistic setting where:

  1. The original training data is unavailable, requiring the distillation to happen on a different dataset (in this case, the Nemotron-4 CT dataset). This motivates the teacher correction phase.

  2. Depth pruning decisions need to be informed by downstream task performance, not just validation loss, because the two can diverge in ways that matter for the final model's utility. This motivates the Winogrande-based layer selection criterion.

Importantly, the paper does not claim to have invented structured pruning or knowledge distillation. It does not claim that teacher correction is a theoretically novel form of adaptation β€” it is straightforward fine-tuning. The contributions are engineering insights backed by empirical validation: that fine-tuning the teacher on the distillation dataset before using it as a supervisor solves the distribution mismatch problem (demonstrated by the >6% reduction in LM validation loss attributed to teacher correction in the "Insights" section), and that Winogrande accuracy is a better saliency signal than perplexity for depth pruning when downstream task performance matters.

The paper also explicitly positions itself in the lineage of model family creation through compression rather than from-scratch training. The framing in the Introduction draws a direct contrast: "LLM providers often train an entire family of models from scratch... However, training multiple billion-plus parameter models from scratch is extremely time-, data- and resource-intensive." The Minitron approach β€” and this paper's extension of it β€” offers an alternative where only the largest model in a family is trained from scratch, and smaller models are derived through pruning and distillation. The teacher correction phase extends this paradigm to the case where the largest model is developed externally (e.g., by Meta or Mistral) and made available as an open-weight checkpoint, but the training data is not released.

Finally, the paper positions its results as state-of-the-art in the 4B-8B parameter class for models produced through compression. Table 1 systematically compares against Gemma 2 (2B), the original Minitron 4B, Gemma 7B, Mistral 7B, and Llama 3.1 8B, demonstrating that MN-Minitron-8B matches or exceeds all similarly-sized models despite being derived from a larger teacher using far fewer training tokens. Table 2 extends this to instruction-tuned variants. The key competitive claim is not that compression produces better models than pretraining from scratch at the same size β€” the comparisons to Llama 3.1 8B (pretrained on 15T tokens) show competitive but not universally superior results β€” but rather that compression with teacher correction achieves comparable accuracy at a fraction of the training cost, making it the pragmatic choice when time, budget, or data access constrain the alternatives.

3. Technical Approach

3.1 Reader Orientation

This paper presents a practical recipeβ€”not a single algorithm but a pipeline of three sequential stepsβ€”for taking a large, publicly available pretrained language model (like Llama 3.1 8B or Mistral NeMo 12B) and producing a smaller, faster model at roughly half the parameters (4B or 8B respectively) without needing access to the original training data. The problem it solves is the distribution mismatch that breaks standard compression techniques when the original training corpus is unavailable: the core insight is that a lightweight "teacher correction" fine-tuning phase (~100B tokens) adapts the large model to whatever distillation dataset is available before it serves as a supervisor, restoring the effectiveness of the established pruning-and-distillation pipeline and enabling the student model to recover competitive accuracy using 40–150Γ— fewer training tokens than pretraining from scratch.

3.2 Big-Picture Architecture (Diagram in Words)

The system has three sequential stages, each producing a specific artifact that feeds into the next. The flow is visualized in Figure 1 and detailed in Figure 2:

  1. Teacher Correction Phase (input: pretrained base model + distillation dataset β†’ output: "corrected teacher"). The large pretrained model is fine-tuned for ~100B tokens on the Nemotron-4 CT dataset using a low learning rate. This adapts its internal token distribution to match the dataset that will be used for distillation, solving the distribution mismatch that would otherwise produce suboptimal supervision.

  2. Pruning Phase (input: corrected teacher + importance calibration data β†’ output: structurally pruned model). Activation-based importance scores are computed for every structural component (layers, attention heads, MLP neurons, embedding channels) using only forward passes on a small calibration set (1024 samples). Components below a threshold are physically removed from the weight matrices, producing a smaller architecture that inherits the teacher's remaining weights as initialization.

  3. Distillation Phase (input: pruned student + corrected teacher + distillation dataset β†’ output: final compressed model). The pruned model is trained to match the corrected teacher's output distribution by minimizing the forward KL divergence between their logit distributions. This recovers the accuracy lost during pruning using far fewer tokens than training from scratch (94B for the 4B models, 380B for the 8B model).

For depth pruning specifically, there is a layer selection sub-process between the teacher correction and pruning steps: the corrected teacher is evaluated on the Winogrande task with various contiguous layer blocks removed, and the block whose removal causes the least accuracy drop is selected for deletion. This replaces the standard validation-loss-based selection criterion.

3.3 Roadmap for the Deep Dive

  • First, the teacher correction mechanism β€” why distribution mismatch breaks distillation, exactly what fine-tuning is performed, and the empirical evidence (Figure 4) that it works. This is the paper's primary methodological contribution and the foundation on which the rest of the pipeline depends.

  • Second, the importance estimation procedure β€” how activation-based scores are computed for every structural axis (heads, neurons, embedding channels, layers) using only forward passes and a small calibration set. This is the shared machinery that enables both width and depth pruning.

  • Third, width pruning β€” the specific recipe applied to reduce hidden dimensions, MLP intermediate dimensions, and embedding channels while retaining attention heads, producing the Llama-3.1-Minitron-4B-Width and MN-Minitron-8B architectures.

  • Fourth, depth pruning and the Winogrande saliency criterion β€” the novel task-based layer selection metric, the empirical justification for why validation loss fails (Figures 8 and 9), and the contiguous-layer removal strategy inherited from Gromov et al. (2024). This produces the Llama-3.1-Minitron-4B-Depth architecture.

  • Fifth, the distillation retraining procedure β€” the logit-only forward KL divergence objective, the hyperparameter configuration (Table 4), and the design choices (why no cross-entropy loss, why the corrected teacher provides supervision).

  • Sixth, the parallel teacher correction variant β€” the finding that teacher correction can be performed concurrently with distillation (Figure 5), which is a practical optimization for reducing end-to-end wall-clock time.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical engineering paper whose core idea is that a lightweight fine-tuning phase before pruning and distillation solves the distribution mismatch problem that otherwise cripples compression when the original training data is unavailable. The pipeline inherits its structure from the original Minitron work (Muralidharan et al., 2024) and adds two practical modifications: a teacher correction step and a downstream-task-based depth pruning criterion.

Teacher Correction: Solving the Distribution Mismatch

The problem: why uncorrected teachers fail. Knowledge distillation operates by having the student model mimic the teacher's output distribution β€” specifically, the probability vectors produced by the teacher's final softmax layer for each input token. These probability vectors encode the teacher's learned knowledge about which tokens are plausible continuations given the context. However, this encoding is distribution-specific: the teacher's logit values and the resulting probability assignments are calibrated for the token distribution it encountered during pretraining.

When the distillation dataset differs from the pretraining dataset, the teacher encounters tokens and token sequences that are out-of-distribution relative to its training experience. The consequences are two-fold. First, the teacher may assign inappropriately high probabilities to tokens that were frequent in its training data but are rare or contextually inappropriate in the distillation data. Second, the teacher may assign inappropriately low probabilities to tokens that are common in the distillation data but rare in its training data. These miscalibrated probabilities propagate directly to the student through the distillation loss, producing suboptimal supervision. The paper reports that using an uncorrected teacher results in substantially higher LM validation loss during student training compared to using a corrected teacher (Figure 4), quantified as "over a 6% reduction in LM validation loss" attributed to teacher correction in the Insights section.

The hypothesis. The paper hypothesizes that the failure is driven by "the change in distribution of sub-word tokens across the original dataset the teacher model was trained on vs. the dataset being distilled on." This is not about high-level semantic differences (e.g., the teacher was trained on news articles and now sees code). It is about the statistical properties of the token sequences β€” which tokens co-occur, what the base rates of different tokens are, and how the teacher's internal representations map to output probabilities for these tokens. Even if the distillation dataset covers broadly similar topics to the pretraining data, differences in tokenization, formatting conventions, or sampling mix can produce meaningful distribution shifts at the sub-word level.

The correction procedure. Teacher correction is a straightforward fine-tuning step applied to the pretrained base model before any pruning occurs. The procedure is:

  1. Take the pretrained base model (e.g., Mistral NeMo 12B or Llama 3.1 8B as downloaded from Hugging Face).
  2. Fine-tune it on the Nemotron-4 CT dataset for approximately 100B tokens.
  3. Use a low learning rate to avoid catastrophic forgetting while allowing the model's internal token statistics to adapt to the new distribution. The specific recipe: use "one-fifth the peak learning rate, identical batch size, minimum learning rate and decay schedule the original model was trained on."
  4. Use 120 warm-up steps before reaching the peak learning rate.

The resulting model is called the corrected teacher. It serves as the supervision source for all subsequent distillation steps. The correction is applied to the full, unpruned model β€” the pruning step happens afterward, on the corrected teacher.

Why fine-tune before pruning, not after? The paper explores an alternative where the original (uncorrected) teacher is pruned first, and the corrected teacher is obtained in parallel with distillation β€” essentially fine-tuning the teacher on the distillation data while simultaneously using its evolving logits to supervise the student. The results in Figure 5 show that this parallel approach "performs on par with using guidance from a fully corrected teacher." This has practical implications: teacher correction and distillation can be overlapped in time, reducing the end-to-end pipeline latency. However, the paper's main results all use the sequential approach (correct the full teacher first, then prune, then distill) for simplicity.

What teacher correction does and does not do. The correction phase does not substantially change the teacher's accuracy on downstream benchmarks. Table 1 shows that the Mistral NeMo 12B-FT (the corrected teacher) has slightly different accuracies compared to the base Mistral NeMo 12B β€” some tasks improve (e.g., GSM8K goes from 56.4% to 55.7%? No β€” examining Table 1 carefully, the 12B-Base column shows GSM8K at 56.4%, and the 12B-FT column shows 55.7%, a slight decrease; meanwhile HumanEval drops from 23.8% to 23.8%, which is unchanged, and MMLU goes from 69.0% to 70.1%, an increase). The paper notes explicitly: "the correction process has a minor effect on the teacher model's accuracy on downstream tasks, with some tasks improving and some degrading" and hypothesizes "this to be an artifact of the dataset used for fine-tuning."

This is an important point: teacher correction is not improving the teacher β€” it is realigning the teacher's output distribution to match the distillation data. The teacher's fundamental knowledge and reasoning capabilities are largely preserved (the downstream accuracy changes are small and mixed in direction), but its logit distributions become better calibrated for the specific token sequences the student will see during distillation. This is why the technique works despite not producing a "better" teacher by conventional accuracy metrics.

Design choice: lightweight fine-tuning rather than full continued pretraining. The correction uses only ~100B tokens, which is a tiny fraction of the original pretraining budget (15T tokens for Llama 3.1 8B, implying a ratio of approximately 1:150). The paper notes that exploring even lighter approaches β€” such as using LoRA adapters or tuning only LayerNorm parameters β€” is left to future work. The 100B token budget was chosen empirically; the paper does not present an ablation over different correction budgets, so it is unclear whether fewer tokens would suffice.

Empirical validation. The paper's central evidence for teacher correction's effectiveness is Figure 4, which plots LM validation loss against training tokens for the MN-Minitron-8B student during distillation. Two curves are shown: one where the teacher is the original Mistral NeMo 12B (uncorrected), and one where the teacher is the fine-tuned Mistral NeMo 12B (corrected). The corrected-teacher curve is consistently and substantially lower throughout training, with a final gap quantified as "over a 6% reduction in LM validation loss." This is the figure that establishes teacher correction as necessary rather than optional when distilling on a new dataset.

Importance Estimation: The Shared Scoring Machinery

Before any pruning can occur, the system must decide which components to remove. This requires computing an importance score for every structural element that could potentially be pruned: individual layers, attention heads, MLP neurons (individual channels in the intermediate dimension of the feed-forward networks), and embedding channels (individual dimensions in the hidden representations throughout the model).

The paper inherits the purely activation-based importance estimation strategy from the original Minitron work. The key properties of this approach are:

Forward-pass only, no gradients. Unlike some pruning methods that use gradient-based sensitivity (computing how much the loss would change if a component were removed, requiring backward passes), this approach requires only forward propagation. A small calibration dataset of 1024 samples (drawn randomly from the full Nemotron-4 CT dataset) is passed through the model, and the activations produced by each component are recorded. Importance is then computed from the statistical properties of these activations.

Batch and sequence aggregation. For each structural component, activations are collected across all samples in the calibration set (batch dimension) and all token positions in each sample (sequence dimension). These are aggregated to produce a single scalar importance score per component. The specific aggregation functions are: l2-norm (Euclidean norm) across the batch dimension, and mean across the sequence dimension. This means that for a given attention head, the system computes the L2 norm of that head's output activations for each calibration sample, then averages those norms across all token positions, producing a single number representing the head's typical activation magnitude.

Simultaneous multi-axis estimation. The same forward passes compute importance for all axes at once β€” layers, heads, neurons, and embedding channels β€” eliminating the need for separate calibration runs. The specific activation sources for each axis are:

  • Attention heads: activations produced by each head in the multi-head attention (MHA) layer. Each head produces a slice of the full attention output, and its importance is the magnitude of that slice's activations.

  • MLP neurons: activations produced by individual channels in the intermediate (hidden) dimension of the MLP blocks. In a standard transformer MLP with hidden dimension d_ff, there are d_ff neurons, each corresponding to one position in the intermediate activation vector.

  • Embedding channels: activations at individual dimensions of the LayerNorm output. Since LayerNorm operates element-wise along the hidden dimension, each of the d_model dimensions has an associated activation magnitude that can be ranked.

  • Layers: for depth pruning, importance is not computed from activations but through a separate removal-and-evaluate procedure (described in the depth pruning section below). This is explicitly treated as a special case that "we do not combine [depth pruning] with compressing other dimensions."

Single-shot pruning, not iterative. Once importance scores are computed, all components below the threshold for the target architecture are removed simultaneously. The paper does not use iterative pruning (remove a few components, retrain, recompute importance, repeat), following the original Minitron finding that single-shot pruning is sufficient and more efficient.

Why activation magnitude? The underlying assumption is that components producing larger-magnitude activations (as measured by L2 norm) are more "important" to the model's computation β€” they are contributing more signal to subsequent layers. Components with consistently small activations are presumed to be redundant or minimally influential. This is a heuristic, not a theoretically justified importance measure, but the original Minitron work demonstrated it works well in practice for LLMs. The alternatives β€” weight magnitude pruning (removing small weights) or gradient-based sensitivity β€” either require gradient computation (adding cost) or are less effective for structured pruning of large transformers.

Calibration dataset size. The paper uses 1024 samples, which is a small fraction of the full distillation dataset. This keeps the importance estimation cost negligible relative to the subsequent distillation training. The paper does not ablate over calibration set size, so the sensitivity of the pruning decisions to this choice is unknown.

Width Pruning: Compressing Hidden, MLP, and Embedding Dimensions

Width pruning targets three architectural axes that control the model's representational capacity at each layer: the hidden dimension (the size of the residual stream and the output of attention and MLP blocks), the MLP intermediate dimension (the expansion factor in feed-forward layers), and the embedding dimension (which is typically tied to the hidden dimension but can be trimmed in the embedding lookup table and output projection).

The width pruning recipe for each compressed model is specified in the "Pruning" subsection of Section 2 and summarized in Table 3:

Llama-3.1-Minitron-4B-Width (from Llama 3.1 8B):

  • Hidden dimension: 4096 β†’ 3072 (25% reduction)
  • MLP hidden dimension: 14336 β†’ 9216 (36% reduction)
  • Attention heads: 32 (unchanged)
  • Depth: 32 layers (unchanged)
  • Vocabulary: 128256 (unchanged)
  • Total parameters: 4.5B, non-embedding parameters: 3.7B

MN-Minitron-8B (from Mistral NeMo 12B):

  • Hidden dimension: 5120 β†’ 4096 (20% reduction)
  • MLP hidden dimension: 14336 β†’ 11520 (20% reduction)
  • Attention heads: unchanged
  • Depth: 40 layers (unchanged)
  • Vocabulary: 131072 (unchanged)
  • Total parameters: 8.4B, non-embedding parameters: 7.3B

The pruning procedure for each axis is:

  1. Compute importance for all heads, neurons, and embedding channels using the activation-based method described above.

  2. Rank the components on each axis by importance score, producing sorted lists from most to least important.

  3. Trim the weight matrices directly by removing rows and columns corresponding to the lowest-ranked components. For neuron pruning, this means removing entire rows from the MLP weight matrices (the W_up, W_down, and W_gate projections in a gated MLP architecture). For embedding channel pruning, the hidden dimension is reduced throughout the model β€” in the attention projections, the MLP projections, the LayerNorm parameters, and the embedding matrix β€” by removing the least important channels.

  4. No architectural search. The original Minitron work used Neural Architecture Search (NAS) to determine the optimal allocation of the parameter budget across axes (e.g., how much to prune from hidden vs. MLP vs. heads). This paper skips NAS entirely, instead using manual architecture configurations "inspired by the Minitron-4B and Minitron-8B models." This is a pragmatic simplification that trades some potential accuracy for simplicity and reduced compute cost.

Key design choice: retain all attention heads. Both width-pruned models keep the full complement of attention heads from the teacher (32 heads for both Llama 3.1 8B and Mistral NeMo 12B). The paper states explicitly: "For width pruning, we achieve stronger accuracy by retaining attention heads and pruning the other dimensions (MLP intermediate dimension, embedding channels)." This is a non-obvious finding: attention heads are typically considered prime candidates for pruning because multi-head attention has known redundancy. The paper's empirical observation that pruning MLP and hidden dimensions while preserving heads yields better results suggests that for these model sizes and this compression ratio, the attention mechanism is more parameter-efficient than the feed-forward layers. This aligns with broader observations in the literature that LLM feed-forward layers contain substantial redundancy (e.g., the success of mixture-of-experts architectures that expand MLP capacity).

The trimmed model as initialization. After pruning, the remaining weights are preserved exactly as they were in the corrected teacher β€” no random reinitialization, no added noise. This is the critical property that makes pruning effective: the pruned model starts from a set of weights that already encode meaningful representations, rather than from random initialization. The LM validation loss immediately after pruning (before any distillation) is substantially lower than what a randomly initialized model of the same architecture would achieve. Figure 6 demonstrates this clearly: the "Pruning + Distillation" curve starts at a much lower initial loss than the "Random Init + Distillation" curve.

Depth Pruning and the Winogrande Saliency Criterion

Depth pruning removes entire transformer layers, reducing the total number of sequential processing steps. For Llama-3.1-Minitron-4B-Depth, 16 of the original 32 layers are removed (Table 3), halving the model depth while keeping all other dimensions unchanged: hidden dimension remains 4096, MLP hidden dimension 14336, attention heads 32, vocabulary 128256. The total parameter count is 4.5B with 3.5B non-embedding parameters (slightly fewer non-embedding parameters than the width-pruned variant, 3.5B vs. 3.7B, because depth pruning removes fewer embedding-associated parameters).

The problem with validation loss for layer selection. The standard approach in prior depth pruning work (ShortGPT, Shortened LLaMA) is to select layers for removal based on their impact on validation perplexity: remove each layer (or layer block), measure how much the LM loss increases, and remove the layers that cause the smallest increase. The intuition is straightforward β€” layers whose removal minimally degrades next-token prediction are the most redundant.

However, the paper presents a counterexample where this approach fails. The evidence is in Figures 8 and 9, which show two different rankings for the same set of 16 layers to remove from Llama 3.1 8B:

  • By validation loss (Figure 8): Removing 16 non-contiguous layers that individually cause the smallest loss increase produces a lower (better) validation loss than removing layers 16–31 contiguously. The dashed line in Figure 8 (non-contiguous removal) sits below the validation loss values for most contiguous removal windows. This would lead a loss-based criterion to prefer the non-contiguous selection.

  • By downstream task accuracy (Figure 9): On Winogrande, removing layers 16–31 contiguously yields 59.5% accuracy, while removing the 16 layers selected by the loss-based criterion yields only 50.0% β€” essentially random chance for this 5-shot task. The contiguous removal dramatically outperforms the non-contiguous selection on actual task performance.

This disconnect is the paper's justification for using a task-based metric rather than perplexity for depth pruning decisions. The layers that matter least for next-token prediction on a generic validation set are not necessarily the layers that matter least for the reasoning and knowledge tasks that downstream users care about. Perplexity optimizes for local token-level coherence, while task accuracy captures whether the model's aggregate computation produces correct outputs β€” and these can diverge when pruning decisions affect the model's ability to perform multi-step reasoning.

The Winogrande-based selection procedure. The paper selects layers for removal through a direct evaluation approach:

  1. For each candidate contiguous block of 16 layers (e.g., layers 1–16, layers 2–17, ..., layers 17–32), physically remove those layers from the corrected teacher model.

  2. Evaluate the resulting pruned model on the Winogrande benchmark (5-shot) without any retraining.

  3. Select the contiguous block whose removal produces the highest Winogrande accuracy.

  4. Remove those layers permanently, producing the pruned architecture for distillation.

The paper chooses Winogrande specifically because it is a commonsense reasoning task that requires the model to resolve pronoun references using world knowledge β€” it tests a combination of linguistic understanding and reasoning that is broadly representative of the downstream capabilities users care about. The paper does not provide an ablation comparing Winogrande to other possible task metrics (e.g., MMLU, HellaSwag), so it is unclear how sensitive the selection is to the choice of evaluation task.

Contiguous vs. non-contiguous removal. The paper explicitly adopts the contiguous removal strategy from Gromov et al. (2024) ("The Unreasonable Ineffectiveness of the Deeper Layers"), which observed that removing contiguous blocks of layers is more effective than removing individually-selected non-contiguous layers. The paper confirms this finding: "dropping contiguous layers is better than individual, as also observed in Gromov et al." The hypothesis, drawn from that prior work, is that contiguous removal preserves the functional specialization of different depth ranges β€” early layers handle local syntactic processing, middle layers handle semantic composition, and late layers handle task-specific output formatting. Removing layers scattered throughout the depth stack disrupts all of these stages simultaneously, while removing a contiguous block preserves the intact processing pipelines for the remaining depth ranges.

The specific contiguous block chosen for Llama 3.1 8B is layers 16–31. This means the pruned model retains layers 1–15 and layer 32, discarding the entire upper-middle and late-middle section of the network. The paper notes in Figure 8 that "the layers at the beginning and end are the most important" based on validation loss patterns, which aligns with prior findings that the first and last layers of transformers tend to be more critical than middle layers.

Why not use the Block Importance metric? The paper explicitly rejects BI (Block Importance) from Men et al. (2024), stating it "was recently shown to under-perform the validation loss/PPL metric" in the original Minitron work. BI uses a linear approximation (a first-order Taylor expansion) to estimate the impact of layer removal on the loss, avoiding the need to actually remove and evaluate each candidate. This is computationally cheaper than direct evaluation but less accurate in practice.

Depth pruning and throughput. A key practical advantage of depth pruning over width pruning is inference speed. Figure 10 shows that the depth-pruned Llama-3.1-Minitron-4B achieves a 2.7Γ— throughput improvement over Llama 3.1 8B, while the width-pruned variant achieves only 1.8Γ—. This is because depth pruning reduces the sequential computation chain β€” fewer layers means each token passes through fewer transformer blocks, directly reducing latency. Width pruning, by contrast, reduces the per-layer computation but not the number of sequential steps, yielding more modest speedups. This tradeoff (better accuracy from width pruning vs. better throughput from depth pruning) is one of the paper's practical takeaways.

Distillation Retraining: Recovering Accuracy After Pruning

After pruning (whether width or depth), the model has the target parameter count but substantially degraded accuracy β€” the pruning process has removed structural components that the remaining weights depended on. The distillation phase trains the pruned model to recover this lost accuracy, using the corrected teacher's output distribution as the training target.

Training objective: logit-only forward KL divergence. The paper uses only distillation loss; the standard language modeling cross-entropy loss against ground-truth next tokens is omitted entirely. The objective is:

Ldistill=DKL(pteacherβˆ₯pstudent)\mathcal{L}_{\text{distill}} = D_{\text{KL}}(p_{\text{teacher}} \parallel p_{\text{student}})

where pteacherp_{\text{teacher}} and pstudentp_{\text{student}} are the full probability distributions over the vocabulary produced by the teacher and student respectively, for every token position in every training sequence. The KL divergence from the teacher distribution to the student distribution is computed as:

DKL(pteacherβˆ₯pstudent)=βˆ‘v∈Vpteacher(v)log⁑pteacher(v)pstudent(v)D_{\text{KL}}(p_{\text{teacher}} \parallel p_{\text{student}}) = \sum_{v \in V} p_{\text{teacher}}(v) \log\frac{p_{\text{teacher}}(v)}{p_{\text{student}}(v)}

where VV is the vocabulary (128,256 tokens for Llama, 131,072 for Mistral NeMo) and vv indexes individual tokens.

What this computes operationally: For each input token in the training sequence, the teacher and student both produce logit vectors of dimension ∣V∣|V|, which are passed through softmax to produce probability distributions. The KL divergence measures how many extra bits (or nats, depending on the log base) are needed to encode samples from the teacher distribution using a code optimized for the student distribution. Minimizing this divergence forces the student to assign similar probabilities to every token in the vocabulary β€” not just the most likely token β€” as the teacher. The student learns not just what token is correct, but the teacher's full relative preferences among all possible tokens.

Why forward KL and not reverse KL? Forward KL (DKL(pteacherβˆ₯pstudent)D_{\text{KL}}(p_{\text{teacher}} \parallel p_{\text{student}})) penalizes the student heavily when the teacher assigns high probability to a token that the student assigns low probability β€” it forces the student to cover all modes of the teacher's distribution. Reverse KL (DKL(pstudentβˆ₯pteacher)D_{\text{KL}}(p_{\text{student}} \parallel p_{\text{teacher}})) penalizes the student when it assigns high probability to tokens the teacher assigns low probability β€” it allows the student to ignore some modes of the teacher's distribution (mode-dropping). For distillation, forward KL is preferred because the goal is to capture the full breadth of the teacher's knowledge, not to find a single sharp mode. The paper explicitly states this choice follows "the best practices outlined in the Minitron work."

Why no ground-truth cross-entropy loss? Standard distillation recipes often combine the KL divergence loss with the standard next-token prediction loss (cross-entropy with ground-truth labels), using a mixing coefficient to balance them. The paper's decision to omit cross-entropy entirely follows the original Minitron finding that logit-only distillation is sufficient and that adding the LM loss does not improve results for this pipeline. The paper provides an interesting piece of evidence for this: "our compressed model outperforms the teacher on two benchmarks, GSM8k and HumanEval after pruning and distillation... However, retraining is performed using the distillation loss alone." This means the student achieves higher accuracy than the teacher on GSM8K (58.5% vs. 55.7%) and HumanEval (36.2% vs. 23.8%) despite never being trained on ground-truth answers β€” it learned solely from the teacher's output distributions. This is possible because the distillation process, combined with the fact that the teacher correction adapted the teacher to a dataset that may be higher-quality or more focused than the teacher's original training data, can produce a student that is better calibrated for certain tasks than the teacher.

Training hyperparameters (Table 4). The distillation configuration is:

  • Peak learning rate: $1 \times 10^{-4}$ for both Llama-3.1-Minitron and MN-Minitron
  • Minimum learning rate: $1 \times 10^{-5}$ for Llama, $4.5 \times 10^{-7}$ for MN-Minitron (the MN-Minitron model uses a wider decay range)
  • Warm-up steps: 40 steps for Llama, 60 steps for MN-Minitron
  • Learning rate decay schedule: Cosine decay from peak to minimum
  • Global batch size: 1152 for Llama, 768 for MN-Minitron
  • Context length: 8192 tokens for both
  • Total training tokens: 94B for the 4B models, 380B for the 8B model
  • Hardware: 32 NVIDIA DGX H100 nodes (each DGX H100 contains 8 H100 GPUs, so approximately 256 GPUs total)

The 40–150Γ— token reduction. The paper emphasizes that these distillation token budgets are dramatically smaller than pretraining from scratch. Llama 3.1 8B was pretrained on 15T tokens; MN-Minitron-8B uses 380B tokens for distillation, a 40Γ— reduction (15T / 380B β‰ˆ 39.5). Llama-3.1-Minitron-4B uses 94B tokens, a 160Γ— reduction (15T / 94B β‰ˆ 159.6). These comparisons are between the distillation budget and the teacher's original pretraining budget, not between distillation and what it would cost to pretrain a 4B or 8B model from scratch β€” the paper does not estimate from-scratch pretraining costs for the smaller architectures, though they would presumably be somewhat less than the larger teacher's 15T tokens. The paper states "up to 40Γ—" and "150Γ—" in the evaluation section, referring to the comparison against the teacher's 15T tokens.

The order of operations in distillation. During distillation, each training step proceeds as follows:

  1. A batch of token sequences is drawn from the Nemotron-4 CT dataset.
  2. The full corrected teacher model runs a forward pass on this batch, producing logit vectors for every token position. These teacher logits are computed once and can be cached if memory permits, since the teacher is frozen during distillation.
  3. The pruned student model runs a forward pass on the same batch.
  4. Both logit sets are passed through softmax to produce probability distributions.
  5. The forward KL divergence is computed between teacher and student probabilities at each token position.
  6. The gradient of this loss with respect to the student's parameters is computed via backpropagation, and the student's parameters are updated.

This is illustrated in Figure 2, which shows the teacher and student transformer layers running in parallel, with the KL divergence loss computed between their respective LM head outputs. The embedding layer and LM head are shown as shared or structurally parallel β€” in practice, the student has its own (smaller) embedding and LM head after width pruning, or identical ones after depth pruning.

Parallel Teacher Correction: An Operational Optimization

In addition to the sequential pipeline (correct teacher β†’ prune β†’ distill), the paper explores an alternative where teacher correction happens concurrently with student distillation. The configuration is:

  1. Take the original (uncorrected) pretrained model.
  2. Prune it using the importance estimation procedure to produce the student architecture.
  3. Begin distillation, using the original pretrained model as the initial teacher.
  4. Simultaneously, continue fine-tuning the original pretrained model on the distillation dataset (teacher correction).
  5. At each distillation step, use the most recent version of the correcting teacher to provide logit supervision.

Figure 5 compares this approach against the sequential (prune corrected teacher + distill corrected teacher) approach, showing that the two produce nearly identical training curves. The paper concludes that "teacher correction can be performed in parallel with distillation to recover accuracy of the pruned student model."

Why this matters. In the sequential approach, the total wall-clock time is the sum of teacher correction time plus distillation time. In the parallel approach, the teacher correction overlaps with distillation, so the wall-clock time is approximately the maximum of the two (which is likely the distillation phase, since it runs for more tokens in the 8B case: 380B distillation tokens vs. ~100B correction tokens). For practitioners with limited compute windows, this parallelization can substantially reduce time-to-deployment.

The logit distribution during parallel correction. As the teacher continues to be fine-tuned during distillation, its logit distributions evolve. The student is therefore trained on a moving target β€” the teacher at step $t$ of distillation is slightly different from the teacher at step $t-1$. The fact that this does not degrade final performance (per Figure 5) suggests that the teacher's corrections converge quickly relative to the student's learning rate, or that the student is robust to this level of target non-stationarity. The paper does not analyze this dynamic in detail.

Design Choices: Summary and Justifications

Teacher correction over adapting the student directly. An alternative to correcting the teacher would be to train the student with a combination of the uncorrected teacher's distillation loss and some ground-truth cross-entropy loss to compensate for the teacher's miscalibration. The paper does not ablate this alternative, so the relative efficacy is unknown. The teacher correction approach has the advantage of conceptual simplicity: fix the supervision source rather than trying to work around its flaws.

Activation-based importance over gradient-based or weight-based. The purely forward-pass activation method requires no backward passes through the full model, which matters because the models being pruned (8B–12B parameters) are large enough that gradient computation is a meaningful cost. The calibration set of 1024 samples can be processed with a single forward pass per sample, making the total cost negligible compared to the subsequent distillation phase (~100B–380B tokens of training).

Single-shot pruning over iterative. Iterative pruning (remove a few components, retrain slightly, recompute importance) can theoretically produce better importance estimates because it accounts for interactions between components β€” removing one head changes the importance of the remaining heads. However, iterative approaches multiply the number of pruning-and-retraining cycles, which is expensive when each retraining phase involves large-scale distillation. The paper's single-shot approach accepts potentially suboptimal component selection in exchange for dramatically simpler logistics.

Winogrande for depth pruning over validation loss. This is the paper's most novel design choice and the one with the strongest empirical justification (the direct comparison in Figures 8 and 9). The choice reflects a pragmatic recognition that users of compressed models care about task performance, not perplexity, and that the two can diverge for structural pruning decisions. The limitation is that Winogrande represents only one type of downstream capability (commonsense reasoning), and it is possible that tasks requiring different cognitive skills (mathematical reasoning, code generation, factual recall) would suggest different optimal layer removal patterns.

Logit-only distillation over hybrid objectives. The omission of cross-entropy loss is empirically justified by the original Minitron work and validated by the strong downstream results, including cases where the student outperforms the teacher. A potential explanation is that the corrected teacher's output distribution already encodes a better training signal than the hard ground-truth labels for the distillation dataset, either because the teacher generalizes beyond exact next-token prediction or because the distillation dataset's token-level labels are noisier than the teacher's smoothed probability estimates. The paper does not investigate this mechanism directly.

Manual architecture over NAS. The original Minitron work used NAS to determine the optimal per-axis pruning ratios (how much to prune from hidden vs. MLP vs. heads vs. layers). This paper skips NAS entirely and manually specifies architectures. This is a tradeoff of potential accuracy (NAS might find a slightly better allocation of the parameter budget) for simplicity and reproducibility. The paper does not quantify what accuracy is left on the table by this choice.

Retaining attention heads during width pruning. The observation that preserving all attention heads while aggressively pruning MLP and hidden dimensions yields better results is reported as an empirical finding rather than a design choice motivated by prior theory. The paper provides no mechanistic explanation; one possibility is that at the 4B–8B scale, the attention mechanism's parameter count (which scales as $4 \times d_{\text{model}}^2$ for Q, K, V, and O projections) is already near the minimum needed for the number of heads used, while the MLP's intermediate dimension (typically $8 \times d_{\text{model}}$ in Llama architectures) contains substantial redundancy that can be trimmed without degrading the attention computation's ability to route information.

4. Key Insights and Innovations

Innovation 1: Teacher Correction Reframes Distribution Mismatch from an Obstacle to a Solvable Preprocessing Step

The dominant assumption in the knowledge distillation literatureβ€”implicit in the original Minitron work and widespread in practiceβ€”is that distillation works best (or only works well) when the teacher supervises the student on data drawn from the teacher's own training distribution. When practitioners lacked access to the original pretraining data, the default response was either to abandon distillation in favor of conventional fine-tuning from scratch, or to accept degraded results from a miscalibrated teacher providing suboptimal soft targets.

This paper fundamentally reframes that obstacle. Rather than treating distribution mismatch as a reason distillation fails, it treats it as a preprocessing problem: the teacher itself can be adapted to the new data distribution through a lightweight fine-tuning phase before distillation begins. The conceptual move is subtle but importantβ€”teacher correction separates the problem of distribution alignment from the problem of knowledge transfer. The teacher is first made competent on the target data, and only then is asked to supervise.

What makes this more than a trivial "just fine-tune the teacher" observation is the demonstration that this correction phase need not improve the teacher's downstream accuracy to be effective. The corrected Mistral NeMo 12B shows mixed and minor accuracy changes compared to the base model (some tasks improve slightly, others degrade) yet provides substantially better supervision, as evidenced by the >6% reduction in student LM validation loss in Figure 4. The mechanism is not that correction makes the teacher smarterβ€”it is that correction realigns the teacher's output probabilities to be better calibrated for the specific token sequences the student will encounter during distillation. This distinction between accuracy and calibration-for-distillation as separate properties of a teacher model is a conceptual contribution that the paper demonstrates empirically but does not explicitly name; it is the key to understanding why teacher correction works.

Significance beyond the pipeline: This reframing has implications beyond the specific Minitron recipe. Any scenario where a pretrained model is used as a distillation teacher on data it wasn't trained onβ€”domain-specific compression, cross-lingual distillation, distillation from API-based teachers whose training data is unknownβ€”can potentially benefit from a teacher correction phase. The paper establishes that the correction can even happen in parallel with distillation (Figure 5), meaning the total wall-clock time need not increase. This transforms teacher correction from an expensive prerequisite into an optional optimization that trades some implementation complexity for faster deployment.

Prior work contrast: The original Minitron work (Muralidharan et al., 2024) assumed data availability without addressing the alternative. The broader distillation literatureβ€”from Hinton et al. (2015) through LLM-scale applicationsβ€”has focused on distillation objectives, student-teacher architecture matching, and training efficiency, but has not systematically studied what happens when the distillation data distribution diverges from the teacher's training distribution. This paper's teacher correction phase fills that gap with a simple, empirically validated solution.

Qualification: This is an engineering innovationβ€”a practical fix to a practical problemβ€”rather than a theoretical advance. Teacher correction is standard fine-tuning; what is novel is the diagnosis of why uncorrected teachers fail (miscalibrated logits due to sub-word token distribution shift) and the demonstration that a lightweight fix (100B tokens, low learning rate) suffices. The paper does not provide a theoretical framework for predicting when teacher correction will be necessary or how much correction is enough, leaving those as open questions.

Innovation 2: Task-Based Saliency for Depth Pruning Exposes a Failure Mode of Perplexity-Driven Layer Selection

Prior work on depth pruning for LLMsβ€”including ShortGPT (Men et al., 2024), Shortened LLaMA (Kim et al., 2024), and the original Minitronβ€”universally used validation perplexity or its approximations (e.g., Block Importance) as the criterion for selecting which layers to remove. The logic was straightforward: layers whose removal minimally increases next-token prediction loss are the most redundant and should be pruned first.

This paper provides the first clear empirical demonstration that this logic can fail in ways that matter for downstream applications. Figures 8 and 9 present a direct comparison: removing 16 non-contiguous layers selected by a validation-loss-minimization criterion produces better validation loss than removing layers 16–31 contiguously (Figure 8, dashed line), but the non-contiguous removal collapses to random performance on Winogrande (50.0% accuracy, Figure 9) while the contiguous removal retains meaningful capability (59.5%). This is not a small divergenceβ€”it is a qualitative failure where a pruning decision that looks optimal by the standard metric produces a model that is effectively useless on a commonsense reasoning task.

The conceptual contribution is the recognition that perplexity and downstream task accuracy measure different aspects of model quality, and these aspects can be in tension for structural pruning decisions. Perplexity captures local token-level coherenceβ€”whether the model assigns high probability to the actual next token given the preceding context. Task accuracy captures whether the model's aggregate computation across many layers produces a correct final output. When layers are removed, the interaction between these two measures changes in ways that a purely loss-based criterion cannot capture. Layers that contribute minimally to next-token prediction on a generic corpus may nonetheless play a critical role in the multi-step reasoning chains that produce correct answers on tasks like Winogrande.

Why this is more than a "use a different metric" finding: The deeper insight is that structural pruning decisionsβ€”especially depth pruning, which removes entire sequential processing stagesβ€”can affect the model's computational dynamics in ways not reflected in aggregate loss. Removing layers from the middle of the network may disrupt specific reasoning subroutines that the model learned during pretraining, even if the remaining layers can compensate well enough for next-token prediction on generic text. Using a task-based metric like Winogrande to guide pruning is a pragmatic way to ensure that the pruning decisions preserve the reasoning capabilities that users actually care about, rather than optimizing a proxy metric that can diverge from those capabilities.

Prior work contrast: The original Minitron work used only validation loss-based importance for depth pruning and did not compare against task-based alternatives. Gromov et al. (2024) demonstrated the effectiveness of contiguous layer removal but used perplexity as the selection criterion. This paper's introduction of a downstream task metric (specifically Winogrande) as the pruning saliency signal is novel in the depth pruning literature and directly motivated by the observed loss-accuracy disconnect.

Qualification: This is a diagnostic innovationβ€”it identifies a failure mode of existing practice and provides a fixβ€”but the fix itself is limited. The paper evaluates only Winogrande as the task metric; it is unknown whether MMLU, HellaSwag, or GSM8K would suggest different optimal layer removal patterns. The choice of task metric for pruning is itself a design decision that may need to be tuned per deployment scenario. Additionally, the paper applies this metric only to depth pruning, not to width pruning, leaving open whether the same loss-task disconnect exists for neuron, head, or embedding channel removal. The innovation is therefore more about exposing the problem than providing a complete solutionβ€”but exposing the problem is valuable in a literature that had largely assumed perplexity was a sufficient pruning signal.

Innovation 3: Compression Without Access to Original Data Is Not Just Possible but Can Produce State-of-the-Art Results

The paper's headline empirical findingβ€”that MN-Minitron-8B, produced through pruning and distillation without access to Mistral NeMo's original training data, outperforms Llama 3.1 8B (trained from scratch on 15T tokens) across all evaluated benchmarks using 40Γ— fewer training tokensβ€”is more than a strong result. It establishes a new baseline for what is achievable in the "compression without original data" regime.

Prior to this work, the assumption embedded in the Minitron approach and the broader compression literature was that distillation on the original training data was the gold standard, and that using alternative data would produce degraded results. The original Minitron paper's experiments were all conducted with access to the training data. This paper demonstrates that with teacher correction, the degradation can be not just mitigated but eliminatedβ€”the compressed model trained on an entirely different dataset (Nemotron-4 CT) matches or exceeds the performance of models trained from scratch on their original data, and does so at a fraction of the training cost.

The significance of this finding extends beyond the specific models produced. It means that the growing ecosystem of open-weight LLMsβ€”Llama, Mistral, Qwen, Gemma, and othersβ€”can serve as starting points for compressed model families by practitioners who have no access to the pretraining data pipelines. The barrier to creating efficient, task-appropriate model variants drops from "you need the original data or you need to pretrain from scratch" to "you need the open-weight checkpoint and a reasonable distillation dataset." This democratizes model family creation in a way that parallels how open-weight releases democratized fine-tuning.

The 40–150Γ— token reduction as a redefinition of efficiency: When the paper reports that MN-Minitron-8B uses 380B tokens versus Llama 3.1 8B's 15T tokens, it is not claiming that 380B tokens of distillation is equivalent to 15T tokens of pretraining in terms of the information content or diversity of the data. The reduction is possible because distillation starts from a pruned model that already encodes structured knowledge inherited from the teacher, in contrast to pretraining from scratch which must learn all linguistic and world knowledge from raw text. What the paper demonstrates is that this knowledge inheritance survives both the distribution shift (via teacher correction) and the structural compression (via pruning), enabling extreme training efficiency without sacrificing final model quality.

Prior work contrast: The original Minitron work demonstrated the efficiency of pruning + distillation when original data is available. This paper extends that finding to the data-unavailable regime and shows that the efficiency gains are preserved. The comparison against Gemma 2 (2B, trained on 2T tokens), the original Minitron 4B (94B tokens), Gemma 7B (6T tokens), Mistral 7B (8T tokens), and Llama 3.1 8B (15T tokens) in Table 1 makes the efficiency claim concrete: the compressed models use the smallest training budgets in the comparison table while achieving competitive or superior accuracy.

Qualification: This is a practical innovation rather than a theoretical one. The finding does not reveal new principles about why distillation works; it demonstrates that existing principles (knowledge distillation from pruned models) remain effective under a relaxed assumption (original data not required) when combined with a specific fix (teacher correction). The result is strong but boundedβ€”it applies to the specific models, dataset (Nemotron-4 CT), and compression ratios tested. Whether the same efficiency holds for more aggressive compression ratios, different model architectures, or different distillation datasets is not established.

Innovation 4: Width and Depth Pruning Are Complementary Compression Axes with Distinct Accuracy-Efficiency Tradeoffs

The paper produces two distinct 4B models from the same Llama 3.1 8B teacherβ€”one via width pruning (Llama-3.1-Minitron-4B-Width) and one via depth pruning (Llama-3.1-Minitron-4B-Depth)β€”and systematically compares their properties. This comparison surfaces a finding that is intuitive in retrospect but was not previously documented at this scale: width pruning preserves accuracy better, while depth pruning delivers better inference throughput.

The evidence is clear. On nearly every benchmark in Table 1, the width-pruned variant outperforms the depth-pruned variant: MMLU 60.5% vs. 58.7%, HellaSwag 76.1% vs. 73.2%, GSM8K 41.2% vs. 16.8%, Winogrande 73.5% vs. 72.1%. The GSM8K gap is particularly strikingβ€”the depth-pruned model's math reasoning capability collapses relative to the width-pruned model, suggesting that the sequential computation through many layers is important for the multi-step reasoning chains that math problems require. However, Figure 10 shows the inverse pattern for throughput: the depth-pruned variant achieves a 2.7Γ— speedup over Llama 3.1 8B, while the width-pruned variant achieves only 1.8Γ—. The depth-pruned model is faster because it reduces the number of sequential transformer blocks each token must traverse, directly reducing latency; width pruning reduces per-layer computation but not the number of sequential steps.

This tradeoff is not itself surprisingβ€”removing layers reduces sequential depth, which is the primary latency bottleneck in transformer inferenceβ€”but the paper's contribution is quantifying it at the 4B scale for models derived from the same teacher, allowing practitioners to make informed choices. If your deployment constraint is accuracy on reasoning tasks, choose width pruning. If your constraint is throughput or latency, depth pruning offers substantially better speedups, though at the cost of some reasoning capability (which partially recovers after instruction tuning, as Table 2 shows the depth-pruned instruct variant achieving 71.1% on GSM8K vs. 79.8% for the width-pruned instruct variantβ€”the gap narrows but remains significant).

Why this is more than an engineering tradeoff: The finding implies that the choice of pruning axis is not just a matter of which components are most redundant, but a strategic decision about what kind of model you want to deploy. Width and depth pruning produce models with different computational profiles, not just different parameter counts. This has implications for hardware-specific optimization: depth-pruned models benefit more from inference engines that are memory-bandwidth-bound (fewer layers means less parameter loading per token), while width-pruned models may benefit more from compute-bound optimizations. The paper does not explore this hardware dimension, but the throughput numbers in Figure 10 hint at it.

Prior work contrast: The original Minitron work applied both width and depth pruning but did not systematically compare the two axes against each other for the same teacher-to-student compression ratio. Most depth pruning papers (ShortGPT, Shortened LLaMA, Gromov et al.) report only perplexity and select downstream metrics without comparing against a width-pruned variant at the same parameter count. This paper's head-to-head comparison at fixed 4B parameters fills that gap.

Qualification: This is an empirical characterization rather than a new method. The paper does not propose a way to combine width and depth pruning to achieve the best of both worldsβ€”for instance, moderate depth reduction plus moderate width reduction might yield accuracy closer to the width-pruned model with throughput closer to the depth-pruned model. The finding also does not explain why depth pruning disproportionately affects reasoning tasks like GSM8K while leaving tasks like MMLU relatively intactβ€”the mechanism behind this differential impact is an open question.

5. Experimental Analysis

Evaluation Methodology

Dataset. All pruning and distillation experiments use the Nemotron-4 curated continued training (CT) dataset (Parmar et al., 2024). This is a proprietary dataset developed by NVIDIA, distinct from the undisclosed pretraining datasets used for Llama 3.1 8B and Mistral NeMo 12B. The paper does not disclose the CT dataset's total size or composition statistics. A calibration subset of 1024 samples is drawn randomly from the full dataset for importance estimation during pruning. Teacher correction uses approximately 100B tokens from this dataset. Distillation retraining uses 94B tokens for the Llama-3.1-Minitron-4B variants and 380B tokens for MN-Minitron-8B (Table 4).

Base models. The experiments start from two publicly available pretrained models downloaded from Hugging Face:

  • Mistral NeMo 12B (Mistral AI, 2024): a 12.2B-parameter model (10.9B non-embedding parameters) with 40 transformer layers, hidden dimension 5120, MLP hidden dimension 14336, 32 attention heads with head dimension 128, and vocabulary size 131,072. Its original pretraining data and training token count are not publicly disclosed.
  • Llama 3.1 8B (Dubey et al., 2024): an 8B-parameter model (7B non-embedding parameters) with 32 transformer layers, hidden dimension 4096, MLP hidden dimension 14336, 32 attention heads with head dimension 128, and vocabulary size 128,256. Pretrained on 15T tokens from a proprietary data mixture.

These models are chosen because they represent state-of-the-art open-weight models at the time of writing and have different architectures, vocabularies, and training data distributions, allowing the paper to test the generality of the compression recipe across model families.

Metrics. The paper evaluates models using standard benchmarks from the LM Evaluation Harness, reporting accuracy under the following configurations:

  • Base models: 5-shot MMLU (Hendrycks et al., 2021), 5-shot Winogrande (Sakaguchi et al., 2021), 25-shot ARC-Challenge (Clark et al., 2018), 10-shot HellaSwag (Zellers et al., 2019), 0-shot TruthfulQA (Lin et al., 2022), 0-shot on 20% of XL-Sum English (Hasan et al., 2021), and average pass@1 for HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021). For pass@1, the paper uses temperature 0.2 and nucleus sampling with top-p = 0.95.
  • Instruct models: 0-shot (greedy sampling where applicable) on MMLU, GSM8K (Cobbe et al., 2021), GPQA (Rein et al., 2023), HumanEval, MBPP, IFEval (Zhou et al., 2023), BFCLv2 live accuracy (Yan et al., 2024), and MT-Bench with GPT4-Turbo as judge (Wang et al., 2024). IFEval is reported as "the average of prompt and instruction across loose and strict evaluations."

All evaluations are conducted by the authors themselves except entries marked with an asterisk (*) in Tables 1 and 2, which are taken from the corresponding papers. This ensures consistent evaluation protocols across compared models.

Baselines. The paper compares against multiple categories of baselines in Tables 1 and 2:

  • Teacher models: Mistral NeMo 12B Base, Mistral NeMo 12B-FT (the corrected teacher), and Llama 3.1 8B. These represent the uncompressed upper bound.
  • Previous-generation compressed model: Minitron 4B from Muralidharan et al. (2024), which used the original Minitron recipe with access to the training data.
  • Independently trained models at similar scales: Gemma 2 2B (Team et al., 2024; 2.6B total, 2B non-embedding, trained on 2T tokens), Phi-2 2.7B (evaluation numbers marked with *, taken from the corresponding paper), Qwen2 1.5B, Gemma 7B (8.5B total, 7.7B non-embedding, 6T tokens), and Mistral 7B (7.3B total, 7B non-embedding, trained on 8T tokens according to the table).
  • In-context learning baselines: Comparisons in Tables 1 and 2 are primarily against same-sized models rather than alternative compression methods (e.g., the paper does not compare against SliceGPT, Sheared LLaMA, or ShortGPT as baselines for final model quality).

Generation budget / compute accounting. The paper measures training compute indirectly through total training tokens consumed during distillation, which serves as the primary efficiency metric for comparison against from-scratch pretraining. The specific token budgets are: 94B tokens for Llama-3.1-Minitron-4B variants, 380B tokens for MN-Minitron-8B, and approximately 100B tokens for teacher correction. Training is conducted on 32 NVIDIA DGX H100 nodes (approximately 256 H100 GPUs) with global batch sizes of 1152 (Llama) and 768 (MN-Minitron) at context length 8192 (Table 4). The paper does not report total FLOPs, GPU-hours, or wall-clock time for training. Inference throughput is measured in requests per second using TensorRT-LLM in FP8 precision on a single H100 80 GB GPU (Figure 10), with configurations varying input/output sequence lengths.

Cross-validation / statistical protocol. The paper does not employ cross-validation or report confidence intervals for benchmark results. Accuracy numbers in Tables 1 and 2 are reported as single-point estimates without variance information. The depth pruning layer selection procedure evaluates each candidate contiguous layer block by measuring Winogrande accuracy on the pruned (unretrained) model, selecting the block with the highest accuracy. This is a one-shot selection with no cross-validation over calibration data splits. The paper does not discuss whether the layer selection would change if a different calibration set or evaluation task were used.

Main Quantitative Results

Base Model Accuracy: MN-Minitron-8B vs. Similarly-Sized Models

The headline result from Table 1 is that MN-Minitron-8B (8.4B total parameters, 7.3B non-embedding) outperforms all similarly-sized models across every evaluated benchmark. The specific comparisons are:

  • Against the teacher Mistral NeMo 12B: MN-Minitron-8B surpasses the uncorrected 12B teacher on MMLU (69.5% vs. 69.0%), GSM8K (58.5% vs. 56.4%), HumanEval (36.2% vs. 23.8%), and XL-Sum (32.0% vs. 33.4% β€” a small regression). It trails slightly on Winogrande (80.4% vs. 82.2%), Arc-Challenge (64.4% vs. 65.1%), and HellaSwag (83.0% vs. 85.2%). Against the corrected teacher (12B-FT), MN-Minitron-8B trails on MMLU (69.5% vs. 70.1%) and HellaSwag (83.0% vs. 85.3%) but leads on GSM8K (58.5% vs. 55.7%) and HumanEval (36.2% vs. 23.8% β€” matching the uncorrected teacher).

  • Against Llama 3.1 8B: MN-Minitron-8B outperforms on MMLU (69.5% vs. 65.3%), Winogrande (80.4% vs. 77.3%), Arc-Challenge (64.4% vs. 57.9%), HellaSwag (83.0% vs. 81.8%), GSM8K (58.5% vs. 48.6%), TruthfulQA (47.6% vs. 45.0%), XL-Sum (32.0% vs. 30.0%), MBPP (43.8% vs. 42.3%), and HumanEval (36.2% vs. 24.8%). This is a clean sweep across all nine benchmarks. Critically, this is achieved using 380B training tokens versus Llama 3.1 8B's 15T tokens β€” a 40Γ— reduction.

  • Against Mistral 7B and Gemma 7B: MN-Minitron-8B outperforms Mistral 7B on all benchmarks where both are evaluated (MMLU 69.5% vs. 64.1%, Winogrande 80.4% vs. 78.5%, HellaSwag 83.0% vs. 83.2% β€” essentially tied, GSM8K 58.5% vs. 37.0%) and outperforms Gemma 7B on MMLU (69.5% vs. 64%), Arc-Challenge (64.4% vs. 60.3%), GSM8K (58.5% vs. 50%), TruthfulQA (47.6% vs. 45%), and XL-Sum (32.0% vs. 17%), while essentially matching on HellaSwag (83.0% vs. 82%).

  • Against Gemma 2 2B: MN-Minitron-8B substantially outperforms this smaller model on all shared benchmarks: MMLU (69.5% vs. 51.3%), Winogrande (80.4% vs. 70.9%), HellaSwag (83.0% vs. 73.0%), GSM8K (58.5% vs. 23.9%), MBPP (43.8% vs. 29.0%), and HumanEval (36.2% vs. 20.1%).

The efficiency claim β€” achieving this performance with 40Γ— fewer training tokens than Llama 3.1 8B β€” requires careful contextualization. The 380B tokens refer only to the distillation phase, not including the teacher model's original pretraining cost (which is amortized across all uses of Mistral NeMo 12B). The total tokens that went into producing MN-Minitron-8B include Mistral NeMo's undisclosed pretraining budget plus 100B correction tokens plus 380B distillation tokens. The 40Γ— figure compares distillation tokens (380B) to Llama 3.1 8B's pretraining tokens (15T), but this is not a like-for-like comparison of total training cost.

Base Model Accuracy: Llama-3.1-Minitron-4B Variants

Table 1 also shows results for the two 4B variants derived from Llama 3.1 8B using 94B distillation tokens β€” a 160Γ— reduction from the teacher's 15T pretraining budget. The comparison between the two variants and against baselines reveals:

Width-pruned vs. depth-pruned (both 4.5B total parameters):

  • The width-pruned variant outperforms the depth-pruned variant on every benchmark: MMLU (60.5% vs. 58.7%), Winogrande (73.5% vs. 72.1%), Arc-Challenge (55.6% vs. 52.6%), HellaSwag (76.1% vs. 73.2%), GSM8K (41.2% vs. 16.8%), TruthfulQA (42.9% vs. 38.2%), XL-Sum (28.7% vs. 27.2%), and MBPP (32.4% vs. 30.7%). The GSM8K gap is dramatic: 41.2% vs. 16.8%, a 24.4 percentage point difference, indicating that depth pruning disproportionately damages mathematical reasoning capability.

Against the teacher Llama 3.1 8B (both variants use 150Γ— fewer tokens):

  • The width-pruned variant trails the teacher on most metrics: MMLU (60.5% vs. 65.3%), Winogrande (73.5% vs. 77.3%), Arc-Challenge (55.6% vs. 57.9%), GSM8K (41.2% vs. 48.6%), TruthfulQA (42.9% vs. 45.0%), XL-Sum (28.7% vs. 30.0%), MBPP (32.4% vs. 42.3%). It matches on HellaSwag (76.1% vs. 81.8% β€” a meaningful gap) and slightly exceeds on HumanEval (23.3% for the width variant from the Minitron 4B column vs. 24.8% for Llama 3.1 8B, though the width variant's HumanEval is not directly listed in Table 1 under the Llama-3.1-Minitron columns β€” the Minitron 4B column shows 23.3%, and the Llama-3.1-Minitron-4B-Width shows a dash for HumanEval in the n=20 setting).

Against the previous-generation Minitron 4B:

  • Both Llama-3.1-Minitron-4B variants outperform the original Minitron 4B: MMLU (60.5% and 58.7% vs. 58.6%), Winogrande (73.5% and 72.1% vs. 74.0% β€” the original Minitron has a slight edge on Winogrande against the depth variant), Arc-Challenge (55.6% and 52.6% vs. 50.9%), HellaSwag (76.1% and 73.2% vs. 75.0%), GSM8K (41.2% and 16.8% vs. 24.1%). The width variant shows substantial improvement over the original Minitron, particularly on GSM8K and Arc-Challenge.

Against Gemma 2 2B: Both Llama-3.1-Minitron-4B variants substantially outperform Gemma 2 2B on MMLU (60.5% and 58.7% vs. 51.3%), HellaSwag (76.1% and 73.2% vs. 73.0%), and GSM8K (41.2% and 16.8% vs. 23.9%), while roughly tying on Arc-Challenge (55.6% and 52.6% vs. 55.4%) and trailing slightly on Winogrande (73.5% and 72.1% vs. 70.9%).

Instruct Model Accuracy

Table 2 evaluates instruction-tuned variants of the compressed models against similarly-sized instruction-tuned baselines. The key results are:

MN-Minitron-8B instruct:

  • Outperforms Llama 3.1 8B (instruct) on MMLU (70.4% vs. 69.4%), GSM8K (87.1% vs. 83.8%), GPQA (31.5% vs. 30.4%), IFEval (84.4% vs. 80.4%), and BFCLv2 live accuracy (67.6% vs. 44.3%). It essentially matches on MT-Bench (7.86 vs. 7.78) and slightly trails on HumanEval (71.3% vs. 72.6%) and MBPP (72.5% vs. 72.8%).
  • The BFCLv2 gap is particularly large: 67.6% vs. 44.3%, a 23.3 percentage point advantage, suggesting the corrected teacher + distillation pipeline produces strong function-calling capabilities.
  • Against Gemma 2 2B, Phi-2, and Qwen2, MN-Minitron-8B leads across most benchmarks with substantial margins.

Llama-3.1-Minitron-4B instruct variants:

  • The width-pruned instruct variant outperforms the depth-pruned instruct variant on most metrics: MT-Bench (6.88 vs. 6.19), GSM8K (79.76% vs. 71.11%), GPQA (30.36% vs. 32.59% β€” the depth variant leads here), HumanEval (47.0% vs. 42.7%), MBPP (65.1% vs. 60.3%), IFEval (79.54% vs. 66.77%), and BFCLv2 (55.0% vs. 55.89% β€” essentially tied). The IFEval gap (79.54% vs. 66.77%) is noteworthy, suggesting instruction-following capability is better preserved with width pruning.
  • Compared to the original Minitron 4B instruct: Both variants substantially outperform on GSM8K (79.76% and 71.11% vs. 65.1%), IFEval (79.54% and 66.77% vs. 75.3% β€” the width variant leads, the depth variant trails), and BFCLv2 (55.0% and 55.89% vs. 53.1%). The width variant also improves on MMLU (59.89% vs. 59.3%) and MBPP (65.1% vs. 57.4%).
  • Against Gemma 2 2B instruct: The width-pruned variant outperforms on MT-Bench (6.88 vs. 7.44 β€” trailing), MMLU (59.89% vs. 56.9%), and all programming/math benchmarks. On MT-Bench, Gemma 2 2B leads at 7.44 vs. 6.88.

Notable pattern: instruction tuning narrows the width-depth gap for reasoning. In the base models, the GSM8K gap between width and depth pruning was 41.2% vs. 16.8% (a 24.4 point difference). After instruction tuning, the gap narrows to 79.76% vs. 71.11% (an 8.65 point difference). This suggests that instruction tuning partially recovers the reasoning capability lost to depth pruning, though a meaningful gap remains.

Runtime Performance

Figure 10 reports throughput in requests per second for Llama 3.1 8B and the two Llama-3.1-Minitron-4B variants, optimized with TensorRT-LLM in FP8 precision on a single H100 80 GB GPU. The key findings:

  • Llama-3.1-Minitron-4B-Depth achieves an average 2.7Γ— throughput improvement over Llama 3.1 8B. The depth-pruned model processes more requests per second across multiple input/output sequence length configurations. The advantage comes from halving the number of sequential transformer layers (16 vs. 32), which reduces the per-token latency.

  • Llama-3.1-Minitron-4B-Width achieves an average 1.8Γ— throughput improvement over Llama 3.1 8B. The speedup here comes from reduced per-layer computation (smaller hidden dimension, smaller MLP) without reducing the number of sequential steps.

  • FP8 delivers a 1.4Γ— performance boost over BF16 across these models, as noted in the text.

  • The paper notes that the 4B models can use larger batch sizes (64 vs. 32 for the 8B/12B models) due to their smaller memory footprint, which further improves throughput in high-volume serving scenarios. The batch size difference is not directly controlled for in the reported throughput numbers.

These results establish the practical deployment advantage: for latency-sensitive or high-throughput applications where the width-pruned model's 1.8Γ— speedup is insufficient, the depth-pruned variant offers substantially faster inference at the cost of some accuracy degradation, particularly on reasoning tasks.

Ablation Studies and Robustness Checks

Teacher correction vs. uncorrected teacher (Figure 4): The paper compares LM validation loss during distillation of MN-Minitron-8B when supervised by the original Mistral NeMo 12B versus the corrected (fine-tuned) Mistral NeMo 12B. The corrected teacher produces consistently lower validation loss throughout training, with the final gap quantified as "over a 6% reduction in LM validation loss." This is the core evidence that teacher correction is necessary when distilling on a new dataset. The ablation uses only LM validation loss as the metric; it does not report downstream benchmark accuracy for the student trained with each teacher, which would be a more direct measure of practical impact.

Sequential vs. parallel teacher correction (Figure 5): Two approaches are compared for MN-Minitron-8B: (1) pruning and distilling from a fully corrected teacher (sequential), and (2) pruning the original teacher and distilling from a teacher that is being continuously corrected during distillation (parallel). Figure 5 shows that the training curves "largely overlap," and the paper concludes that "teacher correction can be performed in parallel with distillation." The x-axis extends to ~120B training tokens, covering the range where both teacher correction (~100B) and the early phase of distillation overlap. No final benchmark accuracy comparison between the two approaches is provided.

Pruning + distillation vs. alternative strategies (Figure 6): Four configurations are compared for MN-Minitron-8B: (a) random weight initialization + distillation, (b) random pruning (ignoring importance scores) + distillation, (c) importance-based pruning + standard LM cross-entropy loss (no distillation), and (d) the full pipeline: importance-based pruning + distillation loss. The results show: random initialization starts from much higher loss than any pruning-based approach (validating that pruning provides a better starting point), random pruning underperforms importance-based pruning (validating the importance estimation), and distillation-based training outperforms standard LM loss training (validating the distillation objective). This is a comprehensive ablation of the pipeline's components, though it uses only LM validation loss as the metric without downstream task evaluation for the ablated configurations.

Width vs. depth pruning convergence (Figure 7): Training curves for Llama-3.1-Minitron-4B-Width and Llama-3.1-Minitron-4B-Depth are compared throughout the 94B-token distillation phase. The width-pruned variant starts from lower initial loss and maintains a consistent advantage throughout training, with the gap persisting rather than closing. This demonstrates that width pruning provides a better initialization and that the advantage is not recoverable through additional depth-pruning-specific training. The paper does not ablate over the specific pruning ratios (e.g., whether a smaller width reduction combined with some depth reduction would close the gap).

Depth pruning: loss-based vs. task-based layer selection (Figures 8 and 9): This is the most detailed ablation in the paper. Figure 8 shows LM validation loss when removing 1, 2, 8, or 16 layers from Llama 3.1 8B, with the key observation that removing 16 non-contiguous layers selected by loss minimization (dashed line) produces better validation loss than removing 16 contiguous layers (e.g., layers 16–31). Figure 9 then shows Winogrande accuracy for the same pruned configurations, revealing that removing layers 16–31 contiguously yields 59.5% accuracy while removing 16 non-contiguous layers selected by the loss criterion yields only 50.0% (random performance). This demonstrates the disconnect between validation loss and downstream task performance for depth pruning decisions. The paper also shows that dropping layers 1–16 (beginning of network) performs worse than dropping layers 16–31 (middle-to-end), and that the baseline 32-layer model achieves approximately 0.745 Winogrande accuracy. The ablation does not explore other task metrics (MMLU, GSM8K, HellaSwag) for layer selection, nor does it test whether the optimal contiguous block changes when evaluated on a different task.

Distillation token budget: The paper uses 94B tokens for the 4B models and 380B for the 8B model but provides no ablation over the token budget. It is unclear whether training for fewer tokens would yield similar results or whether more tokens would close the remaining gap to the teacher. The convergence plots (Figures 4–7) show loss continuing to decrease at the end of training for some configurations, suggesting further gains might be possible, but this is not explicitly tested.

Teacher correction token budget: The paper uses ~100B tokens for teacher correction on both model families but does not ablate this choice. The "Future Work" section briefly mentions that using fewer tokens or lighter fine-tuning methods (LoRA, LayerNorm-only tuning) would be interesting to explore, but no experiments are presented.

PRM aggregation and the ReSTEM^{EM} revision model ablation: Not applicable to this paper β€” these are from a different study. There is no corresponding ablation for the Minitron pipeline exploring alternative importance estimation methods, distillation objectives, or pruning ratios.

Manual architecture vs. NAS: The paper states it skips the Neural Architecture Search step used in the original Minitron work, manually specifying architectures instead. No ablation compares the manually specified architecture against a NAS-discovered architecture, so the accuracy cost of this simplification is unquantified.

Retaining all attention heads: The paper reports that keeping all attention heads while pruning other dimensions yields stronger accuracy, but does not present an ablation where some heads are pruned to demonstrate the magnitude of the difference. This is stated as an empirical finding without supporting data in the paper.

Calibration dataset size: The importance estimation uses 1024 samples; no ablation over calibration set size is presented.

Critical Assessment

Do the Experiments Support the Central Claims?

Claim: Teacher correction is crucial for distillation to work on a new, unseen dataset. The paper's evidence for this claim is Figure 4, which shows a >6% reduction in LM validation loss when using a corrected versus uncorrected teacher. This supports the claim at the level of training dynamics β€” the student model achieves lower perplexity during training with the corrected teacher. However, the paper does not report downstream benchmark accuracy for the student trained with the uncorrected teacher. The claim that teacher correction is "crucial" for the distillation to "work" would be much stronger if Tables 1 and 2 included a row for MN-Minitron-8B trained without teacher correction, showing degraded accuracy on MMLU, GSM8K, and other benchmarks. As presented, we know the corrected teacher reduces validation loss, but we do not know whether this translates to practically meaningful accuracy differences on the tasks users care about. The >6% reduction in LM validation loss is substantial, and it is reasonable to infer that downstream accuracy would be affected, but the paper does not close this loop empirically.

Claim: The compressed MN-Minitron-8B model is state-of-the-art among 8B-class models. Table 1 shows MN-Minitron-8B outperforming Llama 3.1 8B, Mistral 7B, and Gemma 7B on all evaluated benchmarks. This is a clean sweep and strongly supports the claim for the specific benchmarks evaluated. However, the claim must be qualified in several ways:

  • Benchmark coverage is limited. The evaluation covers nine benchmarks for base models and eight for instruct models. While these are standard in the field, they represent a specific subset of capabilities (commonsense reasoning, factual knowledge, math, code, summarization). Capabilities not tested β€” long-form generation quality, multilingual performance, safety, robustness to adversarial inputs β€” could reveal different rankings.
  • The comparison against Llama 3.1 8B is particularly strong, but Llama 3.1 8B's numbers in Table 1 are not identical to those reported in the Llama 3.1 technical report (Dubey et al., 2024), likely due to differences in evaluation protocols. The paper states "all evaluations are conducted by us, except entries marked with *," which means these are the authors' own evaluations of the Llama 3.1 8B model using their evaluation harness. This is methodologically sound (consistent evaluation) but means the numbers may differ from what Meta reports.
  • The 40Γ— token reduction claim compares distillation tokens (380B) to Llama 3.1 8B's pretraining tokens (15T). This is not a total-cost comparison. MN-Minitron-8B's total training includes: Mistral NeMo 12B's original pretraining (token count undisclosed, but likely multi-trillion) + 100B teacher correction tokens + 380B distillation tokens. The 40Γ— figure is best understood as "the incremental cost to produce an 8B model from an existing 12B model is 40Γ— less than pretraining an 8B model from scratch," which is a valid efficiency metric but not a total-cost comparison.

Claim: The 4B compressed models approach the 8B teacher's accuracy using 150Γ— fewer tokens. Table 1 shows that Llama-3.1-Minitron-4B-Width achieves MMLU 60.5% vs. Llama 3.1 8B's 65.3%, HellaSwag 76.1% vs. 81.8%, and GSM8K 41.2% vs. 48.6%. These are meaningful gaps (4–8 percentage points) on major benchmarks. The 4B model is competitive with but not matching the teacher, and the phrase "approaches the 8B teacher's accuracy" is a fair characterization only if these gaps are considered acceptable. On some benchmarks (HumanEval, MBPP), the gaps are larger. The 150Γ— token reduction is significant, but as with the 8B model, it measures incremental distillation cost, not total model creation cost.

Claim: Width pruning delivers better accuracy; depth pruning delivers better throughput. Figure 7 and Table 1 support the accuracy advantage of width pruning, and Figure 10 supports the throughput advantage of depth pruning. This claim is well-supported. However, the paper does not explore why this tradeoff exists β€” whether it reflects a fundamental property of transformer architectures or is specific to Llama 3.1 8B at this compression ratio. The dramatic GSM8K gap for depth pruning (41.2% vs. 16.8% base, narrowing to 79.76% vs. 71.11% after instruction tuning) is documented but not explained. An ablation that measures the impact of depth pruning on different reasoning tasks would clarify whether this is a general reasoning degradation or specific to mathematical reasoning.

Claim: Winogrande-based layer selection is superior to loss-based selection for depth pruning. Figures 8 and 9 demonstrate a clear case where loss-based selection fails (producing a model with random Winogrande accuracy) while contiguous removal guided by task accuracy succeeds. This supports the claim for Winogrande specifically. The paper does not demonstrate that Winogrande-guided selection produces better results on other downstream tasks (MMLU, GSM8K, HellaSwag) compared to loss-guided selection. It is possible that Winogrande-guided selection is optimal for Winogrande but suboptimal for other tasks, and that a loss-guided selection would produce better aggregate performance across the evaluation suite. The paper's decision to use Winogrande as the sole task metric for layer selection is a design choice whose generality is untested.

Genuine Weaknesses in the Experimental Design

No ablation of the teacher correction token budget. The paper uses ~100B tokens for teacher correction but does not test whether 10B, 50B, or 200B tokens would be more or less effective. The claim that teacher correction is "lightweight" is relative to pretraining budgets but 100B tokens is still a substantial training run. If 10B tokens sufficed, the practical appeal of the method would be much stronger.

No comparison against alternative compression methods. The paper compares against the original Minitron 4B and against from-scratch-trained models, but does not evaluate other structured pruning methods (SliceGPT, Sheared LLaMA, LLM-Pruner, Wanda) applied to the same teacher models. Without these comparisons, we cannot assess whether the Minitron recipe with teacher correction is the best available compression approach or simply a good one. A comparison against at least one alternative pruning method at the same parameter budget would significantly strengthen the paper's claim to state-of-the-art compression.

Teacher correction evaluated only on Nemotron-4 CT. All experiments use a single distillation dataset. The paper's framing implies teacher correction is a general solution for distribution mismatch, but it is unknown whether the effectiveness depends on how similar the distillation dataset is to the original training data. If Nemotron-4 CT happens to be relatively similar to Mistral NeMo's training data in its sub-word token distribution, the results might not generalize to distillation datasets with larger distribution shifts.

Small calibration set for importance estimation (1024 samples) is not ablated. If importance rankings are sensitive to the calibration data, the pruning decisions could be unstable. The paper does not report whether importance scores from different random 1024-sample subsets produce consistent rankings.

Single-point accuracy numbers without variance. All benchmark results in Tables 1 and 2 are reported without confidence intervals or standard deviations. For tasks like GSM8K with 0-shot evaluation and HumanEval with pass@1, sampling variance can be non-trivial. The 0.5–2 percentage point differences between some models (e.g., MMLU comparisons between MN-Minitron-8B and Llama 3.1 8B) may or may not be statistically significant.

No FLOPs or GPU-hour accounting. The paper reports token counts and GPU hardware used but does not compute total FLOPs or GPU-hours for training. This makes it difficult to reproduce the training budget or to compare costs across different approaches. For a paper whose efficiency claims are central, this is a notable omission.

Instruct model comparisons are confounded by different alignment procedures. The paper uses NeMo-Aligner with a specific recipe (math and code SFT, then instruction SFT, then two rounds of RPO) for all its instruct models, but the baseline instruct models (Llama 3.1, Gemma, Qwen) were aligned with different procedures and datasets. Differences in instruct model performance reflect both the base model quality and the alignment recipe. The paper's claim that "our aligned models are consistently better" is a claim about the combination of base model compression + the specific alignment recipe, not about the compressed base model alone.

Missing Experiments That Would Strengthen the Paper

  • Downstream evaluation of the uncorrected-teacher student: Add a row to Table 1 for "MN-Minitron-8B (uncorrected teacher)" to quantify the practical accuracy impact of omitting teacher correction.
  • Teacher correction budget sweep: Test correction at 10B, 50B, 100B, and 200B tokens to establish the minimum effective correction budget.
  • Alternative task metrics for depth pruning layer selection: Compare Winogrande-guided selection against MMLU-guided, GSM8K-guided, and loss-guided selection, reporting final accuracy across all benchmarks for each.
  • Width + depth combined pruning: Produce a model that uses moderate reductions in both width and depth (e.g., hidden 4096β†’3584, layers 32β†’24) to explore whether the accuracy-throughput tradeoff has a more favorable Pareto frontier than either axis alone.
  • Comparison against an alternative pruning method: Apply SliceGPT or Sheared LLaMA to the same teacher models at the same compression ratio and evaluate on the same benchmarks.
  • Reproducibility checks: Report importance score stability across different random calibration subsets, and variance in benchmark evaluations (at minimum for the model's own results).

6. Limitations and Trade-offs

The Teacher Correction Token Budget Is Unexplored

The assumption or constraint. The paper applies a ~100B token teacher correction phase to both the Mistral NeMo 12B and Llama 3.1 8B teachers before distillation, describing this as a "lightweight" fine-tuning step. However, no ablation over the correction token budget is performed. The paper acknowledges this implicitly in the Teacher Correction section of the Methodology:

"Optimizing this process further by using fewer than ∼100B tokens, lighter fine-tuning such as LoRA or tuning layer normalization parameters alone would be an interesting topic for future work."

The 100B token figure is presented as a fixed constant without justification β€” no experiments at 10B, 50B, or 200B tokens to establish whether this is a minimum effective dose, an overshoot, or simply an arbitrary round number.

The consequence. From a practitioner's perspective, 100B tokens is a substantial training run β€” comparable to or exceeding the 94B-token distillation budget for the 4B models. If teacher correction could achieve comparable effectiveness at 20B or 50B tokens, the end-to-end training cost could be meaningfully reduced. Conversely, if 100B tokens is insufficient for larger distribution shifts between teacher training data and distillation data (the paper only tests one distillation dataset, Nemotron-4 CT), practitioners working with more divergent datasets might encounter silently degraded results. Without a budget sweep, the paper provides no guidance on how to select this hyperparameter for new teacher-dataset pairs, leaving a critical cost-quality tradeoff entirely unexplored.

What evidence exists in the paper. The paper provides only a single data point: 100B tokens on Nemotron-4 CT for two model families. Figure 4 shows that this specific budget produces effective teacher correction (the corrected-teacher curve sits substantially below the uncorrected-teacher curve for MN-Minitron-8B). However, the convergence plot in Figure 4 extends to only ~25B tokens (the x-axis shows 0–22.5B, and the two curves begin to separate meaningfully only after ~5B tokens), making it impossible to assess from this figure whether correction largely plateaus at 25B or continues improving through 100B. The paper presents no data on how quickly the teacher's logit calibration improves during correction or whether downstream benchmark accuracy of the final student varies with correction budget.

Mitigation status. The paper does not address this limitation experimentally. The future work section mentions exploring fewer tokens and lighter methods (LoRA, LayerNorm tuning) but provides no data to bound the current requirement. A practitioner implementing this pipeline today must either replicate the 100B token budget (costing substantial compute) or conduct their own budget sweep to determine whether the claimed efficiency gains survive at lower correction budgets. The headline efficiency numbers (40Γ—, 150Γ— token reductions relative to pretraining) exclude the 100B correction tokens from their framing β€” the 380B distillation budget for MN-Minitron-8B is cited, but the total pipeline consumes ~480B tokens when correction is included, a 23% overhead that is not reflected in the efficiency claims.

Width and Depth Pruning Are Never Combined Despite Their Complementary Strengths

The assumption or constraint. The paper studies width pruning and depth pruning as entirely independent compression axes, producing two separate 4B models (Llama-3.1-Minitron-4B-Width and -Depth) from the same 8B teacher. The Methodology section states explicitly that these are not combined:

"We consider depth pruning as a special case and do not combine it with compressing other dimensions."

Width pruning targets the hidden dimension, MLP intermediate dimension, and embedding channels while preserving all layers and attention heads. Depth pruning removes 16 of 32 contiguous layers while leaving all other dimensions unchanged. No model explores a hybrid strategy β€” for example, reducing depth from 32 to 24 layers while simultaneously reducing the hidden dimension from 4096 to 3584.

The consequence. The paper identifies a clear tradeoff: width pruning preserves accuracy better (Table 1, e.g., GSM8K 41.2% vs. 16.8%, MMLU 60.5% vs. 58.7%) while depth pruning delivers better throughput (Figure 10, 2.7Γ— vs. 1.8Γ— speedup over the 8B teacher). This tradeoff is presented as a binary choice between two extremes, but there is no reason to believe the Pareto frontier is this simple. A moderate combination β€” some depth reduction plus some width reduction β€” might achieve throughput closer to the depth-pruned model (fewer sequential layers) while preserving accuracy closer to the width-pruned model (less disruptive to reasoning chains than halving the layer count). The paper's own evidence suggests depth pruning's accuracy penalty may be non-linear in the number of layers removed: dropping 8 layers might produce substantially less degradation than dropping 16, while still capturing meaningful latency improvements. Without exploring this space, practitioners cannot determine whether the published tradeoff is close to optimal or whether substantial improvements are available at intermediate operating points.

What evidence exists in the paper. The paper provides ample evidence that the two axes have different properties but no evidence about their interaction. Figure 8 shows the effect of dropping 1, 2, 8, and 16 contiguous layers on validation loss, with the 8-layer removal causing less degradation than the 16-layer removal β€” this suggests the accuracy cost of depth pruning is monotonic in the number of layers removed but provides no information about combination with width pruning. Figure 7 shows that width pruning starts from a lower initial loss and maintains its advantage throughout distillation, but does not test whether a hybrid model would start from an intermediate loss and potentially converge to better final accuracy than the depth-pruned variant at a similar throughput. The architecture configurations in Table 3 are presented as fixed manual choices "inspired by the Minitron-4B and Minitron-8B models" from the original Minitron work, not as the output of a systematic exploration of the compression design space.

Mitigation status. The paper does not address this limitation. The decision to treat depth pruning as a "special case" and not combine it with width pruning is stated as a methodological choice but not justified. The original Minitron work did use NAS to explore combined architectures, so the capability to explore this space exists β€” this paper simply chose not to exercise it, prioritizing simplicity (skipping NAS entirely) over architecture optimization. A practitioner facing a deployment scenario where both accuracy and throughput matter (which is most real-world deployments) has no guidance on whether to accept the width-pruned model's 1.8Γ— speedup or the depth-pruned model's significant accuracy penalty, because the space between these extremes is entirely uncharacterized.

The Practical Cost of the 40Γ— and 150Γ— Token Reduction Claims Is Misleading Without Accounting for the Teacher's Pretraining

The assumption or constraint. The paper's headline efficiency claims compare distillation token budgets against the from-scratch pretraining budget of the teacher model. Specifically:

"MN-Minitron-8B... using 40Γ— fewer training tokens (380B vs. 15T)"

and:

"the Llama 3.1-Minitron-4B models perform favorably compared to the teacher Llama 3.1 8B model using 150Γ— fewer training tokens (94B vs. 15T)"

These comparisons frame the distillation budget as the total cost of producing the compressed model, implicitly treating the teacher's pretraining cost as a sunk expenditure that can be amortized across all downstream uses. The total tokens that go into producing a compressed model, however, include: (1) the teacher's original pretraining on undisclosed data (for Mistral NeMo 12B) or 15T tokens (for Llama 3.1 8B), (2) ~100B tokens for teacher correction, and (3) 94B or 380B tokens for distillation.

The consequence. This framing systematically understates the total cost and misleads about the regimes where compression is truly more efficient than training from scratch. The correct comparison for a practitioner deciding whether to compress an existing teacher versus train a small model from scratch is:

  • Total cost of compression = (amortized fraction of teacher pretraining) + 100B correction + 380B distillation
  • Total cost of from-scratch training = pretraining a model of the target size on the target data

The 40Γ— and 150Γ— figures apply only to the incremental cost of distillation versus pretraining a same-sized model as the teacher, not the total cost of compression versus the total cost of training a same-sized model as the student. If a practitioner does not already have the teacher model (e.g., they are building a model family from scratch and deciding whether to train one large model and compress, or train each size independently), the teacher's pretraining cost cannot be amortized away β€” it must be paid. In that regime, the efficiency argument depends on how many compressed models are derived from the same teacher (amortization factor) and whether the compression ratio justifies the teacher's training cost.

Even in the "teacher already exists" scenario (the paper's explicit use case β€” starting from an open-weight checkpoint), the amortization argument only holds if the marginal cost of obtaining the teacher is zero. For open-weight models like Llama 3.1 8B, this is approximately true (download cost is negligible). But the efficiency claim is then not "compression is 40Γ— cheaper than training" but rather "given a free teacher, producing a smaller model costs 380B tokens." This is still valuable β€” it means practitioners can produce small models at a fraction of what from-scratch pretraining would cost β€” but it is a fundamentally different claim from the unqualified "40Γ— fewer training tokens."

What evidence exists in the paper. The paper's own numbers reveal the tension. Table 1 compares MN-Minitron-8B (380B distillation tokens) against Llama 3.1 8B (15T pretraining tokens) and notes the 40Γ— reduction. But the same table also compares against Gemma 7B (6T tokens), Gemma 2 2B (2T tokens), and Mistral 7B (8T tokens) β€” models trained from scratch at the target size class. These comparisons are fair for accuracy but not for total training cost, because the 380B figure excludes the teacher's pretraining. The paper does not provide a total-FLOPs or total-tokens accounting that sums the teacher's pretraining, correction, and distillation costs, nor does it compare this sum against from-scratch pretraining budgets for models at the 4B and 8B scale.

Mitigation status. The paper does not address this. The efficiency claims are stated without qualification about what costs are included or excluded. The text does note that "our approach requires significantly fewer training tokens (up to 40Γ—; 380B instead of 15T tokens) to produce the state-of-the-art MN-Minitron-8B model," but the "instead of" comparison is to the teacher's pretraining budget, not to the cost of training an 8B model from scratch. A careful reader can parse this, but the unqualified framing is likely to mislead. A fair presentation would either (a) report total tokens including teacher pretraining (with amortization assumptions stated), (b) compare against estimated from-scratch pretraining costs for the target size, or (c) explicitly state that the efficiency claim is incremental given an existing teacher.

The Depth Pruning Saliency Criterion Uses a Single Task, Leaving the Generality of the Approach Unestablished

The assumption or constraint. The paper selects which contiguous block of 16 layers to remove from Llama 3.1 8B based on a single downstream task: Winogrande (5-shot accuracy). The justification is that validation loss-based selection fails on this task (Figures 8 and 9), and the paper therefore switches to a task-based metric. However, no evidence is presented that Winogrande-guided selection produces optimal or even good layer removal decisions for other downstream tasks β€” MMLU, GSM8K, HellaSwag, HumanEval, or any of the benchmarks on which the compressed model is ultimately evaluated.

"Based on our empirical analysis (see Figures 8 and 9), we use the Winogrande metric to prune sets of contiguous layers."

The choice of Winogrande specifically is not defended against alternatives; the paper does not explain why this particular task was chosen over MMLU (which tests broader factual knowledge), GSM8K (which tests mathematical reasoning), or any other benchmark.

The consequence. The layer selection procedure is essentially optimizing the pruned model's architecture for performance on a single commonsense reasoning task, then evaluating the resulting model on nine benchmarks spanning reasoning, knowledge, math, code, and summarization. If different tasks are sensitive to the removal of different layer subsets β€” which is plausible given that transformers develop task-specialized circuitry at different depths β€” then the Winogrande-optimal layer removal might be suboptimal for MMLU or GSM8K. The paper's own results hint at this possibility: the depth-pruned model shows dramatically worse GSM8K performance (16.8% vs. 41.2% for width pruning) while the MMLU gap is much smaller (58.7% vs. 60.5%). The paper attributes this to depth pruning generally, but it is possible that a different layer removal pattern β€” one optimized for mathematical reasoning rather than commonsense reasoning β€” would preserve more GSM8K accuracy while sacrificing some Winogrande performance.

More broadly, the paper presents the Winogrande-based criterion as a general improvement over loss-based depth pruning, but it has only demonstrated superiority on the specific task used for selection. A loss-based criterion might actually produce better aggregate performance across the full evaluation suite β€” the paper simply does not test this. The claim that "LM validation loss/PPL-based layer importance fails to produce the most accurate pruned model(s) on downstream tasks" is supported for one downstream task (Winogrande) but not for downstream tasks in general.

What evidence exists in the paper. Figures 8 and 9 compare only one task metric (Winogrande) against validation loss. The paper does not present layer removal evaluations for MMLU, GSM8K, HellaSwag, or other benchmarks that would reveal whether the optimal contiguous block differs by task. Table 1 shows final post-distillation accuracy for the depth-pruned model across nine benchmarks, but these reflect the combined effect of layer selection and distillation, not the effect of layer selection alone. It is possible that distillation partially compensates for suboptimal layer removal (the student learns to route around missing layers), in which case the choice of selection task matters less in practice than the Figure 9 pre-distillation comparison suggests. But without an ablation comparing final accuracy under different layer selection criteria, the practical importance of the Winogrande-based selection β€” as opposed to simply choosing a contiguous block and relying on distillation to recover β€” is unquantified.

Mitigation status. The paper does not address this limitation. The choice of Winogrande is justified by the observed loss-accuracy disconnect in Figures 8 and 9, but the paper does not extend this analysis to determine whether the disconnect is specific to Winogrande or generalizes across tasks. A practitioner replicating this approach on a new teacher model must either (a) accept Winogrande as the saliency metric without evidence that it transfers, (b) conduct their own multi-task layer removal evaluation (which requires evaluating each candidate architecture on all tasks of interest β€” expensive but feasible pre-distillation), or (c) fall back to loss-based selection and accept the risk demonstrated in Figure 9. The paper provides no guidance on this choice.

All Results Are on a Single Proprietary Distillation Dataset, Making Generality to Other Data Distributions Unknown

The assumption or constraint. Every experiment in the paper β€” teacher correction, importance estimation, pruning, and distillation β€” uses the Nemotron-4 curated continued training (CT) dataset, an NVIDIA-proprietary dataset whose composition, size, and statistical properties are not disclosed. The paper's central claim is that teacher correction enables effective compression when the original pretraining data is unavailable and a different dataset must be used for distillation. But this claim is tested with exactly one alternative dataset, and no characterization of how different this dataset is from the teachers' original training data (Mistral NeMo's undisclosed data, Llama 3.1's 15T-token proprietary mixture) is provided.

The paper states in the Teacher Correction section:

"We hypothesize this is due to the change in distribution of sub-word tokens across the original dataset the teacher model was trained on vs. the dataset being distilled on."

But the magnitude of this distribution shift β€” how different Nemotron-4 CT's token distribution is from Mistral NeMo's or Llama 3.1's training data β€” is never quantified. The datasets could be quite similar (both are likely large-scale web-crawled text corpora with similar tokenization), making the distribution shift mild and the teacher correction burden light. Or they could be substantially different, making the 100B-token correction phase necessary and perhaps insufficient under larger shifts.

The consequence. For a practitioner with a specific distillation dataset in mind (e.g., domain-specific text for a medical or legal application, code-heavy data for a programming assistant, or multilingual data for a translation system), the paper provides no evidence about whether teacher correction will work, how many tokens of correction will be needed, or whether the approach fails under large distribution shifts. The hypothesis about sub-word token distribution shift is plausible but untested beyond a single dataset. If Nemotron-4 CT happens to be relatively similar to the teachers' training data in token-level statistics, the paper's results may overstate the effectiveness of teacher correction for more divergent datasets. Conversely, if Nemotron-4 CT is substantially different, the results may understate how little correction is needed for near-distribution scenarios.

The paper also provides no characterization of the Nemotron-4 CT dataset itself β€” its total size, domain composition, language distribution, or overlap with common public corpora. This makes it impossible for practitioners to assess whether their own distillation dataset is likely to be "similar enough" to Nemotron-4 CT for the paper's 100B-token correction recipe to transfer, or whether they should expect to need more (or less) correction.

What evidence exists in the paper. The only evidence about distribution shift comes from Figure 4, which shows that using the uncorrected teacher on Nemotron-4 CT produces substantially higher student validation loss than using the corrected teacher. This demonstrates that some distribution mismatch exists β€” enough that correction matters β€” but does not characterize the mismatch's magnitude or nature. The paper does not report token distribution statistics, vocabulary overlap metrics, or any other quantitative measure of dataset similarity that would allow practitioners to calibrate their expectations. The teacher correction is applied with identical hyperparameters (100B tokens, low learning rate, 120 warm-up steps) to both Mistral NeMo 12B and Llama 3.1 8B, suggesting the paper treats the approach as dataset- and model-agnostic, but this claim is supported by only two model-data pairs, both using the same distillation dataset.

Mitigation status. The paper does not address this limitation. The dataset is described only by citation to Parmar et al. (2024) and the statement that it is the "Nemotron-4 curated continued training (CT) dataset." No analysis of distribution shift, no experiments with alternative distillation datasets, and no guidance on how to assess whether teacher correction will be sufficient for a new dataset are provided. A practitioner with a custom distillation dataset is flying blind: the paper demonstrates that the approach works on Nemotron-4 CT for two teacher models, but provides no evidence that this generalizes.

The Depth-Pruned Model's Reasoning Collapse Is Documented but Neither Explained Nor Resolved

The assumption or constraint. Depth pruning from 32 to 16 layers on Llama 3.1 8B produces a dramatic collapse in GSM8K accuracy: the base depth-pruned model achieves 16.8% versus 41.2% for the width-pruned variant and 48.6% for the 8B teacher (Table 1). This is a 31 percentage point drop from the teacher, compared to only a 7.4 point drop for the width-pruned variant. The paper notes this in the Insights section:

"Reasoning ability for base variants appears to be impacted significantly for the depth pruned version, with GSM8K accuracy at 16.8% compared to 41.24% for the width pruned version. However, the gap reduces with instruct tuning."

After instruction tuning (Table 2), the gap narrows to 71.11% (depth) versus 79.76% (width) on GSM8K β€” still an 8.65 point gap but substantially improved from the 24.4 point base-model gap. The paper presents this as an observation without attempting to explain the mechanism: why does halving depth disproportionately damage mathematical reasoning, and why does instruction tuning partially recover it?

The consequence. For practitioners whose primary use case involves mathematical reasoning, code generation, or multi-step logical inference, the depth-pruned model is substantially weaker than the width-pruned model at the same parameter count, and the paper provides no guidance on whether this is an inherent limitation of depth pruning or a correctable artifact of the specific layers removed. If the GSM8K degradation is caused by the removal of specific layers that implement crucial reasoning subroutines (e.g., layers that perform arithmetic or track variable bindings), then a different contiguous block selection β€” perhaps optimized for GSM8K rather than Winogrande β€” might substantially recover the lost capability. If the degradation is inherent to having fewer sequential processing steps (a "depth bottleneck" that prevents the model from performing sufficient serial computation), then no layer selection will fix it, and practitioners who need reasoning capability should avoid depth pruning entirely at aggressive ratios.

The instruction tuning recovery is interesting but puzzling: it suggests that the depth-pruned model retains the knowledge needed for mathematical reasoning (since fine-tuning can partially elicit it) but cannot access it effectively in a zero-shot or few-shot setting. This pattern β€” knowledge is present but not retrievable without task-specific training β€” is characteristic of models whose forward computation has been disrupted in ways that task-specific fine-tuning can route around. But the paper does not investigate whether the recovery comes from the math-specific SFT phase of the alignment recipe (suggesting the problem is retrieval, not capability) or from general instruction-following improvements.

What evidence exists in the paper. The GSM8K collapse is documented in Table 1 (16.8% vs. 41.2% vs. 48.6%) and Table 2 (71.11% vs. 79.76% vs. 83.8%). The paper does not provide layer-level analysis β€” no probing of what computations the removed layers performed, no comparison of the depth-pruned model's internal representations to the teacher's, and no test of whether different layer removal choices change the GSM8K degradation pattern. The Insights section describes the observation but offers no mechanistic hypothesis. The gap on other reasoning-heavy benchmarks is less extreme: MMLU drops from 65.3% to 58.7% (a 6.6 point gap), and HumanEval drops from 24.8% to a value not directly reported for the depth variant in the base-model n=20 setting. This suggests the degradation is not uniform across reasoning tasks, making the lack of analysis more consequential β€” if the degradation were uniform, it could be dismissed as a general capacity loss, but its task-specificity demands explanation.

Mitigation status. The paper partially mitigates the practical impact by demonstrating that instruction tuning substantially narrows the gap (GSM8K 71.11% vs. 79.76% after alignment), and by providing both width-pruned and depth-pruned variants so practitioners can choose based on their accuracy-throughput priorities. However, the paper does not explain the mechanism, does not explore whether the degradation can be further reduced through better layer selection or training, and does not characterize which types of reasoning are most affected. A practitioner who needs strong reasoning performance and high throughput is left with an unresolved tension: choose the width-pruned model (better accuracy, 1.8Γ— speedup) or the depth-pruned model (worse accuracy, 2.7Γ— speedup), with no understanding of whether a third option (different layer removal, hybrid width+depth pruning, or specialized training) might resolve the tradeoff.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper is best understood as a practical re-enablement rather than a paradigm shift. It does not propose new compression theory, a new pruning algorithm, or a novel distillation objective. Instead, it identifies and solves the specific engineering obstacle that was preventing the established Minitron compression recipe from working in the increasingly common scenario where the original pretraining data is unavailable β€” and in doing so, it converts a "cannot be done" assumption into a "here is the standard recipe" workflow.

The magnitude of this contribution should be judged by what it unlocks, not by its conceptual novelty. Before this work, a practitioner who downloaded Llama 3.1 8B from Hugging Face and wanted to produce a 4B variant faced an unappealing set of options: (1) pretrain a 4B model from scratch on whatever data they had (prohibitively expensive), (2) apply pruning and distillation using their available dataset and accept degraded results from a miscalibrated teacher (the "suboptimal guidance" the paper documents), or (3) abandon compression entirely and use the 8B model as-is despite its deployment cost. This paper demonstrates that option (2) can be rehabilitated into a competitive strategy through a single additional step β€” teacher correction β€” that costs ~100B tokens and can even be overlapped with distillation to avoid increasing wall-clock time. The result is that the Minitron pipeline, which previously required data access that most practitioners lacked, now works with data they do have. This is an engineering unlock: it does not expand the theoretical frontier of what compression can achieve, but it dramatically expands the set of practitioners who can achieve it.

The conceptual reframing that makes this possible is the separation of teacher accuracy from teacher calibration-for-distillation. The paper's central empirical finding β€” that teacher correction produces a teacher whose downstream accuracy is essentially unchanged (Table 1: Mistral NeMo 12B-FT vs. 12B-Base show mixed, minor differences) yet whose supervision quality improves substantially (Figure 4: >6% reduction in student validation loss) β€” demonstrates that these are distinct properties. A teacher can be "accurate" (producing correct answers) while being "miscalibrated" (producing logit distributions that are suboptimal for distillation on a new data distribution). This distinction is not theoretically novel β€” it follows from the well-known sensitivity of logit-based distillation to the teacher's output distribution β€” but the paper is the first to systematically demonstrate it as the primary failure mode when distilling LLMs on unseen data, and to show that a lightweight fine-tuning phase specifically targeting calibration (not accuracy) resolves it. This finding has implications beyond the Minitron pipeline: any scenario where a pretrained model serves as a distillation teacher on out-of-distribution data β€” domain-specific compression, cross-lingual distillation, distillation from API-based models with unknown training data β€” should now include a teacher calibration assessment and potentially a correction phase.

The paper also introduces a diagnostic finding that should shift how depth pruning research evaluates its saliency criteria. The disconnect between validation loss and downstream task accuracy in Figures 8 and 9 β€” where loss-optimal layer removal produces random performance on Winogrande while a different removal pattern preserves meaningful accuracy β€” exposes a failure mode of perplexity-driven pruning that the depth pruning literature had not systematically documented. Prior work (ShortGPT, Shortened LLaMA, the original Minitron) used validation loss or its approximations as the default saliency metric under the implicit assumption that lower perplexity implies better downstream performance. This paper provides a clear counterexample, demonstrating that this assumption can break for structural pruning decisions in ways that produce qualitatively useless models. The implication is not that Winogrande is the "correct" metric β€” the paper itself does not establish this β€” but rather that the choice of saliency metric for depth pruning is a consequential design decision that should be validated against the downstream tasks of interest, not assumed away by optimizing perplexity. This is a methodological contribution: it adds a new validity check (does your pruning saliency correlate with your evaluation metric?) to the depth pruning research protocol.

The paper also provides the first systematic accuracy-throughput characterization of width versus depth pruning at the same compression ratio and parameter count, derived from the same teacher. The finding that width pruning preserves accuracy better (Table 1) while depth pruning delivers superior throughput (Figure 10: 2.7Γ— vs. 1.8Γ—) is intuitive but was not previously quantified in a controlled comparison. This gives practitioners a concrete basis for choosing between compression strategies based on their deployment constraints, rather than treating pruning axis as an implementation detail. It also surfaces the open question of whether intermediate operating points (combined width and depth reduction) exist on a more favorable part of the Pareto frontier β€” a question the paper raises by its silence on combined pruning.

The research directions that become more attractive after this work include: (1) lightweight teacher calibration methods that achieve correction at even lower token budgets (the paper explicitly flags LoRA and LayerNorm-only tuning as unexplored alternatives), (2) multi-task saliency criteria for depth pruning that avoid the single-task overfitting risk inherent in the Winogrande-based approach, (3) systematic characterization of when distribution shift breaks distillation (what statistical distance metrics predict the need for teacher correction?), and (4) combined width-depth compression strategies that explore the full architecture design space rather than treating the two axes independently. The research directions that become less attractive include: (1) from-scratch pretraining of small models as the default approach for obtaining efficient deployment variants when a larger open-weight model already exists (the 40–150Γ— token reduction makes compression the clearly preferable starting point, at least for the compression ratios tested), and (2) perplexity-only evaluation of depth pruning methods without downstream task validation (the Figure 8-9 disconnect establishes that perplexity can be misleading for structural decisions, raising the bar for what counts as adequate evaluation in depth pruning research).

Finally, the paper reconciles a tension between the original Minitron work's strong results (achieved with access to original training data) and the practical reality that most open-weight models do not release their training data. The original Minitron paper demonstrated that compression could work, but left open the question of whether it would work for practitioners who lack the original data. This paper closes that gap: yes, it works, provided you first correct the teacher. The reconciliation is empirical rather than theoretical β€” it demonstrates feasibility rather than explaining why β€” but it transforms Minitron from a method that works in controlled settings to a method that works in the setting most practitioners actually inhabit.

Follow-Up Research This Work Enables

Characterizing the minimum effective dose for teacher correction. The paper applies a ~100B token correction phase without ablating the budget, leaving open the question of how little correction is sufficient. A direct follow-up would train a series of MN-Minitron-8B students using teachers corrected for 5B, 10B, 25B, 50B, and 100B tokens (and an uncorrected baseline), and evaluate both the student's final downstream accuracy (MMLU, GSM8K, HellaSwag, HumanEval β€” not just validation loss) and the teacher's logit calibration as a function of correction budget. The practical payoff is substantial: if 10B tokens suffice, the correction overhead drops from 20% of the distillation budget to 2%, making the pipeline meaningfully cheaper. A variant of this experiment would test whether correction effectiveness depends on the similarity between the teacher's original training data and the distillation dataset β€” using multiple distillation datasets with known statistical distances from the teacher's training distribution (e.g., the same domain, a related domain, an unrelated domain) to map out how the required correction budget scales with distribution shift. The paper's hypothesis about "sub-word token distribution" shift motivating teacher correction could be tested directly by computing token frequency correlations between datasets and seeing whether they predict correction need.

Multi-task depth pruning saliency criteria. The paper uses Winogrande alone to select which 16 contiguous layers to remove from Llama 3.1 8B, but evaluates the resulting model on nine benchmarks. A direct follow-up would compare the final (post-distillation) accuracy of depth-pruned models whose layer removal was guided by different single-task metrics (Winogrande, MMLU, GSM8K, HellaSwag, HumanEval) and by a multi-task aggregate (e.g., average accuracy across all tasks, or a Pareto-optimal selection that avoids catastrophic degradation on any single task). The experiment would reveal whether (a) the choice of saliency task matters substantially after distillation recovers accuracy, (b) Winogrande-guided selection happens to be near-optimal across tasks or is overfit to commonsense reasoning, and (c) multi-task selection provides meaningful improvements over single-task selection. The experiment would also test the paper's implicit claim that task-based selection is superior to loss-based selection in general (not just for the specific task used in selection) by including a loss-guided baseline whose final downstream accuracy is evaluated across all nine benchmarks. This would close the loop on the Figure 8-9 finding by showing whether the pre-distillation disconnect between loss and task accuracy survives distillation or is washed out by retraining.

Combined width and depth pruning at fixed parameter budgets. The paper treats width and depth pruning as independent alternatives, both targeting ~4.5B parameters from Llama 3.1 8B, but never explores intermediate architectures that apply moderate reductions to both axes. A direct follow-up would produce a family of 4.5B models spanning the width-depth spectrum: e.g., (width-only: hidden 3072, 32 layers), (mostly-width: hidden 3328, 28 layers), (balanced: hidden 3584, 24 layers), (mostly-depth: hidden 3840, 20 layers), (depth-only: hidden 4096, 16 layers). Each model would be evaluated for downstream accuracy (the full Table 1 suite) and inference throughput (TensorRT-LLM FP8 on H100). The goal is to map the actual Pareto frontier between accuracy and throughput, testing whether the binary tradeoff the paper presents (width = better accuracy, depth = better throughput) is a genuine dichotomy or whether intermediate points achieve near-depth throughput with near-width accuracy. The original Minitron work included a NAS phase that could in principle discover such architectures; this experiment would determine whether the paper's decision to skip NAS left meaningful gains on the table. A second dimension of this experiment would test whether the optimal width-depth allocation depends on the teacher model and compression ratio β€” e.g., does Mistral NeMo 12B β†’ 8B benefit from depth pruning in ways that Llama 3.1 8B β†’ 4B does not?

Diagnosing the depth pruning reasoning collapse. The paper documents that depth pruning Llama 3.1 8B from 32 to 16 layers causes GSM8K accuracy to drop from 48.6% to 16.8% (a 31.4 point gap), while width pruning to the same parameter count causes a much smaller drop to 41.2% (a 7.4 point gap). A mechanistic follow-up would investigate why mathematical reasoning is disproportionately affected. Specific experiments: (1) Evaluate the depth-pruned model's accuracy as a function of the number of reasoning steps required in GSM8K problems (stratifying problems by the number of operations needed), testing the hypothesis that the degradation scales with required sequential computation depth. (2) Probe the internal representations of the pruned model at each remaining layer during math problem solving, comparing against the teacher to identify which computational stages (number parsing, operation selection, intermediate value tracking, final answer extraction) are disrupted by layer removal. (3) Test whether the specific layers removed (16–31) contain specialized circuitry for arithmetic or symbolic manipulation by evaluating the teacher with those layers ablated (zeroed or skipped) on math-specific probing tasks. (4) Compare the depth-pruned model's GSM8K degradation when layers 1–16 are removed versus 16–31, testing whether the collapse is specific to removing middle-to-late layers (as the paper's contiguous selection does) or is a general property of halving depth regardless of which layers are removed. This would determine whether the collapse is a "depth bottleneck" (insufficient serial computation regardless of which layers remain) or a "capability loss" (removal of specific layers that had learned arithmetic subroutines), with different implications for whether the problem is fundamental to depth pruning or addressable through better layer selection.

Stress-testing teacher correction under controlled distribution shifts. The paper tests teacher correction on a single distillation dataset (Nemotron-4 CT) whose relationship to the teachers' original training data is uncharacterized. A rigorous follow-up would construct a controlled experiment: take a teacher model where the original training data is available (e.g., a model trained on a public corpus like FineWeb or Dolma), deliberately distill on datasets at varying known distribution distances (same corpus, different domain subset, entirely different domain, different language), and measure both the uncorrected-teacher degradation and the correction budget required to recover. The output would be a "correction requirement curve" β€” a mapping from distribution distance (quantified by vocabulary overlap, token frequency correlation, or embedding space divergence) to necessary correction tokens. This would give practitioners a principled way to estimate their correction budget rather than blindly applying 100B tokens, and would test the paper's hypothesis that sub-word token distribution shift is the underlying mechanism. A variant would compare teacher correction against alternative approaches for handling distribution mismatch, including: mixing a small amount of in-distribution data into the distillation dataset, using a weighted combination of corrected and uncorrected teacher logits, or applying logit temperature scaling to the uncorrected teacher. If any of these cheaper alternatives achieves comparable results, the practical case for full teacher correction weakens.

Scaling the approach to larger compression ratios and smaller target models. The paper compresses 12B β†’ 8B (1.5Γ— reduction) and 8B β†’ 4B (2Γ— reduction). A natural extension tests more aggressive ratios: 8B β†’ 2B (4Γ—), 12B β†’ 4B (3Γ—), and 12B β†’ 2B (6Γ—). The key question is whether the teacher correction + pruning + distillation recipe continues to work at higher compression ratios, or whether there is a threshold beyond which the pruned model's capacity is insufficient to absorb the teacher's knowledge regardless of calibration quality. The original Minitron work demonstrated some results at higher compression ratios, but always with original data access; testing whether teacher correction remains sufficient when the student is much smaller would establish the practical limits of the approach. A related extension would apply the recipe to a smaller teacher (e.g., 4B β†’ 2B or 2B β†’ 1B) to determine whether the benefits of pruning + distillation over from-scratch training persist at smaller scales where from-scratch training is cheaper in absolute terms. This would map out the regime where compression is the clearly dominant strategy versus the regime where the overhead of teacher training (even if amortized) makes from-scratch training competitive.

Practical Applications and Downstream Use Cases

Rapid model family creation from any open-weight checkpoint. The most direct application enabled by this work is the ability to take any publicly released large language model β€” Llama 3.1, Mistral, Qwen, Gemma, or future releases β€” and produce a family of smaller, faster variants targeting different deployment scales, without needing access to the pretraining data pipeline. Concretely, an organization that has standardized on Llama 3.1 8B for its application can now produce a 4B variant (for edge deployment or high-throughput serving) using ~194B total tokens (100B correction + 94B distillation) on whatever reasonable text corpus they have available, rather than needing to pretrain a 4B model from scratch or accept the inference cost of the 8B model everywhere. The 2.7Γ— throughput improvement for the depth-pruned variant (Figure 10) means serving costs drop proportionally β€” a deployment handling 1M queries/day on Llama 3.1 8B could handle the same volume with roughly one-third the GPU allocation using the 4B depth-pruned model, or could serve 2.7Γ— the query volume on the same hardware. The 1.8Γ— improvement for the width-pruned variant offers a more moderate cost reduction with better accuracy preservation (particularly on reasoning tasks; Table 1 shows GSM8K at 41.2% vs. 16.8% for the base width and depth variants respectively). Organizations can choose their operating point on this accuracy-throughput curve based on their specific latency requirements, accuracy sensitivity, and task distribution.

Cost-efficient distillation for domain-specific deployment. A common deployment pattern is taking a general-purpose LLM and adapting it to a specific domain (medical, legal, financial, code) through fine-tuning. This paper's pipeline enables a more efficient variant: rather than fine-tuning the full 8B model and deploying it at full cost, a practitioner can first apply teacher correction on their domain data (~100B tokens), then prune to 4B, then distill on the same domain data (~94B tokens), producing a domain-adapted 4B model at roughly half the inference cost. The teacher correction phase serves double duty here β€” it both calibrates the teacher for distillation and begins the domain adaptation β€” and the distillation phase transfers both the teacher's general knowledge and its domain-specific calibration to the smaller model. The paper's instruction tuning results (Table 2) demonstrate that this pipeline produces models that are competitive with or superior to larger models on domain-relevant tasks: MN-Minitron-8B achieves 67.6% on BFCLv2 (function calling) versus Llama 3.1 8B's 44.3%, and 87.1% on GSM8K versus 83.8%. A medical domain deployment could target similar gains on clinical benchmarks while reducing serving costs by the 1.8–2.7Γ— factors reported in Figure 10. The key practical requirement is that the practitioner has a domain-relevant text corpus for the correction and distillation phases β€” but this is typically available (medical records, legal documents, code repositories) even when the original pretraining data is not.

On-device and edge deployment with depth-pruned models. The 2.7Γ— throughput improvement for depth pruning (Figure 10) is particularly significant for on-device deployment scenarios where GPU memory and compute are tightly constrained β€” mobile phones, laptops, embedded systems, and edge servers. A 4B depth-pruned model derived from Llama 3.1 8B requires roughly half the GPU memory for parameter storage (4.5B vs. 8B parameters) and processes tokens through half as many transformer layers (16 vs. 32), directly reducing both the memory footprint and the per-token latency. The paper's findings about instruction tuning partially recovering depth pruning's reasoning degradation (GSM8K going from 16.8% base to 71.11% instruct) mean that a depth-pruned model that has been instruction-tuned for the target application may achieve acceptable accuracy despite the base-model regression. This enables use cases where a full 8B model cannot meet latency or memory constraints β€” real-time voice assistants, on-device code completion, mobile translation β€” to deploy a model derived from a state-of-the-art teacher rather than settling for a weaker from-scratch small model. The paper's comparison against Gemma 2 2B (Table 1) shows that even the depth-pruned 4B model substantially outperforms the smaller from-scratch model on most benchmarks, suggesting that compression from a strong teacher is preferable to using a weaker but natively small model when both fit within the deployment constraints.

Rapid prototyping of model variants for A/B testing. For organizations running production LLM services, the ability to quickly produce model variants at different size-speed tradeoffs enables systematic A/B testing of the accuracy-latency Pareto frontier with real user traffic. Rather than committing to a single compressed model based on benchmark evaluations alone, a team can produce a width-pruned variant (better accuracy, 1.8Γ— speedup) and a depth-pruned variant (worse accuracy, 2.7Γ— speedup) from the same teacher using the same pipeline, deploy both alongside the full teacher, and measure actual user satisfaction, task completion rates, and latency tolerance. The paper's ~94B token distillation budget for 4B models means each variant costs a manageable amount of compute to produce β€” small enough that generating multiple candidates for testing is feasible. The parallel teacher correction variant (Figure 5) further reduces turnaround time by overlapping correction with distillation. This application is particularly valuable because benchmark evaluations (Tables 1 and 2) may not capture the latency-sensitivity of real users or the specific task mix of a production deployment; the ability to cheaply generate and test multiple model variants lets the deployment team optimize for their actual objective function rather than proxy metrics.