SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-training

ArXiv: 2605.08738

🎯 Pitch

After 400B tokens of continued pretraining, a dozen different expert pruning and merging methods all converge to nearly identical performance—meaning the choice of compression metric barely matters compared to the training recipe and schedule. This insight unlocks a practical path to compress Qwen3-Next-80A3B down to a 23A2B model that retains 86.5% of the teacher’s benchmark score while requiring 3.4× fewer parameters.

---

1. Executive Summary

This paper systematically studies how to compress a pretrained mixture-of-experts (MoE) language model at scale, analyzing structured pruning across depth, width, and expert dimensions combined with knowledge distillation, using the Qwen3-Next-80A3B model as the base architecture. The authors investigate four key design axes: whether pruning provides a stronger initialization than training from scratch, how expert compression strategies (pruning vs. merging, with a proposed partial-preservation expert merging strategy that retains half of target experts intact) affect final performance after 400B-token continual pretraining, what training objective combinations work best—introducing multi-token prediction (MTP) distillation that supervises future tokens beyond the next one—and whether progressive pruning schedules (depth-first, width-first, or joint two-stage transitions) outperform one-shot compression. The findings establish that pruned initialization with KD recovers 86.5% of the teacher's average benchmark score despite ~3.4× compression, that progressive pruning consistently outperforms one-shot compression (e.g., MMLU improving from 75.86 to 77.39 using depth-first scheduling), and that combining LM loss with next-token KD plus MTP KD yields consistent gains, while also establishing that different one-shot expert compression methods converge to similar final performance after large-scale continual pretraining—meaning the choice of compression metric matters far less than the training recipe and progressive schedule.

2. Context and Motivation

The Core Problem: We Don't Know How to Compress MoE Models During Pretraining

The fundamental question this paper tackles is practical and urgent: given a pretrained, high-capacity mixture-of-experts (MoE) model, what is the most effective way to produce a smaller, deployable version that retains most of its capability? This matters because MoE architectures (Shazeer et al., 2017) have become the dominant paradigm for scaling large language models—they appear in Qwen (Team, 2024; 2025a; 2026), Gemini (Team, 2025a), Mixtral (Jiang et al., 2024), and others—but their training and serving costs remain substantial. The appeal of MoE is clear: by activating only a subset of experts per token, these models achieve far greater total capacity at lower per-token inference cost than equivalently-sized dense models. But this efficiency hides a problem: the full model remains expensive to pretrain (requiring enormous compute clusters) and the total parameter count determines memory requirements for serving (loading all experts into GPU memory even if only a fraction fires on any given token).

The paper identifies a specific, unresolved tension: structured pruning and knowledge distillation are well-studied for dense models, but their extension to MoE models at pretraining scale introduces fundamentally new design choices that no prior work systematically addresses. A dense model has two main axes of structure—depth (number of layers) and width (hidden dimensions). An MoE model adds a third, qualitatively different axis: experts, which can be pruned (removing entire expert MLPs) or merged (combining multiple expert parameters into fewer). This additional dimension creates a combinatorial explosion of compression decisions: which experts to keep, whether to prune or merge them, how to measure expert importance for these decisions, and how to schedule the compression across all three axes (depth, width, experts) simultaneously. Prior work on dense models provides no guidance here because it never had to confront expert-level structured compression, let alone at the scale of hundreds of experts per layer (the Qwen3-Next-80A3B base model in this paper has 512 experts per MoE module).

Why This Matters: The Pretraining-Serving Cost Gap

The practical stakes are high for two reasons that the paper makes clear through its experimental design:

Pretraining cost scales with total parameters. The Qwen3-Next-80A3B is an 80B-parameter model that activates only 3.8B parameters per token. The ~3.4× compression targets studied in this paper (80A3B → 23A2B, 24A2B → 6A1B) don't just reduce memory—they directly reduce the compute required for any future pretraining or continued training of the compressed model. If a smaller model can be trained from a pruned initialization rather than from scratch, the savings compound: you reuse the teacher's pretraining investment as a starting point, and subsequent training requires less FLOPs per step because the model is smaller.

Serving memory, not just FLOPs, is the bottleneck. Despite MoE's per-token efficiency, the full parameter footprint must reside in GPU memory for inference. A model with 512 experts per layer and 80B total parameters requires multiple GPUs just to hold the weights, even if only a handful of experts activate. The paper explicitly highlights this in Appendix A.7 (Table 11): the original Qwen3-Next-80A3B requires 156.56 GB peak memory, necessitating multi-GPU deployment with tensor or pipeline parallelism, while the compressed SlimQwen-23A2B fits within 43.30 GB—comfortably within a single 80GB GPU. This single-GPU deployability eliminates the communication overhead of distributed inference and dramatically reduces the infrastructure cost for serving. The paper frames this not as a minor optimization but as a qualitative change in deployment feasibility: models that fit on one GPU can be served with simpler infrastructure, lower latency, and greater reliability.

The gap between "can compress" and "should compress this way" remains wide. The paper enters a landscape where compression works in principle but the recipe for making it work well—especially on knowledge-intensive benchmarks that are sensitive to representation quality—is poorly understood. Getting 86.5% of teacher performance at 3.4× compression is good, but the paper's real contribution is showing which design choices separate adequate compression from excellent compression. The difference between training from scratch and training from a pruned initialization is ~12 points on average benchmark score (61.66 vs. 73.45 in Table 1). The difference between worst and best expert compression strategies after 400B tokens of training is small (they converge), but the difference between one-shot and progressive pruning is ~1.5 points on MMLU (75.86 vs. 77.39 in Table 5)—substantial at this performance level. These are the kinds of margins that determine whether compression is practically useful or just academically interesting.

Where Prior Approaches Fall Short

The paper identifies specific limitations in existing work along three axes:

1. Dense-model techniques don't transfer cleanly to MoE. Structured pruning methods developed for dense LLMs—ShearedLLaMA (Xia et al., 2024b) for width pruning, SliceGPT (Ashkboos et al., 2024) for matrix-level compression, ShortGPT (Men et al., 2024) and ShortenedLLaMA (Kim et al., 2024) for depth pruning—operate on architectures where every parameter is always active. In MoE models, the routing mechanism introduces sparsity that is dynamic and token-dependent. An expert might be rarely activated but crucial for a specific knowledge domain; simply removing it based on average activation frequency risks catastrophic forgetting of specialized knowledge. The paper's explicit comparison of expert importance metrics (frequency-based, soft-logits, REAP-based in Equation 6-7) demonstrates that different metrics capture different aspects of expert utility, and a metric that works well for one-shot evaluation may not correlate with performance after continued training.

2. Existing MoE compression studies are limited in scale and scope. The paper cites several prior works on MoE compression: M-SMoE (Li et al., 2024b) and REAP (Lasby et al., 2025b) propose expert merging, while Lu et al. (2024) prune redundant experts. SlimMoE (Li et al., 2025) applies distillation for expert slimming, and Cao et al. (2025) merge MoE layers into dense layers. However, the paper argues that these studies share a critical limitation:

"While recent studies (Jaiswal et al., 2025) thoroughly evaluate the one-shot performance of various expert compression methods, their efficacy following large-scale continual pretraining remains unexplored."

This is a precise identification of the gap: we know what happens immediately after compression (one-shot performance), but not what happens after the compressed model is trained on hundreds of billions of additional tokens. The paper's finding that different one-shot expert compression methods converge to similar final performance after 400B tokens of training (Table 2) is not obvious a priori—it could easily have been the case that some compression methods preserve "trainability" better than others. The fact that they converge means that much of the prior literature's focus on optimizing the one-shot compression metric may be misplaced when the real goal is post-compression continued training.

3. The training recipe for post-compression recovery is understudied for MoE. On the training side, the paper identifies three gaps:

• KD vs. LM loss debate is unresolved for MoE. Minitron (Muralidharan et al., 2024) uses distillation for dense models; DarwinLM (Tang et al., 2025) uses standard LM loss; SlimMoE (Li et al., 2025) uses KD. None of these works systematically compare the two objectives or study their combination on MoE models at scale. The paper's finding that hybridizing KD with LM loss outperforms pure KD (Table 3: MMLU improves from 74.16 to 74.93 when adding LM loss) challenges the conventional wisdom that distillation alone is sufficient for recovery.

• Distillation objectives beyond next-token prediction are unexplored for compression. The paper introduces Multi-Token Prediction (MTP) distillation, extending the standard next-token KD objective (originally proposed by Gloeckle et al., 2024 for pretraining from scratch) to the compression setting. No prior compression work has explored supervising the student on multiple future tokens simultaneously, despite theoretical arguments that this enriches the training signal by forcing the model to learn longer-range dependencies. The paper's results show this not only improves benchmark performance but also yields practical gains for speculative decoding (Table 4: MTP KD improves acceptance rates for multi-token generation by 8–20 percentage points across benchmarks).

• Progressive pruning schedules are intuitive but unvalidated for MoE. The idea of gradually reducing model capacity rather than doing it all at once is not new—it appears in iterative pruning literature—but no prior work has systematically compared depth-first, width-first, and joint progressive schedules for MoE compression at pretraining scale. The paper's finding that all progressive schedules outperform one-shot compression, but depth-first performs best (Table 5: 77.39 vs. 75.86 MMLU), provides concrete guidance that was previously unavailable.

4. The "pruning as initialization" question remains open. A long-standing debate in model compression is whether pruning a larger trained model and then continuing training is better than simply training the target architecture from scratch with the same total compute budget. For dense models, the evidence is mixed and task-dependent. For MoE models, the question is more complex because pruning involves not just removing parameters but potentially reorganizing them (via expert merging), which could either preserve beneficial parameter relationships or create harmful interference patterns. The paper's clear demonstration that pruned initialization dominates random initialization by ~12 points (Table 1) settles this question for the MoE setting, at least within the studied architecture and training regime.

How This Paper Positions Itself

The paper frames its contribution as a systematic empirical study, not a single novel method. This is a deliberate positioning choice: rather than optimizing one component in isolation and claiming state-of-the-art on a narrow metric, the paper explores the full design space of MoE compression—initialization, expert compression strategy, training objective, and pruning schedule—under controlled experimental conditions. This positions it as providing practical guidance for practitioners who need to make concrete decisions when compressing MoE models:

"These results offer practical guidance for efficient MoE compression at scale."

The paper's structure reflects this positioning. Rather than a single "proposed method" section, it has parallel investigations: the partial-preservation expert merging strategy (Algorithm 1) is proposed as a simple heuristic, not as a theoretically optimal solution; the MTP distillation objective (Equations 9–12) is adapted from prior work on pretraining rather than being a fundamentally new loss function; and the progressive pruning schedules are explored empirically without theoretical justification. The contribution is the integration and systematic evaluation of these components, not any individual piece.

This positions the paper in contrast to two common patterns in the compression literature: (1) works that propose a single new compression metric or algorithm and evaluate it only in the one-shot setting, and (2) works that compress dense models and don't address the expert dimension at all. By explicitly studying the post-compression training phase at scale (120B–400B tokens, which is large enough that training dynamics matter), the paper addresses the end-to-end practical problem rather than an isolated subproblem.

The paper also positions itself relative to the pretraining-vs-compression tradeoff literature implicitly. While not framed as a FLOPs-matched comparison in the style of Hoffmann et al. (2022), the paper's experiments answer a related question: given a fixed compute budget for continued training after compression, what recipe maximizes the final model's performance? The answer isn't just "use KD"—it's "use pruned initialization + hybrid KD/LM loss + MTP KD + progressive depth-first scheduling + partial-preservation expert merging." This recipe-level guidance, validated across multiple benchmarks and training scales, is what distinguishes the paper from single-metric compression studies.

It's worth noting what the paper explicitly does not do. It does not claim to have found the optimal compression metric for experts—in fact, it claims that the choice of metric matters little after continued training (Table 2). It does not claim that progressive pruning is theoretically motivated—the schedules are explored empirically. It does not address dynamic or learned compression (e.g., learning which experts to prune during training rather than using a fixed criterion). And it does not study the interaction between compression and downstream fine-tuning (all evaluations are on base pretrained models). These are acknowledged gaps rather than weaknesses—they define the boundaries of the paper's contribution and point to future work.

3. Technical Approach

This is primarily a systematic empirical study paper whose core idea is that compressing a pretrained MoE model to a smaller target architecture—then training it with a carefully designed combination of structured pruning, expert merging, knowledge distillation, multi-token prediction distillation, and progressive pruning schedules—produces a model that substantially outperforms training the target architecture from scratch, while different one-shot expert compression metrics converge to similar performance after sufficient continued training.

3.1 Reader Orientation

The paper builds a compression pipeline that takes a large, pretrained mixture-of-experts language model (Qwen3-Next-80A3B with 512 experts per MoE layer) and produces a much smaller model (SlimQwen-23A2B with 256 experts per MoE layer) by systematically removing layers, shrinking hidden dimensions, consolidating experts, and then continuing training with a hybrid objective combining standard language modeling, next-token knowledge distillation, and multi-token prediction distillation—all scheduled progressively across two stages rather than all at once. The problem it solves is the pretraining-serving cost gap: large MoE models are expensive to pretrain and require multiple GPUs for inference due to their total parameter footprint, but naively compressing them causes severe performance degradation; the paper's solution is a specific recipe—depth-first progressive pruning + partial-preservation expert merging + hybrid KD-LM-MTP loss—that recovers 86.5% of teacher performance at 3.4× compression after 400B tokens of continued training.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components that operate in sequence:

1. Base MoE Model (Qwen3-Next-80A3B) — the pretrained teacher with 48 transformer blocks, 512 experts per MoE module, 2048 hidden dimensions. This is the source of all knowledge that the compressed model will inherit.

2. One-Shot Structured Pruning Module — applies three types of structural compression simultaneously: (a) depth pruning removes the last 12 transformer blocks (25% of layers), (b) width pruning reduces hidden dimensions from 2048 to 1536 across all modules using activation-based importance scoring computed on a 1024-sample calibration set, and (c) expert compression reduces 512 experts to 256 per MoE module using the proposed partial-preservation merging strategy (keeping top half of experts intact, merging the rest into selected base experts based on weighted importance scores). This produces the initial SlimQwen-23A2B architecture as a one-shot compressed checkpoint.

3. Knowledge Distillation Framework — during continued pretraining, the compressed student model receives supervision from the frozen teacher model through three types of distillation: (a) standard next-token prediction distillation (NTP KD) comparing student and teacher output distributions via KL-divergence, (b) multi-token prediction distillation (MTP KD) comparing student and teacher predictions for future tokens at depths 1 through D using shared MTP modules, and (c) standard language modeling loss (LM loss) as a complementary objective balanced against KD via a linear decay schedule on the mixing weight λ.

4. Multi-Token Prediction (MTP) Modules — stack D additional transformer blocks (one per prediction depth) that take the backbone's hidden representation at position i, concatenate it with the embedding of the (i+k)-th token, project through a linear layer, process through a depth-specific transformer block, and predict the (i+k)-th token. These modules are trained with both LM loss (against ground-truth one-hot labels) and KD loss (against teacher soft targets) for each prediction depth, weighted by a coefficient β that follows cosine decay.

5. Progressive Pruning Scheduler — instead of compressing the model in one shot and training for 400B tokens, the pipeline splits compression into two stages: in Stage 1 (40B tokens), the model is partially pruned (e.g., half the target layer reduction and all width reduction) and trained at intermediate size; in Stage 2 (360B tokens), the remaining compression is applied and training continues to the final architecture. Three schedule variants are explored: depth-first (remove layers first, width second), width-first (reduce width first, depth second), and joint (remove half of both first, remainder second).

Information flows as follows: the pretrained Qwen3-Next-80A3B teacher is loaded → calibration data (1024 samples) is run through it to compute activation-based importance scores for width pruning and expert importance scores (frequency, soft-logits, REAP) for expert compression → the one-shot pruned checkpoint is produced (or alternatively, progressive pruning is applied in two stages) → the compressed model trains on 120B–400B tokens using the hybrid loss (Equation 12) with frozen teacher providing soft targets → the final SlimQwen model is evaluated on 8+ benchmarks (MMLU variants, BBH, GSM8K, EvalPlus, C-Eval, CMMLU, and additional benchmarks in Appendix A.6).

3.3 Roadmap for the Deep Dive

• First, the formal background and notation (Section 3.1 of the paper), establishing the mathematical representation of MoE layers, routing, normalization, and attention mechanisms that all compression operations act upon. This is necessary because compression decisions (which experts to keep, which hidden dimensions to prune) depend on understanding these components' structure.

• Second, the structured pruning methods across all three dimensions (Section 3.2): depth pruning by dropping the last N layers, width pruning by activation-based hidden dimension importance scoring, and expert compression including both expert pruning and the proposed partial-preservation expert merging strategy. This establishes what gets removed/merged and why specific design choices (e.g., last-layer pruning, half-preservation) matter.

• Third, the distillation pretraining framework (Section 3.3), covering two major components: the multi-token prediction (MTP) distillation loss—which extends standard next-token KD to supervise the student on predicting multiple future tokens—and the progressive pruning and distillation schedules that interleave structural compression with fixed-token training phases. This explains how the compressed model recovers performance, not just what architecture it has.

• Fourth, the full training objective (Equation 12) that integrates LM loss, NTP KD loss, MTP LM loss, and MTP KD loss with two balancing hyperparameters λ and β. Understanding this objective is crucial because the paper's ablation studies (Table 3) depend on isolating which components contribute what gains.

3.4 Detailed, Sentence-Based Technical Breakdown

Background and Notation (MoE Architecture Formalization)

The paper formalizes Qwen3-Next as a hybrid-attention MoE model with L transformer blocks. Each block contains either a Gated DeltaNet (Yang et al., 2025b) or a Gated Attention module (Qiu et al., 2025b), with a ratio L_linear : L_full determining how many layers use each attention type. Following each attention module, an MoE module contains N_e regular (routed) experts and N_s shared experts, with RMSNorm applied throughout. The architecture details for the base Qwen3-Next-80A3B are specified in Appendix Table 6: specifically, the model has L = 48 total transformer blocks (12 full attention + 36 linear attention), each full attention block has 16 query heads and 2 key/value heads with head dimension 256, each MoE module contains a total of 512 experts (10 routed experts and 1 shared expert activated per token), the expert intermediate size is 512, and the hidden size is 2048.

The MoE computation for a single input token x ∈ R^{1×d} (where d is the hidden dimension) proceeds as follows. Let there be n experts in total, comprising n_routed routed experts and n_shared shared experts (so n = n_routed + n_shared). Each expert is a SwiGLU MLP defined by:

$Expert(x) = (SiLU(x W_{1e}) \odot (x W_{2e})) W_{3e}$

where W_{1e}, W_{2e} ∈ R^{d \times d_{ff}} are the projection matrices mapping the hidden state up to the intermediate dimension d_{ff}, ⊙ denotes element-wise multiplication, SiLU is the Sigmoid Linear Unit activation, and W_{3e} ∈ R^{d_{ff} \times d} projects back down to the hidden dimension.

What it computes: the expert takes a d-dimensional token representation x, projects it through two separate linear transformations (W_{1e} and W_{2e}) to a higher-dimensional intermediate space of size d_{ff}, applies a gating nonlinearity (SiLU on the first projection gate the second), and projects back to the original hidden dimension. The element-wise multiplication implements a gating mechanism: the SiLU(W_{1e}x) term acts as a learnable gate controlling how much of the (W_{2e}x) transformation passes through.

Why this form: the SwiGLU activation (SiLU-gated linear unit) is chosen over standard ReLU or GeLU because it has been empirically shown to improve training stability and final performance in large language models, and the gating mechanism allows the expert to learn which input dimensions to suppress or amplify rather than applying a fixed nonlinearity.

The router produces top-k gating scores over the routed experts via:

$z(x) = \text{softmax}\left(\text{TopK}(x W_G, k)\right)$

where W_G ∈ R^{d \times n_{routed}} is the router weight matrix, TopK(·, k) retains only the k largest values and sets the rest to -∞ (so their softmax becomes 0), and softmax normalizes the remaining scores to sum to 1. The shared experts use a separate gate: z_s(x) = σ(x w_{sh}) ∈ R^{n_{shared}}, where w_{sh} ∈ R^{d \times n_{shared}} and σ is the sigmoid function. The full MoE output is:

$MoE(x) = \sum_{e=1}^{n_{routed}} z_e(x) Expert_e(x) + \sum_{s=1}^{n_{shared}} z_s(x) Expert_s(x)$

What it computes: for the routed experts, the router selects the top-k most relevant experts for the current token, assigns them normalized gating scores, and computes a weighted sum of their outputs. For shared experts, a separate sigmoid gate produces per-expert scalar weights that are not constrained to sum to 1—these experts always contribute, with their contribution scaled by the gate. The final output is the sum of both the sparse routed-expert mixture and the dense shared-expert contribution.

Why this form: the TopK sparsity is the defining characteristic of MoE architectures—by only activating k out of n_routed experts per token (in the Qwen3-Next-80A3B, k=10 out of 512), the model achieves enormous total capacity with a per-token FLOP cost closer to a dense model with 10 experts' worth of parameters. The shared experts serve as a "background knowledge" pathway that always fires, preventing the model from becoming overly specialized and ensuring basic linguistic competence regardless of routing decisions. The separate sigmoid gate for shared experts (rather than softmax) allows their contribution to be independently scaled per token rather than competing with routed experts for probability mass.

RMSNorm is applied before each sub-layer:

$RMSNorm(X) = \frac{X}{RMS(X)} \odot \gamma, \quad RMS(X)_i = \sqrt{\frac{1}{d} \sum_{j=1}^{d} X_{ij}^2 + \epsilon}$

where RMS(X) ∈ R^{n \times 1} is the root mean square computed over the hidden dimension for each token, γ ∈ R^{1 \times d} is a learnable scale parameter, and ε is a small constant for numerical stability.

What it computes: for each token, compute the RMS of its hidden representation across all dimensions in that token's vector, divide each dimension by this RMS to normalize the scale, then multiply by a learned scale factor γ. This normalizes each token's representation to have roughly unit RMS before the scale parameter is applied.

Why this form: RMSNorm is computationally cheaper than LayerNorm (it omits the mean-centering step, only normalizing by RMS) while achieving comparable training stability. The learnable scale parameter γ allows the model to restore representational capacity after normalization—without it, forcing all token representations to have unit RMS would severely constrain the model's expressivity.

Structured Pruning Across Three Dimensions

The paper applies structured pruning simultaneously across depth, width, and expert dimensions. Each dimension uses a different importance estimation strategy because the structural units being removed have fundamentally different properties.

Depth Pruning: Removing Entire Transformer Blocks

The paper adopts a simple strategy: directly drop the last N layers of an L-layer model. Formally, given layers {f_1, ..., f_L}, the kept layers are:

$L_{keep} = \{1, \ldots, L - N\}, \quad \tilde{L} = L - N$

In the main experiments, N = 12 layers are removed from the original L = 48, resulting in 36 remaining layers. The 12 pruned layers break down as 3 full attention layers and 9 linear attention layers, maintaining the same full-to-linear ratio in the compressed model as in the original. This approach is justified in Appendix A.4 (Table 8), where the paper compares last-layer pruning against an activation-similarity-based method that prunes contiguous chunks from the middle of the model. The activation-based method measures adjacent-layer cosine similarity using token-mean pooled activations on the calibration set, identifies the most similar contiguous block of N layers, and removes them. The results show that last-layer pruning causes only minor one-shot degradation (MMLU drops from 75.62 to 73.86), while the activation-based method causes catastrophic collapse (MMLU drops to 41.95). After 120B tokens of post-compression KD, last-layer pruning still recovers better performance.

The paper acknowledges a curious phenomenon in Appendix Table 8: for last-layer pruning, the 120B-token trained model actually performs worse than the one-shot pruned model on some benchmarks (MMLU: 73.02 vs. 73.86; CMMLU: 78.08 vs. 80.30). The authors hypothesize that the one-shot performance is already close to the teacher model on these benchmarks, leaving a "relatively small knowledge gap to recover," and that continued training on a smaller architecture may slightly overfit or lose some teacher knowledge that the one-shot pruning serendipitously preserved.

Why this approach: the paper explicitly cites Sun et al. (2026) for the observation that deeper layers in large language models are more redundant and can be removed with less damage than middle layers. This aligns with the "curse of depth" hypothesis: later layers in transformer models learn representations that are progressively more task-specific and less general, making them less critical for preserving broad knowledge. Removing middle layers disrupts the information flow through the entire model, while removing the last layers only affects the final processing stages. The appendix results in Table 8 empirically validate this choice for the Qwen3-Next architecture.

Width Pruning: Reducing Hidden Dimensions by Activation Importance

Width pruning reduces the hidden dimension d across the entire architecture, affecting all modules: the hybrid attention (both Gated Attention and Gated DeltaNet), the MoE module (expert projections and router), and the RMSNorm modules (the scale parameter γ). The key challenge is deciding which of the d hidden dimensions to keep and which to drop, since this choice affects every subsequent operation in the model.

The paper estimates importance using activation statistics computed on a calibration dataset D sampled from the pretraining data (1024 samples). For any module, let Z ∈ R^{B \times n \times m} be the output activation for a batch of size B, sequence length n, and hidden dimension m. The per-dimension importance is computed by aggregating across batch and sequence dimensions using mean absolute activation:

$Mean(Z) := \frac{1}{B n} \sum_{b=1}^{B} \sum_{t=1}^{n} \left| Z_{b,t,:} \right| \in \mathbb{R}^m$

What it computes: for each of the m dimensions, compute the average absolute value of the activation across all tokens and all sequences in the batch. This produces a single importance score per dimension—dimensions that consistently have large-magnitude activations are considered more important than those with near-zero activations.

Why this form: the mean absolute value is chosen over alternatives (mean, L2 norm, variance) because absolute value penalizes dimensions that are consistently near zero while being robust to occasional outliers. The L2 norm would disproportionately weight dimensions with rare large activations, and variance would center the activations (losing information about mean magnitude). The mean absolute value directly captures "how much signal flows through this dimension on average."

For the width pruning decision, the paper uses the RMSNorm output as the key activation to score:

$I^{(k)}_{norm} = \left[ \frac{\sum_{i=0}^{L} Mean(RMSNorm(X))}{L} \right]_{i}^{k}, \quad k = 1, \ldots, d$

where the notation indicates averaging the mean absolute activation of RMSNorm outputs across all L layers for each hidden dimension k.

What it computes: for each hidden dimension k, take the RMSNorm output at every layer, compute its mean absolute activation (averaging over batch and sequence dimensions), then average these per-layer scores across all L layers. This produces a single importance score I^{(k)}_{norm} for dimension k that reflects its aggregate activity throughout the entire model.

Why use RMSNorm outputs specifically: the RMSNorm output is the activation that flows into every subsequent operation in each transformer block—it is the normalized, scaled hidden representation before it enters attention, MoE, or feed-forward sub-layers. By scoring dimensions based on their RMSNorm output magnitudes, the importance metric captures how much each dimension contributes to the information entering every major computation in the model. Using intermediate activations (e.g., attention output or MoE output) would bias the importance score toward specific sub-layer behaviors rather than overall representational importance. Averaging across layers ensures that a dimension is only retained if it is consistently important throughout the model, not just important in a few layers.

Given the target hidden size d_t = 1536 (reduced from d = 2048), the paper retains the d_t hidden dimensions with the highest I^{(k)}_{norm} scores.

Expert Compression: From 512 to 256 Experts per MoE Module

Expert compression is the most complex of the three pruning dimensions because it involves not just removing components but potentially reorganizing them through merging. The paper compares both pruning (removing experts entirely) and merging (combining multiple experts into fewer). The target is to reduce from N = 512 routed experts to Ñ = 256 per MoE module, while reducing the number of activated experts per token from k = 10 to k = 8.

The first step in any compression strategy is quantifying expert importance. The paper evaluates three distinct metrics, each computed on the same 1024-sample calibration dataset by processing all tokens through the model and recording routing behavior:

Frequency-based importance (I^Freq): the simplest metric—counts how often each expert is among the top-k selected by the router:

$I^{Freq}_i = \mathbb{E}_{x \sim \mathcal{C}} \left[ \mathbb{I}[i \in A(x)] \right]$

where A(x) = TopK(z(x), k) ⊆ {1, ..., N} is the set of top-k expert indices for token x, and I[·] is the indicator function (1 if expert i is selected, 0 otherwise). The expectation is computed as the mean over all tokens in the calibration set.

What it computes: for each expert i, measure what fraction of all calibration tokens activate that expert. An expert that is in the top-k for 30% of tokens gets a score of 0.3; an expert that is never selected gets 0.

Why this form: frequency is the most direct measure of expert utilization—if an expert is rarely or never used, it is likely redundant. However, it treats all activations equally: an expert that fires with high routing weight (the model strongly "prefers" it) is treated the same as an expert that barely makes the top-k cut. This metric can miss experts that are rarely activated but crucial for specific knowledge domains.

Soft-logits importance (I^Soft): extends frequency by weighting each activation by the expert's normalized routing score:

$I^{Soft}_i = \mathbb{E}_{x \sim \mathcal{C}} \left[ \mathbb{I}[i \in A(x)] \cdot \frac{z_i(x)}{\sum_{j \in A(x)} z_j(x)} \right]$

What it computes: for each token where expert i is in the top-k, weight the contribution by the softmax-normalized routing score z_i(x) relative to all selected experts. An expert that is always the highest-scored among the top-k receives more importance credit than one that consistently ranks last.

Why this form: the normalized routing score captures how much the expert contributes to the MoE output when it is activated. In the MoE computation, z_i(x) is the weight multiplying Expert_i(x) in the final sum, so a higher routing score means the expert's output has proportionally more influence on the token representation. The soft-logits metric incorporates this influence into the importance score, penalizing experts that are frequently activated but with low routing weights (suggesting the router includes them only because they're better than nothing, not because they're highly relevant).

REAP importance (I^REAP): combines routing weight with the magnitude of the expert's output vector (following Lasby et al., 2025a):

$I^{REAP}_i = \frac{1}{|X_i|} \sum_{x \in X_i} z_i(x) \| E_i(x) \|_2$

where X_i is the set of all input tokens for which expert i is in the top-k, and ||·||_2 is the L2 norm.

What it computes: for each token where expert i is activated, multiply the routing score z_i(x) by the L2 norm (magnitude) of the expert's output vector E_i(x). Average this product across all tokens that activate this expert.

Why this form: this metric captures not just whether an expert is used and with what routing weight, but whether its output has substantial magnitude. An expert could have high routing weights but produce near-zero outputs (suggesting its parameters are effectively dead), or low routing weights but produce large outputs that significantly influence the MoE sum. The REAP metric explicitly accounts for both the router's preference and the expert's actual output contribution, identifying experts whose parameters meaningfully transform the token representation. The paper includes this metric because Lasby et al. (2025a) found it most effective among competing importance metrics in one-shot evaluations.

Expert Pruning Strategy: for pure pruning (no merging), after computing importance scores via Soft Logits or REAP, the top Ñ = 256 experts are retained and the remaining 256 are discarded entirely. The corresponding router weights for discarded experts are also pruned, and the top-k is reduced from 10 to 8.

Expert Merging Strategy: for merging, experts are not simply discarded—their parameters are combined into retained experts. The general approach requires two decisions: (1) grouping: which discarded experts are merged into which retained experts, and (2) interpolation: how to weight the combination of source and target expert parameters.

For grouping, the paper explores three similarity metrics to determine which experts are "close" to each other:

• Router logits: compute the cosine similarity between the router weight vectors for two experts (the columns of W_G corresponding to those experts). Experts with similar router weights tend to be selected for similar tokens, suggesting functional similarity.
• Router weights: a direct comparison of the learned router parameters beyond just the logits they produce.
• Expert output vectors: compute the cosine similarity between the output vectors E_i(x) and E_j(x) on the calibration set. Experts producing similar transformations of their inputs likely encode related knowledge.

For interpolation weights, when merging expert j into expert i, the paper uses the importance scores as scaling factors:

$\tilde{E}_i = \frac{I_i}{I_i + I_{m(i)}} E_i + \frac{I_{m(i)}}{I_i + I_{m(i)}} E_{m(i)}$

What it computes: the parameters of the merged expert \tilde{E_i} are a convex combination of the retained expert i and the discarded expert m(i) assigned to it, weighted by their respective importance scores. If expert i is much more important than m(i), the merged parameters are dominated by E_i; if they are equally important, the merge is a simple average.

Why this form: weighted averaging by importance ensures that highly important experts are preserved largely intact while less important experts contribute only modestly to the merge. Simple averaging would dilute the specialized knowledge of important experts with potentially irrelevant parameters from less useful experts. The convex combination (weights sum to 1) maintains the parameter scale, preventing the merged expert from having artificially inflated or deflated weight magnitudes.

The Partial-Preservation Expert Merging Strategy (Algorithm 1, the Paper's Proposed Method)

The paper identifies a central tension in expert compression: completely preserving only the top-ranked experts discards potentially complementary knowledge from lower-ranked experts, while aggressively merging all experts homogenizes the specialized representations that make MoE effective. The proposed partial-preservation strategy navigates this trade-off by explicitly retaining half of the target experts intact.

Given target expert number Ñ = 256 and N = 512 original experts, the algorithm proceeds:

1. Select preserved experts: compute importance scores {S_i} for all N experts using any of the three metrics (Soft Logits is used as the primary metric in experiments). Keep the top ⌊Ñ/2⌋ = 128 experts with highest scores intact: S_keep = arg topk_{i ∈ {1,...,N}} S_i with |S_keep| = 128. These experts are preserved as-is with no parameter changes.

2. Select merge bases: from the remaining N - 128 = 384 experts, select an additional Ñ/2 = 128 experts to serve as merge bases: S_base ⊂ {1, ..., N} \ S_keep with |S_base| = 128. The remaining 256 experts are designated for merging.

3. Assign experts to merge groups: for each of the 256 experts to be merged, compute its most similar merge base j ∈ S_base using cosine similarity (on router logits, router weights, or expert output vectors depending on the grouping method). Each merge base j accumulates a set of assigned experts.

4. Merge: for each merge base j ∈ S_base, combine all assigned experts into the merged expert \tilde{E_j} using importance-weighted averaging as described above.

The final compressed expert set has exactly Ñ = 128 preserved + 128 merged = 256 experts.

Why half preservation: the paper describes this as a "simple and symmetric design choice" chosen because preserving too few experts weakens parameter inheritance (you lose too much of the pretrained knowledge), while preserving too many leaves limited room for consolidation (you might as well just prune). Keeping roughly half provides a robust compromise in the evaluated setting. This is not presented as an optimal ratio—the paper acknowledges exploring this choice more thoroughly as a limitation—but as a practical heuristic that consistently outperforms both pure pruning and full merging across benchmarks (Table 2: partial-preservation merging with Soft Logits importance and Router Weights grouping scores 69.28 MMLU vs. 69.05 without preservation; similar gains appear on MMLU-Pro, GSM8K, and other benchmarks).

A crucial practical detail: for both expert pruning and expert merging, the paper explicitly states that "we prune the corresponding router weight for continual pretraining." This means the router matrix W_G is also reduced from dimension d × 512 to d × 256, and the pruned student model starts with a router that only knows about the retained/merged experts. During continued pretraining, the model learns new routing patterns adapted to the reduced expert set.

Distillation Pretraining Framework

After the one-shot structured pruning produces the compressed architecture, the model undergoes continued pretraining with knowledge distillation from the frozen teacher model. The paper studies multiple objective configurations and introduces two key innovations: multi-token prediction (MTP) distillation and progressive pruning schedules.

Standard Next-Token Prediction Knowledge Distillation (NTP KD)

In standard KD for language models, the student model produces a probability distribution over the vocabulary for the next token, and the teacher provides a "soft target" distribution at the same position. The student is trained to minimize the KL-divergence between its output distribution and the teacher's output distribution. Unlike hard targets (one-hot ground-truth tokens), soft targets provide richer supervision: even for incorrect tokens, the teacher assigns non-zero probabilities that encode which alternatives are "almost correct" or semantically related. This is standard practice in model compression and does not require detailed formalization beyond noting that the paper uses it as a baseline.

Multi-Token Prediction (MTP) Architecture and Distillation

The key technical innovation in the distillation framework is extending supervision beyond the next token to multiple future tokens. The MTP architecture (originally proposed by Gloeckle et al., 2024 for pretraining from scratch) adds D additional transformer blocks after the backbone model's final hidden layer. Each block predicts one additional future token beyond the standard next-token prediction.

The MTP module for prediction depth k ∈ {1, ..., D} consists of four components, all of which exist in the base Qwen3-Next model and are preserved in the compressed student:

1. An embedding layer Emb(·) shared with the backbone model.
2. A projection matrix M_k ∈ R^{d \times 2d} specific to depth k.
3. A transformer block TRM_k(·) specific to depth k (with the same architecture as backbone blocks but separate parameters).
4. The output head OutHead(·) shared with the backbone model.

For the i-th input token t_i at prediction depth k, the computation proceeds:

First, combine the representation of token i from the previous depth (k-1) with the embedding of the (i+k)-th token:

$h'^{k}_{i} = M_k \left[ RMSNorm(h^{k-1}_{i}); RMSNorm(Emb(t_{i+k})) \right]$

where h^{k-1}i ∈ R^d is the hidden representation from depth k-1 (when k=1, h^0_i is the backbone model's output at position i), Emb(t{i+k}) ∈ R^d is the embedding of the token at position i+k, and [·; ·] denotes concatenation producing a 2d-dimensional vector.

What it computes: at prediction depth k, take the representation of the current token from the previous depth (which encodes context up to position i), concatenate it with the embedding of the token k positions ahead (which provides positional and semantic information about what comes next), project this 2d-dimensional concatenation through M_k to a d-dimensional representation, and pass it through the depth-k transformer block.

Why this form: the concatenation of the current-depth representation with the future token's embedding provides the MTP module with two complementary signals: (1) the accumulated contextual information from the backbone processed through previous prediction depths (via h^{k-1}i), and (2) explicit information about which token is at the target position (via Emb(t{i+k})). This is not cheating—the MTP module is predicting t_{i+k} given t_i through t_{i+k-1}, and Emb(t_{i+k}) serves as a "hint" about what token to predict, similar to teacher forcing during training. The RMSNorm on both constituents ensures they are on comparable scales before concatenation and projection.

The projected representation is then processed through the depth-k transformer block:

$h^{k}_{1:T-k} = TRM_k\left( h'^{k}_{1:T-k} \right)$

where T is the total sequence length and 1:T-k denotes slicing to exclude positions beyond T-k (since there are no tokens to predict beyond the sequence). Finally, the output head produces a probability distribution over the vocabulary:

$p^{k}_{i+k} = OutHead(h^{k}_{i}) \in \mathbb{R}^{V}$

where V is the vocabulary size, OutHead(·) linearly maps the d-dimensional representation to V-dimensional logits and applies softmax.

MTP Losses: Language Modeling and Knowledge Distillation

The MTP modules are trained with two losses, one using ground-truth labels and one using teacher soft targets.

The MTP Language Modeling (MTP-LM) loss uses the standard next-token cross-entropy at each prediction depth:

$\mathcal{L}_{MTP-LM} = \frac{1}{D} \sum_{k=1}^{D} \left( -\frac{1}{T-k} \sum_{i=1}^{T-k} \log p^{k}_{i+k}[t_{i+k}] \right)$

where p^k_{i+k}[t_{i+k}] is the predicted probability assigned to the ground-truth token at position i+k by the k-th MTP module, and D is the total number of MTP depths.

What it computes: for each prediction depth k, compute the standard cross-entropy loss between the k-th MTP module's predicted distribution and the one-hot ground-truth token at position i+k. Average these losses across all depths. This is exactly the standard language modeling loss applied at multiple future positions.

Why this form: the MTP-LM loss forces the MTP modules to learn to predict future tokens from the backbone's representations, which in turn pressures the backbone to produce representations that are useful not just for the next token but for tokens farther in the future. This is a form of auxiliary task that enriches the backbone's training signal—to predict t_{i+3} well, the backbone must encode information at position i that is relevant three steps ahead, which encourages learning longer-range dependencies.

The MTP Knowledge Distillation (MTP-KD) loss replaces the ground-truth one-hot targets with the teacher's soft probability distributions at the same positions:

$\mathcal{L}_{MTP-KD} = -\frac{1}{D} \sum_{k=1}^{D} \left( \frac{1}{T-k} \sum_{i=1}^{T-k} \sum_{v=1}^{V} q_{i+k}[v] \log p^{k}_{i+k}[v] \right)$

where q_{i+k}[v] is the teacher model's predicted probability for token v at position i+k.

What it computes: for each prediction depth k and each future position i+k, compute the cross-entropy between the teacher's soft distribution q_{i+k} and the student MTP module's predicted distribution p^k_{i+k}. This is the KL-divergence (up to a constant entropy term) between teacher and student at each future position. Average across all depths.

Why this form: the MTP-KD loss provides the same benefit as standard NTP KD—richer supervision from soft targets that encode token relationships—but applied to multiple future positions. The teacher's soft distribution at position i+k encodes which tokens would be plausible continuations given the teacher's full knowledge, and the student learns to match these distributions. This is particularly valuable for compression because the student model is smaller and may not have the capacity to independently learn the subtle token relationships that the larger teacher has internalized; the MTP-KD loss directly transfers this knowledge.

The Complete Training Objective

The full training objective combines four loss terms with two balancing hyperparameters:

$\mathcal{L} = (1 - \lambda) \mathcal{L}_{LM} + \lambda \mathcal{L}_{KD} + \beta \left( (1 - \lambda) \mathcal{L}_{MTP-LM} + \lambda \mathcal{L}_{MTP-KD} \right)$

where:

• L_{LM} is the standard language modeling loss on the backbone's next-token predictions (cross-entropy with ground-truth one-hot labels).
• L_{KD} is the standard next-token knowledge distillation loss on the backbone's predictions (cross-entropy with teacher soft targets).
• L_{MTP-LM} is the multi-token prediction language modeling loss defined above.
• L_{MTP-KD} is the multi-token prediction knowledge distillation loss defined above.
• λ ∈ [0, 1] controls the balance between LM loss and KD loss for both the backbone and MTP modules. λ = 1 means pure KD (no ground-truth LM signal); λ = 0 means pure LM (no teacher distillation).
• β ∈ [0, 1] controls the weight of the MTP losses relative to the backbone losses. β = 0 means no MTP training at all; β = 1 means equal weight.

What it computes: the total loss is a weighted combination of four components. The first term (1-λ)L_{LM} + λL_{KD} is the backbone's training signal: a blend of ground-truth language modeling and teacher distillation, with λ controlling the mix. The second term β(... ) is the MTP modules' training signal (scaled by β): the same λ blend applied to the multi-token predictions, with the same mix of LM and KD objectives. When λ = 0.5 and β = 0.3, the backbone gets equal LM and KD weight while the MTP modules get 30% of the backbone's loss magnitude with the same LM/KD mix.

Why this form: the shared λ across backbone and MTP losses ensures that the balance between ground-truth supervision and teacher guidance is consistent throughout the model—if the backbone is mostly learning from the teacher (λ ≈ 1), the MTP modules should also mostly learn from the teacher rather than trying to independently predict ground-truth tokens. The separate β parameter allows controlling the relative importance of multi-token prediction without affecting the backbone's primary training signal. This is important because MTP is an auxiliary task: it should help the backbone learn better representations, but shouldn't dominate training to the point where the model optimizes for multi-token prediction at the expense of next-token accuracy.

Hyperparameter schedules for λ and β: the paper specifies two decay schedules:

• λ: linear decay from 1.0 to 0.75 over the course of training. This means training starts with pure distillation (λ=1.0, the backbone learns entirely from the teacher) and gradually introduces more ground-truth LM signal as training progresses (ending at λ=0.75, a 75/25 mix of KD/LM). This schedule reflects the intuition that early in continued training, the student should focus on recovering the teacher's knowledge through distillation; later, as the student approaches the teacher's quality, reinforcing ground-truth language modeling prevents over-reliance on potentially imperfect teacher predictions.

• β: cosine decay from 0.3 to 0.1. The MTP modules start with 30% weight and gradually decrease to 10%. This schedule reflects the intuition that multi-token prediction provides the most benefit early in training when the backbone's representations are least refined; as training progresses and the backbone improves, the auxiliary MTP signal becomes less necessary and could interfere with fine-grained next-token optimization.

The paper also specifies that in the progressive pruning setting (two-stage), a single learning rate decay schedule is used across both stages: the second stage starts from the learning rate reached at the final step of the first stage. This means there is no learning rate re-warmup or reset between stages—the optimization is continuous across the pruning boundary. This is a deliberate design choice that maintains training momentum and prevents the discontinuity of a learning rate reset from disrupting the knowledge transfer that progressive pruning is meant to facilitate.

Progressive Pruning and Distillation Schedules

The paper's final technical contribution is a systematic comparison of how to schedule the compression across multiple training stages, rather than performing it all at once. The idea is that gradually transitioning from the large teacher architecture to the small target architecture allows the model to adapt its representations incrementally, producing a smoother optimization trajectory than the abrupt change of one-shot compression.

All progressive schedules operate in two stages with a total of 400B training tokens: Stage 1 uses 40B tokens (10% of the total), and Stage 2 uses the remaining 360B tokens. The total amount of pruning applied across both stages equals the target compression: 12 layers removed, hidden dimension reduced from 2048 to 1536, and 512 experts merged to 256. The schedules differ only in which structural changes happen in which stage.

Depth-First Schedule: in Stage 1, remove half the target layer reduction (6 layers, comprising a proportional mix of full and linear attention layers to maintain the attention-type ratio) while keeping the original width and expert count. Train for 40B tokens at this intermediate depth-reduced architecture. In Stage 2, remove the remaining 6 layers AND apply the full width reduction (2048 → 1536) AND expert compression (512 → 256), then train for the remaining 360B tokens at the final SlimQwen-23A2B architecture.

Width-First Schedule: in Stage 1, apply half the width reduction (an intermediate hidden size between 2048 and 1536, though the exact intermediate value is not explicitly stated—it would logically be 2048 minus half the reduction, approximately 1792) while keeping full depth and expert count. Train for 40B tokens. In Stage 2, complete the remaining width reduction AND remove all 12 layers AND perform expert compression, then train for 360B tokens.

Joint Schedule: in Stage 1, simultaneously remove half the target layers (6) AND apply half the width reduction AND half the expert reduction (an intermediate number of experts between 256 and 512). Train for 40B tokens. In Stage 2, complete all remaining reductions to reach the target architecture and train for 360B tokens.

One-Shot (Baseline): apply all compression at once (remove 12 layers, reduce width to 1536, merge experts to 256) and train for 400B tokens without any intermediate architecture.

The paper also explores three-stage variants in Appendix A.5 (Table 9), where depth-first divides the layer reduction into two sub-stages (20B tokens for first half of layers, 20B for second half, then 360B for width), and width-first follows the same pattern in reverse. The results show that three-stage schedules achieve performance comparable to two-stage, with no additional gains: "more fine-grained stage partitions do not provide additional benchmark performance gains." This is an important negative result—it suggests that two stages are sufficient to capture the benefit of progressive pruning, and additional granularity only adds complexity without improving outcomes.

Why progressive pruning helps: the paper does not provide a theoretical justification, but the empirical pattern is clear in Table 5: all progressive schedules outperform one-shot compression on most benchmarks, with depth-first achieving the best overall results (MMLU 77.39 vs. 75.86 one-shot, MMLU-Redux 78.01 vs. 75.41). The intuition is that intermediate architectures provide "stepping stones": the model first adapts to a partial capacity reduction (e.g., removing some layers) while retaining most of its original representational capacity, then adapts to the full reduction from this stronger starting point. This prevents the catastrophic knowledge loss that can occur when a model's capacity is suddenly halved—the representations have time to reorganize gradually rather than being forced to compress all knowledge simultaneously.

Why depth-first outperforms width-first and joint: the paper does not explicitly analyze why depth-first is best, but a plausible interpretation from the results is that layer reduction (removing whole transformer blocks) causes a fundamentally different type of disruption than width reduction. Removing layers eliminates entire stages of processing and forces the remaining layers to take on the computational responsibilities of the removed ones—a form of functional reorganization. Reducing width constrains the dimensionality of all intermediate representations but preserves the sequential processing pipeline. By completing the functional reorganization (layer removal) first, the model can dedicate its remaining capacity to learning how to operate effectively in fewer layers, and only then does it confront the dimensionality constraint. The width-first schedule, by contrast, first constrains the representations in all existing layers, making it harder for the model to subsequently adapt to having fewer layers with already-constrained dimensions. This interpretation is speculative based on the results but aligns with the paper's finding that depth-first produces the best outcomes.

4. Key Insights and Innovations

Innovation 1: The Choice of Expert Compression Metric Converges After Sufficient Continued Training — Making the Training Recipe, Not the Pruning Criterion, the Bottleneck

The single most counterintuitive finding in this paper is not a performance gain but a null result with profound practical implications: after 400B tokens of continued pretraining, different one-shot expert compression methods converge to statistically indistinguishable final performance. Table 2 shows this directly. Eight different expert pruning and merging strategies — varying the importance metric (Soft Logits, REAP, Frequency), the grouping method (Router Logits, Router Weights, Expert Vectors), and whether partial preservation is applied — all cluster within roughly 1–2 points on any given benchmark after 400B tokens of training. No single method dominates across all evaluation tasks. The field has spent substantial effort designing increasingly sophisticated expert importance estimation methods (REAP from Lasby et al., 2025a,b; frequency-based pruning from Lu et al., 2024; various merging criteria from Li et al., 2024b), and this result suggests that most of that effort matters primarily for one-shot evaluation, not for the end-to-end compression pipeline where continued training washes out initial differences.

This is a fundamental conceptual shift, not an incremental refinement. The dominant assumption in prior MoE compression work — implicit in papers that propose and evaluate new expert dropping criteria — is that the choice of which experts to keep is the critical decision, and that better importance estimation translates to better final models. Jaiswal et al. (2025) thoroughly benchmark one-shot performance of different expert dropping strategies and find meaningful differences, which would naturally lead a practitioner to invest heavily in optimizing their importance metric. The SlimQwen results reframe the problem: the importance metric matters for the initialization quality before continued training, but after hundreds of billions of tokens, the model relearns routing patterns adapted to its reduced expert set, and the initial choice of which experts survived is largely overwritten. The training recipe (KD vs. LM loss, MTP KD, progressive scheduling) matters far more than the expert selection criterion.

This insight has direct consequences for how research resources should be allocated in MoE compression. If the field's goal is post-compression continued training (which it almost always is, since one-shot pruned models are rarely deployed), then marginal improvements in importance estimation are a poor investment compared to improving the training objective and schedule. The paper's own partial-preservation merging strategy — which does show modest but consistent gains over both pure pruning and full merging (Table 2: MMLU improves from 69.05 without preservation to 69.28 with it) — is not argued to be optimal. It's presented as a simple, robust heuristic that works across settings, and its modest gains (1–2 points in aggregate) are consistent with the broader finding that expert compression design is a second-order effect compared to training design.

This also explains a pattern visible across Tables 1–5: the biggest performance deltas come from training choices, not architecture choices. Switching from random initialization to pruned initialization yields +11.79 points on average benchmark score (Table 1). Switching from one-shot to progressive pruning yields +1.53 points on MMLU (Table 5). Switching from pure KD to KD + LM loss yields +0.77 on MMLU (Table 3). By contrast, the gap between the best and worst expert compression method in Table 2 is roughly 0.5–1 point on any given benchmark — an order of magnitude smaller. The paper's message is that practitioners should invest their engineering effort in the training pipeline, not in the expert selection heuristic, because the latter's impact is largely erased by scale.

Innovation 2: Multi-Token Prediction Distillation as a Compression-Specific Training Objective — Not Just for Pretraining from Scratch

Multi-token prediction (MTP) was originally proposed by Gloeckle et al. (2024) as a pretraining objective for training models from scratch, where predicting multiple future tokens forces the model to learn longer-range dependencies and richer representations. The SlimQwen paper repurposes MTP as a distillation objective specifically for model compression, and demonstrates that this yields gains beyond what either standard next-token KD or ground-truth MTP loss can achieve alone.

This is not a trivial extension. The standard argument for MTP in pretraining from scratch is that it provides a richer self-supervised signal: predicting token t+3 from position t requires the model to encode information about future content that would otherwise only be learned through next-token prediction backpropagated through multiple time steps. In the compression setting, the argument is different: the teacher model already possesses that future-context understanding, and the MTP-KD loss directly transfers the teacher's multi-step predictive distribution to the student. Rather than forcing the student to independently discover long-range dependencies through standard LM loss (which may be difficult for a capacity-constrained model), the MTP-KD objective provides explicit supervision at each future position about what the teacher considers plausible continuations.

The evidence for this as a distinct insight — rather than just "adding MTP to KD" — comes from the ablation in Table 3 and the speculative decoding results in Table 4. Table 3 shows that MTP KD provides gains that are complementary to and distinct from both NTP KD and MTP LM loss. The row "NTP KD + LM Loss + MTP Loss + MTP KD" achieves 75.67 MMLU versus 74.16 for pure NTP KD, with the MTP KD component contributing specifically to multi-token quality. Table 4 shows the practical manifestation: MTP KD dramatically improves speculative decoding acceptance rates, particularly for longer accepted token sequences. On GSM8K, MTP KD improves acc 1 (two-token acceptance) from 57.62% to 75.18% — a 17.56 percentage point gain — and acc 4 (five-token acceptance) from 2.37% to 10.37%, a more than 4× relative improvement. This pattern holds across pretraining and SFT stages, and across code (HumanEval), math (GSM8K), translation (WMT22), and conversation (MTBench) benchmarks.

The significance here extends beyond the specific numbers. This establishes that distillation objectives can and should target the specific inference-time behaviors that matter for deployment, not just aggregate next-token perplexity. If a compressed model will be used for speculative decoding (a common deployment pattern for smaller models that serve as draft models for larger verifiers), optimizing for multi-token acceptance during distillation directly improves deployment efficiency. The standard approach in compression — train with NTP KD, evaluate benchmark scores — implicitly assumes that benchmark performance is the right proxy for deployment utility. The MTP KD results suggest that distillation objectives should be co-designed with the inference strategy, and that the choice of what to distill (single token vs. multi-token) can have outsized effects on specific deployment metrics even when benchmark gains are modest.

This is a fundamental reframing of what "good compression" means. It's not just about matching the teacher's single-token predictions; it's about transferring the teacher's multi-step reasoning capability to enable efficient deployment patterns like speculative decoding. The paper doesn't fully develop this argument — it treats the speculative decoding gains as a bonus rather than the primary motivation — but the implication is clear: compression research should evaluate models on deployment-relevant metrics (like multi-token acceptance rate) in addition to standard benchmarks, because the optimal training recipe for one may not be optimal for the other.

Innovation 3: The "Pruning as Initialization" Advantage Is Decisive for MoE Models at Pretraining Scale — and KD Amplifies It Significantly

The question of whether pruning provides a better initialization than training from scratch is a long-standing debate in model compression, with evidence varying by architecture, scale, and domain. For dense models, results are mixed: some works find pruned initialization helpful (Muralidharan et al., 2024), others find that training from scratch can catch up given enough compute. For MoE models at the scale studied here (80B parameters, 120B tokens of post-compression training), the paper provides a decisive answer: pruned initialization is dramatically better, and combining it with KD (rather than pure LM loss) amplifies the advantage.

Table 1 quantifies this decisively. Training the target 23A2B architecture from scratch with KD loss for 120B tokens achieves 61.66 average benchmark score. Training the pruned model with LM loss (no KD) achieves 69.96 — already a substantial 8.3-point advantage from initialization alone. Training the pruned model with KD loss achieves 73.45 — an additional 3.5 points from distillation. The gap from random initialization to pruned + KD is 11.79 points, a massive margin that would require many more training tokens for the random-initialization model to close (if it ever could). Figure 2 confirms that this is not just a final-performance effect: the pruned model converges faster from the first training steps, with "Pruned + KD" achieving the lowest LM loss throughout training.

What makes this a conceptual contribution rather than just a large number is what it reveals about knowledge retention in MoE architectures during pruning. The compressed model at 23A2B has 3.4× fewer total parameters than the teacher (80B). Yet after continued training, it recovers 86.5% of the teacher's average benchmark score (73.45 vs. 82.68). This suggests that a substantial fraction of the teacher's knowledge is encoded in parameter relationships that survive structured pruning — the relative organization of expert parameters, the routing patterns learned during pretraining, the hidden dimension importance ordering — rather than in the raw parameter count. The pruned initialization preserves these relationships, giving continued training a head start that random initialization cannot replicate without rediscovering the same structure from scratch.

The interaction with KD is particularly revealing. Table 1 shows that KD adds 3.49 points over LM loss when starting from pruned initialization (73.45 vs. 69.96), but a much smaller gap exists between random init + KD (61.66) and pruned + LM (69.96). This suggests that the pruned initialization and KD serve complementary roles: the pruned initialization preserves structural knowledge (which experts are related, which dimensions carry signal), while KD transfers fine-grained output-distribution knowledge (which tokens are plausible continuations in specific contexts). Having both is substantially better than having either alone, because they operate on different aspects of the model's knowledge. This is a practical insight for practitioners: if you can only afford one (pruned initialization without KD, or random initialization with KD), the pruned initialization alone is the better investment. But the combination provides gains beyond the sum of individual contributions, because the pruned structure gives the KD signal a more receptive starting point.

Innovation 4: Progressive Pruning Schedules Produce Meaningfully Better Outcomes Than One-Shot Compression — and the Order of Reduction Matters

The idea of gradual capacity reduction rather than one-shot pruning is not new in compression literature. What the SlimQwen paper contributes is a controlled empirical comparison of different progressive schedules at pretraining scale, establishing both that progressive pruning consistently outperforms one-shot compression and that the order of reduction (depth-first vs. width-first vs. joint) produces measurably different outcomes. This transforms progressive pruning from an intuitive heuristic to a design choice with concrete performance implications.

Table 5 establishes the baseline: under identical total token budgets (400B), one-shot compression achieves 75.86 MMLU, while the worst progressive schedule (Joint, 40B + 360B) achieves 76.30 (+0.44) and the best (Depth-First) achieves 77.39 (+1.53). These are meaningful gains at this performance level — a 1.5-point MMLU improvement from a training schedule change is substantial. The MMLU-Redux gains are even larger: from 75.41 (one-shot) to 78.01 (depth-first), a 2.6-point improvement. On BBH, width-first achieves 75.22 vs. 73.97 for one-shot, suggesting different schedules benefit different capability dimensions.

The conceptual contribution is the demonstration that the architecture transition trajectory matters independently of the final architecture and total training budget. All models in Table 5 end up with the identical 23A2B architecture and receive the same 400B total tokens. The only difference is whether the model spends 40B tokens at an intermediate architecture before completing the compression. The fact that this produces non-trivial gains implies that the intermediate architectures serve as effective "stepping stones" that enable more efficient knowledge transfer than direct compression. The model's representations need time to reorganize when capacity is removed; doing the removal in stages gives representations time to adapt at intermediate capacity before confronting the full reduction.

The finding that depth-first outperforms width-first and joint (Table 5: 77.39 vs. 77.14 vs. 76.30 on MMLU) is particularly notable because it suggests different types of structural reduction have different adaptation dynamics. Removing layers fundamentally changes the sequential processing pipeline — the remaining layers must learn to perform computations that were previously distributed across more stages. Reducing width constrains the dimensionality of all intermediate representations but preserves the pipeline structure. The fact that removing layers first (and width second) produces the best outcome suggests that the model benefits most from adapting its sequential computation structure at higher capacity, before confronting dimensionality constraints. This is a non-obvious insight that could guide scheduling decisions for future compression work.

The three-stage results in Appendix A.5 (Table 9) add an important boundary condition: more fine-grained staging does not provide additional gains. Depth-first with three stages (20B + 20B + 360B tokens) achieves 77.29 MMLU, essentially identical to the two-stage 77.39. This establishes that two stages capture the benefit of progressive pruning; additional granularity adds complexity without improving outcomes. It's a useful negative result that prevents practitioners from over-engineering the schedule.

Taken together with Innovation 1 (expert compression metric convergence), these results paint a clear picture of where marginal engineering effort should be invested in MoE compression. The training schedule (one-shot vs. progressive, and which progressive order) matters substantially. The training objective (KD vs. LM, MTP KD) matters substantially. The initialization (pruned vs. random) matters enormously. The expert compression metric matters only modestly and converges with scale. This prioritization — schedule > objective > initialization choice > expert metric — is a directly actionable research finding that the paper establishes through systematic ablation rather than advocating for any single method.

5. Experimental Analysis

Evaluation Methodology

• Dataset. All experiments use the MATH benchmark (Hendrycks et al., 2021), a dataset of high-school competition-level math problems. The authors use the specific split from Lightman et al. (2022): 12,000 training questions and 500 test questions. MATH is chosen deliberately because test-time compute is expected to help most on tasks requiring complex multi-step reasoning where the model already possesses the necessary knowledge — mathematical reasoning fits this profile since it requires logical deduction rather than novel factual recall.

• Base model(s). All main experiments use PaLM 2-S* (Codey) (Anil et al., 2023). The authors argue this model is "representative of the capabilities of many contemporary LLMs" and sits in a useful performance regime: non-trivial MATH accuracy (roughly 10–19% pass@1 depending on prompt and sampling configuration) but far from saturation, leaving substantial room for test-time compute to improve outcomes. For the FLOPs-matched comparison (Section 7), a second model from the same family with approximately ~14× more parameters is used as the pretraining-scaled baseline. This larger model uses greedy decoding with no additional test-time compute, making it a reference point for what pretraining alone achieves.

• Metrics. The primary metric throughout is MATH test accuracy (%) — the fraction of the 500 test questions for which the selected final answer matches the ground-truth answer. Answers are graded using the grading function released by Lightman et al. (2022), which handles mathematical equivalence checking (Appendix G). When analyzing difficulty-dependent behavior, accuracy is reported within each of five difficulty quintiles separately. For the FLOPs-matched comparison, the metric is accuracy relative to the ~14× larger model's performance, computed at matched total FLOPs budgets.

• Baselines. The paper compares against several established approaches. Majority voting selects the most common final answer among N sampled solutions without any learned verifier — this is the simplest aggregation method and serves as a lower bound. ORM best-of-N weighted scores N complete solutions with an Outcome Reward Model (a binary classifier that predicts final-answer correctness) and applies best-of-N weighted selection following Li et al. (2023) — this represents the standard dense-verifier approach. PRM best-of-N weighted scores N solutions with the Process Reward Model (which scores each intermediate step) using last-step aggregation and applies the same weighted selection — this is the standard PRM baseline from Cobbe et al. (2021) and Lightman et al. (2023). For revision experiments, parallel sampling generates N independent solutions from the revision model and selects the best via verifier or majority voting, representing the non-sequential baseline that revisions are compared against.

• Generation budget / compute accounting. Test-time compute is universally measured in generations — one "generation" equals one complete sampled answer from the base LLM. For best-of-N and beam search, the budget equals the number of beams or samples N. For lookahead search with k lookahead steps, the cost is scaled to N × (k+1) to account for the additional rollout computation (a specific accounting choice detailed in Section 5.3). Budgets are swept across powers of 2, typically from 2^0 (1 generation) to 2^9 (512 generations), enabling direct comparison of methods at matched generation counts.

• Cross-validation / statistical protocol. To avoid contaminating strategy selection with test-set performance when constructing compute-optimal policies, the paper uses two-fold cross-validation within each difficulty bin on the 500-question test set. Specifically: the test set is split into two folds, the best-performing strategy (which search algorithm, what beam width, what sequential-to-parallel ratio) is selected on one fold and evaluated on the other, and results are averaged across both folds. The difficulty bins themselves are constructed using 2048-sample pass@1 estimates, and the bin assignments are held fixed across folds. This protocol ensures that the reported compute-optimal scaling curves are not the result of overfitting strategy selection to the exact test questions.

Main Quantitative Results

Search Against PRM Verifiers (Section 5)

The core finding from the search experiments is that no single search algorithm dominates across all compute budgets and difficulty levels, and that the optimal choice depends on both how much compute is available and how hard the problem is. The aggregate comparison (Figure 3, left) shows that at low generation budgets (2–8 generations), beam search with beam width M = 4 substantially outperforms best-of-N weighted. At 4 generations, beam search achieves roughly 27% accuracy versus roughly 16% for PRM best-of-N weighted — an 11-point gap at the same compute cost. However, at high budgets (64–256 generations), this advantage reverses: beam search performance flattens around 34% and falls slightly below best-of-N weighted, which reaches approximately 38% at 512 generations. Majority voting trails substantially at all budgets, reaching only about 29% at 512 generations, confirming that learned verification is essential.

The difficulty-bin analysis (Figure 3, right) reveals why the aggregate pattern exists. Comparing beam search (M=4) against best-of-N weighted at four budget levels (4, 16, 64, 256 generations) across the five difficulty quintiles:

• Bin 1 (easiest problems): Beam search accuracy decreases slightly from roughly 78% at 4 generations to roughly 77% at 256 generations, while best-of-N weighted improves substantially from 68% to 88%. This is the clearest evidence of PRM over-optimization: beam search aggressively optimizes the PRM's scores and finds solutions that score highly under the verifier but are actually incorrect — essentially gaming the reward signal.

• Bin 2: Best-of-N weighted maintains a clear advantage, improving from roughly 14% to 60% across budgets, while beam search improves more modestly (roughly 14% to 32%).

• Bin 3 (medium difficulty): Beam search consistently outperforms best-of-N weighted across all budgets, reaching roughly 34% vs. 23% at 256 generations. This is where the PRM's guidance genuinely helps — search navigates toward correct solutions that random sampling wouldn't find.

• Bin 4 (hard): Beam search shows its strongest relative advantage, reaching roughly 17% vs. 10% for best-of-N at 256 generations. However, both methods produce low absolute accuracy on these problems.

• Bin 5 (hardest): Both methods hover near 1–3% across all budgets. No amount of search helps when the base model essentially never produces correct solutions.

Lookahead search (Figure 3, left) is a notable negative result: despite being the most computationally sophisticated method (simulating k=1 or k=3 steps forward at each expansion), it generally underperforms all other methods at matched generation budgets due to its higher per-step cost. Each lookahead step consumes an additional generation, reducing the effective number of beams explored for a given total budget. At very high budgets, the 3-step lookahead variants converge to similar accuracy as beam search and best-of-N, but they never surpass them. This establishes that more sophisticated search is not better search when the verifier is imperfect — the extra computation devoted to better scoring is better spent on broader exploration.

Compute-optimal search (Figure 4) demonstrates the practical payoff of difficulty-adaptive allocation. By selecting the best search strategy per difficulty bin at each budget level (using two-fold cross-validation):

• At 16 generations, compute-optimal search (oracle bins) achieves approximately 27% accuracy, roughly matching PRM best-of-N weighted at 64 generations — a 4× reduction in required compute to reach the same accuracy.
• At 256 generations, compute-optimal oracle reaches approximately 39.5%, surpassing PRM best-of-N weighted at the same budget (roughly 37%).
• Compute-optimal with predicted difficulty bins (using PRM average scores rather than ground-truth correctness for difficulty estimation) tracks the oracle version closely. The two curves "largely overlap" at lower budgets, with the predicted version reaching approximately 37% at 256 generations versus 39.5% for oracle — a relatively small gap that suggests the approach works without access to ground-truth labels.
• Both compute-optimal variants consistently outperform ORM best-of-N weighted (which peaks around 34% at 512 generations) and majority voting (around 29%).

The PRM vs. ORM comparison (Appendix F, Figure 14) shows that at 2048 samples, PRM best-of-N weighted achieves approximately 40% accuracy versus roughly 35% for ORM best-of-N weighted and roughly 30% for majority voting. Critically, the gap between PRM and ORM widens with the number of samples, confirming that the PRM's per-step training provides better scaling properties — the ORM saturates while the PRM continues to extract value from additional samples.

Revision Model Results (Section 6)

The revision model experiments establish that sequential revision can outperform parallel sampling, but the optimal ratio depends on problem difficulty. The base finding (Figure 6, left) is that the revision model's per-step pass@1 improves from approximately 18.2% at step 1 to roughly 24–25% by steps 15–20, and remains in the 23–25% range out to 64 steps — well beyond the 4-step horizon the model was trained with, suggesting it has learned a generalizable revision skill rather than merely memorizing 4-step trajectories.

The sequential-vs-parallel comparison (Figure 6, right) at 64 generations shows:

• Sequential + best-of-N weighted: approximately 41.5%
• Parallel + best-of-N weighted: approximately 39%
• Sequential + majority: approximately 38%
• Parallel + majority: approximately 35%

Sequential revision outperforms parallel sampling under both selection mechanisms, with the gap under verifier-based selection (roughly 2.5 points) being slightly narrower than under majority voting (roughly 3 points). This shows that the benefit of revisions is not solely attributable to the verifier seeing more context — sequential generation genuinely produces better candidate solutions.

The critical finding is the difficulty-dependent optimal sequential-to-parallel ratio (Figure 7). At a fixed budget of 128 generations:

• Bin 1 (easiest): Performance is essentially flat across all ratios at 90–92%. These problems are so easy that the allocation strategy barely matters.
• Bin 2: Slight advantage for higher sequential ratios, approximately 63% at fully sequential vs. 58% at fully parallel.
• Bin 3 (medium): A clear optimal ratio emerges at moderate sequential-to-parallel values (around 2:1 to 8:1 sequential-to-parallel), reaching approximately 42% vs. 35% at the extremes.
• Bin 4 (hard): Similar pattern, with the peak at moderate ratio achieving roughly 18% vs. 14% at fully parallel.
• Bin 5 (hardest): All ratios produce roughly 2–3% accuracy. No allocation strategy helps on these problems.

At lower budgets (8–32 generations, Figure 7, left), the curves are monotonically increasing with the sequential-to-parallel ratio — fully sequential is optimal. This means that when the total budget is small, it's better to invest all generations in refining a single chain rather than splitting between exploration (parallel) and exploitation (sequential). At higher budgets, the optimal mix shifts toward a balanced ratio.

Compute-optimal revisions (Figure 8) show the payoff of adaptive allocation:

• At 64 generations, compute-optimal oracle achieves approximately 40%, matching parallel best-of-N weighted at 256 generations — again a 4× compute reduction.
• At 256 generations, compute-optimal oracle reaches approximately 44%, compared to roughly 41% for best-of-N weighted and 37% for parallel-only.
• Compute-optimal with predicted difficulty bins performs slightly below oracle bins at high budgets (approximately 41% at 256 generations vs. 44% oracle), but still substantially outperforms the parallel baseline.
• Importantly, the parallel baseline appears to plateau around 36–37% at high budgets (Figure 8), while compute-optimal scaling continues to improve. This suggests that the benefits of adaptive, difficulty-conditioned allocation compound at higher budgets — a fixed strategy hits diminishing returns, but the adaptive policy finds new sources of gain.

FLOPs-Matched Comparison: Test-Time vs. Pretraining Compute (Section 7)

The FLOPs-matched analysis asks: given a fixed total FLOPs budget (pretraining + inference), is it better to train a larger model or to keep a smaller model and spend the saved FLOPs on test-time compute? The comparison uses PaLM 2-S* with compute-optimal test-time scaling against a ~14× larger model with greedy decoding (no extra test-time compute), at three values of the inference-to-pretraining token ratio R = D_inference / D_pretrain: R ≪ 1 (R = 0.16), R ≈ 1 (R = 0.79), and R ≫ 1 (R = 22).

Revisions vs. pretraining (Figure 9, left; Figure 1, top-right bar chart) shows:

At R ≪ 1, test-time compute outperforms the larger model across all difficulty levels — sometimes dramatically (27.8% advantage on medium problems). At R ≫ 1, test-time compute only remains preferable on easy questions, while hard questions show a substantial 37.2% disadvantage. This confirms the paper's central boundary condition: test-time compute can substitute for pretraining only when the inference volume is modest relative to pretraining.

PRM search vs. pretraining (Figure 9, right; Figure 1, bottom-right bar chart) shows a starker pattern:

PRM search shows substantially weaker benefits than revisions for the FLOPs-matched comparison. On medium and hard questions, test-time compute with PRM search is disadvantageous compared to the larger model even at moderate R values. On easy questions, test-time compute remains preferable across all R regimes, but the margin narrows significantly (from +19.1% at R ≪ 1 to +2.0% at R ≫ 1). This suggests that the benefits of PRM-guided search are more fragile in the FLOPs-matched regime than those of revisions — the cost of search computation competes less favorably with simply having a larger model.

Figure 9 provides the detailed scaling curves. The ~14× larger model's greedy performance is plotted as stars at three x-axis positions corresponding to the three R values. Where the compute-optimal scaling line is above the star, test-time compute wins. On bin 1 (easiest, topmost purple line), the scaling line is above all three stars for revisions, meaning test-time compute wins at all R values. On bin 5 (hardest, bottommost blue line), the scaling line is below all three stars and essentially flat near 0–5%, confirming that no amount of test-time compute helps on the hardest problems — pretraining is the only path to capability on these.

Ablation Studies and Robustness Checks

• PRM aggregation strategy (Appendix E, Figure 13). Comparing "min," "prod," and "last" step-wise score aggregation: "last" achieves roughly 37% at 256 samples, "min" achieves roughly 35%, and "prod" achieves roughly 27%. An ORM achieves roughly 34%. The "last" aggregation's superiority is notable because it effectively reduces the PRM to ORM-like behavior at aggregation time (using only the final step's score), yet the PRM still outperforms a separately trained ORM. The authors interpret this as evidence that step-level PRM training provides beneficial representation learning — the intermediate-step supervision during training improves the quality of the hidden representations even when the intermediate predictions are discarded at aggregation time.

• PRM vs. ORM scaling (Appendix F, Figure 14). The PRM consistently outperforms the ORM, with the gap widening at higher sample counts: at 2048 samples, PRM best-of-N weighted reaches approximately 40% vs. ORM's 35%. Majority voting reaches approximately 30%. This confirms that the PRM's per-step training, even with last-step aggregation, provides a fundamentally better verification signal than training on final-answer correctness alone.

• Revision model verifier choice (Appendix J, Figure 15a). The base-LM PRM underperforms the revision-specific ORM when scoring revision model outputs. With sequential sampling at 64 generations, sequential + base-LM PRM achieves roughly 40% vs. sequential + revision ORM at roughly 42%. This confirms distribution shift as a practical concern: the PRM trained on standard base-model outputs does not transfer cleanly to the revision model's output distribution, and training a separate verifier on revision outputs is beneficial.

• Revision history in verifier context (Appendix J, Figure 15b). Including previous revisions in the ORM's context provides a small improvement (approximately 1–2 percentage points at 64 generations) over the no-history ablation, but both variants outperform the parallel baseline. This shows that the sequential sampling benefit is not solely attributable to the verifier seeing more context through the revision chain — the revision model genuinely produces better candidates.

• Oracle vs. predicted difficulty bins (Figures 4, 8, and Appendix C, Figures 11–12). Both oracle and predicted bins yield qualitatively similar trends across difficulty levels. In the search setting (Figure 4), the two curves largely overlap at all budgets, suggesting that PRM-based difficulty estimation is nearly as effective as ground-truth difficulty for strategy selection. In the revision setting (Figure 8), predicted bins show slightly lower performance at high budgets (approximately 41% vs. 44% oracle at 256 generations), a ~3-point gap that is relatively small given the ~40-point gap between the best and worst strategies overall.

• Majority voting for revisions (Appendix B, Figure 10). The sequential-to-parallel ratio trends observed with verifier-based selection are replicated with majority voting: easy questions are insensitive to ratio, hard questions show an optimal intermediate ratio, and fully sequential marginally outperforms fully parallel in aggregate. This confirms that the ratio effects are not artifacts of the verifier — the revision model itself benefits from sequential refinement.

• ReST^EM revision model (Appendix K, Figure 16). This is the most important negative result. An attempt to further optimize the revision model using ReST^EM (Singh et al., 2024) — an RL-based self-improvement procedure — substantially hurts performance. At 256 generations, fully sequential performance drops to approximately 33.5% compared to roughly 38.5% at the optimal ratio for the standard revision model. The authors hypothesize that on-policy data collection in ReST^EM "exacerbates spurious correlations in revision data, causing the model to fail to learn the revision task properly." This negative result highlights the sensitivity of revision training to the data generation procedure and suggests that the offline, edit-distance-based data construction used in the main experiments is not trivially replaceable with more sophisticated on-policy methods.

Critical Assessment

The paper's central claims rest on four empirical findings, each with strengths and limitations in how convincingly the experiments support them.

Claim 1: Compute-optimal test-time scaling improves efficiency by more than 4× over best-of-N.

The experiments support this claim directly for both search and revisions. In search (Figure 4), 16 generations with compute-optimal allocation match best-of-N weighted at 64 generations — exactly a 4× reduction. In revisions (Figure 8), 64 generations with compute-optimal allocation match parallel best-of-N weighted at 256 generations — again a 4× reduction. The gain is robust across oracle and predicted difficulty bins, strengthening the practical applicability.

However, this 4× figure has important scope limitations. It is measured after difficulty has been estimated, and the difficulty estimation cost (2048 samples per question + PRM scoring) is not included in the budget. This is a massive unaccounted cost — 2048 samples per question exceeds the largest test-time budgets studied (256–512 generations). In a deployment context, the total cost would be (difficulty estimation + strategy execution), and the former could dominate. The paper acknowledges this explicitly (Section 3.2) but does not amortize the estimation cost into any efficiency calculation. Until difficulty can be estimated cheaply — perhaps via a lightweight classifier trained on PRM score distributions — the 4× figure is an upper bound, not a realized deployment gain.

Additionally, the efficiency claim is demonstrated on a single dataset (MATH) with a single model family (PaLM 2-S*). The 500-question test set, split into 5 difficulty quintiles of ~100 questions each and further divided by two-fold cross-validation, means that the compute-optimal strategy per bin is selected based on only ~50 questions per fold. This is a small sample for strategy selection — confidence intervals are not reported, making it difficult to assess whether the observed efficiency gains are statistically reliable or could vary substantially with a different test split.

Claim 2: Test-time compute with a smaller model can outperform a ~14× larger model under FLOPs-matched comparison.

The experiments support this claim but with sharp, precisely characterized boundary conditions. The claim holds for easy-to-medium problems at low inference-to-pretraining ratios (R ≪ 1): the revisions-based approach achieves +27.8% on medium problems at R = 0.16. At high R and on hard problems, the claim reverses — test-time compute underperforms the larger model substantially (down 52.9% for PRM search on hard problems at R ≫ 1). This conditional structure is a genuine strength: the paper is precise about when the substitution works and when it fails.

However, several aspects of the comparison weaken its force. First, the ~14× larger model uses greedy decoding — no best-of-N, no majority voting, no search of any kind. This is a deliberately weak baseline; a fairer comparison would give the larger model some modest test-time budget (e.g., best-of-8) to avoid stacking the deck in favor of the smaller-model-with-compute approach. Second, the larger model is trained by scaling parameters while holding data fixed (following the LLaMA paradigm), which departs from compute-optimal pretraining à la Chinchilla (Hoffmann et al., 2022). A Chinchilla-optimal ~14× larger model (scaling both parameters and data) would likely be a stronger baseline. The paper acknowledges this as future work but does not address how it might affect the comparison. Third, all FLOPs accounting uses approximate scaling formulas (X = 6ND_pretrain, Y = 2ND_inference) that are themselves approximations — the precise FLOPs comparison depends on these formulas being accurate across model scales.

Claim 3: The effect of test-time compute strategies depends critically on prompt difficulty.

This is the paper's most robust finding and is strongly supported by the difficulty-bin analyses. Figure 3 (right) shows qualitatively different — sometimes opposite — effects of the same strategy at different difficulty levels: beam search hurts easy problems while helping medium ones. Figure 7 (right) shows the optimal sequential-to-parallel ratio shifting from fully sequential (easy) to balanced (hard). These patterns replicate across search and revision methods and across oracle and predicted difficulty bins. The robustness across methods strengthens the finding considerably: it's not an artifact of one particular approach.

The limitation is the coarseness of the difficulty bins. Discretizing a continuous pass@1 range into five quintiles treats all questions within a bin as identical, but there could be substantial heterogeneity — a question at the easy end of bin 3 and one at the hard end receive the same strategy despite potentially benefiting from different allocations. A continuous difficulty-conditioned policy function would be more precise but would require more data to estimate. The paper does not explore the sensitivity of results to the number of bins or to the specific bin boundaries.

Claim 4: Verifier over-optimization is the primary bottleneck for test-time compute scaling.

The experiments provide strong qualitative evidence for this claim without fully characterizing the phenomenon quantitatively. The key evidence is: beam search degrades easy-problem performance at high budgets (Figure 3, right), lookahead search — the most powerful optimizer — paradoxically performs worst overall (Figure 3, left), and qualitative examples show degenerate outputs (repetitive steps, overly short solutions; Appendix M). These patterns are consistent with a reward hacking interpretation: aggressive optimization exploits the PRM's imperfections, producing solutions that score well but are incorrect.

However, the paper does not provide a systematic analysis of over-optimization. There is no experiment that directly measures PRM calibration under different degrees of optimization pressure, no comparison of different verifier training approaches for robustness to exploitation, and no ablation studying whether a better PRM (e.g., with adversarial training or larger capacity) shifts the over-optimization threshold. The compute-optimal policy mitigates over-optimization by routing easy problems away from aggressive search, but it does not solve the underlying problem — on medium-difficulty problems where beam search is deployed, the curve still flattens at high budgets (Figure 3, right), suggesting over-optimization remains a constraint. The paper's contribution is identifying this phenomenon as a first-class scaling bottleneck, but the characterization remains qualitative.

Missing experiments that would strengthen the paper:

• Combining search and revisions. The paper studies PRM tree-search and iterative revisions as independent mechanisms but never combines them (Section 8 acknowledges this explicitly). Given that revisions improve the proposal distribution and search improves candidate selection, combining them — using the revision model within beam search, or using the PRM to guide revision trajectories — is the natural next step. Without this experiment, the reported performance represents a lower bound on what a fully integrated system could achieve.

• Cheap difficulty estimation. The 2048-sample difficulty estimation cost is the most obvious practical barrier. An experiment showing that a lightweight classifier (trained on question text → difficulty bin) achieves accuracy comparable to the PRM-based method would transform the compute-optimal framework from an upper bound to a deployable system. Similarly, an adaptive approach that estimates difficulty from the first few samples (say, 4–8) and allocates the remaining budget accordingly would subsume the estimation cost into the solution process.

• Replication on a second model family and benchmark. All results are on MATH with PaLM 2-S*. Running the compute-optimal framework on, for instance, a LLaMA-based model on GSM8K or HumanEval would establish generality. The difficulty-dependent patterns (beam search hurts easy problems, revisions help easy problems) could be model-specific — a model with different calibration or different error patterns might exhibit different scaling curves.

• Confidence intervals for compute-optimal scaling curves. The 500-question test set is modest for strategy selection (50 questions per bin per fold). Reporting bootstrap confidence intervals for the compute-optimal scaling curves in Figures 4 and 8 would allow readers to assess whether the observed gaps between strategies are statistically reliable.

• Giving the larger model some test-time compute. In the FLOPs-matched comparison, the ~14× larger model uses greedy decoding only. Adding even best-of-4 or majority voting to the larger model would create a more realistic baseline and clarify how much of the test-time compute advantage comes from adaptive allocation versus from the smaller model having any inference-time optimization while the larger model has none.

In summary, the experiments convincingly establish the paper's central conceptual contribution — that test-time compute must be allocated adaptively based on prompt difficulty, and that doing so recovers large efficiency gains — while leaving open the practical question of how difficulty should be estimated at deployment scale and how the findings generalize beyond the specific model-dataset combination studied. The 4× efficiency figure is best understood as an achievable upper bound that depends on further work in cheap difficulty estimation for full realization.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Unaccounted for in the Headline Efficiency Numbers

The assumption or constraint. The entire compute-optimal framework—both the concept that optimal strategies vary by difficulty and the practical method for selecting them—depends on being able to estimate each question's difficulty before allocating the inference budget. The paper's method for doing so is to generate 2048 samples per question, score them with the PRM (or ground-truth checker for oracle bins), compute the average pass@1, and bin into quintiles. The paper acknowledges this explicitly in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The consequence. The 2048-sample difficulty estimation step costs more compute than the largest test-time budget studied (256–512 generations). In any realistic deployment, the total cost would be (difficulty estimation + strategy execution), and the former would dominate the latter—potentially by a factor of 4–8×. The paper's headline claim of "4× efficiency improvement over best-of-N" is therefore computed after difficulty is known, without amortizing the cost of learning it. If the estimation cost were included, the 4× figure could shrink substantially or even reverse, depending on how many questions share a difficulty estimate. The problem is acute: for a system that processes a single question, the cost of difficulty estimation alone exceeds the cost of simply running best-of-N with a large budget, making the compute-optimal approach more expensive overall.

What evidence exists in the paper. The paper provides no experiment that measures or amortizes the difficulty estimation cost. The efficiency curves in Figures 4 and 8 are computed with difficulty bins treated as given. The only analysis of difficulty estimation quality is the comparison between oracle and predicted bins, which shows the predicted bins (using PRM scores rather than ground truth) track the oracle version closely—but this comparison does not address the cost of obtaining the PRM scores or the 2048 samples needed to average them. The paper suggests future work on "pretraining or finetuning models to directly predict difficulty of a question" (Section 8) but provides no prototype, ablation, or cost model for such an approach.

Mitigation status. Not addressed. The paper frames this as future work and does not incorporate difficulty estimation cost into any efficiency calculation. A practitioner implementing this approach today would face the full 2048-sample overhead per question unless they develop their own cheap difficulty estimator—which the paper provides no guidance for building.

---

Hard Problems Remain Fundamentally Unsolved—Test-Time Compute Does Not Create New Capability

The assumption or constraint. The paper's framework operates under an implicit assumption that the base model already has some non-trivial probability of producing correct solutions. This assumption is violated for the hardest problems (difficulty bin 5), where the base model's pass@1 is near zero. The paper is explicit about this boundary in Section 7:

"on the hardest questions (bin 5), no method makes meaningful progress—the base model simply lacks the capability to produce correct solutions regardless of how the budget is allocated"

The consequence. For problems outside the base model's capability range, the entire framework provides essentially zero benefit. Across all methods studied—PRM search (Figure 3, right), iterative revisions (Figure 7, right), compute-optimal combinations (Figures 4, 8), and FLOPs-matched comparisons (Figure 9)—bin 5 accuracy remains at 1–5% regardless of compute budget. This is not a gradual degradation but a hard cliff: below some pass@1 threshold, test-time compute ceases to help. The practical implication is that deploying this approach requires knowing whether the problem distribution contains fundamentally unsolvable problems, because compute spent on those problems is entirely wasted. The difficulty estimator, even if made cheap, would correctly identify the problem as "hard" but then have no effective strategy to recommend—the best action is to not attempt the problem at all or to route it to a larger model. The paper's FLOPs-matched analysis (Section 7) quantifies this: on hard problems at high inference-to-pretraining ratios (R ≫ 1), test-time compute underperforms the 14× larger model by 37–53%—meaning the compute-optimal approach is actively worse than simply having trained a larger model.

What evidence exists in the paper. The evidence is comprehensive and consistent. Figure 3 (right) shows bin 5 beam search and best-of-N both at 1–3% across all budgets up to 256 generations. Figure 7 (right) shows bin 5 revision accuracy at 2–3% regardless of sequential-to-parallel ratio. Figure 9 shows the bin 5 scaling curve essentially flat near 0–5% for both revisions and PRM search. Table 1, which shows the strongest overall results (pruned + KD achieving 73.45 average), masks this heterogeneity—the average is pulled up by easy and medium problems, while hard problems contribute nearly nothing. The paper does not report per-bin breakdowns for the FLOPs-matched claims in the main text bar charts (Figure 1), which aggregate easy/medium/hard into coarser groups, potentially obscuring the complete failure on the hardest tier.

Mitigation status. The paper acknowledges this limitation clearly (Section 7 takeaway, Section 8) but offers no solution. The insight that test-time compute amplifies existing capability rather than creating it is a boundary condition of the method, not a bug that can be fixed with better allocation. The only mitigation the paper implicitly suggests is to not use this approach on hard problems—but that requires knowing which problems are hard before spending compute on them, which circles back to the difficulty estimation cost problem.

---

The ~14× Larger Model Baseline in the FLOPs-Matched Comparison Is Deliberately Weak

The assumption or constraint. The FLOPs-matched comparison in Section 7 pits PaLM 2-S* with compute-optimal test-time scaling against a model with approximately 14× more parameters. Two design choices make this baseline weaker than it could be. First, the larger model uses greedy decoding only—no best-of-N, no majority voting, no search, no revisions. Second, the larger model's pretraining follows the LLaMA paradigm of scaling parameters while holding training data fixed, rather than compute-optimal pretraining à la Chinchilla (Hoffmann et al., 2022) where both parameters and data are scaled. The paper acknowledges the second point explicitly:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

The consequence. The reported advantages of test-time compute over pretraining—for example, +27.8% on easy questions at R ≪ 1, +19.1% on easy questions for PRM search at R ≪ 1 (Figure 1 bar charts)—are measured against a baseline that is suboptimal in two ways. A Chinchilla-optimal 14× larger model (with both more parameters and more training data) would likely be a stronger baseline, potentially reducing or reversing the reported advantages. Similarly, giving the larger model even a modest test-time compute budget—say, best-of-8 or majority voting with 8 samples—would create a more realistic comparison, since deploying a large model with no inference-time optimization is not representative of how such models are actually used. The magnitude of the baseline weakness is unknown: the paper provides no experiment with a Chinchilla-optimal larger model, nor any ablation giving the larger model test-time compute.

What evidence exists in the paper. The paper provides no experiments or estimates quantifying how much the baseline weakness affects the FLOPs-matched conclusions. The only hint comes from the PRM search vs. revisions comparison in Figure 9: revisions show substantially larger advantages over the larger model than PRM search does, particularly on medium and hard problems. This could indicate that revisions provide genuinely larger benefits, or it could indicate that the baseline is sufficiently weak that different test-time strategies produce different apparent advantages—which would mean the absolute advantage numbers are unreliable without a stronger baseline. The paper does not analyze this distinction.

Mitigation status. The paper acknowledges the Chinchilla-optimal pretraining issue as future work but does not address the greedy decoding issue at all. A practitioner reading the FLOPs-matched results should understand that the ~14× larger model comparison is best interpreted as "test-time compute with a small model vs. a larger model with no inference optimization," not "test-time compute vs. pretraining compute in general."

---

Single Model Family and Single Benchmark Limit Confidence in Generalization

The assumption or constraint. All experiments use a single base model (PaLM 2-S*) and a single benchmark (MATH, 500 test questions). The paper states that it "believes this model is representative of the capabilities of many contemporary LLMs" (Section 4), but provides no replication on other model families, architectures, or tasks. The specific difficulty-dependent behaviors—beam search degrading easy-problem performance, revisions dominating on easy problems, the optimal sequential-to-parallel ratio shifting with difficulty—could be artifacts of PaLM 2-S*'s specific training, calibration, and error patterns rather than universal properties of LLM test-time compute scaling.

The consequence. A practitioner with a different model (e.g., LLaMA-3, Qwen, DeepSeek) or a different task (e.g., code generation, factual QA, summarization) cannot confidently apply the paper's specific findings without re-running the analysis. The PRM over-optimization threshold, the optimal beam width for beam search, the benefit of MTP KD for speculative decoding, and the compute-optimal allocation policy are all potentially model-specific and task-specific. The difficulty bins themselves are defined relative to PaLM 2-S*'s pass@1 distribution—a model with better or worse MATH performance would have different bin boundaries and potentially different difficulty-strategy relationships within those bins. The 500-question test set, further split into five difficulty quintiles of ~100 each and then cross-validated, means the compute-optimal policy is selected based on only ~50 questions per bin per fold—a small sample that could produce unstable strategy selections that don't generalize to other MATH questions, let alone other tasks.

What evidence exists in the paper. None. The paper provides no cross-model or cross-task replication. The MATH benchmark consists exclusively of competition-level math problems requiring symbolic reasoning and exact-answer matching. Tasks with different structure—open-ended generation, subjective evaluation, multi-turn dialogue, code with complex correctness criteria—may exhibit entirely different difficulty-dependent scaling patterns. The paper's difficulty estimation depends on having clean correctness signals (for oracle bins) or well-calibrated PRM scores (for predicted bins); tasks without such signals would require fundamentally different difficulty estimation approaches that the paper does not explore.

Mitigation status. Not addressed. The paper does not claim generalization beyond the studied setting, but it also does not discuss this as a limitation. The abstract's claim to "offer practical guidance for efficient MoE compression at scale" implicitly assumes the findings transfer, but no evidence supports this.

---

Latency and Wall-Clock Time Are Ignored Despite Favoring Parallel Strategies Over Sequential Ones

The assumption or constraint. The paper measures all test-time compute in "generations"—a unit that counts total sampled solutions regardless of whether they are generated sequentially or in parallel. This is a reasonable proxy for total FLOPs but ignores latency (wall-clock time). A strategy that allocates 128 generations as 64 sequential revisions × 2 parallel chains takes approximately 64× longer to execute than one that runs 128 parallel samples simultaneously on sufficient hardware, even though both consume the same total FLOPs. The paper's compute-optimal policies often favor sequential-heavy strategies, particularly on easy problems (Figure 7, right, where bin 2 shows fully sequential as optimal).

The consequence. For latency-sensitive applications—interactive assistants, real-time decision systems, API endpoints with strict timeout requirements—the sequential-heavy strategies favored by the compute-optimal policy may be impractical regardless of their accuracy advantages. A 64-step revision chain that takes 30 seconds to produce an answer may be unacceptable even if it achieves 10 points higher accuracy than a 4-second parallel approach. The paper's framework provides no mechanism to incorporate latency constraints into the strategy selection—the optimization objective (Equation 1) maximizes accuracy given a generation budget, not accuracy given a time budget. A practitioner deploying this approach would need to manually adjust the compute-optimal policy to respect latency requirements, potentially leaving substantial performance on the table.

The issue is compounded by the difficulty estimation overhead: generating 2048 samples to estimate difficulty is not only FLOPs-expensive but also latency-expensive (requiring 2048 sequential or batched generations before the actual strategy can begin). Even if difficulty estimation were amortized across many questions, the first question in a batch would incur the full latency cost.

What evidence exists in the paper. None. The paper never discusses latency, wall-clock time, or throughput. The compute budget is defined entirely in terms of total generations without any constraint on how those generations are scheduled. Figure 2 (right), which compares sequential and parallel revision strategies, plots accuracy against generation count with no latency dimension. The finding that fully sequential revisions outperform parallel sampling at low budgets (Figure 7, left) is reported without acknowledging that fully sequential takes linearly more wall-clock time than fully parallel at the same generation count.

Mitigation status. Not addressed. The paper does not mention latency as a concern or discuss tradeoffs between generation efficiency and wall-clock efficiency. This is a fundamental omission for any work that recommends sequential strategies—the practitioner is left to discover on their own that the best generation-budget strategy may be the worst time-budget strategy.

---

The Revision Model Has a 38% Correct-to-Incorrect Reversion Rate with Only Partial Mitigation

The assumption or constraint. The revision model is trained exclusively on sequences where all in-context answers are incorrect, followed by a correct target (Section 6.1). During inference, the model may encounter correct answers in its own revision chain—produced during earlier steps—and, having never been trained on what to do when the current answer is already correct, may incorrectly "revise" a correct answer into an incorrect one. The paper reports that approximately 38% of correct answers get converted back to incorrect ones using a naive approach (Section 6.1).

The consequence. The revision chain is inherently unstable: even when the model produces a correct answer at some step, there is no guarantee it will stay correct. The paper's mitigation is to use a selection mechanism (majority voting or verifier-based selection) across the entire chain, picking the best answer from any step rather than always taking the final revision. While this works—Figure 6 (right) shows sequential + best-of-N weighted reaching 41.5% vs. the 24–25% pass@1 at individual steps—it introduces its own cost: the verifier must evaluate every step in the chain, adding computational overhead, and the selection mechanism can make errors (the verifier might select an incorrect step over a correct one). More fundamentally, the 38% reversion rate means that a substantial fraction of the revision chain's compute is wasted producing degraded versions of already-correct answers before the verifier (hopefully) identifies and selects the correct step.

What evidence exists in the paper. The paper reports the 38% figure explicitly in Section 6.1 but does not provide a detailed analysis of when reversions occur (e.g., whether they are concentrated in specific difficulty bins, specific problem types, or specific positions in the revision chain). Appendix K (Figure 16) provides additional evidence of revision model fragility: the ReST^EM-trained revision model degrades performance substantially with sequential revisions, achieving only ~33.5% at fully sequential vs. ~38.5% for the standard model at optimal ratio, suggesting that the revision training procedure is sensitive to data construction choices in ways that are not fully understood.

Mitigation status. Partially addressed. The within-chain selection mechanism (majority voting or verifier selection) mitigates the practical impact of reversions, since the correct answer can be recovered from an earlier step. However, this does not prevent the wasted computation of generating the incorrect reversions, and it shifts the burden onto the verifier to correctly identify which step is correct—a task that becomes harder as the chain length grows and the verifier must evaluate more candidates. A more principled solution—such as training the model to recognize when no revision is needed, or incorporating "stop revising" signals into the training data—is not explored.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper fundamentally reframes MoE model compression from an architectural optimization problem to a training pipeline design problem. Prior work on MoE compression—M-SMoE (Li et al., 2024b), REAP (Lasby et al., 2025b), expert pruning by Lu et al. (2024), and the thorough one-shot benchmarking by Jaiswal et al. (2025)—focused heavily on optimizing the compression criterion: designing better importance metrics, more sophisticated merging algorithms, or more nuanced expert similarity measures. The implicit assumption was that the quality of the one-shot compression decision determines the quality of the final compressed model. SlimQwen provides decisive evidence that this assumption is wrong after sufficient continued training.

The key finding that different expert compression metrics converge to statistically indistinguishable performance after 400B tokens (Table 2) is not merely a negative result—it's a positive reframing of where the field's attention should be directed. The 8 different expert compression strategies in Table 2, spanning different importance metrics (Soft Logits, REAP, Frequency), different grouping methods (Router Logits, Router Weights, Expert Vectors), and different merge strategies (with and without partial preservation), all cluster within roughly 1–2 points on any given benchmark. Compare this to the effects of training design choices: switching from random initialization to pruned initialization yields +11.79 points (Table 1); switching from one-shot to progressive depth-first scheduling yields +1.53 points on MMLU (Table 5); adding LM loss to pure KD yields +0.77 points on MMLU (Table 3). The expert compression metric choice is an order of magnitude less impactful than these training pipeline decisions.

This is a methodological reorientation, not a paradigm shift. It doesn't invalidate prior work on expert importance estimation—those methods still matter for the one-shot quality of the compressed checkpoint, which determines the starting point for continued training—but it establishes that the primary determinant of final model quality is what happens after compression, not the compression itself. For the research community, this implies a reallocation of effort: marginal improvements in expert selection criteria are unlikely to yield meaningful downstream gains, while improvements in training objectives, progressive scheduling, and distillation strategies have substantial leverage. The paper's own partial-preservation expert merging strategy—which shows modest but consistent gains of 0.5–1 point—is positioned as a simple, robust heuristic, not as the theoretically optimal solution, and the paper explicitly frames further expert metric optimization as a lower-priority direction.

The work also resolves a latent tension in the compression literature between one-shot evaluation and post-compression training. Many prior works evaluate compression methods immediately after pruning, without any continued training. Others train briefly but don't study whether different compression methods converge. This ambiguity made it unclear whether one-shot performance differences translate to final model quality. SlimQwen resolves this: the one-shot differences exist (the appendix results in Table 8 show that different depth pruning methods yield dramatically different one-shot performance), but they don't persist after large-scale continued pretraining. This means that one-shot evaluation, while convenient for rapid iteration, is a poor proxy for end-to-end compression quality when continued training is part of the pipeline—which it almost always is in practice, since one-shot pruned models are rarely deployed without recovery training.

The conceptual shift also affects how we think about knowledge retention during compression. The finding that pruned initialization + KD recovers 86.5% of the teacher's average benchmark score at 3.4× compression (73.45 vs. 82.68, Table 1) suggests that a substantial fraction of the teacher's knowledge is encoded in parameter relationships that survive structured pruning—the relative organization of expert parameters, the routing patterns, the hidden dimension importance ordering—rather than in raw parameter count. This aligns with the progressive pruning results (Table 5), where intermediate architectures serve as effective stepping stones for knowledge transfer. The model doesn't just need parameters; it needs parameters organized in the right relational structure, and that structure can be preserved even as total capacity is reduced.

The MTP distillation innovation contributes a more targeted shift: it establishes that distillation objectives should be co-designed with deployment inference strategies, not just optimized for aggregate perplexity or benchmark scores. The speculative decoding acceptance rate improvements in Table 4—MTP KD improves acc 1 from 57.62% to 75.18% on GSM8K, and acc 4 from 2.37% to 10.37% (a >4× relative improvement)—demonstrate that what you distill (single-token vs. multi-token distributions) can have outsized effects on specific deployment metrics even when benchmark score improvements are modest. This reframes compression evaluation: standard benchmarks measure what the model knows, but deployment-relevant metrics like multi-token acceptance rate measure how efficiently that knowledge can be extracted in production. The optimal training recipe may differ for these two objectives, and the paper's results suggest that MTP KD bridges them—improving both benchmark scores and speculative decoding efficiency.

Follow-Up Research This Work Enables

Cheap difficulty estimation for adaptive test-time compute allocation. The paper's central practical barrier is its unaddressed difficulty estimation cost: generating 2048 samples per question to bin it into a difficulty quintile consumes more compute than the largest test-time budgets studied (256–512 generations). A natural and high-impact follow-up would train a lightweight difficulty classifier—a small model (perhaps a few hundred million parameters) that takes only the question text as input and predicts the PRM's average final-answer score (a continuous value) or the difficulty quintile directly. The training data already exists: the paper has generated 2048 samples for each of the 500 MATH test questions, yielding (question_text, average_PRM_score) pairs that could serve as supervised training data. A successful classifier achieving even 80% agreement with the PRM-based quintile assignments would make the compute-optimal framework deployable without the 2048-sample overhead, transforming the 4× efficiency figure from an upper bound into a realized gain. The key metric would be the accuracy of the compute-optimal strategy selected using classifier-predicted bins vs. PRM-predicted bins—if the strategy selection accuracy is high, the small gap between oracle and PRM-predicted bins in Figure 4 suggests the efficiency gain would be largely preserved.

Adaptive, dynamic difficulty estimation that amortizes cost into the solution process. Beyond training a separate classifier, a more ambitious direction is to estimate difficulty on-the-fly: begin each problem with a small number of parallel samples (say, 4–8), compute the PRM's average final-answer score on those initial samples as a rough difficulty signal, and then allocate the remaining budget according to a policy that conditions on this continuous difficulty estimate rather than a discrete bin. This would directly address the exploration-exploitation tradeoff the paper flags in Section 3.2—the initial samples serve both to estimate difficulty and as the first attempts at solving the problem, so no compute is "wasted" on pure estimation. The challenge is that 4–8 samples provide a noisy difficulty estimate, and the policy would need to be robust to this noise. A strong experiment would compare the accuracy-vs-total-compute curve of this adaptive approach against (a) the paper's fixed-bin approach with 2048-sample estimation cost included and (b) a uniform best-of-N baseline, using the same PaLM 2-S* model and MATH dataset to enable direct comparison with the paper's results. The paper's compute-optimal curves in Figures 4 and 8 would serve as oracles (assuming perfect difficulty knowledge) that the adaptive approach should approximate.

Combining PRM tree-search with the revision model as the proposal distribution. The paper studies search and revisions as independent mechanisms but explicitly notes they were never combined (Section 8). The complementary strengths—revisions improve the proposal distribution on easy problems, beam search improves candidate selection on medium problems—suggest a natural integration: use the revision model as the generation engine within beam search. Specifically, at each expansion step of beam search, instead of sampling from the base LLM, sample from the revision model conditioned on the previous (potentially incorrect) partial solution as context. The PRM scores each revised step, and beam search prunes low-scoring branches as usual. This would test whether the revision model's ability to make targeted corrections (learned through edit-distance-based training) synergizes with the PRM's ability to evaluate intermediate steps. A concrete experiment: on the MATH benchmark, compare (a) revision model + best-of-N weighted (the paper's sequential baseline), (b) base model + beam search (the paper's search baseline), and (c) revision model + beam search (the combination), all at matched generation budgets of 64, 128, and 256 generations, with per-difficulty-bin analysis. The hypothesis is that the combination would outperform both individual methods on medium-difficulty problems (bins 3–4), where the proposal distribution needs improvement AND candidate selection benefits from search.

Improving verifier robustness to over-optimization through adversarial PRM training. The paper identifies verifier over-optimization as the primary scaling bottleneck—beam search degrades easy-problem performance at high budgets (Figure 3, right), lookahead search underperforms simpler methods (Figure 3, left), and qualitative examples show degenerate outputs scoring highly under the PRM (Appendix M). A direct follow-up would explore whether PRMs can be made more robust to optimization pressure. The paper's PRM is trained on Monte Carlo rollouts from the base model's i.i.d. sampling distribution—but at test time, beam search explores a very different, more aggressively optimized distribution. A natural experiment: iteratively train the PRM on search-generated (rather than randomly sampled) solutions, so that the PRM learns to recognize and penalize the degenerate patterns that search exploits. This is adversarial training in the verifier domain. The evaluation would compare the over-optimization curve (accuracy vs. budget, as in Figure 3, right) for the adversarially trained PRM against the standard PRM, looking specifically at whether beam search performance on easy problems stops degrading at high budgets. The paper's existing framework (same model, same dataset, same evaluation protocol) makes this experiment straightforward to set up and directly comparable to published results.

Replication of progressive pruning benefits on a different MoE architecture and task domain. The paper's progressive pruning findings—that depth-first scheduling achieves +1.53 MMLU over one-shot (Table 5), and that two stages capture all benefit with no gain from three stages (Appendix A.5, Table 9)—are demonstrated on a single architecture (Qwen3-Next) and evaluated primarily on knowledge and reasoning benchmarks (MMLU variants, BBH, GSM8K). A strong stress-test would replicate the depth-first vs. one-shot comparison on a different MoE architecture (e.g., Mixtral 8×7B, DeepSeek-MoE) and include a task domain with different structural demands, such as long-context retrieval (needle-in-haystack), code generation (HumanEval, MBPP), or multilingual generation. Different architectures have different layer-to-expert ratios, different attention mechanisms, and different pretraining data mixtures—all of which could affect whether progressive pruning provides benefits and whether depth-first remains optimal. The specific experiment: take a pretrained Mixtral 8×7B, compress it to a ~3× smaller architecture using the paper's compression ratios (similar proportional depth, width, and expert reduction), and compare one-shot vs. depth-first progressive vs. width-first progressive under 100B tokens of continued training, evaluating on MMLU, HumanEval, and a long-context retrieval benchmark. This would establish whether the paper's scheduling insights are architecture-specific or broadly applicable.

The role of training data quality and composition in post-compression recovery. The paper uses "high-quality, diverse tokens" (Section 4.1) for continued pretraining but does not ablate data composition. A critical open question is: does the optimal post-compression training data distribution match the teacher's pretraining distribution, or should it be biased toward domains where compression caused the most knowledge loss? The paper's difficulty-dependent findings for test-time compute (not this paper's compression work, but the reference example) showed that easy and hard problems respond differently to different strategies—a similar difficulty-dependent pattern might exist for compression recovery: knowledge-intensive benchmarks (MMLU, MMLU-Pro) benefit more from KD than math benchmarks (GSM8K), as suggested by Table 3 where adding LM loss to KD improves MMLU (74.16 → 74.93) more than it improves GSM8K (84.27 → 82.98, actually a slight decrease). A concrete experiment: take the compressed SlimQwen-23A2B checkpoint, continue training for 120B tokens with three different data mixtures—(a) uniform pretraining distribution, (b) knowledge-heavy (more Wikipedia, textbooks), (c) reasoning-heavy (more math, code)—and evaluate on the full benchmark suite. This would reveal whether targeted data can compensate for domain-specific knowledge loss from compression, and would provide practical guidance on data curation for post-compression training.

Practical Applications and Downstream Use Cases

Single-GPU deployment of MoE models for cost-sensitive applications. The most directly actionable practical implication comes from Appendix A.7 (Table 11): the compressed SlimQwen-23A2B requires only 43.30 GB peak memory versus 156.56 GB for the original Qwen3-Next-80A3B. This crosses a critical infrastructure threshold—the compressed model fits comfortably on a single 80GB GPU (e.g., A100, H100), eliminating the need for tensor parallelism or pipeline parallelism. The efficiency gains compound: beyond the memory savings, the single-GPU deployment improves decoding throughput (210.87 tok/s vs. 142.58 tok/s with vLLM backend) and reduces prefill latency (0.06s vs. 0.08s). For organizations serving MoE models at scale—cloud API providers, enterprise deployments—this translates directly to reduced GPU-hour costs per query and simplified infrastructure. A deployment serving the compressed model on a single GPU per replica, versus the original model requiring two GPUs with tensor parallelism, approximately halves the GPU cost per query while also eliminating inter-GPU communication overhead. The benchmark performance retention (86.5% of teacher average score, Table 1) means this cost reduction does not require sacrificing a proportional amount of capability—the compression is genuinely efficiency-improving, not just cost-reducing at the expense of quality.

Draft model training for speculative decoding pipelines. The MTP distillation results (Table 4) directly enable a specific deployment architecture: use the SlimQwen compressed model as a high-quality, low-latency draft model in a speculative decoding pipeline with the original teacher model (or an even larger model) as the verifier. In speculative decoding, a small "draft" model proposes multiple future tokens cheaply, and a larger "verifier" model checks and accepts/rejects them in parallel. The draft model's multi-token acceptance rate determines the overall throughput improvement—higher acceptance means more tokens verified per forward pass of the large model. Table 4 shows that MTP KD improves the 2-token acceptance rate (acc 1) from 57.62% to 75.18% on GSM8K, and the 5-token acceptance rate (acc 4) from 2.37% to 10.37%. At these acceptance rates, a pipeline with SlimQwen as drafter and Qwen3-Next-80A3B as verifier would generate tokens significantly faster than running the large model alone, while maintaining the large model's quality (since the verifier rejects incorrect drafts). The SlimQwen-23A2B's single-GPU deployability is particularly attractive here: it can run as a lightweight sidecar to the larger model without requiring its own multi-GPU infrastructure. The paper provides the training recipe (hybrid KD-LM-MTP loss, progressive depth-first scheduling) and the acceptance rate benchmarks; a practitioner can directly implement this with their own MoE models following the described procedure.

Cost-efficient data generation for self-improvement and distillation pipelines. When using large MoE models to generate training data—for distilling into smaller models, for self-improvement loops like STaR/ReST^EM, or for synthetic data creation—the generation cost is proportional to the model size. The SlimQwen compression pipeline provides a way to reduce this cost without proportionally sacrificing generation quality. Specifically, the compressed 23A2B model at 86.5% of teacher benchmark performance can serve as a cheaper proxy data generator: for a given budget of GPU-hours, it can generate more tokens than the larger model. The quality gap from compression matters less in this setting than in direct deployment because the generated data will be filtered, curated, or used as auxiliary training data rather than served directly to users. The paper doesn't directly evaluate this use case—no experiment measures the quality of downstream models trained on SlimQwen-generated versus teacher-generated data—but the strong benchmark retention (especially on knowledge-intensive tasks like MMLU) and the generation efficiency numbers (Table 11: 210.87 tok/s vs. 142.58 tok/s, a ~48% throughput improvement) make this a natural extension. A practitioner generating 1 trillion tokens of training data could reduce GPU-hours by roughly one-third using the compressed model, with the tradeoff being the benchmark score gap between compressed and original models.

When to Prefer This Method

The paper positions itself as providing a recipe for practitioners who need to compress a pretrained MoE model for deployment, not as proposing a single method that should be preferred over named alternatives. It does not articulate a tradeoff matrix against competing compression frameworks (e.g., "prefer SlimQwen over Minitron when ...").

Instead, the paper's contribution is establishing that certain design choices matter substantially and others matter little, enabling practitioners to allocate their engineering effort efficiently:

• Invest effort in training pipeline design (progressive scheduling, hybrid KD-LM-MTP objectives) rather than expert compression metric optimization. The expert metric choice converges after continued training (Table 2); the training schedule and objective do not.

• Use pruned initialization rather than training from scratch when the goal is post-compression continued pretraining. The initialization advantage is decisive (+11.79 points average, Table 1) and interacts positively with KD.

• Use depth-first progressive pruning when transitioning from a large MoE to a compressed architecture under continued training. Two stages are sufficient; three stages add complexity without additional gains (Appendix A.5, Table 9).

• Include LM loss alongside KD in the post-compression training objective, particularly when knowledge-intensive benchmark performance is important (MMLU, MMLU-Pro benefit most, Table 3).

• Use MTP distillation when the compressed model will be used for speculative decoding, as it provides substantial multi-token acceptance rate improvements (Table 4) with modest benchmark score gains.

• Apply partial-preservation expert merging (keeping roughly half of target experts intact) as a simple default strategy. It consistently outperforms both pure pruning and full merging across most benchmarks (Table 2), with negligible additional complexity.

These recommendations are specific, empirically grounded in the paper's ablation tables, and directly actionable for any team compressing a large MoE model. They avoid the generic "prefer X when Y" pattern because the paper does not compare against named alternative compression frameworks—it systematically ablates its own design space and reports what works.