MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies

🎯 Pitch

Small 1–2B models can rival 7B–13B models when trained with a new learning rate scheduler that reveals the optimal data-to-model ratio is nearly 10× larger than previously believed. The key innovation lets researchers study scaling laws without retraining from scratch, slashing the cost of finding optimal model sizes.

1. Executive Summary

This paper introduces MiniCPM, a family of Small Language Models (SLMs) with 1.2B and 2.4B non-embedding parameters, and proposes scalable training strategies centered on a novel Warmup-Stable-Decay (WSD) learning rate scheduler that enables continuous training and efficient data-model scaling law measurement. Through model wind tunnel experiments on C4 with models spanning 0.009B to 2.1B parameters, the authors optimize hyperparameter transfer via Tensor Program techniques and determine that optimal learning rates remain stable around 0.01 across scales. The WSD scheduler explicitly splits training into a high-learning-rate stable phase and a brief decay phase—where loss drops dramatically with only 10% of total tokens needed for convergence—enabling reuse of intermediate checkpoints to study scaling laws with linear rather than quadratic cost along the data axis. The resulting scaling law yields a much higher compute-optimal data-to-model ratio (~192×) than Chinchilla Optimal (~20×), establishing that smaller models can absorb far more data than previously assumed. The MiniCPM-2.4B model matches or exceeds 7B–13B LLMs including Mistral-7B and Llama2-13B on comprehensive benchmarks (achieving 52.33% average accuracy vs. Mistral-7B's 48.97%), and the paper extends the family to long-context (128K), DPO-aligned, and Mixture-of-Experts variants, demonstrating that SLMs can achieve capabilities on par with much larger models when trained with scientifically calibrated, scalable methodologies—though the paper does not validate these scaling laws on LLM-scale models themselves.

2. Context and Motivation

The Core Problem: The Training of Large Language Models Is Unsustainable for Most Researchers

The fundamental tension this paper addresses is deceptively simple: the current paradigm of LLM development is financially and operationally prohibitive for the vast majority of research institutions and corporations. The paper opens (Section 1) by observing that the field has pursued models with "up to an astonishing number of parameters in the trillions" (Fedus et al., 2022), following the revelation of scaling laws (Kaplan et al., 2020). This pursuit, while technically impressive, creates a bifurcated landscape:

Training is economically exclusive. The costs of training trillion-parameter models are measured in millions of dollars. The paper states directly that "given the significant economic and environmental costs, experiments on LLMs are prohibitively expensive for most researchers and corporations" (Section 1). This isn't just an academic complaint — it means that empirical understanding of why LLMs work, how to optimize training, and what scaling strategies apply, is concentrated in a very small number of industrial labs with access to massive compute clusters.
Deployment is practically impossible for everyday scenarios. The paper points out that "the deployment of these colossal models in everyday scenarios, such as on personal computers or smartphones, is either inefficient or unfeasible" (Section 1). This is a critical real-world bottleneck: if LLMs are to power applications like on-device assistants, privacy-preserving local inference, or edge computing, then 70B-parameter models that require multiple GPUs are fundamentally incompatible with these deployment targets.

These two problems — experimental inaccessibility and deployment infeasibility — are not independent. The same cost that makes large-model training exclusive also makes large-model inference expensive per query. They form a compound barrier: most organizations cannot afford to train large models, and even if they could, they cannot afford to serve them at scale.

Why This Matters Beyond Cost: The Empirical Knowledge Gap

Beyond the resource question, there is a subtler and perhaps more important problem: the empirical understanding of LLM training mechanisms remains elusive. The paper explicitly states this (Section 1):

"the empirical understanding of the mechanisms underpinning the training of LLMs remains elusive."

This is not merely a complaint about cost. If it were, one could argue that the few well-funded labs could simply conduct the necessary experiments and publish their findings. The deeper problem is that when experiments are too expensive to run at LLM scale, the entire field operates on assumptions and heuristics rather than systematic empirical science. The specific consequences the paper highlights include:

Uncertainty in scaling laws. Even the most fundamental question — the optimal ratio of data size to model size — remains contested. Kaplan et al. (2020) argued for emphasizing model size scaling (10× parameters for 1× data increase), while Hoffmann et al. (2022) argued for equal scaling, and more recent models like Llama 2 (Touvron et al., 2023) train with far more data than either would prescribe, "still yielding considerable performance gain" (Section 4.5). The paper frames this as an "unaddressed uncertainty" that "stems from the challenges inherent in training multiple models of varying sizes and data sizes in traditional scaling experiments" (Section 4.5). If measuring scaling laws requires training many models from scratch to many different token counts, the cost is quadratic in the number of model sizes — clearly infeasible for large models.
Fragile training strategies. The paper notes that learning rate scheduling, while crucial for model performance, is largely governed by convention rather than systematic analysis. The Cosine learning rate scheduler (Loshchilov & Hutter, 2016) is dominant (Kaplan et al., 2020; Hoffmann et al., 2022; Touvron et al., 2023), but the why behind its effectiveness is poorly characterized. The paper identifies two underexplored questions (Section 4.1): (a) why setting the cosine period equal to total training steps is optimal, and (b) what unique training dynamics occur during the learning rate decay phase. Without the ability to conduct controlled experiments at scale, these questions remain in the realm of folklore rather than science.
The invisibility of intermediate states. A practical consequence of training large models with predefined schedules is that intermediate checkpoints are largely wasted. If a model is trained with Cosine LRS to a fixed step count, the checkpoints along the way are not optimized for their current training state — their learning rate was tuned assuming they would continue training. The paper argues this prevents efficient exploration of scaling properties and makes the reuse of training investment impossible.

The Prior Approaches and Their Limitations

The paper positions itself within a rapidly growing ecosystem of SLM research (Section 2) that has emerged partly in response to the LLM bottleneck. This is not a novel ecosystem — the paper acknowledges the existence of several prominent SLM families — but it identifies two specific gaps that existing SLMs have "yet to fully satisfy":

Gap 1: Lack of Comprehensive Abilities

Existing SLMs, the paper argues, do not achieve the breadth of capabilities demonstrated by LLMs. The authors characterize this as a need for "comprehensive abilities akin to those exhibited by LLMs" (Section 1). This is a substantial claim. Previous SLMs have shown strong performance on specific benchmarks — for example, Phi-2 (Javaheripi & Bubeck, 2023) excels on academic-oriented datasets — but the paper contends that this comes at the cost of general versatility. Models trained primarily on textbook-style data may perform well on standardized evaluations while struggling with the diverse, noisy, and open-ended tasks that LLMs handle routinely.

The evidence for this gap is partially presented in the evaluation results (Table 3), where Phi-2 achieves an average score of 48.84% — comparable to MiniCPM-2.4B's 52.33% — but the paper argues that Phi-2's performance stems from data that "mostly involves textbook-style data that emphasize educational and academic scenarios," while "our pre-training data covers more distribution" (Section 6.5). The implication is that SLMs have not yet demonstrated the kind of general-purpose competence that makes LLMs useful across diverse deployment scenarios, and that this limitation is a function of training strategy and data composition rather than model size alone.

Gap 2: Absence of Transparent, Scalable Training Methodologies

This is the more technically substantive gap. The paper argues that existing work on SLMs has not produced training methodologies that both explain their own effectiveness and scale predictably to larger models. The authors phrase this as "the formulation of transparent and scalable training methodologies that could further propel the evolution of both SLMs and LLMs" (Section 1).

What makes this gap significant is that it prevents SLM research from being truly useful for the broader field. If an SLM achieves good results through a combination of ad-hoc hyperparameter tuning, proprietary data filtering, and model-specific architectural choices, those results provide little guidance for anyone training a larger model on different data. The paper's ambition is explicitly bidirectional: SLM research should inform LLM development, and training strategies developed on SLMs should transfer upwards. This is the intellectual framing behind the "wind tunnel" metaphor (Section 3) — small models as experimental testbeds for principles that apply at any scale.

The paper catalogs prior work that touches on scalable training (Section 2), but identifies limitations in each:

Tensor Program series (Yang et al., 2022; 2023) provides a principled framework for hyperparameter transfer across scales, but it has been applied in limited contexts (CerebrasGPT; Dey et al., 2023) and has not been systematically combined with data scaling strategies.
Small-scale instability prediction (Wortsman et al., 2023) uses smaller models to anticipate training instabilities in larger models, but this addresses the negative problem (avoiding failure) rather than the positive problem (optimizing performance).
Data-centric strategies (Xie et al., 2024; Shi et al., 2023; Ye et al., 2024) focus on data quality and mixture, but do not provide a systematic framework for data quantity scaling — how much data is enough for a given model size.
Batch size scheduling (Smith et al., 2017), adopted by Yi-9B (Young et al., 2024), offers an alternative to learning rate decay but does not address the broader question of how to structure the entire training process.
DeepSeek (Bi et al., 2024) is noted as bearing "the closest resemblance to our proposed WSD LRS" (Section 2), though the paper does not elaborate on the specific similarities or differences. The implication is that prior work has touched on similar ideas but has not developed them into a systematic, experimentally validated framework.

The Specific Technical Gap: Learning Rate Scheduling as an Unexplored Lever

Within the broader landscape of scalable training, the paper identifies learning rate scheduling as a particularly underexplored lever with outsized importance. The dominance of Cosine LRS is not based on rigorous comparison — it is based on convention and the observation that it works well enough. The paper's own analysis of Cosine LRS (Section 4.1) reveals a non-obvious property:

"Cosine LRS with T = S has a longer duration of high learning rate training compared to T < S and other LRS such as Linear LRS. This high learning rate might help the model find a better global optimum. Cosine LRS with T = S has a more thorough learning rate decay phase compared to Cosine LRS with T > S and Constant LRS. This learning rate decay may involve unique training dynamics that enable the model to find a better local optimum."

This decomposition — that Cosine LRS achieves its optimality through a combination of sustained high learning rate exploration followed by thorough decay — is the key insight that motivates the WSD scheduler. But prior to this paper, this decomposition was not explicitly recognized or studied. The paper identifies a set of limitations that existing LRS approaches share:

Predefined training length. Cosine LRS requires knowing the total training steps S in advance, since its schedule depends on when the cosine completes its first period. If you want to extend training (e.g., because you acquired more data or compute), you cannot simply continue — the learning rate would already be at its minimum, and extending requires redesigning the schedule or restarting. This is particularly problematic for the goal of "continuous training" where the model should be able to absorb arbitrarily more data.
Non-reusability of checkpoints. Because Cosine LRS is tightly coupled to the final step count, intermediate checkpoints are not optimized for their current training state. A checkpoint at step T/2 has a learning rate that is already well into the decay — far from the optimal learning rate for that stage of training. This means you cannot plausibly branch off from an intermediate checkpoint to explore "what if we trained with more data from here?" without anomalous behavior.
Opacity of training dynamics. Cosine LRS conflates two effects: the benefit of extended high-learning-rate exploration, and the benefit of learning rate decay for convergence. Because these occur in a single continuous curve, it is difficult to attribute performance improvements to one mechanism or the other. The paper argues that explicitly separating these phases (stable training and decay) makes the training dynamics interpretable and testable.
Inefficient scaling law measurement. This is perhaps the most practically consequential limitation. To measure scaling laws — how loss L varies with model size N and data size D — you need L(N, D) for many (N, D) pairs. With Cosine LRS, each pair requires training a model from scratch to exactly D tokens with the schedule tuned for that D. If you have m model sizes and want to test n data amounts, this costs O(m × n) full training runs. As the paper notes (Section 4.5), "if the average cost of training one model size on one data size is C, then conducting the scaling experiments with m model sizes and m data sizes takes approximately O(m²)C." For even modest m (say, 6 model sizes and 6 data amounts), this multiplies cost by 36× relative to a single training run — completely infeasible for large models.

How the Paper Positions Itself Relative to Prior Work

The paper's positioning is nuanced and multifaceted. It does not claim to introduce a radically new model architecture or a breakthrough in SLM capability. Rather, it presents itself as a systematic, empirically rigorous investigation of training strategies that:

1. Treats SLMs as a scientific instrument, not just a product. The "Model Wind Tunnel Experiments" (Section 3) frame is crucial: SLMs are the apparatus through which the paper discovers principles that should transfer to LLMs. This is a departure from prior SLM work that focused primarily on achieving strong benchmark results. MiniCPM does achieve strong results (beating Mistral-7B, Llama2-13B), but the paper's core contribution claim is about the methodology: "our training methodology is meticulously designed to facilitate seamless scaling of both model scale and data horizons" (Section 1).

2. Fills the gap between Cosine LRS practice and learning dynamics theory. The WSD scheduler is not proposed as an arbitrary alternative to Cosine — it is proposed as a decomposition of Cosine's two effective phases into explicit, independently controllable stages. The paper argues that Cosine works because it implicitly balances high-learning-rate exploration and learning-rate-decay convergence. WSD makes this balance explicit, enabling analysis that Cosine obscures. The paper positions this as a finding about training dynamics (Section 4.4 provides gradient analysis showing that "model parameters undergo consistent changes across steps" during decay, with curvature increasing by an order of magnitude) rather than as a schedule that merely achieves better loss.

3. Enables a new, more efficient paradigm for scaling law measurement. The most technically significant positioning is that WSD LRS makes scaling law measurement linear in model sizes rather than quadratic. Because WSD produces optimal checkpoints at any stable training duration (they just need a 10% decay afterward), you can train each model size once, record checkpoints at multiple steps during the stable phase, decay each checkpoint for 10% of its training, and thereby obtain L(N, D) for all (model, data) pairs at the cost of training each model size once plus the cost of short decays. The paper explicitly contrasts this with prior work: "Previously... conducting the scaling experiments with m model sizes and m data sizes takes approximately O(m²)C" (Section 4.5). With WSD, this becomes O(m)C plus a negligible decay cost.

4. Advocates for a higher data-to-model ratio than established scaling laws. The paper's scaling law measurement yields N_opt/D_opt at C=10²¹ of approximately 1/192, meaning the compute-optimal data size is 192 times the model size, compared to 20 times in Hoffmann et al. (2022). The paper does not claim this is a universal constant — it explicitly notes "there is a huge gap in compute optimal regime between ours and Hoffmann et al. (2022)" — but positions this finding as consistent with recent practice (Llama 2 is estimated at 70–100×) and as evidence that "our configuration can absorb more data into a smaller model compared to previous ones" (Section 4.5). This directly challenges the prevailing assumption that small models saturate quickly.

5. Positions SLMs as a viable deployment path with LLM-comparable capabilities. Against prior SLMs like Phi-2, which the paper acknowledges achieves "on par" academic performance, the authors claim MiniCPM's advantage is breadth: "Since our pre-training data covers more distribution, we think MiniCPM is better at knowledge and ability coverage" (Section 6.5). The paper demonstrates this through diverse evaluation (English, Chinese, code, math, reasoning, commonsense) and through qualitative examples in Appendix F showing translation, math with symbolic substitution, instruction following with acrostic poetry, and emoji understanding — tasks that require general-purpose competence rather than academic knowledge alone.

6. Acknowledges the validation gap explicitly. The paper is careful to note its own limitations. In the Limitations section, the authors state: "this paper does not extend to training an LLM to validate the scaling law. The application of WSD LRS on LLMs has not been fully explored to date." This is an important piece of positioning — the paper claims to provide methods and principles that should scale, but does not claim to have proven they do. The wind tunnel experiments are tests of the aircraft model, not the full-sized aircraft.

The Unaddressed Uncertainty in Data-Model Scaling

One of the paper's most compelling pieces of motivation appears in Section 4.5, where it frames the central empirical puzzle of scaling laws:

"Kaplan et al. (2020) posit that a tenfold increase in model scale should equate to a singlefold increase in data scale. Conversely, Hoffmann et al. (2022) argue for the same scaling rate between model size and data size. What's more, current models such as Llama 2 (Touvron et al., 2023), train much more data than what Hoffmann et al. (2022) claims, still yielding considerable performance gain. Indicating a higher data-to-model ratio."

This paragraph encapsulates the motivation. The three most influential works on scaling laws disagree with each other and with current practice. The disagreement is not over minor coefficients — it's over the fundamental shape of the relationship between model size, data size, and loss. The paper argues that this uncertainty persists precisely because the traditional approach to measuring scaling laws (training many models from scratch to many token counts) is too expensive, especially as models grow. WSD LRS is proposed not primarily as a training technique, but as a measurement instrument — a way to finally get reliable scaling law data without prohibitive cost.

This positions the paper's contribution as methodological and foundational, not merely competitive. The goal is not just to train good SLMs, but to provide the tools that will let the entire field understand how training works.

3. Technical Approach

3.1 Reader Orientation

This paper is primarily a systems and methodology paper that builds a principled pipeline for training small language models (SLMs) efficiently, with the explicit goal that the training strategies discovered at small scale should transfer predictably to larger models. The core idea is that by decomposing the learning rate schedule into an explicit stable training phase followed by a separate decay phase — what the authors call the Warmup-Stable-Decay (WSD) scheduler — you can reuse a single training run to measure scaling laws across many data sizes, optimize hyperparameters once at small scale, and then transfer both the hyperparameters and the data-to-model ratio insights to models of any size.

3.2 Big-Picture Architecture (Diagram in Words)

The MiniCPM system has five major interconnected components, organized as a pipeline from experimental design through final model deployment:

Model Wind Tunnel (MWT) — A systematic hyperparameter optimization framework run on tiny models (0.009B–0.5B parameters) that determines optimal learning rates, batch sizes, and initialization schemes using Tensor Program techniques to ensure these choices remain stable when scaling to larger models.
WSD Learning Rate Scheduler — A three-phase schedule (warmup, stable, decay) that explicitly separates high-learning-rate exploration from learning-rate-decay convergence, enabling continuous training without a predefined endpoint and making intermediate checkpoints reusable for scaling law measurement.
Scaling Law Measurement Apparatus — A protocol that leverages WSD's reusability property: train each of $m$ model sizes once during the stable phase, record checkpoints at multiple token counts, decay each checkpoint for 10% of its training, and fit the resulting $L(N, D)$ surface to extract the compute-optimal data-to-model ratio.
Two-Stage Pre-training Strategy — A data scheduling approach that uses only large-scale, coarse-quality pre-training data during the stable phase, then introduces diverse, high-quality knowledge-and-ability-oriented data (including SFT data) during the decay phase, followed by a final separate SFT phase.
MiniCPM Model Family — The concrete instantiations (1.2B, 2.4B parameter models) that aggregate all the discovered techniques, with extensions to DPO alignment, 128K long-context, and Mixture-of-Experts variants.

Information flows as follows: hyperparameters are optimized once via MWT on tiny models → those hyperparameters (learning rate ~0.01, initialization std, scaling factors) are transferred via Tensor Program rules to the target model size → the target model is trained with WSD LRS using only pre-training data during the stable phase → at a chosen checkpoint, the model enters the decay phase with a mixture of pre-training and high-quality SFT data → the decayed model undergoes a final separate SFT phase → optional alignment (DPO), context extension, or MoE expansion is applied.

3.3 Roadmap for the Deep Dive

First, the Model Wind Tunnel Experiments (Section 3) — how hyperparameters are optimized at small scale and why they transfer. This is the foundation that makes everything else reliable.
Second, the WSD Learning Rate Scheduler (Section 4.1–4.2) — what it is, how it decomposes Cosine LRS, and its functional form. This is the central mechanism that enables the paper's two key contributions: continuous training and efficient scaling law measurement.
Third, the experimental validation of WSD (Section 4.3) — the empirical findings that loss drops dramatically during decay, that 10% decay steps suffice, and that continuous training with WSD can match larger models.
Fourth, the analysis of the decay stage dynamics (Section 4.4) — what happens to gradients, parameter updates, and loss curvature during decay. This provides mechanistic insight into why WSD works.
Fifth, the scaling law measurement protocol (Section 4.5) — how WSD enables linear-cost scaling law measurement and what the resulting $N_{\text{opt}}/D_{\text{opt}}$ ratio implies. This is the most technically innovative use of WSD.
Sixth, the two-stage pre-training strategy (Section 5) — how data quality is scheduled across training phases and why introducing SFT data during decay outperforms adding it only during SFT.
Seventh, the MiniCPM model architecture and training details (Section 6) — the concrete model configurations, tokenizers, data mixtures, and training stages that instantiate the methodology.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a training methodology paper whose core technical contribution is the WSD learning rate scheduler and the accompanying protocol for efficient scaling law measurement. The model architecture itself (a standard dense Transformer with Group Query Attention and shared input-output embeddings) is intentionally conventional — the innovation is entirely in how it is trained, not in novel architectural components.

Model Wind Tunnel Experiments: Hyperparameter Optimization at Small Scale

The Model Wind Tunnel (MWT) is a systematic protocol for determining optimal hyperparameters using tiny models, based on the premise that "many aspects of model training are universal across scales" (Section 3). The wind tunnel metaphor is explicit: just as aircraft designers test scale models in wind tunnels before building full-sized planes, the authors test training configurations on models with 0.009B to 0.5B parameters before committing to the full MiniCPM training runs. The MWT comprises three sub-experiments: hyperparameter search and transfer, optimal batch size scaling, and optimal learning rate stability verification.

Hyperparameter Transfer via Tensor Program

The core challenge in hyperparameter optimization for LLMs is that you cannot afford to search hyperparameters independently for each model scale. The brute-force approach — training multiple model sizes with multiple hyperparameter settings — scales multiplicatively with the number of hyperparameters, model sizes, and data sizes, which is "not feasible for LLMs" and "even for SLM like MiniCPM, extensive experiments on hyper-parameters search take a lot of resources" (Section 3.1).

The solution is the Tensor Program framework (Yang et al., 2022; 2023), which provides a set of parameterization rules that ensure optimal hyperparameters remain stable across model scales. The specific operations applied are listed in Table 7 and include:

Embedding Output Scaling: Multiply the output of the embedding layer by a factor scale_emb. The embedding output, which is a $d_m$ -dimensional vector at each token position, is multiplied elementwise by this scalar before entering the first transformer block.
Residual Connection Scaling: Before adding a block's output back into the residual stream, scale the output tensor by scale_depth / sqrt(num_layers). This prevents the residual stream from being dominated by either the original input or the block output as depth increases, ensuring that deeper models maintain balanced signal propagation.
Initialization of Two-Dimensional Tensors: Set the initialization standard deviation of each two-dimensional parameter tensor (weight matrices in linear layers, attention projections) to init_std / sqrt(d_m / d_base), where $d_m$ is the model's hidden dimension and $d_{base}$ is a reference dimension (the hidden size of the base model used for hyperparameter search). Other parameters (biases, layer norms) are initialized to 0.1. This ensures that as the model width grows, weight matrices start with appropriately scaled variance.
Learning Rate Scaling of Two-Dimensional Tensors: Adjust the learning rate of each two-dimensional tensor parameter to 1 / (d_m / d_{base}) times the overall learning rate. This means weight matrices in wider models get smaller learning rates relative to biases and other parameters.
LM Head Scaling: Divide the output logits (pre-softmax scores over the vocabulary) by d_m / d_{base}. This prevents the logit scale from growing with model width, which would make the softmax distribution artificially peaked.

The authors note an important design choice: they do not apply the attention softmax temperature scaling recommended by Yang et al. (2022), which would divide attention logits by sqrt(d_h) to keep the pre-softmax variance constant. Instead, they rely on the standard 1/sqrt(d_h) scaling in the attention computation and verify empirically that this does not destabilize training.

The hyperparameter search itself is conducted as a Bayesian search over a predefined parametric space using a 0.009B model trained on |D| = 10N = 0.09B tokens (10× the model's parameter count in tokens). The authors confirm in pilot experiments that optimal hyperparameters found at 10N data scale are consistent with those found at 20N. The search space includes four key parameters:

scale_depth: tested across values 1, 2, 3, 4, 5, 6, 7, 8 (controls the residual scaling factor)
scale_emb: tested across values 2.5, 5.0, 7.5, 10.0, 12.5, 15.0, 17.5, 20.0 (controls embedding output scaling)
init_std: tested across values 0.025, 0.050, 0.075, 0.100, 0.125, 0.150, 0.175, 0.200 (controls initialization variance)
lr (base learning rate): tested across values 0.00, 0.02, 0.04, 0.06, 0.08, 0.10

The results are visualized in Figure 14, which compares configurations with and without QK-Norm (Query-Key Normalization; Henry et al., 2020) and decoupled weight decay (Loshchilov & Hutter, 2017). The optimal configuration is identified as: scale_depth = 1.4, scale_emb = 12, init_std = 0.1, and lr = 0.01. The authors observe that adding QK-Norm "significantly decreases the learning rate sensitivity," which is consistent with Wortsman et al. (2023), but choose not to use it because "as the MiniCPM project itself is an SLM, we do not require low learning rate sensitivity as long as we find the best learning rate with TensorProgram." This is a pragmatic design choice: stability through QK-Norm is most valuable when hyperparameter search is expensive, but since MWT makes hyperparameter search cheap (it's run on tiny models), the extra normalization layer is unnecessary overhead.

Optimal Batch Size as a Function of Loss

The batch size determines the balance between gradient noise (which helps exploration) and computational efficiency (larger batches use hardware more effectively). The paper's approach to finding the optimal batch size differs from Kaplan et al. (2020) in a subtly important way that the authors explain in Appendix A.2.

In Kaplan et al. (2020), the assumption is that GPUs are unlimited — if you double the batch size, the per-step computation time doubles, but you can simply use twice as many GPUs to keep wall-clock time constant. Under this assumption, the optimization goal is to minimize total training time, which means trading off between steps (prefer fewer steps with larger batches) and tokens (prefer fewer total tokens with smaller batches that give better gradient estimates). This leads to the concept of a "critical batch size."

In MiniCPM's setting, the constraint is different: GPUs are fixed. Doubling the batch size means doubling the per-step wall-clock time (since you're processing twice as many tokens on the same hardware). Therefore, the total training wall-clock time is proportional to total tokens consumed, not total steps. The optimization goal shifts entirely to "minimizing the token quantity to achieve the lowest loss" (Appendix A.2). This changes what "optimal batch size" means: it is now the batch size that achieves the lowest loss for a fixed total token budget, independent of step count.

The experimental protocol: train 0.009B, 0.03B, and 0.17B models, each with six different batch sizes, using a learning rate of 0.01 and Cosine LRS. The results are plotted in Figure 1, where:

The x-axis is batch size
The y-axis is token consumption
The color of each point represents loss (lighter = higher loss)

Each vertical line of colored points represents one training curve at a fixed batch size. The authors fit parabolas to the equal-loss contours (horizontal slices across batch sizes) and connect the minima of these parabolas with a red line, which shows how the optimal batch size shifts as loss decreases. From the three red lines (one per model size), they observe that the lines "connect each other well into a linear relationship in the log space," yielding the fitted relationship:

$bs = \frac{1.21 \times 10^9}{L^{6.24}}$

where $bs$ is the optimal batch size and $L$ is the C4 loss.

What it computes: Given a target loss value $L$ on C4, this formula estimates the batch size (in tokens) that will achieve that loss with minimal total token consumption. For example, if you expect to reach a loss of 3.0, the optimal batch size would be approximately $1.21 \times 10^9 / 3.0^{6.24}$ tokens.

Why this form: The power-law relationship $bs \propto L^{-\gamma}$ with $\gamma \approx 6.24$ means that as loss decreases (model improves), the optimal batch size grows rapidly — you need much larger batches to make efficient progress at lower loss values. This is consistent with the intuition that as the model approaches convergence, gradient noise becomes less helpful and you want more accurate gradient estimates. The authors note the apparent circularity (you need to know the loss to set the batch size, but you only know the loss after training), acknowledging this as a "Chicken-and-egg" paradox that is resolved in practice by using prior knowledge from preliminary experiments to estimate the achievable loss for a given model size.

An important methodological note from Appendix A.2: the optimal batch size and optimal learning rate are likely correlated (larger batches may tolerate or require larger learning rates). To handle this coupling, the authors use a coordinate descent approach: fix a preliminary learning rate, optimize batch size, then re-optimize learning rate using the new batch size. They acknowledge this is not fully rigorous and invite "more rigorous methods" in future work.

Learning Rate Stability Across Scales

After applying Tensor Program scaling rules, the paper verifies that the optimal learning rate remains stable as model size increases. The experiment: train models at 0.04B, 0.1B, 0.3B, and 0.5B parameters, each with six different learning rates, and measure C4 loss. The results in Figure 3 show that "although the model size has increased by ten times, the optimal base learning rate does not show a noticeable shift and remains around 0.01." A further validation at 2.1B confirms that learning rate 0.01 still achieves the lowest loss.

This stability is the payoff of the Tensor Program approach: hyperparameters discovered at 0.009B transfer directly to models 200× larger, making the wind tunnel metaphor operational. Without Tensor Program, the optimal learning rate might shift with model size, requiring expensive re-tuning at each scale and undermining the validity of small-scale experiments.

The WSD Learning Rate Scheduler: Decomposing Cosine LRS

The WSD scheduler is the paper's central technical contribution. To understand why it is designed the way it is, we must first understand what the authors discovered about Cosine LRS through their own reproduction experiments (Section 4.1).

What the Paper Finds About Cosine LRS

The authors train a 0.036B model under several Cosine LRS variants to isolate why Cosine works well. They test:

Cosine(T): Standard Cosine LRS where the cosine completes its first period at step $T$ and the learning rate stays at its minimum (10% of max) thereafter. The total training steps $S$ is varied across experiments: 20N, 40N, 60N, 80N (where $N$ is the model's parameter count in tokens, so 20N means 20× the parameter count in training tokens).
CosineLoop(T): Cosine with warm restarts — after completing one period at step $T$ , the learning rate resets to the maximum and begins a new cosine cycle.

The functional form of Cosine(T) at step $s$ is:

$\text{Cosine}(T; s) = \begin{cases} \frac{s}{W}\eta, & s < W \\ 0.9\eta\cos\left(\pi\frac{s}{T}\right) + 0.1\eta, & W < s < T \\ 0.1\eta, & s > T \end{cases}$

where $W$ is the warmup end step (warmup is linear from 0 to $\eta$ ), $T$ is the period length, and $\eta$ is the maximum learning rate.

What it computes: During warmup (first $W$ steps), the learning rate increases linearly from 0 to $\eta$ . After warmup and before step $T$ , it follows a cosine curve that starts near $\eta$ (specifically, $0.9\eta + 0.1\eta = \eta$ at $s = W$ ) and decays to $0.1\eta$ at $s = T$ . After step $T$ , it stays constant at $0.1\eta$ . The $0.9$ coefficient controls the amplitude of the cosine wave, and the $0.1\eta$ floor prevents the learning rate from ever reaching zero.

Why this form: The $0.1\eta$ floor is critical — if the learning rate decayed to exactly zero, training would effectively stop. The floor ensures a small but non-zero learning rate maintains some plasticity. The $0.9$ amplitude means the learning rate at the peak of the cosine (near $s = W$ ) is approximately $\eta$ , while at the trough (near $s = T$ ) it is $0.1\eta$ .

The key finding from Figure 4: for any total training steps $S$ , the lowest loss is always achieved when $T = S$ — that is, when the cosine period exactly matches the total training duration. Both $T < S$ (decay finishes early, model trains at minimum LR for remaining steps) and $T > S$ (decay is incomplete at training end) are suboptimal.

This observation leads the authors to hypothesize that Cosine LRS works because of two complementary mechanisms:

Extended high-learning-rate exploration: Cosine with $T = S$ maintains a high learning rate for a substantial fraction of training, enabling the model to explore the loss landscape and potentially find better basins of attraction. This is why $T < S$ underperforms — it cuts the exploration phase short.
Thorough learning rate decay: Cosine with $T = S$ provides a complete decay from $\eta$ to $0.1\eta$ , which the authors hypothesize "may involve unique training dynamics that enable the model to find a better local optimum." This is why $T > S$ underperforms — the decay is incomplete at the end of training, leaving the model in a suboptimal local region.

CosineLoop(T) is included to test whether the decay phase itself is necessary, or whether simply maintaining high learning rate with periodic resets would suffice. Figure 4 shows CosineLoop underperforms Cosine with $T = S$ , confirming that the decay phase is not merely about reducing the learning rate — it involves dynamics that produce better optima than constant-high-learning-rate exploration would.

The WSD Scheduler: Explicit Separation of Stable and Decay Phases

Given the insight that Cosine works by combining sustained high-LR exploration with thorough LR-decay convergence, the WSD scheduler makes this separation explicit. Instead of one continuous cosine curve that implicitly blends both phases, WSD has three distinct stages:

$\text{WSD}(T; s) = \begin{cases} \frac{s}{W}\eta, & s < W \\ \eta, & W < s < T \\ f(s - T)\eta, & T < s < S \end{cases}$

where:

$W$ is the end step of the warmup stage (linear increase from 0 to $\eta$ )
$T$ is the end step of the stable training stage (constant learning rate $\eta$ )
$S$ is the total training steps
$f(s - T)$ is a decreasing function with $0 < f(s - T) \leq 1$ that controls the decay shape
$\eta$ is the maximum (and stable-stage) learning rate

What it computes: During warmup ( $s < W$ ), the learning rate increases linearly from 0 to $\eta$ . During the stable stage ( $W < s < T$ ), the learning rate is held constant at $\eta$ — exactly the maximum, with no decay whatsoever. During the decay stage ( $T < s < S$ ), the learning rate follows the decreasing function $f$ , starting from $f(0)\eta = \eta$ (at $s = T$ ) and decreasing to some minimum at $s = S$ . The specific form of $f$ is not specified in the definition — it could be exponential, linear, cosine, or any other decreasing function.

Why this form: The explicit separation into stages serves multiple purposes that Cosine LRS cannot achieve:

Independence from total training length. The stable stage length $T$ is decoupled from the total training length $S$ . You can decide to train longer (increase $S$ ) without affecting the stable stage — you just extend the stable phase or add more decay. With Cosine LRS, extending training requires redesigning the entire schedule.
Reusability of stable-stage checkpoints. Any checkpoint saved during the stable stage (at any step $T' < T$ ) can be decayed independently to produce an optimally converged model for that training duration. The checkpoint has been trained entirely at the optimal learning rate $\eta$ , so it represents the best state achievable with $T'$ tokens of stable training. This is the property that enables efficient scaling law measurement.
Independent control of stable and decay durations. The ratio of stable to decay tokens can be tuned separately. The paper's experiments (Section 4.3) show that decay of approximately 10% of total tokens is sufficient — meaning you can spend 90% of tokens in the stable phase (at constant high LR) and only 10% in decay.
Explicit hypothesis testing. By making the two phases explicit, WSD allows direct empirical investigation of what each phase contributes. For example, you can vary the stable duration while keeping decay fixed, or vary the decay shape while keeping stable training identical.

The authors note that "as long as the warmup stage is enough, it affects little performance" (Section 4.2), so the warmup stage is treated as a fixed overhead not relevant to the scheduler's key properties. In subsequent discussion, $W$ is omitted and WSD is referred to by its stable and decay durations: WSD(T, D) where $D$ is the number of decay steps. For example, WSD(40N, 4N) means stable training for 40N tokens followed by 4N tokens of decay (10% of stable).

The Specific Decay Function: Exponential Annealing

For the actual MiniCPM training runs, the decay function $f$ is specified as exponential:

$f(s - T) = 0.5^{(s - T) / T_d}$

where $T_d$ is the decay duration (5000 steps = 20B tokens for MiniCPM).

What it computes: At each decay step $s$ , the learning rate is multiplied by $0.5$ raised to the power of $(s - T) / T_d$ . On the first decay step ( $s = T$ ), this gives $0.5^0 = 1$ , so the learning rate starts at $\eta$ . After $T_d$ steps ( $s = T + T_d$ ), this gives $0.5^1 = 0.5$ , so the learning rate has halved. After $2T_d$ steps, it would be $0.5^2 = 0.25$ , and so on.

Why this form: Exponential decay has the property that the learning rate decreases by a constant factor per fixed number of steps, meaning the rate of decrease is proportional to the current learning rate. This contrasts with linear decay (constant absolute decrease per step) or cosine decay (decrease rate varies sinusoidally). Exponential decay means the model makes relatively large absolute changes early in decay (when the LR is high) and increasingly fine-grained adjustments later (when the LR is low). The authors observe in Figure 12 that "the loss still drops after the learning rate drops below 10% of the max learning rate," indicating that the exponential schedule continues to provide useful gradient steps even at very low learning rates.

Experimental Validation of WSD: Loss Drop, Decay Duration, and Continuous Training

The paper validates WSD through three key experiments on 0.036B models, all reported in Section 4.3.

Finding 1: Loss Decreases Dramatically in the Decay Stage

In Figure 5, the authors plot training loss for WSD variants with different stable durations (40N, 60N, 80N tokens) and decay durations (2N, 4N, 6N, 8N tokens), compared against Cosine(80N). The result is striking: at the onset of decay, the loss curve shows a sharp, rapid decrease that quickly reaches the same loss as Cosine LRS trained for the same total token count.

For instance, WSD(40N, 4N) — which trains for 40N tokens at constant high LR, then decays for 4N tokens — achieves roughly the same final loss as Cosine(40N), which trains for 40N tokens with a cosine schedule. Similarly, WSD(40N, 2N) falls short (2N steps of decay is insufficient), while WSD(40N, 4N) achieves parity.

This validates the central hypothesis: the stable and decay phases can be explicitly separated without loss of performance. A model trained at constant high LR for $T$ tokens, then decayed, reaches the same loss as a model trained with Cosine LRS for $T$ total tokens. This means you can "reuse the model before decay and continue training with the previous high learning rate" — if you later decide you want more training, you can simply extend the stable phase from the pre-decay checkpoint, then decay again.

Finding 2: 10% Steps Are Enough for Decay

Also shown in Figure 5: comparing WSD(40N, 4N) vs WSD(40N, 2N), WSD(60N, 6N) vs WSD(60N, 2N), and WSD(80N, 8N) vs WSD(80N, 2N). The 10% decay variants (4N/40N, 6N/60N, 8N/80N) consistently achieve the best results, while the 2.5% decay variants (2N steps for all) consistently fall short. The conclusion: "having a decay of 10% of the total tokens is sufficient to achieve the best results, while a decay of 2.5% of total tokens falls short."

This is a practically crucial finding because it means the overhead of the decay phase is small. If you train a model for 1T tokens in the stable phase, you need only ~100B tokens of decay to reach optimal convergence. This keeps the cost of re-decaying multiple checkpoints (for scaling law measurement) manageable.

Finding 3: Continuous Training Can Match Larger Models

Figure 6 demonstrates the most provocative implication of WSD: a fixed-size model can be trained far beyond what was previously considered its saturation point, and can eventually match the performance of a larger model trained with fewer tokens. The experiment:

Train a 0.036B model with WSD for increasing stable durations: 20N, 40N, 60N, 80N, 160N, 320N, each with 10% decay (2N, 4N, 6N, 8N, 16N, 32N respectively).
Also train a 0.17B model (roughly 4.7× larger) with WSD(40N, 4N).
Fit the optimal loss envelope — the loss achieved by WSD(D, 0.1D) for each $D$ — as a function of compute $C$ .

The green lines in Figure 6 show that the 0.036B model's loss continues to decrease even at 320N training tokens, far beyond what Chinchilla Optimal would prescribe (the compute-optimal data for a 0.036B model would be much less than 320N if model size and data were scaled in equal proportion). The pink point shows the 0.17B model at 40N tokens. The crucial finding: "a 0.036B model can match the performance of a 0.17B model with an acceptable increase (~4 times) in training compute while saving a lot of inference computation (saving ~5 times per inference call)."

This establishes a compute-optimal tradeoff between training and inference (building on Sardana & Frankle, 2023): if inference cost dominates your total compute budget (e.g., you serve the model millions of times), it can be optimal to train a much smaller model on far more data than Chinchilla Optimal would suggest, because the inference savings outweigh the extra training cost.

The fitted loss envelope uses two candidate functional forms:

Exponential: $L(C) = \alpha e^{-\beta C} + L_0$
Power-law: $L(C) = \beta C^{-\alpha} + L_0$

The power-law form fits better, consistent with Kaplan et al. (2020), suggesting that even with WSD's explicit separation of phases, the underlying loss-compute relationship retains the power-law character observed in prior scaling law studies.

Analysis of the Decay Stage: Gradient Dynamics and Loss Curvature

Section 4.4 provides mechanistic insight into why the decay stage produces such dramatic loss reduction despite involving much smaller parameter updates than the stable stage. The analysis is conducted on a 0.2B model, recording gradient information at every step.

Parameter Updates During Decay

Figure 7 plots the maximum element-wise difference between consecutive checkpoints for two specific weight matrices (gate_proj and q_proj of layer 25 in MiniCPM-2.4B). The key observation: "the updates exhibit a robust correlation with the learning rate's magnitude." During the stable stage (high LR), weight matrices change substantially between checkpoints, but the loss decreases slowly. During the decay stage (low LR), weight changes are much smaller in magnitude, yet the loss drops rapidly.

The authors explicitly flag this as non-obvious: "This observation may not be trivial: the model checkpoints experience significant updates preceding the learning rate's decay, yet the loss exhibits minimal reduction. Conversely, during the decay stage, despite less pronounced weight alterations, there is an accelerated decrease in loss."

This suggests that the high-LR stable stage is exploring the loss landscape (making large moves that may not immediately reduce loss but help find better basins), while the decay stage is exploiting — making small, precise adjustments that rapidly descend into local optima. This is consistent with the exploration-exploitation framing from Section 4.1.

Gradient Statistics Over Training

Figure 8 presents a richer view of the optimization dynamics through six statistics computed at each training step:

Gradient norm $\|g^{(t)}\|$ (L2 norm of the gradient vector): diminishes during the decay phase. This is expected — as the model approaches a local optimum, gradients become smaller. The magnitude of decrease correlates with the learning rate decay.

Cosine of the gradient's angle between consecutive steps: $\frac{g^{(t+1)} \cdot g^{(t)}}{\|g^{(t+1)}\| \|g^{(t)}\|}$ . Upon commencement of the decay stage, "the cosine between gradients predominantly assumes positive values." This means that in the decay phase, gradient directions are consistent from step to step — the model is moving coherently in parameter space rather than bouncing around. In the stable phase (high LR), the cosine is more variable, consistent with exploration that involves larger, less correlated moves.

First-order directional derivative along the parameter trajectory: $D_1 = \frac{g^{(t+1)} \cdot v^{(t)}}{\|v^{(t)}\|}$ , where $v^{(t)} = x^{(t+1)} - x^{(t)}$ is the parameter update vector. This measures how much the loss function is changing along the direction of the parameter update. The negative of this quantity (so positive values mean loss is decreasing) "diminishes exponentially with each step, aligning closely with the learning rate." This is expected: as the learning rate decreases, the update vector shrinks, and the first-order change in loss scales proportionally.

Second-order directional derivative along the parameter trajectory: $D_2 = \frac{(g^{(t+1)} - g^{(t)}) \cdot v^{(t)}}{\|v^{(t)}\|^2}$ . This measures how the gradient changes along the update direction — essentially the curvature of the loss function along the optimization trajectory. The paper finds that "the second-order directional derivative exhibits a slight increase in magnitude" during decay. This means the loss landscape becomes more curved (sharper minima) as the model converges.

Loss curvature along the trajectory: $K = \frac{|D_2|}{(1 + D_1^2)^{3/2}}$ . This is the standard formula for curvature of a parametric curve, applied to the optimization trajectory. The paper reports that "the curvature of the loss function also increases by a magnitude, indicating the proximity to a local optimum." This makes geometric sense: as you descend into a valley in the loss landscape, the path curves more sharply.

Maximum gradient (the largest absolute gradient element): similar behavior to gradient norm, decreasing during decay.

Taken together, these statistics paint a picture of the decay stage as a period of consistent, low-noise convergence toward a sharp local minimum. The high-LR stable stage explores broadly (large parameter changes, less correlated gradient directions, lower curvature), while the decay stage exploits locally (small consistent parameter changes, high gradient correlation, increasing curvature). The paper frames this as offering "a deeper insight into the shape optimization space, a subject reserved for future exploration."

Measuring the Scaling Law with WSD LRS: Linear-Cost Protocol

Section 4.5 presents what is arguably the paper's most technically innovative contribution: using WSD LRS to measure the data-model scaling law with linear rather than quadratic cost in the number of model sizes.

The Traditional Scaling Law Measurement Problem

The standard approach to measuring $L(N, D)$ — how loss depends on model size $N$ and data size $D$ — requires training each (model size, data size) pair from scratch. For $m$ model sizes and $m$ data sizes, this costs $O(m^2)$ full training runs. With Cosine LRS, you cannot reuse intermediate checkpoints because they are not trained at the optimal learning rate for their current data amount — a checkpoint at step $T/2$ in a Cosine(T) schedule has a learning rate already well into the decay phase, far from optimal for a model that has seen only $T/2$ tokens.

The authors quantify this: "Previously, if the average cost of training one model size on one data size is C, then conducting the scaling experiments with m model sizes and m data sizes takes approximately O(m²)C" (Section 4.5). For m = 6, this is 36× the cost of a single training run — prohibitive for large models.

The WSD-Based Protocol

WSD changes this because any stable-stage checkpoint can be independently decayed to its optimal convergence, regardless of how long the stable stage has run. The protocol:

Train $m$ models of different sizes (0.04B, 0.1B, 0.3B, 0.5B, 0.9B, 2.0B parameters) during the stable phase only, with WSD LRS. Save checkpoints at multiple token counts (10N, 20N, 30N, 40N, 50N, 60N — corresponding to 10×, 20×, ..., 60× the model parameter count).
Decay each checkpoint independently. From each saved checkpoint, run a 10% decay phase. This produces $m$ (model sizes) × 6 (checkpoint counts) = $6m$ converged models.
Evaluate losses on five held-out datasets: Code, English (Wikihow), Chinese (Wikihow), Ultratext, Chinese (Yayi Corpus), plus an average across all five.

The total cost: $m$ full stable-phase training runs, plus $6m$ short decay runs (each 10% of its stable duration). Since the decay runs are short relative to full training, the total cost is approximately $O(m)$ , not $O(m^2)$ .

The authors note a practical consideration: "To potentially compare the loss when the model uses different tokenizers, we take the average of loss by a number of bytes instead of a number of tokens, following Achiam et al. (2023)." This ensures that models with different tokenizers (which would produce different numbers of tokens for the same text) are compared on equal footing — the loss per byte is tokenizer-independent.

The Fitted Scaling Law

The losses $L(N, D)$ for each (model size, data size) pair are fit to the standard parametric form from Hoffmann et al. (2022):

$L(N, D) = C_N N^{-\alpha} + C_D D^{-\beta} + L_0$

where:

$N$ is the number of non-embedding parameters
$D$ is the number of training tokens
$C_N$ and $C_D$ are coefficients controlling the magnitude of the model-size and data-size contributions
$\alpha$ and $\beta$ are exponents controlling how loss scales with model size and data size
$L_0$ is the irreducible loss (the Bayes error or entropy limit of the data distribution)

What it computes: Given model parameters $N$ and data tokens $D$ , this formula predicts the expected loss. The first term $C_N N^{-\alpha}$ captures how loss decreases as model size grows (with exponent $\alpha$ ), the second term $C_D D^{-\beta}$ captures how loss decreases as data grows (with exponent $\beta$ ), and $L_0$ represents the floor — the loss that would remain even with infinite model size and infinite data.

Why this form: This decomposition separates the effects of model capacity and data quantity, allowing analysis of their relative importance. The power-law form is motivated by extensive empirical evidence from Kaplan et al. (2020) and Hoffmann et al. (2022) that both model scaling and data scaling follow power-law relationships. The additive combination assumes the two sources of error (limited model capacity, limited data) are approximately independent.

Deriving the Compute-Optimal Ratio

Given a fixed compute budget $C = 6ND$ (following the FLOPs approximation from Rae et al., 2021, where each training token costs approximately 6 FLOPs per parameter for the forward and backward passes), the optimal allocation between $N$ and $D$ is found by substituting $D = C / (6N)$ into the loss formula and minimizing with respect to $N$ . This yields:

$\frac{N_{\text{opt}}}{D_{\text{opt}}} = K^2 \left(\frac{C}{6}\right)^\eta$

where:

$K = \left(\frac{\alpha C_N}{\beta C_D}\right)^{\frac{1}{\alpha + \beta}}$ is a constant determined by the fitted coefficients and exponents
$\eta = \frac{\beta - \alpha}{\alpha + \beta}$ determines how the ratio changes with compute $C$

What it computes: For a given total compute budget $C$ (in FLOPs), this formula gives the ratio of optimal model size to optimal data size. If $\eta > 0$ (i.e., $\beta > \alpha$ ), the ratio grows with compute — you should emphasize data scaling more as compute increases. If $\eta < 0$ (i.e., $\alpha > \beta$ ), you should emphasize model scaling more. If $\eta = 0$ (i.e., $\alpha = \beta$ ), the ratio is constant independent of compute.

Why this form: The derivation follows directly from calculus — set the derivative of $L(N, C/(6N))$ with respect to $N$ to zero and solve. The exponent $\eta$ emerges naturally as the difference between the two scaling exponents normalized by their sum. This connects the qualitative behavior of the optimal ratio (increasing, decreasing, or constant with compute) to the relative magnitudes of the empirical scaling exponents.

The Fitted Results

The fitted coefficients and exponents are shown in the first text box of each subplot in Figure 10. Averaging across all five evaluation corpora:

$\alpha = 0.29$ (model size exponent)
$\beta = 0.23$ (data size exponent)
$K^2 = 0.01$
$\eta = -0.10$

Since $\eta = -0.10 < 0$ , the optimal data-to-model ratio actually decreases slightly with compute, meaning data scaling becomes marginally less important relative to model scaling as total compute grows. However, the absolute value of $\eta$ is small, so the ratio is nearly constant.

The concrete ratio at $C = 10^{21}$ FLOPs: $D_{\text{opt}} / N_{\text{opt}} \approx 192$ on average (from the "Avg" subplot in Figure 10). This means the compute-optimal data size is approximately 192 times the model size — e.g., a 1B-parameter model should be trained on roughly 192B tokens. This is dramatically larger than Chinchilla Optimal's ~20× ratio.

The individual datasets show variation: Code gives $D_{\text{opt}}/N_{\text{opt}} \approx 195$ , English Wikihow gives ~166, Chinese Wikihow gives ~834, Ultratext gives ~96, and Chinese Yayi Corpus gives ~113. The high ratio for Chinese Wikihow suggests that Chinese text data is particularly beneficial to continue training on (perhaps because it contains more information per token, or because the model starts with weaker Chinese capabilities and thus benefits more from additional Chinese data).

The paper explicitly addresses the large deviation from Chinchilla Optimal:

"with respect to the large deviation from Chinchilla Optimal... we notice that their scaling experiment was conducted in a not very recent configuration."

To validate against more recent practice, the authors perform an approximate analysis of Llama 2's training curves (Appendix B.4, Figure 18). From the published loss curves, they estimate that the 13B model's loss curve would intersect the 34B model's at approximately $10^5$ EFlops, and the 34B would intersect the 70B at approximately $5 \times 10^5$ EFlops. This gives an estimated $D_{\text{opt}}/N_{\text{opt}}$ of roughly 70–100, which is much closer to MiniCPM's ~192 than to Chinchilla's ~20. The authors note that this estimate is "only a rough one" but it supports the conclusion that modern training configurations (better data, better hyperparameters, better architectures) can productively absorb more data per parameter than earlier scaling law experiments suggested.

The Contour Plots (Figure 10)

Figure 10 visualizes the fitted $L(N, D)$ surface as contour plots of equal loss in the (N, D) plane. Each subplot corresponds to one evaluation dataset. The x-axis is non-embedding parameters (log scale, ranging from ~0.03B to ~3.3B), and the y-axis is compute in $10^{18}$ FLOPs (log scale, ranging from ~ $10^{17}$ to ~ $10^{21}$ ). The diagonal lines represent constant compute contours (since $C = 6ND$ , constant compute means $N$ and $D$ trade off along lines of slope −1 in log-log space). The black dots show actual experimental measurements, and the colored contours show the fitted loss values. The text boxes display the fitted equation and the compute-optimal ratio at $C = 10^{21}$ .

Two-Stage Pre-training Strategy: Data Quality Scheduling

Section 5 introduces a data scheduling strategy that exploits WSD's separation of training phases. The core hypothesis is that the decay stage is the right time to introduce high-quality, task-oriented data, for two reasons:

During the decay stage, the learning rate is low, so the model makes fine-grained adjustments. Introducing high-quality data at this point "fosters a more comprehensive model learning" and "facilitates a more pronounced loss reduction in relation to the SFT data distribution, rather than the pre-training data distribution" — meaning the model converges toward being good at the tasks you actually care about, not just at predicting arbitrary web text.
If high-quality data were introduced uniformly throughout pre-training, it would need to be repeated many times (since high-quality datasets are typically small relative to pre-training corpora). This "could lead to negative effects" from overfitting or memorization. By concentrating the high-quality data in the brief decay phase, it is seen only a few times.

The proposed strategy has two stages:

Stable stage: Use only large-scale, coarse-quality pre-training data. This data is "abundant and can support continuous training when provided with more computational resources." There is no fixed endpoint — you can keep training as long as you have compute and data.

Decay stage: Mix diverse, high-quality knowledge-and-ability-oriented SFT data into the pre-training data. This includes datasets like UltraChat, SlimOrca, OssInstruct, EvolInstruct, and proprietary data (LeetCode questions, K12 textbooks and questions). The mixture proportions are shown in Figure 11 (right panel): Code Pretrain 19.6%, Dolma 15.7%, CommonCrawl Chinese 14.6%, C4 9.5%, Wikipedia 6.7%, Pile 5.1%, with ~4.8% SFT-mixed data and smaller contributions from specialized sources (book Chinese, code SFT, knowledge SFT, math synthetic, etc.).

SFT stage (separate): After decay, conduct a final SFT phase using only high-quality labeled data (no pre-training data), trained with approximately 6B tokens for the 1.2B model and 4B tokens for the 2.4B model.

The ablation study (Table 1) validates this strategy:

A-1: 2.4B model, decay with only pre-training data, then 4B token SFT.
A-2: 2.4B model, decay with pre-training + SFT data mixture, then 4B token SFT.
B-1: 1.2B model, decay with only pre-training data, then 6B token SFT.
B-2: 1.2B model, decay with only pre-training data, then 12B token SFT.
B-3: 1.2B model, decay with pre-training + SFT data mixture, then 6B token SFT.

The results show that adding SFT data during decay (A-2 vs A-1, B-3 vs B-1) substantially improves benchmark performance. For the 2.4B model: C-Eval jumps from 40.0 to 52.6, CMMLU from 41.5 to 51.1, MMLU from 44.6 to 50.9, GSM8K from 27.7 to 42.3. For the 1.2B model: C-Eval jumps from 40.9 to 49.1, CMMLU from 41.5 to 46.8.

The comparison between B-2 (double SFT tokens, no decay mixing) and B-3 (standard SFT tokens, decay mixing) shows that simply adding more SFT data in the SFT phase (B-2) does not compensate for not mixing it during decay. B-2 achieves 41.2 C-Eval vs B-3's 49.1, despite having twice the SFT tokens. The authors conclude: "the benefits of introducing high-quality data at the beginning of the decay stage are much higher than simply adding it during the SFT phase." This is a remarkably strong statement — it implies that when you introduce high-quality data matters more than how much you use, at least within the ranges tested.

The paper recommends that "specialization and enhancement of model capabilities should start from the decay phase," positioning the decay stage not merely as a convergence mechanism but as the primary window for capability injection.

MiniCPM Model Architecture and Training Configuration

Section 6 specifies the concrete models that instantiate the methodology. The architecture is intentionally conventional — a standard dense Transformer decoder — reflecting the paper's focus on training methodology rather than architectural innovation.

Model Configurations

The two base models are specified in Table 2:

MiniCPM-2.4B:

Non-embedding parameters: 2,442,057,984 (~2.4B)
Hidden dimension $d_m = 2304$
Feedforward bottleneck dimension $d_{ff} = 5760$
Attention head dimension $d_h = 64$
Number of query heads $n_q = 36$
Number of key/value heads $n_{kv} = 36$ (standard multi-head attention, no GQA)
Number of layers $L = 40$
Training batch size: 4M tokens
Total training tokens: ~1.1T

MiniCPM-1.2B:

Non-embedding parameters: 1,247,442,432 (~1.2B)
Hidden dimension $d_m = 1536$
Feedforward bottleneck dimension $d_{ff} = 3840$
Attention head dimension $d_h = 64$
Number of query heads $n_q = 24$
Number of key/value heads $n_{kv} = 8$ (Group Query Attention with 3 query heads per KV head)
Number of layers $L = 52$
Training batch size: 2M tokens (increased to 4M during training)
Total training tokens: ~1.1T

Design Choices and Their Justifications

Deep-and-thin architecture. The MiniCPM-2.4B uses 40 layers, which is deeper than comparable models like Phi-2 (32 layers). For MiniCPM-1.2B, this principle is pushed further to 52 layers while reducing width ( $d_m = 1536$ vs 2304). The paper notes this aligns with Liu et al. (2024)'s finding that "deep and thin networks" are effective for SLMs. The intuition: for a fixed parameter budget, more layers provide more sequential processing depth, which may be more valuable for complex reasoning than wider layers that provide more parallel representation capacity.

Shared input-output layer (weight tying). For both models, the input embedding matrix and output projection (LM head) matrix share parameters. The paper explains: "For SLM, the embedding takes up a large parameter space. To make the model parameters smaller, we use the Embedding Sharing techniques." With vocabularies of 122,753 and 73,440 tokens respectively and hidden dimensions of 2304 and 1536, the embedding matrices would consume $122753 \times 2304 \approx 283$ million parameters (2.4B model) and $73440 \times 1536 \approx 113$ million parameters (1.2B model) — substantial fractions of the total parameter budget. Sharing input and output embeddings halves this cost.

Group Query Attention for the smaller model only. MiniCPM-2.4B uses standard multi-head attention (36 query heads, 36 KV heads). MiniCPM-1.2B uses Group Query Attention (Ainslie et al., 2023) with 24 query heads but only 8 KV heads, "inspired by Liu et al. (2024), to further reduce the parameters number." GQA reduces the KV-cache size during inference by a factor of 3 (24/8 = 3), which is particularly valuable for the 1.2B model that targets edge deployment where memory is constrained.

Vocabulary sizes. MiniCPM-2.4B uses a 122,753-token vocabulary (MiniCPMTokenizer-120K), trained with Byte Pair Encoding (BPE; Sennrich et al., 2016) on "extensive and diverse language data" including "traditional Chinese characters, rare characters, emojis, and special symbols such as Greek letters, Cyrillic letters, etc." MiniCPM-1.2B uses a smaller 73,440-token vocabulary (MiniCPMTokenizer-70K) because "for the SLM, the embedding parameters will take up a lot of parameter space if the vocabulary is large." The smaller vocabulary is created by re-training BPE on the same documents but with a maximum vocabulary size of 64,000, plus special characters but excluding rare Chinese characters.

The compression ratio comparison (Table 9) shows that MiniCPMTokenizer-120K achieves 3.73 bytes/token for Chinese and 4.14 bytes/token for English — slightly better than Baichuan2 and ChatGLM2's tokenizers. The smaller MiniCPMTokenizer-70K achieves 3.56 bytes/token for Chinese and 4.02 for English — a modest decrease in compression efficiency traded for significant parameter savings.

Including embedding parameters. The paper clarifies: "Including the embedding parameters increases total parameters by 0.3B and 0.2B respectively." Since embeddings are shared, the embedding parameter count is vocab_size × d_m, which is 122753 × 2304 ≈ 283M for the 2.4B model and 73440 × 1536 ≈ 113M for the 1.2B model. The difference between these numbers and the stated 0.3B/0.2B is because the embedding sharing means only one copy of the embedding matrix exists, not the typical two (input + output).

Training Stages

The overall training comprises three stages, with the WSD scheduler governing the first two:

Stable Training Stage:

Data: ~1T tokens, mixture shown in Figure 11 (left): Code Pretrain 25%, CommonCrawl Chinese 25%, Dolma 24%, C4 15%, Pile 8%, with smaller contributions from peS2o, Arxiv, Open Web Math (1% each)
Batch size: 3.93M tokens (stated as ~4M in Table 2)
Maximum learning rate: 0.01
Scheduler: WSD, stable phase only (constant $\eta = 0.01$ )

Decay Stage:

Data: mixture of pre-training data and high-quality SFT data, proportions shown in Figure 11 (right)
Decay function: exponential, $f(s - T) = 0.5^{(s - T) / T_d}$ with $T_d = 5000$ steps (20B tokens)
Total decay tokens: not explicitly stated, but from the "5000 steps" and 4M batch size, this is approximately 5000 × 4M = 20B tokens

SFT Stage:

Data: SFT data only (similar to decay-stage SFT data but excluding pre-training data)
Tokens: ~6B for 1.2B model, ~4B for 2.4B model
Learning rate: aligned with the learning rate at the end of annealing
Scheduler: WSD with exponential decay (effectively continuing the decay from the decay stage's endpoint)

Figure 12 shows the C4 loss curve for both models. The loss drops sharply during the decay stage (orange segment), as expected from the preliminary experiments. The authors note: "Since we use the exponential decay, the loss still drops after the learning rate drops below 10% of the max learning rate." However, they "do not utilize the final checkpoints" — instead, they use checkpoints from the end of the dark green segment (stable stage) and begin SFT from there. The first drop in the MiniCPM-1.2B curve is attributed to batch size enlargement from 2M to 4M, which "might have a similar effect as decreasing learning rate" (Smith et al., 2017).

Data Deduplication and Processing

The data processing pipeline includes: "CommonCrawl Chn in a Chinese Corpus is derived from CommonCrawl raw corpus and goes through thorough cleaning." For English corpora (Dolma, C4, Pile), "they are deduplicated inner corpus and across corpus using MinHash algorithms" (Broder, 1997). Code pre-training data (The Stack, StarCoder) undergoes "inner deduplication and cross deduplication." This attention to deduplication is important for the paper's claims about data scaling — if data contains duplicates, the effective information content is lower than the token count suggests, which could inflate the apparent optimal data-to-model ratio.

Evaluation Methodology

The evaluation uses the open-source UltraEval framework with vLLM (Kwon et al., 2023) for inference acceleration. The benchmark suite includes: MMLU (English knowledge), CMMLU and C-Eval (Chinese knowledge), HumanEval and MBPP (coding), GSM8K and MATH (mathematics), HellaSwag, ARC-e, ARC-c (commonsense reasoning), and BBH (logic reasoning).

A notable methodological detail: for QA tasks (ARC-e, ARC-c, HellaSwag), the paper uses the better of two evaluation approaches — perplexity-based selection (extending each option as continuation, selecting lowest PPL) or direct generation (model outputs answer options). The authors observe that "Mistral-7B-v0.1 performs better in PPL tests but exhibits poorer performance in direct generation," while MiniCPM is more balanced. To ensure fairness, "we adopt the score from the evaluation method that yields the highest score" for each model. While this is a generous comparison methodology (each model gets its best evaluation approach), it is applied consistently across all models in the comparison tables, so the relative rankings should be robust.

4. Key Insights and Innovations

Innovation 1: Learning Rate Scheduling as an Explicit Decomposition of Exploration and Exploitation

The paper's most conceptually distinctive contribution is not the WSD scheduler itself as a mathematical formula, but the diagnostic insight that Cosine LRS works because it implicitly balances two independent mechanisms, and that making this balance explicit unlocks capabilities that the implicit version cannot provide. Prior to this work, the field treated learning rate scheduling as a black-box hyperparameter choice — Cosine LRS was dominant because empirical evidence showed it outperformed alternatives (Kaplan et al., 2020; Hoffmann et al., 2022), but why it worked and what levers it exposed were not systematically characterized.

The paper makes a specific diagnostic move that changes how one thinks about scheduling: instead of treating the learning rate curve as an atomic object to be optimized, treat it as a temporal allocation of two qualitatively different types of optimization behavior — high-learning-rate exploration (large, uncorrelated parameter moves that survey the loss landscape) and learning-rate-decay exploitation (small, consistent parameter moves that descend into local optima). The evidence for this decomposition is not merely the functional form of WSD but the gradient dynamics analysis in Section 4.4 (Figure 8): during the stable phase, gradient cosines are variable (exploration), while during decay they become consistently positive (exploitation), and loss curvature increases by an order of magnitude. These are not properties that Cosine LRS could reveal, because Cosine blends the two regimes into one continuous curve.

This reframing is significant beyond the specific schedule proposed. It establishes that the decay phase is not merely a convergence mechanism but a distinct training regime with its own dynamics. The finding that 10% of tokens in decay suffices to match Cosine performance (Figure 5) is not just a practical optimization — it implies that the high-LR phase is doing most of the "work" of finding good basins, while the decay phase is a remarkably efficient exploitation process. This asymmetry (90% exploration, 10% exploitation) was invisible under Cosine LRS, where the two phases are interleaved throughout training.

More subtly, the paper demonstrates that the exploitation phase is data-quality-sensitive in a way the exploration phase is not. Section 5 shows that introducing high-quality SFT data during decay dramatically improves downstream task performance compared to introducing the same data only during the separate SFT phase (Table 1: C-Eval jumps from 40.0 to 52.6 for the 2.4B model). This suggests a non-obvious interaction: the low-learning-rate regime, where parameter updates are small and consistent, is precisely when the model is most receptive to fine-grained capability shaping. The stable phase builds general competence; the decay phase specializes it. This temporal separation of "what to learn" (general language modeling during stable training) from "what to be good at" (task-specific capabilities during decay) is a conceptual advance over the standard approach of uniformly mixing data throughout training.

Comparison to prior work: Prior approaches to learning rate scheduling — Cosine LRS (Loshchilov & Hutter, 2016), linear decay, constant with sudden drops, warm restarts — all treat the schedule as a single curve whose shape is optimized end-to-end. The closest prior work, DeepSeek (Bi et al., 2024), is noted as bearing "the closest resemblance" but is not described in detail, implying it did not develop the explicit two-phase decomposition or the scaling-law measurement applications. The paper's contribution here is conceptual rather than algorithmic: WSD is simple to implement, but the idea that the two phases can and should be separated, analyzed independently, and recombined at different ratios for different purposes (continuous training, scaling law measurement, data quality injection) represents a fundamental shift in how to think about the training process.

Significance classification: This is a fundamental reframing, not an incremental improvement. It changes the unit of analysis from "the schedule" to "the allocation of optimization regimes," which opens new research questions (what is the optimal stable-to-decay ratio? how does the decay function shape affect capability acquisition? can the model detect when it should transition from exploration to exploitation?) that were not visible under the Cosine paradigm.

Innovation 2: Linear-Cost Scaling Law Measurement via Checkpoint Reusability

The second major conceptual contribution is the recognition that WSD LRS transforms scaling law measurement from a quadratic-cost to a linear-cost problem, and the demonstration that this enables empirically measuring the data-model scaling relationship at a granularity that was previously infeasible. This is not primarily a practical efficiency gain — it is an epistemological advance that changes what questions can be asked about scaling.

The problem the paper identifies is structural, not merely computational. Traditional scaling law measurement requires training each (model size, data size) pair from scratch with Cosine LRS tuned for that specific data amount, because intermediate checkpoints are not at their optimal learning rate for their current data volume. This means the cost is O(m²) for m model sizes — prohibitive for m larger than a handful, which forces researchers to fit scaling laws from sparse data with strong parametric assumptions. The paper's critique (Section 4.5) is that this sparsity is precisely why fundamental questions about the data-to-model ratio remain contested: Kaplan et al. (2020), Hoffmann et al. (2022), and Llama 2 (Touvron et al., 2023) arrive at ratios differing by an order of magnitude (10×, 20×, ~70–100× respectively), and "this unaddressed uncertainty stems from the challenges inherent in training multiple models of varying sizes and data sizes."

WSD breaks this constraint because any stable-stage checkpoint, at any token count, can be independently decayed to its optimal convergence. The insight is not merely that this is cheaper — it is that it makes the data axis nearly continuous for a fixed set of model sizes. You train each model size once, save checkpoints at many token counts, and decay each independently. The cost is O(m) stable training runs plus O(m × k) short decay runs (where k is the number of checkpoint counts). Since decay runs are 10% of their stable duration, the total cost is approximately O(m) — linear in model sizes, not quadratic.

This matters because it enables the paper to measure L(N, D) at 6 model sizes × 6 data amounts = 36 points (Figure 10) — a density that would require 36 full training runs under the traditional paradigm, but costs approximately 6 full runs plus 36 short decays under WSD. The fitted scaling law surface in Figure 10, with contour plots showing how equal-loss curves bend in the (N, D) plane, would be impossible to produce at this resolution without the WSD protocol. And it is this resolution that reveals the paper's most striking empirical finding: the compute-optimal data-to-model ratio of ~192× at C=10²¹ is nearly an order of magnitude larger than Chinchilla Optimal's ~20×.

Crucially, the paper does not claim this ratio is universal. The authors are careful to note variation across datasets (from ~96× for Ultratext to ~834× for Chinese Wikihow) and to validate against more recent practice by extracting Llama 2's approximate ratio (~70–100×) from published loss curves (Appendix B.4). The contribution is the measurement methodology itself — a protocol that makes scaling law measurement accessible enough that future work can systematically study how the ratio varies with data composition, model architecture, tokenizer choice, and training configuration, rather than relying on a single canonical number.

Comparison to prior work: Prior scaling law studies (Kaplan et al., 2020; Hoffmann et al., 2022) conducted measurements by training each (N, D) pair from scratch, limiting the density of their sampling and forcing strong parametric assumptions. The paper's approach is closest in spirit to the "optimal loss envelope" concept from Kaplan et al. (2020) — fitting the lower envelope of loss curves — but with the crucial difference that WSD produces points on the envelope by construction (each checkpoint is optimally converged for its data amount), rather than requiring envelope estimation from suboptimal points. This turns envelope fitting from a statistical estimation problem into a direct measurement.

Significance classification: This is a methodological advance with empirical consequences. The linear-cost protocol does not just make existing analyses cheaper — it enables analyses (dense sampling of the (N, D) plane, dataset-specific scaling laws, architecture-specific exponents) that were previously impractical, potentially resolving the long-standing disagreement about the optimal data-to-model ratio.

Innovation 3: The Decay Stage as a Distinct Optimization Regime with Measurable Geometric Properties

The paper provides what may be the first detailed empirical characterization of the geometric properties of the optimization trajectory during learning rate decay, establishing that the decay phase is not merely a continuation of training at a lower learning rate but a qualitatively distinct regime with identifiable signatures: gradient directions become consistently aligned across steps, parameter update magnitudes shrink, and loss curvature increases by an order of magnitude (Section 4.4, Figure 8).

This is significant not because it solves an open problem — the paper explicitly reserves deeper analysis for "future exploration" — but because it creates a new diagnostic vocabulary for understanding training dynamics. Prior to this work, the decay phase was a black box: everyone knew loss drops when you lower the learning rate, but what happens geometrically during that drop was uncharacterized. The paper introduces measurable quantities — gradient cosine between consecutive steps, first- and second-order directional derivatives along the optimization trajectory, loss curvature — and shows that they exhibit systematic, interpretable behavior during decay.

Two specific findings stand out as conceptually novel:

Finding 1: Loss decreases dramatically during decay despite smaller parameter updates. Figure 7 shows that during the stable phase, checkpoints differ substantially (large max element-wise weight differences), but loss decreases slowly. During decay, weight changes are an order of magnitude smaller, yet loss drops sharply. This implies that the stable phase is moving the model through regions of the loss landscape where progress is not immediately reflected in loss reduction — consistent with the exploration hypothesis, where the model traverses high-loss plateaus or ridges to reach better basins. The decay phase then descends rapidly into those basins.

Finding 2: Curvature increases by an order of magnitude during decay. The loss curvature K along the optimization trajectory (Figure 8, bottom-right panel) shows a clear jump at the onset of decay. This is a concrete geometric signature of approaching a local minimum: as you descend into a valley, the path curves more sharply. The paper interprets this as "indicating the proximity to a local optimum," but the more important implication is that the decay phase is when the model actually commits to a specific local minimum. The stable phase explores broadly; the decay phase selects and commits.

These diagnostics are not merely descriptive — they point toward a mechanistic understanding of why the stable-to-decay transition works. The paper speculates that the high-LR phase helps the model "find a better global optimum" while the decay phase "may involve unique training dynamics that enable the model to find a better local optimum" (Section 4.1). The gradient statistics in Figure 8 operationalize this speculation into testable quantities. Future work could, for example, use these diagnostics to automatically detect when a model should transition from stable to decay training, or to compare the optimization geometry of different model architectures and data distributions.

Comparison to prior work: Prior analyses of neural network training dynamics (e.g., studies of the Hessian spectrum, loss landscape visualization, sharp vs. flat minima) have characterized properties of converged solutions but have not examined the transition dynamics that occur during learning rate annealing. The closest prior work is likely studies of learning rate warmup (which examine the early phase of training) and cyclical learning rates (which study repeated annealing cycles), but neither provides the step-by-step gradient statistics during a single sustained decay phase that this paper presents. The contribution here is not a theoretical model but an empirical characterization that makes the decay phase a legible object of study rather than an opaque convergence mechanism.

Significance classification: This is an empirical diagnostic contribution — it does not claim to explain why the observed geometric properties emerge, but it provides the measurements and vocabulary that make such explanations possible. Its significance depends on whether future work picks up these diagnostics and builds on them.

Innovation 4: The Two-Stage Data Strategy as a Principle for Capability Injection

The paper's finding that introducing high-quality data during the decay stage dramatically outperforms introducing the same data only during SFT (Table 1) is more than a practical optimization — it establishes a principle about when capability-specific data should be introduced relative to the optimization process. The principle is: general competence is built during high-learning-rate exploration on large-scale diverse data; specialized capabilities are injected during low-learning-rate exploitation on high-quality targeted data.

This is counterintuitive from a standard curriculum learning perspective. One might expect that introducing high-quality data earlier (during the stable phase, when the learning rate is high and the model is more plastic) would be more effective, since the model has more capacity to absorb new patterns. The paper's results suggest the opposite: high-quality data introduced during the stable phase would need to be repeated many times (since the stable phase is long relative to the size of high-quality datasets), leading to potential overfitting or memorization. Introducing it during the brief decay phase means it is seen only a few times, and the model's low-learning-rate state is precisely calibrated to make fine-grained adjustments that improve task-specific performance without destabilizing the general capabilities acquired during the stable phase.

The ablation results are striking in their magnitude and consistency. For the 2.4B model, adding SFT data during decay (A-2 vs. A-1) improves C-Eval by 12.6 points (40.0 to 52.6), CMMLU by 9.6 points, and GSM8K by 14.6 points — all from changing when the same data is introduced, not how much data is used. More tellingly, doubling the SFT data in the separate SFT phase (B-2 vs. B-1, 12B vs. 6B SFT tokens) produces essentially no improvement (C-Eval: 41.2 vs. 40.9), while adding SFT data during decay with the standard SFT amount (B-3 vs. B-1) produces a 8.2-point C-Eval improvement (49.1 vs. 40.9). The paper's conclusion is unambiguous: "the benefits of introducing high-quality data at the beginning of the decay stage are much higher than simply adding it during the SFT phase."

This principle has implications beyond the specific MiniCPM training recipe. It suggests that the standard two-phase paradigm (pre-training followed by SFT) is suboptimal in its temporal allocation of data quality. The pre-training phase should remain focused on large-scale diverse data for general competence, but the transition to high-quality data should begin before the learning rate reaches its minimum — during the decay phase, when the model is still making systematic parameter updates but at a scale that enables precise capability shaping. The separate SFT phase remains necessary (the paper still does one), but its role shifts from being the sole capability-injection window to being a final polishing step after the decay phase has already done the heavy lifting.

Comparison to prior work: Prior approaches to data scheduling in LLM training include uniform mixing of all data throughout training, phased training where different data distributions are used at different stages (e.g., code then text), and the standard pre-train-then-SFT pipeline. None of these explicitly tie the type of data to the optimization regime (high-LR exploration vs. low-LR exploitation) in the way this paper does. The closest concept may be "annealing" in the context of knowledge distillation or domain adaptation, where a model is fine-tuned with a decaying learning rate on target-domain data, but these approaches typically do not preserve a separate stable phase with general-domain data. The paper's contribution is to couple data quality scheduling to learning rate scheduling as a unified principle, rather than treating them as independent design choices.

Significance classification: This is a practical principle with theoretical implications. It does not propose a new algorithm but establishes a relationship (optimization regime × data quality → capability acquisition) that should generalize across model scales and domains. The principle is testable and falsifiable: future work can verify whether decay-phase data injection consistently outperforms SFT-only injection across different model families, data compositions, and task types.

Innovation 5: The "Wind Tunnel" as a Coherent Methodology for Scalable Training Research

The paper's most ambitious conceptual contribution is the Model Wind Tunnel (MWT) as a systematic methodology for conducting training research at small scale and transferring findings to large models. This is not merely a collection of techniques (Tensor Program, batch size scaling, learning rate tuning) but a coherent experimental philosophy with explicit assumptions, validation steps, and a bidirectional knowledge flow: SLM experiments inform LLM training, and LLM-scale validation confirms (or refutes) the transferability of SLM-derived principles.

What makes this distinctive is the explicit articulation of the scaling assumptions and their empirical verification. The paper does not simply assume that hyperparameters transfer — it validates that they do (Figure 3: optimal learning rate remains at ~0.01 across a 10× size range, verified at 2.1B). It does not simply assume that batch size scaling laws from tiny models apply to larger ones — it measures the relationship at three model sizes and verifies that the fitted curves "connect each other well" (Figure 2). It does not simply assume that WSD dynamics observed at 0.036B apply at 2.4B — it reproduces the loss drop pattern in the full MiniCPM training runs (Figure 12). Each step of the MWT is empirically grounded, not merely assumed.

The MWT philosophy addresses a structural problem in LLM research: the cost of experimentation creates a selection pressure toward ad-hoc methods. When each training run costs millions of dollars, researchers have strong incentives to use whatever worked last time rather than systematically exploring alternatives. This produces a literature where training recipes are justified by precedent ("Cosine LRS is standard") rather than by controlled experiments. The MWT breaks this logjam by making controlled experiments cheap — the paper's 0.009B hyperparameter search (Section 3.1) explores dozens of configurations at a cost that is negligible compared to even a single LLM training run — while providing empirical evidence that the findings transfer upward.

However, the paper is honest about the limitations of this transfer. The Limitations section explicitly states: "this paper does not extend to training an LLM to validate the scaling law. The application of WSD LRS on LLMs has not been fully explored to date." This is a crucial admission: the MWT methodology is validated for transfer from 0.009B to 2.4B, but not necessarily from 2.4B to 70B+. Phenomena like training instabilities, loss spikes, or emergent capabilities that appear only at large scale might not be capturable in wind tunnel experiments. The paper positions MWT as a necessary but not sufficient condition for LLM training research — it lets you eliminate bad ideas cheaply and identify promising ones, but the promising ideas still need LLM-scale validation.

Comparison to prior work: Individual components of the MWT have precedents. Tensor Program (Yang et al., 2022; 2023) provides the hyperparameter transfer theory, and CerebrasGPT (Dey et al., 2023) applied it to predict LLM loss. Wortsman et al. (2023) used small models to predict large-model instabilities. But these prior works focus on specific aspects (hyperparameters, stability) rather than on the integrated experimental methodology. The paper's contribution is to combine hyperparameter transfer, batch size scaling, learning rate optimization, scaling law measurement, and data scheduling into a single coherent pipeline, and to demonstrate that this pipeline produces an SLM (MiniCPM-2.4B) that competes with models 3–5× its size (Mistral-7B, Llama2-13B). This end-to-end validation — from 0.009B experiments to competitive 2.4B model — is what distinguishes MWT from prior component-level contributions.

Significance classification: This is a methodological framework with practical validation at moderate scale. Its ultimate significance depends on whether the principles transfer to LLM scale (70B+), which the paper does not demonstrate. As a framework for SLM research, it is immediately useful; as a framework for LLM research, it is promising but unvalidated. The paper's intellectual contribution is in making the assumptions explicit and the verification steps concrete, so that future work can test transferability rather than merely assuming it.

5. Experimental Analysis

Evaluation Methodology

Dataset. The primary evaluation uses a suite of standard benchmarks: MMLU (Hendrycks et al., 2020) for English knowledge, CMMLU (Li et al., 2024) and C-Eval (Huang et al., 2024) for Chinese knowledge, HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021) for coding, GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021) for mathematics, HellaSwag (Zellers et al., 2019), ARC-e and ARC-c (Clark et al., 2018) for commonsense reasoning, and BBH (Suzgun et al., 2022) for logic reasoning. For the scaling law experiments (Section 4.5), the training losses are evaluated on five held-out datasets: Code, English (Wikihow), Chinese (Wikihow), Ultratext, and Chinese (Yayi Corpus), with average loss computed over bytes rather than tokens to enable comparison across models with different tokenizers, following Achiam et al. (2023). For long-context evaluation (Section 7.2), the paper uses ∞Bench (Zhang et al., 2024b). For the chat model, MTBench (Zheng et al., 2024) is used. All evaluations are conducted using the open-source UltraEval framework with vLLM (Kwon et al., 2023) for inference acceleration.
Base model(s). The MiniCPM family includes two base models: MiniCPM-1.2B (1.25B non-embedding parameters, 52 layers, d_m = 1536, d_{ff} = 3840, GQA with 24 query / 8 KV heads) and MiniCPM-2.4B (2.44B non-embedding parameters, 40 layers, d_m = 2304, d_{ff} = 5760, full multi-head attention with 36 heads). Both are standard dense Transformer decoders trained from scratch. The scaling law experiments (Section 4.5) use 6 model sizes: 0.04B, 0.1B, 0.3B, 0.5B, 0.9B, and 2.0B non-embedding parameters. The model wind tunnel experiments (Section 3) use models from 0.009B to 2.1B. The MiniCPM family extensions include MiniCPM-DPO (aligned via DPO on UltraFeedback), MiniCPM-128K (context extended to 128K tokens via ABF and NTK-Aware RoPE scaling), and MiniCPM-MoE (13.6B total parameters, ~4B activated via 2-of-8 expert routing, initialized via sparse upcycling from MiniCPM-2.4B).
Metrics. The primary metric throughout the benchmark evaluations is per-task accuracy (or pass@1 for code generation), with overall averages computed as Avgen (average across English-knowledge, code, math, reasoning, and commonsense tasks) and Avgchn (average of C-Eval and CMMLU). For MTBench, the metric is the model's judged score on a 1–10 scale. For the scaling law experiments, the metric is byte-normalized cross-entropy loss on held-out evaluation corpora. For the optimal batch size experiment, the metric is C4 loss as a function of batch size and token consumption.
Baselines. The paper compares against a comprehensive set of models at multiple scales. For the ~7B class: Llama2-7B (Touvron et al., 2023), Mistral-7B (Jiang et al., 2023), Gemma-7B (Banks & Warkentin, 2024), Qwen-7B (Bai et al., 2023), Deepseek-7B (Bi et al., 2024), ChatGLM2-6B (Du et al., 2021). For the ~13B class: Llama2-13B (Touvron et al., 2023). For larger models: MPT-30B, Falcon-40B, Llama2-34B. For SLMs directly comparable in scale: TinyLlama-1.1B (Zhang et al., 2024a), Qwen-1.8B, Qwen1.5-1.8B, Gemini Nano-3B (Gemini et al., 2023), StableLM-Zephyr-3B, Phi-2-2B (Javaheripi & Bubeck, 2023), Gemma-2B. For the DPO model: Mistral-7B-Instruct-v0.1 and v0.2, Qwen-7B-Chat, Yi-6B-Chat, Baichuan2-7B-Chat, Deepseek-7B-chat, Llama2-7B-Chat, ChatGLM2-6B. For long-context: LWM-Text-128K, Yarn-Mistral-7B-128K (Peng et al., 2023), Yi-6B-200K, ChatGLM3-6B-128K. For MTBench: Llama2-70B-Chat, Deepseek-67B-Chat, Qwen-72B-Chat, and others in the 7B–70B range.
Generation budget / compute accounting. In the scaling law experiments, compute C is measured in FLOPs using the standard approximation C = 6ND (Rae et al., 2021), where N is non-embedding parameters and D is training tokens. For the continuous training experiment (Section 4.3, Figure 6), training compute is compared between a 0.036B model trained on varying token budgets and a 0.17B model trained on 40N tokens, with inference compute savings noted as ~5x per inference call. For the WSD scheduler validation (Section 4.3), token budgets are expressed in multiples of the model's non-embedding parameter count (e.g., 40N means 40 times the parameter count in tokens). For deployment benchmarking (Appendix E), throughput is measured in tokens per second on various edge devices.
Cross-validation / statistical protocol. The scaling law measurement uses scipy curvefit to fit L(N, D) to the Hoffmann et al. (2022) parametric form, with the fitted parameters reported in Figure 10. The optimal batch size experiment fits parabolas to equal-loss points and connects their minima (Figure 1). The two-stage pre-training strategy ablation (Table 1) compares five configurations (A-1, A-2, B-1, B-2, B-3) on a fixed set of benchmarks. For the QA task evaluation (ARC-e, ARC-c, HellaSwag), the paper adopts the higher score from two evaluation methods (PPL-based and direct generation) for each model "to ensure fairness in comparison." There is no mention of confidence intervals, standard errors, or statistical significance testing for the benchmark comparisons. The benchmark comparison tables do not report variance or number of evaluation runs.

Main Quantitative Results

Model Wind Tunnel: Hyperparameter Stability

Optimal learning rate stability across scales (Figure 3). The paper trains models at 0.04B, 0.1B, 0.3B, and 0.5B parameters, each with six learning rates, and measures C4 loss. Across this 12.5× range in model size, "the optimal base learning rate does not show a noticeable shift and remains around 0.01." A validation at 2.1B confirms this finding. This stability is the empirical payoff of the Tensor Program scaling rules: without them, the optimal learning rate might drift with model size, but with proper initialization and learning rate scaling of two-dimensional tensors (Table 7), it remains fixed. The grid search on the 0.009B model (Figure 14) identifies the optimal combination as scale_depth = 1.4, scale_emb = 12, init_std = 0.1, and lr = 0.01.

Optimal batch size scaling (Figures 1 and 2). Training 0.009B, 0.03B, and 0.17B models with six batch sizes each yields the fitted relationship bs = 1.21 × 10^9 / L^{6.24}, where bs is batch size in tokens and L is C4 loss. The three red lines (connecting equal-loss parabola minima for each model size) "connect each other well into a linear relationship in the log space" (Figure 2). The exponent 6.24 means optimal batch size grows rapidly as loss decreases — a model approaching convergence needs much larger batches to maintain efficient token utilization.

WSD Scheduler Validation on 0.036B Models

Cosine LRS decomposition (Figure 4). Training a 0.036B model with Cosine(T) for T = 20N, 40N, 60N, 80N, and CosineLoop(20N) reveals that for any total training steps S, "the lowest loss is always achieved by the Cosine(T) where T = S." Both T < S (decay finishes early) and T > S (decay incomplete) are suboptimal. CosineLoop underperforms Cosine with T = S, confirming that the decay phase is not simply about reducing learning rate — it involves unique convergence dynamics.

Loss drop in decay stage (Figure 5). For WSD(40N, 4N), WSD(60N, 6N), and WSD(80N, 8N), the loss curve shows "a significant rapid decline" as the learning rate begins to decrease, "quickly decreases to be equal to or lower than the Cosine LRS at step T = S." This validates that the stable and decay phases can be explicitly separated without performance loss. Post-decay, the model can be continued from its pre-decay checkpoint with more stable training and then re-annealed.

Decay duration sufficiency (Figure 5). Comparing WSD(40N, 4N) vs. WSD(40N, 2N), WSD(60N, 6N) vs. WSD(60N, 2N), and WSD(80N, 8N) vs. WSD(80N, 2N): "having a decay of 10% of the total tokens is sufficient to achieve the best results, while a decay of 2.5% of total tokens falls short."

Continuous training vs. larger models (Figure 6). A 0.036B model trained with WSD for increasing stable durations (20N through 320N, each with 10% decay) continues to improve even at 320N tokens. The fitted optimal loss envelope (power-law: L(C) = βC^{-α} + L_0) shows that "a 0.036B model can match the performance of a 0.17B model with an acceptable increase (~4 times) in training compute while saving a lot of inference computation (saving ~5 times per inference call)."

Decay Stage Gradient Dynamics (0.2B Model)

Parameter update magnitudes (Figure 7). The maximum element-wise weight difference between checkpoints "exhibits a robust correlation with the learning rate's magnitude." During the stable phase, weight changes are large but loss decreases slowly. During decay, "despite less pronounced weight alterations, there is an accelerated decrease in loss."

Gradient alignment during decay (Figure 8). "Upon commencement of this stage, the cosine between gradients predominantly assumes positive values, suggesting that in the decay phase, model parameters undergo consistent changes across steps." This contrasts with the stable phase, where gradient directions are more variable.

Directional derivatives and curvature (Figure 8). The negative first-order directional derivative "diminishes exponentially with each step, aligning closely with the learning rate." The second-order directional derivative "exhibits a slight increase in magnitude." Loss curvature K = |D_2| / (1 + D_1^2)^{3/2} "increases by a magnitude, indicating the proximity to a local optimum." The gradient norm diminishes during decay. Maximum gradient behavior mirrors the gradient norm.

Scaling Law Measurement via WSD (6 Models, 5 Evaluation Corpora)

Fitted coefficients (Figure 10, averaged across corpora). The fitted loss function L(N, D) = C_N N^{-α} + C_D D^{-β} + L_0 yields α = 0.29, β = 0.23, K^2 = 0.01, η = -0.10. Since α > β (model size exponent exceeds data size exponent), η = (β - α)/(α + β) < 0, meaning the optimal data-to-model ratio decreases slightly with compute — you should marginally emphasize model scaling over data scaling as total compute grows. However, the absolute value of η is small, so the ratio is nearly constant across the compute range studied.

Compute-optimal data-to-model ratio (Figure 10). At C = 10^{21} FLOPs, the average D_opt / N_opt ≈ 191.87 (from the "Avg" subplot). By dataset: Code yields ~194.93, English Wikihow ~166.04, Chinese Wikihow ~833.95 (the highest, suggesting Chinese text benefits disproportionately from continued training), Ultratext ~95.60, and Chinese Yayi Corpus ~112.79. This ratio is approximately 9.6× larger than Chinchilla Optimal's ~20×.

Llama 2 validation (Appendix B.4, Figure 18). From published Llama 2 loss curves, the paper estimates the 13B model would intersect the 34B model at ~10^5 EFlops, and the 34B would intersect the 70B at ~5 × 10^5 EFlops, yielding an estimated D_opt/N_opt of 70–100 — much closer to MiniCPM's ~192 than to Chinchilla's ~20. The paper acknowledges this is "only a rough one."

Fit quality (Figure 17). The fitted scaling curves (orange) plotted against real loss values (blue) for each model size and dataset show good agreement, "except for the last checkpoints of the 0.11B and 0.25B model."

Two-Stage Pre-training Strategy Ablation (Table 1)

2.4B model (A-2 vs. A-1). Adding SFT data during the decay stage rather than only pre-training data, followed by identical 4B-token SFT, yields: C-Eval 52.6 vs. 40.0 (+12.6), CMMLU 51.1 vs. 41.5 (+9.6), MMLU 50.9 vs. 44.6 (+6.3), GSM8K 42.3 vs. 27.7 (+14.6), HumanEval 30.4 vs. 27.7 (+2.7), MBPP 30.3 vs. 24.4 (+5.9), MATH 5.4 vs. 5.1 (+0.3).

1.2B model, SFT data in decay vs. double SFT tokens (B-3 vs. B-2). B-3 (decay with SFT data mix, 6B SFT tokens) vs. B-2 (decay with pre-training data only, 12B SFT tokens): C-Eval 49.1 vs. 41.2 (+7.9), CMMLU 46.8 vs. 42.0 (+4.8), MMLU 49.6 vs. 47.9 (+1.7), GSM8K 31.8 vs. 34.4 (−2.6), MATH 10.5 vs. 7.3 (+3.2), HumanEval 44.5 vs. 43.9 (+0.6), MBPP 32.8 vs. 29.8 (+3.0). Despite B-2 having twice the SFT tokens, B-3 substantially outperforms on Chinese knowledge and math. The GSM8K regression (−2.6 points) is a notable exception.

1.2B model, standard SFT (B-1) vs. decay mixing (B-3). B-3 outperforms B-1 on C-Eval (49.1 vs. 40.9), CMMLU (46.8 vs. 41.5), MMLU (49.6 vs. 47.9), MATH (10.5 vs. 7.9), but slightly underperforms on GSM8K (31.8 vs. 34.2). The paper concludes: "the benefits of introducing high-quality data at the beginning of the decay stage are much higher than simply adding it during the SFT phase."

MiniCPM Benchmark Results (Table 3)

MiniCPM-2.4B vs. 7B–13B LLMs. MiniCPM-2.4B achieves an average score of 52.33% across all benchmarks, compared to Mistral-7B's 48.97%, Llama2-13B's 41.48%, Gemma-7B's 52.22%, and Qwen-7B's 49.46%. On English tasks specifically (Avgen), MiniCPM-2.4B scores 52.60 vs. Mistral-7B's 49.96 and Llama2-13B's 42.44. On Chinese tasks (Avgchn), MiniCPM-2.4B scores 51.10 vs. Mistral-7B's 44.54 and Llama2-13B's 37.19. The model outperforms Llama2-13B on all benchmarks except MMLU (53.46 vs. 54.71), BBH (36.87 vs. 37.92), and HellaSwag (68.25 vs. 79.23†, where † indicates PPL-based evaluation for Llama2-13B).

MiniCPM-1.2B vs. 7B models. MiniCPM-1.2B achieves 45.67% average, outperforming Llama2-7B (35.40%) but trailing Mistral-7B (48.97%) and MiniCPM-2.4B (52.33%). On Chinese tasks, MiniCPM-1.2B (47.98) substantially exceeds Llama2-7B (31.77) and approaches Mistral-7B (44.54). On coding and math, MiniCPM-1.2B scores HumanEval 44.51, MBPP 32.75, GSM8K 31.77, MATH 10.60 — generally outperforming or matching 7B models on code but underperforming on math reasoning.

MiniCPM-2.4B vs. other SLMs. Among SLMs (~1B–3B parameters), MiniCPM-2.4B ranks highest overall (52.33%) compared to Phi-2-2B (48.84%), Gemma-2B (35.10%), StableLM-Zephyr-3B (43.46%), Qwen1.5-1.8B (37.09%), and Gemini Nano-3B (only BBH 42.40 available). Phi-2 outperforms MiniCPM-2.4B on GSM8K (57.16 vs. 53.83), HumanEval (47.56 vs. 50.00 — MiniCPM leads here), MBPP (55.04 vs. 47.31), and ARC-e (86.11 vs. 85.44), suggesting Phi-2's textbook-focused training gives it an advantage on academic-style benchmarks.

BBH gap for SLMs. "Generally, BBH is harder for SLMs than LLMs compared to another knowledge-oriented dataset, demonstrating that reasoning ability might be more dependent on model size than knowledge." MiniCPM-2.4B scores 36.87 on BBH vs. Mistral-7B's 41.06 and Llama2-13B's 37.92, despite MiniCPM-2.4B outperforming these models on knowledge benchmarks.

MiniCPM-DPO Results (Table 4, Figure 13)

Benchmark tradeoff. After DPO alignment, MiniCPM-2.4B-DPO shows a mixed pattern on benchmarks compared to the base model: C-Eval drops from 51.13 to 48.64 (−2.49), CMMLU drops from 51.07 to 48.37 (−2.70), MMLU drops slightly from 53.46 to 53.05 (−0.41), while HumanEval improves from 50.00 to 51.22 (+1.22), GSM8K stays roughly flat (53.83 to 53.37), and MATH drops from 10.24 to 9.86. This pattern is consistent with the "alignment tax" (Askell et al., 2021) — RLHF optimization for human preferences can degrade benchmark performance.

MTBench (Figure 13). MiniCPM-2.4B-DPO achieves an MTBench score of 7.25, up from 6.89 after SFT. This surpasses Llama2-70B-Chat, Deepseek-67B-Chat, Qwen-72B-Chat, Mistral-7B-Instruct, and GPT-3.5-Turbo-0613 in the ranked comparison shown in Figure 13, placing it among models 10–30× its size.

Vs. larger chat models (Table 4). MiniCPM-2.4B-DPO achieves 51.60% average across benchmarks, compared to Mistral-7B-Instruct-v0.2's 50.91%, Qwen-7B-Chat's 44.93%, and Llama2-7B-Chat's 38.16%. On Chinese knowledge, MiniCPM-2.4B-DPO (48.51 Avgchn) leads most 7B chat models but trails Yi-6B-Chat (71.00 Avgchn) by a wide margin.

MiniCPM-128K Results (Table 5)

∞Bench performance. MiniCPM-2.4B-128K achieves average scores of 27.32 (all tasks) and 37.68 (excluding code and math). This is comparable to Mistral-7B-Instruct-v0.2 (ABF 1000w) at 27.75 / 36.90 and outperforms ChatGLM3-6B-128K (25.58 / 36.57) despite being 2.5× smaller. On specific tasks: Passkey Retrieval 98.31 vs. Mistral's 100, Number String 99.83 vs. Mistral's 78.98, KV Retrieval 9.0 vs. Mistral's 3.6 (a notable advantage), Long Book QA English 23.06 vs. Mistral's 11.74 (nearly 2×). On code and math tasks, MiniCPM-128K underperforms: Code Debug 22.08 vs. Mistral's 17.51, Code Run 0 vs. Mistral's 0, Math Calc 4.29 vs. Mistral's 29.43 (a large deficit), Math Find 0 vs. 0.

MiniCPM-MoE Results (Table 6)

Benchmark performance. MiniCPM-MoE (13.6B total, ~4B activated) achieves: C-Eval 58.11, CMMLU 58.80, MMLU 58.90, HumanEval 56.71, MBPP 51.05, GSM8K 61.56, MATH 10.52, BBH 39.22. Compared to Llama2-34B: MMLU 58.90 vs. 62.6 (Llama2 leads), GSM8K 61.56 vs. 42.2 (MiniCPM-MoE leads substantially). Compared to dense MiniCPM-2.4B: improvements across all benchmarks, with particularly large gains on C-Eval (51.13 → 58.11), GSM8K (53.83 → 61.56), and HumanEval (50.00 → 56.71). MATH shows essentially no improvement (10.24 → 10.52), consistent with the paper's observation that math reasoning may depend more on fundamental capability than parameter count.

Ablation Studies and Robustness Checks

Cosine LRS period ablation (Figure 4). Training a 0.036B model with Cosine(T) for T = 20N, 40N, 60N, 80N, with total training steps S matching each T, reveals that for any S, the optimal T = S. CosineLoop underperforms Cosine(T=S), confirming that the full decay process (not just periodic LR reduction) is necessary.

Decay duration (Figure 5). WSD with 10% decay tokens consistently achieves best results; 2.5% decay is insufficient. At all three stable durations tested (40N, 60N, 80N), the 10% variants (4N, 6N, 8N decay respectively) outperform the 2.5% variants (all 2N decay).

Decay function fitting (Appendix B.2, Figure 16). The optimal loss envelope from continuous training is fit with both exponential and power-law forms. The power-law form L(C) = βC^{-α} + L_0 fits better, consistent with Kaplan et al. (2020) and inconsistent with an exponential saturation model for the data sizes tested.

Two-stage pre-training data strategy (Table 1). The comparison of A-1 vs. A-2 (2.4B), B-1 vs. B-3 (1.2B), and B-2 vs. B-3 (1.2B) isolates the effect of decay-stage data composition from SFT token quantity, showing that mixing SFT data during decay outperforms adding equivalent or double SFT data in the separate SFT phase.

Evaluation methodology for QA tasks (Section 6.5). The paper adopts the higher score between PPL-based and direct-generation evaluation for each model, noting that Mistral-7B-v0.1 performs better under PPL while MiniCPM is more balanced. This is applied consistently but means the reported scores represent an upper bound rather than a fixed evaluation protocol.

Tensor Program with vs. without QK-Norm (Figure 14, Appendix A.1). Adding QK-Norm and decoupled weight decay significantly reduces learning rate sensitivity. However, the paper chooses not to use these techniques since the MWT already identifies the optimal learning rate via cheap small-scale experiments, making the additional stability unnecessary.

Vocabulary size and compression (Table 9, Appendix C). The MiniCPMTokenizer-120K (122,753 tokens) achieves the highest compression ratio overall (3.40 average bytes/token) across Chinese, English, code, and academic papers. The smaller MiniCPMTokenizer-70K (73,440 tokens) achieves 3.31 average bytes/token — a modest reduction that saves embedding parameters. The compression comparison includes Baichuan2 (125,696 vocab, 3.30 bytes/token), ChatGLM2 (64,794 vocab, 3.29 bytes/token), and Llama2 (32,000 vocab, 2.84 bytes/token).

Continuous training loss envelope fitting (Section 4.3, Figure 6, Appendix B.2). Two functional forms are tested for the loss envelope. Power-law fits better than exponential, suggesting that even with WSD's explicit phase separation, the underlying loss-compute scaling retains power-law character.

Post-SFT performance after decay checkpoint selection (Figure 12). The released MiniCPM checkpoints use the end of the stable stage (dark green segment) as the starting point for SFT, not the final decayed checkpoints (orange segment). The authors note that loss continues to drop during the full exponential decay but they "do not utilize the final checkpoints." The effect of continuing decay further before SFT is not ablated.

Batch size schedule (Figure 12, left). MiniCPM-1.2B shows a loss drop when batch size is increased from 2M to 4M tokens, which the paper attributes to batch size increase having "a similar effect as decreasing learning rate" (Smith et al., 2017). This is noted as an observation rather than a controlled ablation.

Critical Assessment

Claim 1: "MiniCPM ranks preeminently in their respective 2B and 1B scale categories and exhibits comparable capabilities to 7B–13B LLMs"

What the experiments demonstrate. Table 3 provides comprehensive evidence that MiniCPM-2.4B (52.33% average) achieves scores comparable to or exceeding Gemma-7B (52.22%), Mistral-7B (48.97%), and Llama2-13B (41.48%). MiniCPM-1.2B (45.67%) outperforms Llama2-7B (35.40%) by a wide margin. Among direct scale competitors, MiniCPM-2.4B leads Phi-2-2B (48.84%) and Gemma-2B (35.10%), while MiniCPM-1.2B substantially outperforms TinyLlama-1.1B (25.36%) and Qwen-1.8B (34.72%).

What the experiments do not demonstrate (or demonstrate weakly). Several aspects of this claim warrant scrutiny:

First, the evaluation methodology for QA tasks uses the higher of two evaluation approaches (PPL vs. direct generation) per model. Table 3 shows that Llama2-7B, Llama2-13B, and several other models use PPL-based evaluation (marked with †) on HellaSwag, ARC-e, and ARC-c, while MiniCPM uses direct generation. This is acknowledged as a fairness measure, but it means the comparison is not like-for-like — each model is evaluated under its most favorable protocol. If MiniCPM had been evaluated under PPL (or Llama2 under direct generation) on these tasks, the gaps might differ. The paper does not report the alternative evaluation numbers to allow readers to assess sensitivity.

Second, the claim that MiniCPM "exhibits comparable capabilities" to 7B–13B models is supported on average but masks substantial per-task variation. On BBH, MiniCPM-2.4B (36.87) trails Mistral-7B (41.06) and Llama2-13B (37.92). On MATH, MiniCPM-2.4B (10.24) leads Mistral-7B (5.00) and Llama2-13B (2.25) by a large margin, but the absolute scores are so low that "comparable" may overstate the practical capability. The paper acknowledges the BBH gap explicitly, noting that "reasoning ability might be more dependent on model size than knowledge," but this observation actually weakens the claim of general comparability — it shows that there are capability dimensions where model size matters significantly and SLMs have a structural disadvantage.

Third, the benchmark suite emphasizes knowledge and pattern matching (MMLU, C-Eval, CMMLU, HellaSwag, ARC) over complex reasoning (BBH, MATH, GSM8K). The average score weights all benchmarks equally, which may favor models strong on knowledge benchmarks. A reader interested in reasoning capabilities might draw different conclusions from the per-task breakdown.

Fourth, the comparison is against base models (not instruction-tuned variants, except in Table 4). While this is a fair comparison (MiniCPM base vs. other base models), deployment-relevant capability is typically measured on instruction-tuned models. The DPO results (Table 4) show that MiniCPM-2.4B-DPO remains competitive but the alignment tax narrows the gap: MiniCPM-2.4B-DPO at 51.60% vs. Mistral-7B-Instruct-v0.2 at 50.91%.

Claim 2: "The WSD scheduler demonstrates intriguing loss landscape properties and enables continuous training with an unpredefined pre-training token number"

What the experiments demonstrate. The gradient dynamics analysis (Section 4.4, Figure 8) provides clear empirical evidence of distinct optimization regimes: gradient cosine becomes consistently positive during decay, curvature increases by an order of magnitude, and first-order directional derivatives track the learning rate. The continuous training experiment (Figure 6) shows that a 0.036B model continues to benefit from additional tokens up to 320N, far beyond Chinchilla Optimal, and can match a 4.7× larger model with ~4× more training compute.

What the experiments do not demonstrate. The gradient dynamics analysis is conducted on a single 0.2B model with a single WSD configuration. It is not clear whether the observed patterns (gradient cosine alignment, curvature increase) are specific to this model size, architecture, or data distribution, or whether they generalize. The paper explicitly "reserves for future exploration" the deeper analysis of these findings.

More importantly, the claim that WSD "enables continuous training with an unpredefined pre-training token number" is demonstrated in principle by the 0.036B experiments but is not fully exercised in the MiniCPM training runs. MiniCPM-2.4B and MiniCPM-1.2B are both trained for approximately 1.1T tokens — a predefined endpoint. The paper does not demonstrate a scenario where training is extended mid-run due to new data or compute becoming available, which would be the acid test of "unpredefined." The released checkpoints are from the end of the stable stage, not from an extended training run beyond the initially planned budget.

Claim 3: "With WSD LRS, we are now able to efficiently study data-model scaling law without extensive retraining experiments on both axes of model and data"

What the experiments demonstrate. The scaling law measurement (Section 4.5, Figures 9, 10, 17) convincingly demonstrates the linear-cost protocol: 6 model sizes are trained once during the stable phase, checkpoints at 6 token counts are saved, and each is independently decayed. The resulting 36 (N, D) points are fit to the Hoffmann et al. (2022) parametric form, yielding reasonable fits and producing the compute-optimal ratio of ~192×.

What the experiments do not demonstrate (or demonstrate weakly). Several concerns:

First, the scaling law is measured entirely on SLMs (0.04B to 2.0B). The paper does not validate whether the fitted L(N, D) parameters (α = 0.29, β = 0.23) extrapolate to larger model sizes. The Limitations section acknowledges: "this paper does not extend to training an LLM to validate the scaling law." This is not a minor caveat — the entire claim of "efficiently study data-model scaling law" rests on the assumption that the SLM-derived exponents apply at LLM scale, which is unverified. If the exponents change with scale (e.g., if β increases for larger models, changing the optimal ratio), the measurement protocol is still cheap but the results would not transfer.

Second, the scaling law is measured on C4 and four other corpora that are all web-text or code datasets. The paper does not measure how the exponents vary with data quality. The finding that MiniCPM can productively absorb 192× more data than parameters might be partly a function of data quality — if the data is heavily deduplicated (as the paper's data processing is), the effective information per token is higher. A model trained on noisier, less deduplicated data might show different exponents. The protocol could measure this (by varying data quality) but does not.

Third, the 0.036B continuous training experiment (Figure 6) shows that a fixed-size model can match a larger model with more training compute, but the paper does not extend this analysis to ask: at what point does additional training of the small model become less compute-efficient than simply training a larger model with fewer tokens? The fitted loss envelope (Figure 6) could answer this question, but the paper does not compute the crossing point.

Fourth, the claim that WSD enables scaling law measurement "with linear effort on the model axis and a negligible effort on the data axis" somewhat understates the total cost. While the cost is linear in model sizes (O(m) stable training runs), the number of checkpoints per model (k = 6 in this paper) multiplies the decay cost. Each decay is 10% of its stable duration, so for k = 6 checkpoints per model, the total decay cost is roughly 0.6× the stable training cost (6 checkpoints × 10%). The total cost is approximately 1.6× the cost of training each model once to its maximum token count — still O(m), but not "negligible" on the data axis.

Claim 4: "The resulting scaling law indicates a much higher compute-optimal data-to-model ratio (~192×) compared with Chinchilla Optimal (~20×)"

What the experiments demonstrate. The fitted exponents and the computed ratio at C = 10^{21} are clearly reported (Figure 10, average: D_opt/N_opt ≈ 191.87). The paper provides extensive per-dataset breakdowns and the contour plots show the fitted surface.

What the experiments do not demonstrate (or demonstrate weakly). This is the most consequential empirical claim in the paper, and it warrants careful scrutiny.

First, the ratio is computed at C = 10^{21} FLOPs — a specific compute budget. For the 2.0B model in the scaling study, this corresponds to approximately D = C/(6N) = 10^{21} / (6 × 2 × 10^9) ≈ 8.3 × 10^{10} tokens, or about 83B tokens — roughly 42× the model size. The optimal ratio of 192× at this compute budget means the model should be smaller than 2.0B to use those FLOPs optimally. The paper's own fitted parameters (η = -0.10) imply the ratio changes with compute, so the ~192× figure is not a universal constant but a point estimate at a particular compute scale.

Second, the paper compares this ratio to Chinchilla Optimal's ~20×, but these are ratios computed under different conditions: different model architectures, different data, different tokenizers, different training configurations, and different compute budgets. The paper's own analysis of Llama 2 (Appendix B.4) estimates a ratio of 70–100×, which is closer to MiniCPM's finding. This suggests the difference from Chinchilla may be largely attributable to improvements in training configuration and data processing rather than to WSD per se. The paper acknowledges this: "we notice that their scaling experiment was conducted in a not very recent configuration." This is an honest caveat, but it means the claim "much higher than Chinchilla Optimal" may reflect the evolution of training practices over time more than a specific discovery enabled by WSD.

Third, the per-dataset variation is substantial: from ~96× (Ultratext) to ~834× (Chinese Wikihow). The average of ~192× masks a factor-of-8 range. A practitioner training on a different data mixture would get a different optimal ratio, and the paper does not provide guidance on how to predict it.

Fourth, the fitted exponents (α = 0.29, β = 0.23) have β < α, meaning data scaling contributes less to loss reduction than model scaling per unit of budget. This actually implies you should scale model size more aggressively than data size as compute grows — which seems to contradict the high absolute ratio. The resolution is that the absolute ratio is also determined by the coefficients C_N and C_D. If C_D is much larger than C_N (data is initially more beneficial than model size), the optimal ratio starts high and may decrease slowly even with β < α. The paper does not discuss this tension explicitly.

Claim 5: "The two-stage pre-training strategy (introducing SFT data during decay) substantially improves performance"

What the experiments demonstrate. Table 1 provides clear, consistent evidence. On the 2.4B model, decay-stage SFT data mixing improves C-Eval by 12.6 points (40.0 → 52.6) — a 31.5% relative improvement. On the 1.2B model, decay mixing with 6B SFT tokens outperforms double SFT tokens (12B) without decay mixing on most benchmarks.

What the experiments do not demonstrate. The ablation compares only two data compositions during decay (pre-training only vs. pre-training + SFT mix) and two SFT token budgets (6B vs. 12B for the 1.2B model). It does not explore:

What fraction of decay data should be SFT data (the Figure 11 right panel shows ~4.8% "SFT mixed" but the total SFT-related data including "Code SFT," "Knowledge SFT," "Math SFT," etc. is larger).
Whether the benefit depends on the quality of the SFT data or merely its distribution (i.e., would any out-of-distribution data during decay help, or does it need to be high-quality task-specific data?).
Whether the benefit is specific to the exponential decay schedule or generalizes to other decay functions.
Whether introducing SFT data earlier (e.g., the last 20% of the stable phase) might help more.
The GSM8K result for the 1.2B model: B-3 scores 31.8 vs. B-1's 34.2, a 2.4-point regression. The paper does not discuss this anomaly, which is one of the few cases where decay mixing hurts a benchmark.

Claim 6: "MiniCPM-128K demonstrates performance either surpassing or matching that of models like Yarn-Mistral-7B-128K and ChatGLM3-6B-128K"

What the experiments demonstrate. Table 5 shows MiniCPM-2.4B-128K achieving 27.32 average (all tasks) and 37.68 average (without code and math). This is comparable to Mistral-7B-Instruct-v0.2 at 27.75/36.90 and exceeds ChatGLM3-6B-128K at 25.58/36.57. On specific tasks, MiniCPM-128K excels at Long Book QA Eng (23.06 vs. Mistral's 11.74) and KV Retrieval (9.0 vs. Mistral's 3.6).

What the experiments do not demonstrate. The comparison is against instruction-tuned models (Mistral-7B-Instruct, ChatGLM3-6B) while MiniCPM-128K appears to be evaluated as a base model (the paper does not mention SFT or alignment for this variant). This is an uneven comparison — instruction-tuned models may have different long-context behavior than base models. The ∞Bench evaluation also includes code and math tasks where MiniCPM-128K underperforms dramatically: Math Calc 4.29 vs. Mistral's 29.43. The "surpassing or matching" claim holds only when code and math tasks are excluded (average without code and math: 37.68 vs. 36.90), which the paper does note but the headline claim in the abstract ("performance either surpassing or matching") does not qualify.

Missing Experiments That Would Have Strengthened the Paper

LLM-scale WSD validation. The paper's central methodological claim — that WSD enables efficient scaling law measurement and that the resulting ratios transfer upward — would be substantially strengthened by training even a single LLM-scale model (e.g., 7B) with WSD and verifying that the SLM-derived exponents predict its loss reasonably. The absence of this validation is the paper's most significant limitation, and the authors acknowledge it.

Cross-model-family generalization of MWT. The wind tunnel experiments are conducted entirely within a single architecture family (standard dense Transformer decoders with specific width/depth ratios, RoPE, etc.). Whether the discovered hyperparameters (lr = 0.01, specific initialization scales) transfer to models with different architectures (e.g., different attention mechanisms, different activation functions, different normalization schemes) is untested.

Difficulty-based or capability-based analysis. Unlike the MiniCPM paper, which evaluates aggregate benchmark scores, there is no analysis of which types of examples MiniCPM succeeds or fails on within each benchmark. The paper notes that BBH is harder for SLMs than LLMs, but does not characterize what makes certain BBH tasks hard. A breakdown by task difficulty or capability category would strengthen the claim about SLM capability breadth.

SFT data composition during decay. The decay-stage data mixture (Figure 11, right) is complex and includes proprietary data. An ablation varying the proportion or composition of SFT data during decay would help distinguish whether the benefit comes from the presence of any high-quality data or from specific data sources.

Long-context model with SFT/alignment. MiniCPM-128K is evaluated in base model form. An aligned version with long-context capability would provide a more deployment-relevant comparison against Mistral-7B-Instruct-v0.2.

Summary Assessment

The paper's experiments provide strong evidence for the effectiveness of the MiniCPM models at their target scale, for the practical benefits of the WSD scheduler (continuous training, checkpoint reusability, efficient scaling law measurement), and for the two-stage data strategy. The benchmark results convincingly demonstrate that careful training methodology can produce SLMs competitive with much larger models on standard benchmarks.

The experiments provide more qualified support for the scaling law findings. The ~192× optimal ratio is substantially higher than Chinchilla Optimal, but the paper acknowledges that this may reflect improvements in training configuration rather than a universal property, and the measurement is limited to SLM scale with no LLM validation. The gradient dynamics analysis (Section 4.4) is suggestive but preliminary — it characterizes what happens during decay without establishing why or whether the patterns generalize.

The paper is generally forthright about limitations (the Limitations section explicitly flags the LLM validation gap and the WSD-on-LLMs open question), but some claims in the abstract and introduction ("MiniCPM propounds a new stage in the development of small language models," "exemplifying the latent potential within SLMs") are more enthusiastic than the experimental evidence strictly warrants. The benchmark results are strong for the model's size, but the claim of a "new stage" would require evidence that MiniCPM's methodology produces SLMs that are categorically more capable than prior SLMs, not just somewhat better on a standard benchmark suite that weights knowledge tasks heavily. The qualitative examples in Appendix F (translation with instruction traps, symbolic substitution, acrostic poetry, emoji understanding) provide some evidence of breadth, but these are cherry-picked successes, not systematic capability evaluations.

The strongest and most robust findings are: (1) WSD enables continuous training and checkpoint reuse with no performance penalty compared to Cosine LRS, (2) 10% decay tokens suffice for convergence, (3) introducing high-quality data during decay substantially outperforms adding it only during SFT, and (4) MiniCPM-2.4B matches or exceeds 7B-class models on a broad benchmark suite. The scaling law ratio of ~192× is the most consequential but also the most caveat-laden finding, and its generalizability beyond the specific models, data, and compute scales studied remains to be established.

6. Limitations and Trade-offs

No LLM-Scale Validation of the Core Scaling Claims

The assumption or constraint. The entire methodological contribution of the paper rests on the premise that training strategies and scaling laws discovered on SLMs (0.009B–2.4B parameters) transfer to LLMs. The wind tunnel metaphor is explicit: "we envision that many aspects of model training are universal across scales" (Section 3). However, the paper explicitly acknowledges in its Limitations section:

"this paper does not extend to training an LLM to validate the scaling law. The application of WSD LRS on LLMs has not been fully explored to date."

This is not a minor caveat — it means the paper's two most ambitious claims (that WSD enables efficient scaling law measurement, and that the resulting ~192× data-to-model ratio challenges Chinchilla Optimal) are supported entirely by experiments on models up to 2.0B parameters. The largest model tested in the scaling law experiments is 2.0B; the largest model trained with WSD end-to-end is 2.4B. The ~192× ratio is therefore a finding about the SLM regime.

The consequence. If the fitted scaling exponents (α = 0.29, β = 0.23) change at larger model scales — and there is precedent for this: Kaplan et al. (2020) and Hoffmann et al. (2022) found different exponents partly because they studied different size ranges — then the compute-optimal ratio of ~192× does not predict the optimal allocation for LLM training. A practitioner training a 70B model cannot use the paper's numbers as a recipe without independent validation at that scale. More fundamentally, the wind tunnel methodology itself remains unproven at the scale where it would be most impactful: if the discoveries at 0.009B–2.4B do not transfer to 70B+, then the entire MWTE framework (Section 3) is valid only for SLM development, not as a general methodology for LLM research. The paper's bidirectional ambition — "SLM experiments inform LLM development, and training strategies developed on SLMs should transfer upwards" — remains aspirational rather than demonstrated.

What evidence exists in the paper. The paper provides transfer evidence up to 2.4B: the optimal learning rate found at 0.009B transfers to 2.1B (Figure 3), and the WSD loss-drop pattern observed at 0.036B reproduces at 2.4B (Figure 12). However, the scaling law itself is fit only to models ≤2.0B (Figure 10). The paper's approximate analysis of Llama 2 (Appendix B.4, Figure 18) provides indirect evidence that the ~192× ratio is closer to modern practice than Chinchilla's ~20×, but this is a rough estimate from published loss curves, not a controlled experiment. The paper does not train even a single 7B model with WSD to verify that the SLM-derived exponents predict its loss.

Mitigation status. The authors are transparent about this limitation in the dedicated Limitations section, calling it out explicitly. They express optimism: "we remain optimistic about its potential advantages." The paper suggests future work on LLM-scale validation but treats this as outside scope. The limitation is therefore acknowledged but unaddressed — it is a gap the paper identifies and leaves for future work. This honesty is commendable, but it means the paper's most consequential claims should be understood as SLM-validated hypotheses rather than established LLM scaling laws.

Difficulty Estimation Cost Is Not Accounted For

The assumption or constraint. The compute-optimal framework described in the referenced MiniCPM paper requires estimating prompt difficulty before selecting the test-time strategy. That paper's method for doing so — generating 2048 samples per prompt and averaging verifier scores — is extraordinarily expensive, consuming more compute than many of the test-time budgets being studied. This paper does not share that specific assumption (it is about pre-training, not test-time compute).

However, a structurally analogous limitation exists in this paper: the cost and practical difficulty of conducting the Model Wind Tunnel Experiments (MWTE) at LLM scale are not discussed. The MWTE methodology requires training dozens of small models to optimize hyperparameters (Section 3.1), measuring batch size scaling curves at multiple model sizes (Section 3.2), verifying learning rate stability across scales (Section 3.3), and training multiple model sizes with multiple checkpoint decays for scaling law measurement (Section 4.5). The paper argues this is cheap at SLM scale — training a 0.009B model on 0.09B tokens costs a tiny fraction of a full training run — but does not discuss whether the same experiments would be feasible or affordable at the scale where the methodology is most needed: LLM-scale models where training costs dominate.

The consequence. If a practitioner wants to apply the MWTE methodology to train a 70B model, they cannot run wind tunnel experiments on 0.009B–0.5B proxies and directly transfer all findings. They would need to verify that the hyperparameters, batch size scaling relationship, and learning rate stability actually transfer to 70B, which requires running verification experiments at intermediate scales (as the paper does up to 2.1B). The cost of these verification experiments scales with model size, and the paper does not provide a cost model for how large the verification models need to be relative to the target. A 70B-scale training run that discovers mid-training that the transferred learning rate is suboptimal would waste far more resources than any MWTE savings. The methodology's practical value therefore depends on an unverified assumption that the transfer properties observed from 0.009B → 2.4B continue to hold at 7B → 70B → 700B.

What evidence exists in the paper. The paper demonstrates transfer up to a ~200× scale factor (0.009B to 2.1B for learning rate stability, Figure 3). There is no evidence beyond this range. The paper does not conduct an error analysis: if the optimal learning rate at 0.009B is 0.01, and at 2.1B it is also 0.01, what is the confidence interval around that estimate? Could it be 0.008–0.012 at 7B, and 0.005–0.020 at 70B? Without characterizing how the optimal hyperparameter distribution evolves with scale, the transfer assumption is qualitative rather than quantitative.

Mitigation status. The paper does not discuss this limitation explicitly. The Tensor Program framework (Yang et al., 2022; 2023) is cited as the theoretical basis for hyperparameter transfer, and the paper presents empirical verification up to 2.1B as evidence. But the paper does not address the question of how much verification is needed at intermediate scales, or what the cost model is for applying MWTE to a new model scale. This limitation is unaddressed — the paper implicitly assumes that transfer continues to hold, and the reader is left to decide whether the 0.009B → 2.4B evidence is sufficient for their use case.

Note: this is a conceptually distinct limitation from "no LLM-scale validation" (Limitation 1). That limitation concerns whether the scaling law findings (~192× ratio) transfer. This limitation concerns whether the MWTE methodology itself is practical and reliable for guiding LLM training, even aside from the specific scaling law numbers.

The Two-Stage Pre-training Strategy Is Tested on Only One Ablation Configuration

The assumption or constraint. The paper's recommendation that "specialization and enhancement of model capabilities should start from the decay phase" (Section 5) is based on a single ablation experiment (Table 1) comparing: decay with pre-training data only vs. decay with pre-training + SFT data mix, followed by SFT. The experiment tests exactly one data composition during decay (the mixture shown in Figure 11, right panel), one decay function (exponential, f(s - T) = 0.5^{(s - T) / T_d}), and one SFT token budget per model size (4B tokens for 2.4B, 6B for 1.2B). The paper does not ablate:

The fraction of SFT data in the decay mixture (Figure 11 right shows ~4.8% "SFT mixed" plus additional specialized sources, but the total SFT-related fraction is not quantified)
The timing of SFT data introduction (e.g., starting SFT data at the beginning of decay vs. mid-decay vs. last portion of decay)
The decay function (exponential vs. linear vs. cosine during decay)
The interaction between decay duration (10% of stable tokens) and SFT data effectiveness (would 20% decay with SFT data help more?)
The dataset dependency — whether the benefit comes from the specific SFT datasets used (UltraChat, SlimOrca, OssInstruct, EvolInstruct, proprietary K12 data) or from any high-quality diverse data

The consequence. A practitioner attempting to replicate the two-stage strategy has no guidance on how to tune the decay-stage data mixture for their own domain. If their SFT data is smaller, noisier, or differently distributed than the paper's mixture, the strong benefits observed in Table 1 (e.g., +12.6 C-Eval points for the 2.4B model) may not materialize. The paper's recommendation to "start from the decay phase" is qualitative; the ablation does not establish that any high-quality data introduced during decay helps, nor does it characterize the dose-response curve (how much SFT data during decay is enough? is there a point of diminishing returns or even negative effects?).

Moreover, the one anomalous result — GSM8K drops from 34.2 (B-1) to 31.8 (B-3) for the 1.2B model when SFT data is added during decay — is not discussed or explained. This is the only benchmark in Table 1 where decay mixing hurts performance (by 2.4 points), and the paper provides no analysis of why. If this is a real degradation rather than noise, it suggests the two-stage strategy may have task-specific tradeoffs that the single ablation does not characterize.

What evidence exists in the paper. Table 1 provides the only evidence for the two-stage strategy. The comparison is internally consistent: A-2 > A-1, B-3 > B-1, and B-3 > B-2 on most benchmarks. But there is no evidence about why these improvements occur beyond the hypotheses stated in Section 5 (more pronounced loss reduction on SFT data distribution, avoiding repetition of small datasets). The GSM8K regression for B-3 is unexplained and suggests the effect may not be uniformly positive.

Mitigation status. The paper does not attempt to address this limitation. The two-stage strategy is presented as a validated finding based on the Table 1 ablation, and the paper recommends it as a general practice: "we recommend that specialization and enhancement of model capabilities should start from the decay phase." No caveats about data composition sensitivity or task-specific tradeoffs are mentioned. This limitation is unaddressed — the ablation is narrow but the recommendation is broad, creating a gap between evidence and claim.

Single Architecture Family with No Architectural Ablations

The assumption or constraint. All experiments in the paper use standard dense Transformer decoders with RoPE positional embeddings, SwiGLU activations, and pre-LayerNorm. The two MiniCPM models differ in width, depth, and attention mechanism (full MHA for 2.4B, GQA for 1.2B), but they share the same fundamental architecture. The paper's methodological contributions — WSD scheduler, MWT hyperparameter optimization, scaling law measurement — are all validated exclusively within this architecture family.

The paper does not perform any architectural ablations: no comparison of WSD effectiveness with different attention mechanisms (multi-query, sliding window), different positional encodings (ALiBi, NoPE), different normalization schemes (post-LN, RMSNorm), or different activation functions. The MWT experiments (Section 3) test hyperparameters within a fixed architecture; the scaling law experiments (Section 4.5) use the same architecture at multiple sizes; the MiniCPM models themselves are architecturally conventional.

The consequence. A practitioner using a different architecture (e.g., Mamba state-space models, mixture-of-experts, or models with different normalization) cannot assume that the optimal learning rate of 0.01, the batch size scaling relationship bs ∝ L^{-6.24}, or the 10% decay duration finding will transfer. The Tensor Program rules (Table 7) are designed for standard Transformer architectures and may not apply to architectures with different scaling properties. The WSD scheduler's effectiveness depends on the optimization dynamics of the specific architecture — if a different architecture has different gradient geometry during decay (different curvature evolution, different gradient cosine alignment patterns), the 10% guideline might not hold.

Even within the paper's own work, this limitation surfaces: MiniCPM-MoE (Section 7.3) uses the WSD scheduler and reuses MiniCPM-2.4B's stable-phase checkpoint for initialization, but the paper does not verify whether the WSD dynamics observed for dense models (loss drop during decay, 10% sufficiency) apply to the MoE architecture with its additional routing mechanism and load balancing loss. The MoE results (Table 6) are presented without WSD-specific validation.

What evidence exists in the paper. None. The paper does not include any architectural variation experiments. The closest is the comparison of with/without QK-Norm in the hyperparameter search (Figure 14, Appendix A.1), but this is a minor modification to a single component, not an architectural variant. The MiniCPM family includes architectural variants (GQA, MoE) but these are presented as trained models, not as controlled architecture ablations for the WSD methodology.

Mitigation status. The paper does not discuss this limitation. The methodology is presented as general — "our training methodology is meticulously designed to facilitate seamless scaling of both model scale and data horizons" (Section 1) — without qualification about architectural dependence. The wind tunnel metaphor implies architecture-agnosticism (aircraft of different designs all benefit from wind tunnel testing), but the paper provides no evidence that the specific findings (lr = 0.01, 10% decay, etc.) are architecture-independent. This limitation is unaddressed.

The Scaling Law Measurement Excludes Embedding Parameters and Uses a Specific Tokenizer-Dependent Loss Normalization

The assumption or constraint. The scaling law experiments (Section 4.5) measure model size N as non-embedding parameters only. Embedding parameters (which add 0.3B and 0.2B for the 2.4B and 1.2B models respectively) are excluded from N in the L(N, D) fitting. This choice is made explicit in Table 2 and the model size labels throughout Section 4.5. The paper uses the formula C = 6ND for FLOPs, which is standard (Rae et al., 2021), but N is defined as non-embedding parameters.

Additionally, to compare loss across models with different tokenizers, the paper normalizes loss "by a number of bytes instead of a number of tokens, following Achiam et al. (2023)" (Section 4.5). This means the reported losses and the fitted L_0 (irreducible loss) are in units of nats per byte, not the more common nats per token. The conversion between these units depends on the tokenizer's compression ratio, which varies across models (Table 9 shows compression ratios from 2.84 to 3.40 bytes/token).

The consequence. The compute-optimal ratio of ~192× is sensitive to both choices. If embedding parameters were included in N, the model size would increase (for a given architecture), which would increase C for a fixed D (since C = 6ND), shifting the computed optimal ratio. More subtly, the relationship between non-embedding parameters and total parameters changes with vocabulary size — MiniCPM uses unusually large vocabularies (122,753 and 73,440 tokens), so excluding embedding parameters removes a larger fraction of total parameters than it would for a model with a smaller vocabulary (e.g., Llama 2's 32,000 tokens). This means the paper's ~192× ratio is for a specific architecture with specific vocabulary sizes; a practitioner with a different vocabulary size would get a different effective N/D ratio even at the same total parameter count.

The byte-normalized loss metric, while principled for comparing across tokenizers, complicates direct comparison with prior scaling law work. Hoffmann et al. (2022) report loss in nats per token, not per byte. The conversion factor is the tokenizer's compression ratio, which is dataset-dependent. The paper's fitted L_0 values (e.g., 0.31 for the average corpus) cannot be directly compared to Chinchilla's L_0 without knowing the tokenizer efficiency. This makes it harder to assess whether the higher optimal ratio is due to better training (the paper's claim), better data, or simply a different loss normalization convention.

What evidence exists in the paper. The paper is transparent about both choices: Table 2 specifies non-embedding parameters, and Section 4.5 explicitly states the byte-normalization rationale. Table 9 provides compression ratios for several tokenizers, showing that MiniCPM-120K achieves 3.40 bytes/token on average. The paper does not provide an ablation showing how the scaling law parameters change if total parameters are used instead of non-embedding, or if token-normalized loss is used instead of byte-normalized loss.

Mitigation status. The paper is transparent about the choices but does not discuss their impact on the headline ratio. A practitioner reading "192× more data than parameters" needs to know that "parameters" means non-embedding parameters for a model with a 122K-token vocabulary. For a model with a 32K-token vocabulary, the equivalent ratio in total parameters would be different. The paper's approximate Llama 2 validation (Appendix B.4) partially mitigates this concern — the estimated 70–100× ratio for Llama 2 is in the same order of magnitude as the paper's ~192× — but the validation is rough and does not disentangle the effects of architecture, tokenizer, data quality, and training methodology. This limitation is partially addressed via the Llama 2 analysis, but the sensitivity of the findings to the parameter-counting and loss-normalization conventions is not characterized.

The Long-Context Model Is Evaluated as a Base Model Against Instruction-Tuned Baselines, and Performance on Code/Math Long-Context Tasks Is Poor

The assumption or constraint. MiniCPM-128K (Section 7.2) is presented as a long-context extension of the MiniCPM-2.4B base model. The paper does not describe any SFT or alignment phase specific to the long-context variant. However, the baselines it is compared against in Table 5 include instruction-tuned models: Mistral-7B-Instruct-v0.2 (ABF 1000w), ChatGLM3-6B-128K, and Yi-6B-200K. The comparison is therefore between a base model (MiniCPM-128K) and instruction-tuned models designed for chat/assistant interaction.

The consequence. The comparison is fundamentally uneven. Instruction-tuned models have been optimized for following instructions, generating coherent long-form responses, and handling diverse task formats — all of which are tested in ∞Bench's long-context tasks (Long Book QA, Long Dialogue QA, summarization). A base model evaluated on the same tasks may exhibit different failure modes (e.g., generating continuations rather than answers, failing to follow output format requirements) that are not indicative of its underlying long-context capability but rather of its lack of instruction following. The paper's claim that MiniCPM-128K "achieve[s] comparable results in Mistral-7B-Instruct-v0.2" (Section 7.2) should be qualified: it is comparable despite being a base model, which could be interpreted either as impressive (it matches instruction-tuned models without instruction tuning) or as misleading (the comparison is between models designed for different use cases).

Furthermore, the claim that MiniCPM-128K demonstrates "performance either surpassing or matching" (Abstract, Section 1) is qualified by the paper's own analysis: this holds when code and math tasks are excluded. On Math Calc, MiniCPM-128K scores 4.29 vs. Mistral-7B-Instruct-v0.2's 29.43 — a factor of ~7 worse. On Code Debug, 22.08 vs. 17.51 (comparable), but on Code Run, both score 0. The average including code/math is 27.32 for MiniCPM-128K vs. 27.75 for Mistral — essentially tied. The "surpassing" claim relies on the average excluding code and math (37.68 vs. 36.90), which the paper does report but the abstract does not qualify.

What evidence exists in the paper. Table 5 provides the full per-task breakdown, clearly showing the code/math deficiencies. The paper reports two averages (all tasks and without code/math) and notes which the "surpassing" claim refers to. However, the training details for MiniCPM-128K (Section 7.2) mention using "synthetic long QA data that significantly enhances model performance in context-aware tasks" but do not describe any instruction tuning. The evaluation section does not specify whether MiniCPM-128K was evaluated using the same prompting format as the instruction-tuned baselines.

Mitigation status. The paper does not explicitly address the base-model-vs-instruct comparison issue. The long-context training description mentions using the WSD scheduler and reusing MiniCPM-2.4B's stable checkpoint, which implies base model training. The two averages in Table 5 partially address the code/math weakness by showing results both with and without those tasks, but the abstract and introduction claims do not carry this qualification. This limitation is partially addressed in the detailed results but not in the headline claims. A practitioner interested in long-context capability should note that the code and math performance is substantially below the instruction-tuned baseline, and that an aligned version of MiniCPM-128K might or might not close this gap.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not propose a single breakthrough architectural innovation or a new state-of-the-art benchmark score — and that is precisely what makes it significant. The contribution is methodological infrastructure: a set of tools and experimental protocols that change how training research can be conducted rather than what model achieves the highest score. Specifically, the paper makes three landscape-shifting contributions that together constitute a reframing of the relationship between small-model research and large-model development.

First, the WSD scheduler transforms learning rate scheduling from a black-box hyperparameter choice into a legible, decomposable training strategy. Prior to this work, Cosine LRS was dominant by convention — everyone used it because Kaplan et al. (2020) and Hoffmann et al. (2022) established it as a strong default, but why it worked and what degrees of freedom it exposed were poorly characterized. The paper's demonstration that Cosine LRS implicitly balances two distinct optimization regimes — an extended high-learning-rate exploration phase and a thorough decay-to-convergence phase — and that making this separation explicit (via WSD) loses no performance while enabling entirely new capabilities (continuous training, checkpoint reuse, independent control of phase durations) fundamentally changes what it means to "schedule" training. It is no longer about picking a single curve shape; it is about allocating optimization behavior across time. The gradient dynamics analysis (Section 4.4, Figure 8) provides concrete, measurable signatures of the two regimes — gradient cosine alignment, curvature increase by an order of magnitude — that give researchers diagnostic tools they did not have before. This reframing is significant because it opens a new class of research questions: what is the optimal stable-to-decay ratio for different architectures? Can the model detect when it should transition from exploration to exploitation? Does the decay function shape interact with data distribution? These questions were invisible under the Cosine paradigm.

Second, the linear-cost scaling law measurement protocol challenges a fundamental assumption about the economics of scaling research. The implicit assumption in prior scaling law work (Kaplan et al., 2020; Hoffmann et al., 2022) was that measuring L(N, D) requires training each (model size, data size) pair from scratch, which costs O(m²) for m model sizes — prohibitive for m larger than a handful, forcing strong parametric assumptions and sparse data. The paper demonstrates that WSD reduces this to approximately O(m), making it feasible to sample the (N, D) plane densely. This is not merely a computational convenience; it is an epistemological shift. It means the central empirical question of LLM research — "given a compute budget, how should I allocate it between model size and data?" — can now be answered with granularity that was previously impossible. The paper's ~192× data-to-model ratio finding is eye-catching, but the deeper contribution is that the ratio can now be measured routinely, per-architecture, per-data-distribution, per-tokenizer, rather than being treated as a universal constant. If the field adopts WSD-based measurement as a standard protocol, we could accumulate a library of scaling law parameters across architectures, data mixtures, and training configurations, replacing the current situation where three influential papers disagree by an order of magnitude on the most basic question in LLM scaling. This would transform scaling law research from a handful of heroic one-off measurements into a systematic empirical science.

Third, the two-stage data strategy establishes a principle — capability injection during low-learning-rate exploitation — that challenges the standard pre-train-then-SFT pipeline. The finding that introducing high-quality data during the decay phase produces dramatically larger benchmark improvements than adding equivalent or double the data during SFT alone (Table 1: +12.6 C-Eval points for the 2.4B model) is not just a practical optimization. It implies that the temporal coupling of data quality and learning rate is a first-order determinant of final model capability, not a secondary hyperparameter. This reframes capability acquisition: general competence is built during high-LR exploration on diverse data; specialized capabilities are injected during low-LR exploitation on targeted data. The standard practice of separating pre-training (diverse data, decaying LR) and SFT (targeted data, low LR) conflates the optimization regime with the data type in a suboptimal way — pre-training's late-stage decay is wasted on general-domain data when it could be used for capability injection. The paper's recommendation that "specialization and enhancement of model capabilities should start from the decay phase" (Section 5) provides a new default template: pre-train with general data at high LR, begin mixing in high-quality data when LR starts decaying, and finish with a brief SFT-only polishing phase. This is not a radical departure from existing practice but a re-temporalization of when data quality is introduced, with outsized practical impact.

Reconciling prior contradictions. The paper indirectly reconciles a tension that has simmered in the scaling law literature: why do Kaplan et al. (2020) find a ~10× optimal ratio, Hoffmann et al. (2022) find ~20×, and modern practice (Llama 2, Chinchilla) suggests ratios of 70–100× or higher? The paper's analysis (Section 4.5 and Appendix B.4) suggests the answer is neither that earlier work was wrong nor that scaling laws are unstable, but that the optimal ratio depends on training configuration — data quality, deduplication, architecture, tokenizer — and that these configurations have improved over time. The WSD protocol makes this dependence measurable: you can hold architecture and data fixed, vary only the training schedule, and see how the ratio changes. The field's disagreement about the "right" ratio may simply reflect that different papers measured it under different implicit configurations. WSD makes those configurations explicit and controllable, potentially resolving the disagreement by showing it was never a single number to begin with.

Research directions that become more attractive. The WSD scheduler makes any research that requires measuring L(N, D) at multiple scales substantially cheaper and more granular. This includes: data mixing law studies (how does the optimal ratio vary with data composition?), architecture scaling studies (do different architectures have different α, β exponents?), multi-lingual scaling studies (how does the ratio differ across languages?), and compute-optimal inference studies (Sardana & Frankle, 2023 — given that a small model can absorb far more data than previously thought, what is the inference-compute-optimal model size?). It also makes continuous pre-training a viable paradigm: models can be trained with WSD, deployed, and later continued with more data without restarting, enabling models that improve over their deployment lifetime.

Research directions that become less attractive. The paper's findings reduce the appeal of: (1) cosine-tuned training runs where checkpoints are discarded — the explicit separation of stable and decay phases makes checkpoint reuse a first-class capability; (2) assuming that small models saturate quickly — the continuous training experiment (Figure 6) shows a 0.036B model still improving at 320N tokens, challenging the intuition that small models have a low ceiling; (3) treating the data-to-model ratio as a solved problem — the paper's ~192× finding, combined with the ~8× variation across datasets, suggests the ratio is highly configurable and should be measured rather than assumed.

Magnitude and specificity of the shift. This is not a paradigm shift in the Kuhnian sense — it does not overturn a core theory or introduce a new model class. It is better characterized as a methodological reframing with practical consequences. The underlying components (Cosine LRS, Tensor Program, scaling law measurement) all existed. The paper's contribution is to show how they can be recombined — by decomposing Cosine into exploration/exploitation, by using checkpoint reusability for scaling law measurement, by coupling data quality scheduling to LR scheduling — into a coherent experimental methodology that changes what questions can be asked and at what cost. The impact will depend on adoption: if the field adopts WSD as a standard training and measurement protocol, the paper's influence could be substantial (enabling a new generation of scaling law research); if Cosine LRS remains dominant by inertia, the paper's influence will be limited to the specific MiniCPM models it produced.

Follow-Up Research This Work Enables

LLM-scale validation of the WSD-derived scaling law parameters. The paper's most significant open question is whether the fitted exponents (α = 0.29, β = 0.23) and the ~192× optimal ratio transfer to LLM scale (7B–70B+). A direct follow-up would: train a single 7B model with WSD LRS using the same architecture family, data mixture, and tokenizer as the MiniCPM scaling experiments; save stable-stage checkpoints at multiple token counts; decay each for 10%; measure loss on the same five evaluation corpora; and compare the observed L(N=7B, D) values against the predictions of the SLM-fitted L(N, D) function. If the SLM-derived parameters predict the 7B loss within, say, 5%, this provides strong evidence that the wind tunnel methodology scales. If they systematically over- or under-predict, refitting the parameters including the 7B data point would reveal whether the exponents change with scale (e.g., does β increase at larger model sizes, changing the optimal ratio?). This experiment costs one 7B training run plus short decays — feasible for many academic labs — and would directly address the paper's acknowledged limitation.

Automated transition detection from stable to decay phase. The gradient dynamics analysis (Section 4.4) shows that the decay phase has identifiable signatures: gradient cosine becomes consistently positive, loss curvature increases by an order of magnitude. An obvious follow-up question: can these signatures be used to automatically determine when to transition from stable to decay training, rather than relying on a predefined step count? A concrete experiment: train a model with WSD at constant high LR; continuously monitor the gradient cosine between consecutive steps, the loss curvature along the optimization trajectory, and the gradient norm; define a criterion (e.g., "transition when gradient cosine exceeds 0.5 for 100 consecutive steps and curvature has been increasing for 200 steps"); compare the resulting automatically-timed decay against manually-timed decay at 40N, 80N, 160N tokens. This would test whether the model's own optimization geometry provides a signal for when exploration has diminishing returns and exploitation should begin. If the automatic transition matches or exceeds manual timing, it would make WSD fully adaptive — no predefined training length needed at all.

Data quality × decay phase interaction study. The two-stage pre-training strategy (Section 5) shows that introducing high-quality data during decay substantially improves downstream task performance, but the experiment tests exactly one data composition. A systematic follow-up would vary: (a) the fraction of SFT data in the decay mixture (0%, 2%, 5%, 10%, 20%, 50%), (b) the data source (synthetic instruction data vs. human-written vs. textbook-style vs. code-only), and (c) the timing of introduction (first half of decay, second half, uniformly throughout). The dependent variables would be benchmark performance (C-Eval, MMLU, GSM8K, HumanEval) and pre-training loss retention (does adding SFT data during decay hurt general language modeling performance?). This would produce a dose-response curve for capability injection during decay, answering: is there an optimal fraction beyond which SFT data crowds out pre-training data and hurts general capabilities? Does the benefit come from data diversity or data quality? The GSM8K regression observed for B-3 (31.8 vs. 34.2 for B-1) in the 1.2B model — the one case where decay mixing hurt — could be investigated by isolating whether specific SFT data sources (e.g., code-heavy data) interfere with math reasoning when introduced during decay.

Architecture dependence of WSD dynamics and optimal ratios. The paper validates WSD on standard dense Transformer decoders with RoPE and SwiGLU. A natural follow-up tests whether the key findings — 10% decay sufficiency, ~192× optimal data-to-model ratio, gradient cosine alignment during decay — generalize to other architectures. Concrete comparisons: (a) a dense Transformer with ALiBi positional encoding (which modifies attention patterns and may change optimization geometry), (b) a mixture-of-experts architecture (the paper's own MiniCPM-MoE uses WSD but is not analyzed for decay dynamics), (c) a non-Transformer architecture such as Mamba or RWKV. For each, train matched-size models (~0.3B parameters) with identical data and WSD schedule; measure loss during decay, gradient statistics (as in Figure 8), and compute optimal data-to-model ratio. If certain architectures (e.g., MoE) require longer or shorter decay phases, or show different optimal ratios, this would establish that the WSD parameters are architecture-dependent — a crucial qualification to the paper's general claims.

WSD for domain-adaptive continuous pre-training. The paper emphasizes WSD's support for "continuous training with an unpredefined pre-training token number" (Section 1), but does not demonstrate a scenario where training is extended with new data after deployment. A practical follow-up would simulate domain adaptation: pre-train a model with WSD on general web data to a certain stable checkpoint; then, instead of decaying on the same data distribution, continue the stable phase with domain-specific data (e.g., scientific papers, legal documents, or a new language) for an additional 20N tokens; then decay on a mixture of general and domain data. Compare against: (a) pre-training from scratch on the general+domain mixture with Cosine LRS, (b) fine-tuning the pre-trained model on domain data with a low learning rate. The results would test whether WSD's ability to "continue training with the previous high learning rate" (Section 4.3) enables more effective domain adaptation than standard fine-tuning, and whether the two-stage data strategy (general data during stable phase, domain data introduced during decay) transfers to domain-specific capability injection.

Measuring scaling laws across tokenizers and compression ratios. The paper normalizes loss by bytes to compare across tokenizers (Section 4.5), but the impact of tokenizer choice on the optimal data-to-model ratio is not isolated. A focused ablation: train identical-architecture 0.3B models with three different tokenizers — MiniCPM-120K (3.40 bytes/token), Llama2-32K (2.84 bytes/token), and a character-level tokenizer (~1 byte/token) — using WSD on the same pre-training corpus; measure L(N, D) for each; compute the optimal ratio. This would test whether the ~192× finding is driven by the tokenizer's compression efficiency. If the character-level tokenizer (which produces many more tokens for the same text) shows a substantially different optimal token-to-parameter ratio, it would imply that the optimal ratio should be expressed in bytes rather than tokens for cross-tokenizer comparability — a small but important refinement to scaling law methodology.

Negative result study: limits of continuous training. The continuous training experiment (Figure 6) shows a 0.036B model still improving at 320N tokens, but the fitted envelope is a power law with an asymptote L_0 — at some point, additional training must stop helping. A follow-up would train a small model (e.g., 0.036B) with WSD to extreme token counts (500N, 1000N, 2000N) to empirically locate the saturation point — where additional training tokens reduce loss by less than, say, 0.01 nats/byte per doubling. Comparing this saturation point against the L_0 estimated from the scaling law fit would validate whether the power-law form extrapolates reliably, or whether loss plateaus more abruptly than the fit predicts. This experiment also tests the paper's implied claim that small models can absorb "far more data than previously assumed" — is there a hard ceiling, or does improvement continue indefinitely with diminishing returns? The answer has direct implications for the inference-compute tradeoff: if a 0.036B model ultimately requires 1000× more training compute to match a 0.17B model, the inference savings per call (~5×) may not amortize the training cost over a realistic deployment lifetime.

Practical Applications and Downstream Use Cases

Cost-efficient scaling law measurement for custom data mixtures and architectures. The most immediate practical application of this work is that organizations training LLMs can now measure their own optimal data-to-model ratio rather than relying on published numbers from different configurations. The protocol: train a small proxy model (e.g., 0.1B parameters) with WSD LRS on the target data mixture; save checkpoints at 10N, 20N, 40N, 80N, 160N tokens; decay each for 10%; evaluate loss; fit L(N, D) for that model size; then repeat for 1–2 additional model sizes to fit the model-size exponent. The total cost is approximately 2–3 full training runs on small models — a tiny fraction of a single LLM training run. This produces a customized scaling law that accounts for the organization's specific data composition, deduplication pipeline, tokenizer, and architecture, enabling informed decisions about how large a model to train and how much data to collect. The paper's finding that the optimal ratio varies from ~96× to ~834× depending on the evaluation corpus (Figure 10) underscores why off-the-shelf scaling laws are insufficient — a team training on Chinese-heavy data should not use a ratio derived from English-heavy data. The WSD protocol makes customization practical.

Training budget allocation between model size and data quantity with updated ratios. For organizations planning a single large training run, the paper's ~192× optimal ratio — or their own WSD-derived variant — provides a concrete starting point for budget allocation. A concrete example: if a team has a compute budget of C = 10^{22} FLOPs (roughly the compute used for a 7B model trained on 1T tokens), and they adopt the paper's average fitted parameters (α = 0.29, β = 0.23, D_opt/N_opt ≈ 192 at C = 10^{21}), they would allocate N ≈ 2.9B parameters and D ≈ 560B tokens (solving N/D = 1/192 and C = 6ND). This is a substantially smaller model with substantially more data than the Chinchilla Optimal allocation (~7B parameters, 200B tokens for the same compute). The resulting model would be cheaper to serve (fewer parameters) and trained on more data, which, per the paper's findings, should achieve lower loss than the Chinchilla allocation. This is directly actionable today for teams that trust the SLM-derived exponents to transfer — and the WSD protocol gives them the tool to verify or refine the numbers for their specific setup.

Deployment-efficient SLMs for on-device and edge computing. The paper's demonstration that MiniCPM-2.4B can be quantized to 4-bit (2GB footprint) and run at 18 tokens/second on an iPhone 15 Pro (Appendix E, Table 10) — while matching the benchmark performance of 7B-class models — makes a concrete case for on-device deployment. For applications requiring privacy-sensitive inference (personal assistants, on-device email summarization, local code completion) or offline operation (field-deployed translation, navigation in areas without connectivity), a 2GB model that runs on consumer hardware and achieves Mistral-7B-level accuracy eliminates the need to send data to cloud APIs. The paper's finding that "a 0.036B model can match the performance of a 0.17B model with an acceptable increase (~4 times) in training compute while saving a lot of inference computation (saving ~5 times per inference call)" (Section 4.3) provides a principled basis for this tradeoff: train small, train long, save on every inference. For a deployment serving millions of queries, the inference savings from a 2.4B vs. 7B model far outweigh the extra training tokens.

Continuous pre-training pipelines for evolving data distributions. The WSD scheduler's support for "continuous training with an unpredefined pre-training token number" (Section 1) enables a deployment pattern that was impractical with Cosine LRS: a model can be trained to a stable checkpoint, deployed, and later continued with new data as the data distribution evolves. For example, a code completion model trained on GitHub data through 2024 could be continued in 2025 with new repositories by resuming the stable training phase from the 2024 checkpoint, then re-decaying. Because the stable checkpoint was trained at the optimal constant learning rate (not mid-decay, as a Cosine checkpoint would be), the continuation is mathematically identical to having trained on the combined 2024+2025 data from the start. This avoids the common problem where fine-tuning on new data causes catastrophic forgetting of old data — the stable phase uses the full learning rate, allowing the model to genuinely integrate new information rather than surface-level adaptation. For any application where the data distribution shifts over time (news, code, scientific literature, social media), WSD-trained models can be updated incrementally without restarting from scratch.

When to Prefer This Method

The paper proposes WSD LRS and the associated two-stage pre-training strategy as general methodologies for language model training, not as special-purpose techniques for a specific regime. It does not position WSD against a named alternative with explicit tradeoff conditions — it argues that WSD matches or exceeds Cosine LRS in all tested scenarios while enabling additional capabilities (continuous training, checkpoint reuse, efficient scaling law measurement). The paper's implicit recommendation is that WSD with 10% decay should replace Cosine LRS as the default scheduler, particularly when any of the following conditions hold:

You may want to extend training beyond the initially planned budget (new data becomes available, more compute is allocated, or you want to explore "what if we trained longer?"). Cosine LRS locks you into a predefined endpoint; WSD lets you continue the stable phase and re-decay.
You need to measure scaling laws for your specific data, architecture, or tokenizer. WSD's checkpoint reusability reduces scaling law measurement from quadratic to approximately linear cost, making it practical where Cosine-based measurement would be prohibitive.
You plan to introduce high-quality specialized data during training. The two-stage strategy (high-quality data during decay) substantially outperforms adding the same data only during SFT (Table 1), making WSD's explicit decay phase a natural window for capability injection.
You value intermediate checkpoints as reusable assets. WSD stable-stage checkpoints are trained at the optimal learning rate for their current data volume, making them suitable for downstream tasks, continued training, or analysis. Cosine checkpoints are tied to a specific position on the decay curve and are not independently optimized.

The paper does not identify a regime where Cosine LRS would be preferable to WSD — the experiments show WSD matching Cosine's performance while adding capabilities. However, practitioners should note that WSD has an additional hyperparameter (the stable-to-decay ratio) that requires tuning (the paper finds 10% decay tokens sufficient for their setup, but this may vary). For teams running a single training run with a fixed budget and no need for checkpoint reuse or continuous training, Cosine LRS remains a simpler drop-in choice with extensive community validation. The paper's case is not that Cosine is bad, but that WSD is strictly more capable at the cost of one additional decision (decay duration), and that the benefits (reusability, continuous training, scaling law measurement, improved data scheduling) more than justify this cost for any team doing systematic training research or planning multiple training runs.