ArXiv: 2602.02472
🎯 Pitch
Unlike prior progressive learning methods that only expand model width at the very start of training—sacrificing most potential cost savings—SPARKLING enables stable 2× width expansion halfway through pre-training. It solves the resulting catastrophic loss spikes by preserving activation statistics via RMS rescaling, while counterintuitively boosting performance by deliberately injecting gradient asymmetry to break the feature copying deadlock. This mid-stage approach slashes training cost by up to 35% with no loss in final model quality.
1. Executive Summary
This paper proposes SPARKLING (balancing Signal Preservation And symmetry breaking for width-progressive Learning), a framework for expanding model width midway through pre-training—a regime prior progressive-learning work avoids due to severe instability. The experiments train OLMoE-style MoE models on 200B tokens, expanding width at 100B tokens across three axes: expert intermediate dimension, hidden dimension, and both jointly. SPARKLING operationalizes two complementary principles—Signal Preservation via RMS-scale consistency (rescaling fan-in weights to preserve activation statistics across expansion) and Symmetry Breaking via asymmetric optimizer-state resetting and asymmetric learning rate re-warmup (selectively stimulating new parameters while leaving the original parameters’ schedule intact). Under 2× width expansion, SPARKLING matches or exceeds the downstream performance of the model trained from scratch at the target width while reducing training cost by up to 35% (a 1.49× wall-clock speedup on joint expansion), establishing that mid-stage width growth is practically viable—but only when both forward activation stability and backward gradient coupling are explicitly managed.
2. Context and Motivation
The Core Problem: Mid-Stage Width Expansion Is Both Essential and Unstable
Progressive Learning (PL) offers a compelling alternative to training large models from scratch: start with a small model, then expand it partway through training, so that the total FLOPs spent on smaller parameters early on translates to computational savings. The existing PL literature has focused overwhelmingly on depth expansion—stacking layers, inserting blocks, or gradually growing the network deeper (Gong et al., 2019; Kim et al., 2024; Du et al., 2024; Wu et al., 2024). These methods have demonstrated notable success, but they address only one axis of model scaling. Width—the hidden dimension, the number of attention heads, the intermediate size of feedforward or MoE expert layers—is the other fundamental dimension for scaling parameter count (Kaplan et al., 2020), and it has received far less systematic attention in the PL context.
The gap is not accidental: width expansion during pre-training is genuinely harder than depth expansion, and the difficulty is sharply concentrated in the regime where it matters most. To understand why, we need to be precise about the timing of expansion within the training trajectory.
Why Early Expansion Undermines the Point of Progressive Learning
Consider what it means to expand partway through pre-training. If a model is trained for total tokens and expansion happens at tokens, then the small model trains for tokens and the expanded model trains for tokens. The computational savings come from the fact that : you spend most of the pre-training budget on a cheaper architecture. But this only yields meaningful savings if is a substantial fraction of —if you expand at 5% of total tokens, the small model barely gets a chance to amortize its lower cost, and the savings are negligible. For PL to be practically worthwhile, expansion must occur in the middle of training, not the beginning.
The paper makes this point explicitly:
"previous investigations have been largely limited to expansion during the initial portion of training, e.g., less than 10–30% of training tokens... Such early expansion offers negligible computational advantages over training the target-width model from scratch and fundamentally undermines the primary motivation of PL—reducing training costs."
And yet, the few prior works that address width expansion cluster precisely in this early-training regime (Du et al., 2024; Shen et al., 2022). The reason is no accident: expanding width at 50% or 70% of training is empirically unstable. The paper reports that naive width expansion at mid-stage triggers loss spikes and poor recovery, which prior researchers presumably observed and then backed away from, choosing to expand earlier when the optimization landscape is more forgiving. This creates a frustrating situation: width expansion is studied only in the regime where it's least useful, while the regime where it's most useful remains unexplored and widely considered risky.
The Two Hidden Mechanisms Behind Mid-Stage Instability
The paper identifies two distinct, interacting failure modes that emerge specifically when width is expanded deep into pre-training. Neither is obvious from the existing PL literature because prior work either expanded too early to encounter them or addressed only one in isolation.
Failure Mode 1: Activation-Scale Mismatch (Signal Disruption). When a model has trained for billions of tokens, its internal representations settle into a finely tuned operating regime. Each layer's input and output activations have characteristic root-mean-square (RMS) magnitudes; residual connections are balanced so that the main stream and branch outputs mix in calibrated proportions. If a width expansion perturbs these RMS scales—for example, by naively appending randomly initialized new channels to a weight matrix without adjusting for the changed fan-in or fan-out dimensionality—the post-expansion hidden states drift away from the pre-expansion scale manifold. Subsequent layers receive out-of-regime inputs, and the residual dynamics become either overwhelmed (the branch output dominates the stream) or nearly identity (the branch is too weak to contribute). The paper notes:
"RMS-scale mismatch alters layerwise signal magnitudes and propagates through residual streams; this destabilizes optimization even when no instantaneous loss spike occurs at the moment of expansion."
This is a subtle but critical point: the damage from RMS mismatch may not register as an immediate loss spike. The forward function might be perfectly preserved at the instant of expansion, but the statistics that the optimizer relies on are corrupted, and the model degrades over subsequent steps. This explains why naive "function-preserving" initializations that produce loss continuity at the expansion point can nonetheless lead to worse final convergence (Figure 1): they preserve the output but not the internal activation statistics that govern long-term optimization dynamics.
Failure Mode 2: Gradient Symmetry Under Copy-Based Expansion. The natural instinct for width expansion is to copy existing channels: duplicate rows or columns of weight matrices so that the forward function is perfectly preserved. This is the standard "function-preserving" approach from Net2Net (Chen et al., 2016) and has been widely adopted in subsequent width-growth works (Chen et al., 2022; Shen et al., 2022; Wang et al., 2024; Han et al., 2025). The appeal is clear: if the expanded model produces exactly the same output as the pre-expansion model, there is no loss discontinuity and training can resume smoothly.
However, the paper identifies a fundamental limitation: copying preserves the forward function but creates duplicated parameter subspaces in the backward pass. When a weight row or column is duplicated, the two copies receive identical gradients because they process the same inputs and contribute symmetrically to the output. With symmetrically initialized optimizer states (either copied or reset to zero), the two copies receive identical updates at every step, remaining permanently coupled. The paper calls this a symmetry lock:
"despite increased parameters, the model remains in the original lower-dimensional subspace. The expanded neurons fail to learn distinct features, making width scaling inefficient unless the symmetry is explicitly broken."
This symmetry is not optimizer-specific. The paper proves that both element-wise optimizers (AdamW) and spectral-style optimizers (Muon, which uses Newton-Schulz orthogonalization) preserve the symmetry under copy initialization (Section 4.1). The orthogonalization step in Muon—often assumed to decorrelate updates—cannot break the symmetry because it operates as a polynomial map of the Gram matrix, and the Gram matrix of duplicated columns remains block-constant. This is a non-obvious theoretical result that explains why prior heuristics (uneven splitting, symmetric perturbations) provide only partial mitigation: they introduce some initial asymmetry but don't fundamentally alter the coupled update dynamics that keep copied subspaces aligned.
Why This Problem Matters
The significance of mid-stage width expansion spans both practical deployment and scientific understanding.
Practical: Training Cost at Scale. Pre-training frontier LLMs costs tens to hundreds of millions of dollars in compute. A 35% reduction in training FLOPs (the paper's reported saving under joint width expansion at 2×, Table 2) translates to substantial absolute savings. But the savings are only realizable if width expansion can be performed reliably in mid-training; if it's restricted to early training (where the cost reduction is negligible), width-progressive learning provides no practical benefit over training from scratch. The paper establishes that such savings are achievable, but only with the right combination of interventions.
Methodological: Width as a Neglected Scaling Axis. The PL literature is heavily skewed toward depth. This is partly historical (depth expansion is easier to implement stably) and partly conceptual (depth is often associated with representational capacity in a more intuitive way). But width matters enormously in modern architectures: Mixture-of-Experts models scale primarily through width (more experts, larger expert hidden dimensions, wider attention), and the Transformer's expressivity depends critically on the dimensionality of key, query, and value projections. A PL framework that can only grow depth but not width is fundamentally incomplete for modern LLM training. The paper takes a step toward filling this gap.
Scientific: Understanding the Limits of Function Preservation. The function-preserving paradigm (Chen et al., 2016) has been influential because it provides a clean mathematical guarantee: if the expanded model computes exactly the same function, training can resume without any discontinuity. The paper challenges this paradigm by showing that function preservation is neither necessary nor sufficient for successful width expansion. It is not necessary because RMS-preserved random or zero initialization often outperforms copying (Figure 1f), and it is not sufficient because copy-based expansion suffers from symmetry lock that function preservation does not address. This reorients the research question from "how do we perfectly preserve the forward function?" to "how do we preserve the optimization regime while enabling the expanded capacity to diversify?"—a more nuanced framing that connects PL to broader questions in optimization dynamics and feature learning.
Prior Work: What Exists and Where It Falls Short
The paper organizes prior approaches along two axes—forward initialization and backward optimization dynamics—and argues that existing work addresses at most one of these in isolation.
Forward-Initialization Approaches
Function-Preserving Parameter Mapping (Net2Net). The foundational idea (Chen et al., 2016) is to copy existing neurons and then adjust subsequent layer connections to exactly preserve the mapping. Variants include advanced initialization schemes like AKI (Chen et al., 2022), which aim to provide better starting points for the new parameters beyond simple copying. More recent work applies these ideas to MoE models (Zhang et al., 2024) and to various Transformer architectures. However, these methods are evaluated almost exclusively at early expansion points (within the first 10–30% of training) and do not address the statistical and symmetry challenges that emerge during mid-stage expansion.
Temporary Masking. Some approaches (Yao et al., 2024) introduce new parameters gradually by masking them out initially and incrementally unmasking during subsequent training. While this provides a softer transition, it doesn't resolve the fundamental redundancy issue: when masks are finally removed, the copied parameters may still be symmetric. Moreover, the masking schedule introduces additional hyperparameters and doesn't address the RMS-scale mismatch that can destabilize downstream layers.
Heuristic Symmetry-Breaking at Initialization. Recognizing that pure copying creates redundancy, several works introduce asymmetric perturbations at expansion time: uneven splitting (assigning different scaling factors to original and copied channels; Chen et al., 2016; Wang et al., 2024; Du et al., 2024), symmetric ± perturbations that cancel in the forward pass (Yuan et al., 2023; Wu et al., 2021; Liu et al., 2019), or weight-norm rebalancing to equalize gradient contributions (Yuan et al., 2023). The paper evaluates these heuristics (Appendix 14, Figure 9) and finds them consistently weaker than the SPARKLING framework. The reason, in hindsight, is that one-shot initialization asymmetry is quickly washed out by subsequent symmetric gradient updates; what's needed is persistent asymmetry in the optimization dynamics, not just the initial state.
Backward-Optimization Approaches
Accelerated Decay Schedules. Wang et al. (2024) propose using faster learning rate decay for expanded models, arguing that they start closer to local optima and thus need less exploration. While intuitively plausible, this treats all parameters uniformly and doesn't address the specific coupling between copied subspaces.
Dynamics-Preserving Growth Operators. Shen et al. (2022) design expansion procedures that aim to preserve not just the forward function but also the loss trajectory (gradient dynamics) through the expansion point. This is a step toward considering backward dynamics, but it operates only at initialization and doesn't provide ongoing mechanisms to maintain diversity.
Learning Growth Operators. Some works (Wang et al., 2023; Pan et al., 2023; Evci et al., 2022) attempt to learn how to expand—predicting which connections to add or training expansion operators that maximize gradient alignment. These methods add complexity and training overhead, making them less practical for large-scale pre-training where simplicity and reliability are paramount.
The Missing Synthesis
The paper's key diagnosis is that prior work treats forward initialization and backward optimization as separate problems, each with its own set of heuristics. But mid-stage width expansion requires simultaneously satisfying two constraints that are in tension with each other:
- Signal Preservation pushes toward copy-based initialization, because copying perfectly preserves forward statistics (including RMS scale) and avoids any loss discontinuity.
- Symmetry Breaking pushes away from copying, because copying creates gradient coupling that prevents the new capacity from being useful.
Any framework that prioritizes only one of these constraints will fail: pure copying yields symmetry lock (Figure 2, "Copy Opt."), while pure random/zero initialization causes RMS mismatch and activation drift (Figure 1, "Naive Init, No Scaled"). The paper's contribution is to recognize this tension explicitly and provide mechanisms that satisfy both simultaneously: RMS-preserved rescaling handles the statistics (allowing non-copy initializations to be stable, or copy initialization to be properly scaled), while asymmetric optimizer-state resetting and re-warmup handle the symmetry (targeting only the new parameters in the backward pass without disrupting the forward function at expansion time).
How This Paper Positions Itself
The paper positions SPARKLING not as an incremental improvement over existing width-expansion methods, but as a conceptual reframing of what makes width expansion succeed or fail. The core intellectual move is to shift attention from the expansion moment itself (where function preservation was the dominant concern) to the ongoing optimization dynamics after expansion (where activation statistics and gradient symmetry dominate). This reframing has several implications for how the paper relates to prior work:
Against the function-preservation orthodoxy. The paper doesn't dismiss function preservation—it uses copy-based initialization as a component of SPARKLING—but it argues that function preservation alone is neither the mechanism that matters most nor the principle that should guide design. The empirical demonstration that RMS-preserved random or zero initialization can outperform copying (Figure 1f) is a direct challenge to the Net2Net heritage.
Unifying forward and backward perspectives. While prior work addresses either initialization or post-expansion scheduling, SPARKLING provides a unified framework where each component addresses a specific failure mode:
- RMS-preserved scaling → prevents activation-statistic drift (forward concern)
- Copy-based initialization → provides the strongest forward continuity
- Asymmetric optimizer-state reset → breaks gradient symmetry (backward concern)
- Asymmetric learning rate re-warmup → selectively stimulates new parameter learning (backward concern)
None of these components is entirely novel in isolation (RMS-based scaling appears in initialization theory; optimizer resetting is used in transfer learning; re-warmup is standard after architecture changes), but the paper's contribution is showing that they must be combined in a specific, principled way to enable mid-stage width expansion, and that each addresses a distinct aspect of the underlying failure modes.
Establishing width expansion as practically viable. The most important positioning is empirical: the paper demonstrates that mid-stage width expansion works—across multiple width axes, with different optimizers, and at a scale that produces meaningful cost savings (20–35% FLOP reduction, up to 1.49× wall-clock speedup). Prior work either expanded too early to demonstrate savings or didn't scale to a regime where the challenges become acute. SPARKLING is the first framework to show that width-progressive learning can be both stable and cost-effective when expansion occurs deep in the pre-training trajectory.
3. Technical Approach
3.1 Reader Orientation
SPARKLING is a recipe—a four-component procedure—that you apply at a single moment during pre-training to safely double (or more generally expand) the width of a Transformer model so that training can resume immediately without instability and the new capacity eventually becomes useful. It solves the problem that naively expanding width at mid-training (say, at 50% of total tokens) causes either a loss spike from disrupted activation statistics or a permanent waste of new parameters because copied neurons receive identical gradients and never diversify, and it solves this by simultaneously preserving the forward-pass signal magnitudes while injecting targeted asymmetry into the backward-pass optimizer dynamics.
3.2 Big-Picture Architecture (Diagram in Words)
The SPARKLING framework is not a training algorithm that runs continuously—it is a one-time intervention applied at the expansion point $t_e$ that modifies the model's parameters, optimizer states, and learning rate schedule, after which standard training resumes. It has four components that execute in a specific order at expansion time:
- RMS-Preserving Parameter Rescaling: immediately before resuming training, every weight matrix whose dimensionality changed is multiplied by a derived scaling factor so that the root-mean-square magnitude of activations at each layer remains identical to the pre-expansion model's. This operates on the parameters themselves.
- Copy-Based Initialization of New Channels: the new rows or columns added to weight matrices are initialized by duplicating existing ones (rather than random or zero), providing the strongest possible forward-function continuity at the expansion moment. This is the parameter initialization strategy.
- Asymmetric Optimizer-State Reset: the optimizer's internal state matrices (momentum and second-moment estimates for AdamW; momentum for Muon) are retained for the original parameters but reset to zero for all newly introduced parameters. This operates on the optimizer, not the model weights.
- Asymmetric Learning Rate Re-Warmup: from the expansion step onward, the original parameters continue on their pre-existing cosine schedule, while the newly introduced parameters follow a separate schedule that re-warms from the current learning rate to a modestly higher peak over a short window, then decays on the same cosine tail. This operates on the per-parameter learning rate.
Conceptually, components 1 and 2 handle the forward pass (what the model computes), while components 3 and 4 handle the backward pass (how the model updates). The forward components ensure that the expanded model produces well-behaved representations from the very first post-expansion forward pass. The backward components ensure that the expanded capacity actually learns distinct features over subsequent training steps rather than remaining a redundant copy.
The critical design insight is that these four components are complementary and individually necessary for mid-stage expansion. RMS-preserved scaling without symmetry breaking yields stable forward statistics but wastes capacity (Figure 1f, copy-copy underperforms other RMS-preserved variants). Symmetry breaking without RMS scaling causes activation drift that degrades late-stage convergence (Figure 1a-e, "Naive Init, No Scaled" curves). Only the full combination delivers both stability and effective capacity utilization.
3.3 Roadmap for the Deep Dive
The explanation follows the logical dependency chain—each component addresses a problem created (or left unsolved) by the previous ones:
-
First: RMS-Preserving Parameter Rescaling (Section 3.4.1). This establishes the mathematical foundation—what does it mean to preserve activation statistics across a width change, and how do we derive the rescaling factors for different expansion types? Understanding this requires analyzing fan-out expansion, fan-in expansion under different initialization assumptions, and RMSNorm expansion, because each case leads to a different rescaling rule. This component is explained first because it operates directly on the model parameters and constrains what initializations are permissible.
-
Second: Copy-Based Initialization and the Symmetry Lock (Section 3.4.2). Once we know how to rescale weights for RMS preservation, we need to choose how to initialize the new channels. Copying is the natural choice for forward continuity, but it creates a fundamental problem: duplicated parameters receive identical gradients and never diversify. I explain why this happens mathematically for both fan-out and fan-in expansion, and why even advanced optimizers like Muon cannot break the symmetry spontaneously. This establishes the necessity of the next two components.
-
Third: Asymmetric Optimizer-State Reset (Section 3.4.3). Given that symmetry is locked in by identical gradients and identical optimizer states, the solution is to make the optimizer states asymmetric while keeping the forward function unchanged. I explain how resetting only the new parameters' optimizer states to zero (while retaining the original parameters' states) breaks the coupling, and why copying or dropping all states fails.
-
Fourth: Asymmetric Learning Rate Re-Warmup (Section 3.4.4). Optimizer-state resetting breaks symmetry at the moment of expansion, but the new parameters start with zero momentum and must integrate gradient signal from scratch. Re-warming the new parameters' learning rate from the current schedule value to a modestly higher peak gives them a transient boost to catch up. I explain the schedule design, the hyperparameter choices, and why re-warming all parameters uniformly is worse.
-
Fifth: Putting It All Together (Section 3.4.5). I trace through the complete expansion procedure step by step, showing how the four components compose into a single expansion operation, and how the paper's experimental design evaluates each component's contribution.
This order builds understanding cumulatively: RMS preservation sets the statistical constraints, copy initialization introduces the symmetry problem, optimizer-state resetting provides a backward-pass solution, and re-warmup accelerates the solution's effect.
3.4 Detailed, Sentence-Based Technical Breakdown
This is primarily an empirical analysis paper with a prescriptive framework. The core idea is that mid-stage width expansion fails for two independent reasons—activation-scale disruption and gradient-symmetry locking—and that a specific, principled combination of parameter rescaling, optimizer-state handling, and learning-rate scheduling can simultaneously resolve both.
3.4.1 RMS-Preserving Parameter Rescaling
The first component of SPARKLING is a set of rules for multiplying the weights of expanded layers by derived scaling factors so that the root-mean-square magnitude of every activation vector remains unchanged across the expansion point. This is the Signal Preservation half of the framework.
Why RMS Preservation Matters, Mathematically
The paper's key argument is that the widely used "function preservation" (FP) criterion is too weak: FP ensures the model computes the same output, but it does not guarantee that the internal statistics that the optimizer relies on remain stable. The root-mean-square of a vector $\bm{h} \in \mathbb{R}^d$ is defined as:
where $\|\bm{h}\|_2$ is the Euclidean norm of the vector and $d$ is its dimensionality.
What it computes: the per-coordinate root-mean-square magnitude of a hidden-state vector, i.e., the typical absolute size of an individual element of $\bm{h}$ after averaging over the dimension.
Why this form: dividing the $\ell_2$ norm by $\sqrt{d}$ normalizes for vector length, making $\mathrm{RMS}$ comparable across layers of different widths. Without this normalization, a wider layer would trivially have a larger $\ell_2$ norm simply because it sums over more elements. The $\mathrm{RMS}$ quantity is the natural scale measure for Transformers because it is invariant under width changes when the per-coordinate distribution is stationary—it is exactly the quantity that RMSNorm normalizes to.
The paper defines the RMS ratio $r$ of a linear layer $\bm{y} = \bm{W}\bm{x}$ as:
where $s_{\text{out}} = \mathrm{RMS}(\bm{y})$ is the output activation scale and $s_{\text{in}} = \mathrm{RMS}(\bm{x})$ is the input activation scale.
What it computes: the factor by which the linear transformation amplifies or attenuates the typical activation magnitude. A value $r > 1$ means the layer tends to grow the signal; $r < 1$ means it tends to shrink it.
Why this matters for expansion: the RMS-preservation constraint is $r^{(\text{post})} = r^{(\text{pre})}$—the ratio must be identical before and after expansion. If this fails, two things go wrong. First, in a residual network with pre-normalization (as used in the paper's OLMoE-based experiments and in modern LLMs like Qwen3, DeepSeek-V3), the hidden state update is:
where $\mathrm{Norm}(\cdot)$ is RMSNorm. The pre-norm stabilizes the input to $f(\cdot)$ (the attention or MLP sublayer) but does not constrain its output scale. Post-expansion, if $\mathrm{RMS}(f(\mathrm{Norm}(\bm{h})))$ changes relative to $\mathrm{RMS}(\bm{h})$, the calibrated mixing between the residual stream and the branch output is disrupted—the branch either overwhelms the stream or becomes negligible. Second, even without residuals, subsequent layers are trained to expect inputs of a certain magnitude; if expansion shifts that magnitude, they receive out-of-distribution inputs and optimization suffers.
The Core Statistical Identity
To derive rescaling factors, the paper leverages the high-dimensional isotropy property of wide neural networks: for a linear layer $\bm{y} = \bm{W}\bm{x}$, the output coordinates $\{y_i\}_{i=1}^{d_{\text{out}}}$ tend to be approximately identically distributed and zero-mean. Taking expectation over the data distribution:
What it computes: in expectation, the squared RMS of the output equals the per-coordinate variance of any individual output element.
Why this is useful: it converts the problem of preserving $\mathrm{RMS}(\bm{y})$ (a vector-level quantity) into the problem of preserving $\mathrm{Var}(y_i)$ (a scalar variance). Under the further assumption that the weights $w_{ij}$ have shared variance $\sigma_w^2$ and the inputs $x_j$ have shared variance $\sigma_x^2$, with all term pairs $(w_{ij}, x_j)$ independent:
What it computes: the per-coordinate output variance decomposes as the input dimension times the weight variance times the input variance—a fan-in aggregation formula.
Operational meaning: if the input scale $\sigma_x^2$ is unchanged by expansion, preserving $\mathrm{Var}(y_i)$ (and hence $\mathrm{RMS}(\bm{y})$) reduces to keeping the product $d_{\text{in}} \sigma_w^2$ constant. When $d_{\text{in}}$ changes during fan-in expansion, the weight variance $\sigma_w^2$ must be adjusted inversely to compensate.
Fan-Out Expansion: Simple, No Rescaling Needed
In fan-out expansion, the output dimension grows from $d_{\text{out}}$ to $d'_{\text{out}} > d_{\text{out}}$ while $d_{\text{in}}$ is unchanged. The expanded weight matrix is formed by stacking new rows on top of the original ones:
where $\bm{W} \in \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}$ is the original weight matrix and $\tilde{\bm{W}} \in \mathbb{R}^{(d'_{\text{out}} - d_{\text{out}}) \times d_{\text{in}}}$ are the newly added rows.
The expanded output is $\bm{y}' = [\bm{y}; \tilde{\bm{y}}]$ where $\bm{y} = \bm{W}\bm{x}$ and $\tilde{\bm{y}} = \tilde{\bm{W}}\bm{x}$.
RMS analysis: since computing $\mathrm{RMS}(\bm{y}')$ involves averaging over $d'_{\text{out}}$ coordinates, and the new coordinates $\tilde{\bm{y}}$ are generated by the same input $\bm{x}$ through new weight rows, the RMS is preserved as long as the new rows are distributionally consistent with the original ones. Concretely, if $\tilde{\bm{W}}$ is initialized by copying rows from $\bm{W}$ or by randomly sampling from the same distribution, the per-coordinate statistics of $\bm{y}'$ match those of $\bm{y}$, and $\mathrm{RMS}(\bm{y}')$ automatically equals $\mathrm{RMS}(\bm{y})$ without any additional scaling.
Practical instances: fan-out expansion occurs in the up and gate projections of an MLP when widening the expert intermediate dimension (these map from the hidden dimension to a larger intermediate dimension) and in the vhead projection of attention when widening the value dimension.
Fan-In Expansion: The Critical Case Where Rescaling Is Necessary
Fan-in expansion is where the input dimension grows from $d_{\text{in}}$ to $d'_{\text{in}} > d_{\text{in}}$ while $d_{\text{out}}$ is unchanged. The expanded weight matrix concatenates new columns to the right of the original ones, and includes an explicit scaling factor $\alpha$ that we need to determine:
where $\bm{W} \in \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}$ is the original weight matrix and $\tilde{\bm{W}} \in \mathbb{R}^{d_{\text{out}} \times (d'_{\text{in}} - d_{\text{in}})}$ are the newly added columns.
The expanded input $\bm{x}' = [\bm{x}; \tilde{\bm{x}}]$ concatenates new input coordinates $\tilde{\bm{x}}$ (from a preceding fan-out layer that was also expanded). The output is:
Why a scaling factor is needed: under fan-in expansion, $d_{\text{in}}$ increases to $d'_{\text{in}}$. If the weights were simply concatenated without scaling ($\alpha = 1$), then according to the variance formula $\mathrm{Var}(y'_i) = d'_{\text{in}} \sigma_{w'}^2 \sigma_{x'}^2$ would be strictly larger than the pre-expansion $\mathrm{Var}(y_i) = d_{\text{in}} \sigma_w^2 \sigma_x^2$, because you're summing over more terms. The scaling factor $\alpha$ must shrink the weights to compensate for the increased fan-in.
The paper derives different $\alpha$ values depending on how the new channels are initialized, because the initialization affects whether the independence assumptions underlying the variance decomposition hold.
Case 1: Random or One-Side Copied. If the newly added fan-in coordinates are initialized by random sampling from the same distribution as the original weights, or if only one side of the new coordinates (either $\tilde{\bm{W}}$ or $\tilde{\bm{x}}$) is copied while the other is random, then the independence across fan-in dimensions is maintained. The per-coordinate variance after expansion is:
Setting this equal to the pre-expansion variance $d_{\text{in}} \sigma_w^2 \sigma_x^2$ and assuming $\sigma_{x'}^2 = \sigma_x^2$ (input scale unchanged) yields:
What this means operationally: every element of the expanded weight matrix (both original and new columns) should be multiplied by $\sqrt{d_{\text{in}} / d'_{\text{in}}}$. Equivalently:
For a $2\times$ width expansion where $d'_{\text{in}} = 2 d_{\text{in}}$, this gives $\alpha = \sqrt{1/2} = 1/\sqrt{2}$.
Why this form: the $\sqrt{d_{\text{in}} / d'_{\text{in}}}$ factor exactly compensates for the increased fan-in. Under random initialization, the variance of a sum of $d'_{\text{in}}$ independent terms grows as $d'_{\text{in}} \sigma^2$. To keep the sum's variance constant when $d'_{\text{in}}$ increases, each term's standard deviation must shrink by $\sqrt{d_{\text{in}} / d'_{\text{in}}}$.
Case 2: Both-Sides Copied. A qualitatively different situation arises when both $\tilde{\bm{W}}$ and $\tilde{\bm{x}}$ are created by copying existing dimensions, because now the independence assumption is violated: copied pairs $(w'_{ij}, x'_j)$ are perfectly correlated duplicates of original pairs, not independent replicas. This requires a more careful variance decomposition.
Let $c$ be the copy ratio such that $d'_{\text{in}} = (1 + c) d_{\text{in}}$. The case $0 < c \leq 1$ corresponds to each copied dimension being duplicated exactly once (e.g., $c = 1$ for $2\times$ expansion where every dimension is copied). The set of new input indices is partitioned into $\mathcal{R}$ (the $c d_{\text{in}}$ copied dimensions) and $\mathcal{S}$ (the remaining $(1-c) d_{\text{in}}$ dimensions, which for $c = 1$ is empty). Because each copied dimension contributes twice with identical value:
The variance becomes:
Setting this equal to the pre-expansion $\mathrm{Var}(y_i) = d_{\text{in}} \sigma_w^2 \sigma_x^2$ and noting that under copying $\sigma_{w'}^2 = \sigma_w^2$ (the scaled weights inherit the original variance up to $\alpha$), we get:
For $2\times$ width expansion where $c = 1$ (every original dimension is copied exactly once), this gives $\alpha = 1/\sqrt{4} = 1/2$.
For $c > 1$ (some dimensions copied multiple times), the variance becomes $\alpha^2 (1+c)^2 d_{\text{in}} \sigma_{w'}^2 \sigma_x^2$, yielding $\alpha = 1/(1+c)$.
Combined, the rescaling rule for both-sides copied fan-in expansion is:
or equivalently in terms of $d_{\text{in}}$ and $d'_{\text{in}}$:
Why the scaling differs from the random case: under both-sides copying, the duplicated coordinates contribute coherently rather than independently. The factor of 4 on the $\mathcal{R}$ sum (instead of the factor of 2 that independent copies would give) comes from the fact that $\mathrm{Var}(2Z) = 4\mathrm{Var}(Z)$ for a random variable $Z$, whereas for independent copies $Z_1$ and $Z_2$, we would have $\mathrm{Var}(Z_1 + Z_2) = 2\mathrm{Var}(Z)$. The $1+3c$ form captures this super-additive variance from duplication. The $c > 1$ regime's $(1+c)^2$ reflects the fact that with multiple copies, the variance grows quadratically in the number of copies (because each additional copy adds one more identical term to the sum, and the variance of a sum of $m$ identical copies scales as $m^2$).
RMSNorm Weight Expansion: No Rescaling Needed
When widening the hidden dimension, the RMSNorm parameter $\bm{\gamma} \in \mathbb{R}^d$ must also be expanded to $\bm{\gamma}' \in \mathbb{R}^{d'}$. The RMSNorm operation (omitting $\epsilon$ for clarity) is:
Applying the same variance analysis:
What this means: the output RMS of the RMSNorm layer is proportional to the RMS of $\bm{\gamma}$. Therefore, preserving the output RMS under width expansion simply requires that the new coordinates of $\bm{\gamma}$ have the same distribution as the original ones. Copying or random sampling from the same distribution naturally achieves this without any additional rescaling.
Why no rescaling: unlike fan-in linear layers, RMSNorm applies an element-wise operation with no summation across dimensions. The output at coordinate $i$ depends only on the input at coordinate $i$; expanding the dimension just adds more independent coordinates with the same per-coordinate statistics. There is no fan-in aggregation to compensate for.
Empirical Validation: RMS-Preserved Scaling Improves Late-Stage Convergence
The paper validates the RMS-preserving approach in Section 3.3 (Figure 1) by comparing "Naive Init, No Scaled" (copy or random initialization with $\alpha = 1$) against "RMS-Preserved Scaled" (applying the derived $\alpha$ values) across five initialization pairs for MoE expert inner-dimension expansion. The key finding:
"Across all initializations, Naive Init, No Scaled consistently yields a smaller immediate loss gap but worse late-stage convergence, whereas RMS-Preserved Scaled recovers steadily and converges to a lower final loss."
This is the critical empirical distinction: function preservation at the expansion moment (smaller immediate loss gap) does not predict better final performance. The RMS-preserved variants, which may cause a slightly larger instantaneous loss increase because they rescale the original weights (disrupting exact forward-function matching), ultimately converge to a lower loss because they maintain the optimization regime's statistical properties. The paper's interpretation is that the immediate loss gap under naive initialization is misleadingly small—the damage is latent in the activation statistics and manifests over subsequent training steps.
The paper also notes a special case: the zero-copy initialization (Appendix 9, Figure 4). When one side is zero-initialized, the paper empirically finds that it should be treated as random under RMS-preserving expansion rather than as strictly loss-preserving at $t_e$. The rationale is that zero-initialized channels become gradient-driven random distributions after the very first update, so their effective statistics quickly resemble those of randomly initialized channels. The RMS-preserved scaling variant that treats the zero side as random yields activation RMS ratios that remain closer to the original baseline after a few post-expansion steps.
3.4.2 Copy-Based Initialization and the Symmetry Lock
The second component of SPARKLING is the choice to initialize new channels by copying existing ones—but this choice creates a problem that the remaining two components must solve.
Why Copy? The Best Forward Continuity
Even though Figure 1f shows that RMS-preserved random or zero initialization can outperform copy-copy in terms of final loss, copy-based initialization remains attractive for two reasons. First, it provides the strongest forward-function continuity at the expansion moment: the expanded model computes exactly the same function as the pre-expansion model (up to the RMS-preserving rescaling, which is a global scaling that doesn't change the function's structure). Second, as Sections 4.3 and 5 show, once the symmetry problem is solved by the backward-pass interventions, copy-copy achieves the best final loss among all initialization strategies—it just needs the symmetry lock to be broken first.
The choice is therefore pragmatic: start with the initialization that provides the best forward continuity, then apply separate mechanisms to fix the backward coupling problem it creates. This decouples the two concerns in a way that a single initialization strategy (like random or zero alone) cannot.
The Symmetry Lock: A Proof Sketch
Consider a $2\times$ fan-out expansion where $\bm{W}' = [\bm{W}; \bm{W}]$ (the new rows are copies of the original rows). The forward pass produces $\bm{y}' = [\bm{y}; \bm{y}]$. If subsequent layers are also copied (which they will be, as part of the paired fan-in expansion), back-propagation preserves symmetry: the gradient with respect to the expanded output is $\partial\mathcal{L} / \partial\bm{y}' = [\bm{g}; \bm{g}]$ where $\bm{g} = \partial\mathcal{L} / \partial\bm{y}$. The gradient with respect to the expanded weights is then:
What this shows: the gradient for the original rows $\nabla_{\bm{W}} \mathcal{L}$ and the gradient for the copied rows $\nabla_{\tilde{\bm{W}}} \mathcal{L}$ are identical. The analogous derivation for fan-in expansion (Appendix 7.3) shows the same result for column copying.
Operational consequence: if the optimizer states (momentum, second-moment estimates) are also initialized symmetrically—either by copying the existing states or by resetting all states to zero—then the two parameter blocks receive identical updates at every step. By induction, $\bm{W}^{(t)} = \tilde{\bm{W}}^{(t)}$ for all $t$. The expanded model has twice the parameters but lives in the original lower-dimensional subspace: the new neurons compute exactly the same function as the original neurons, contributing nothing new.
Why Muon's Orthogonalization Cannot Break the Symmetry
This is a non-obvious theoretical contribution. Muon (Jordan et al., 2024) is a spectral-style optimizer that applies Newton-Schulz orthogonalization to the matrix-valued momentum before using it to update weights. The intuition is that orthogonalization decorrelates the update directions, which might be expected to break symmetry. The paper proves it does not.
The Newton-Schulz iteration can be written as a matrix polynomial applied to the Gram matrix. For a momentum matrix $\bm{X}_k$, the next iterate is:
where $\phi(\cdot)$ is a matrix polynomial—specifically $\phi(\bm{G}) = \frac{1}{2}(3\bm{I} - \bm{G})$ for the standard Newton-Schulz step, or higher-order variants with appropriate coefficients.
Now consider the column-duplicated (fan-in) case where the momentum is initialized as $\bm{X}_0 = [\bm{A}_0, \bm{A}_0]$. Let $\bm{P}_k = \bm{A}_k^\top \bm{A}_k$. Then the Gram matrix is:
Critical observation: the Gram matrix is block-constant—all four blocks are identical. This is a consequence of the columns being duplicates.
Applying the polynomial:
Because $\phi(\cdot)$ is applied to a block-constant matrix and preserves block-exchange symmetry (by construction, as a polynomial of the Gram matrix), the result has the form $[\bm{A}_{k+1}, \bm{A}_{k+1}]$—the two column blocks remain identical. Orthogonalization cannot create asymmetry where none exists in the Gram matrix structure. The symmetry lock is intact.
Why this matters: it means the symmetry problem is fundamental to the copy operation, not an artifact of AdamW's element-wise nature. Even optimizers designed to decorrelate features cannot break a symmetry that is encoded in the parameter duplication itself.
3.4.3 Asymmetric Optimizer-State Reset
Given that the symmetry lock is created by the combination of identical gradients (from the copy structure) and identical optimizer states (from symmetric initialization), breaking the lock requires making at least one of these asymmetric. The gradients cannot be made asymmetric without changing the forward function (which would defeat the purpose of copying), so the intervention must target the optimizer states.
The Intervention
At the expansion point $t_e$, the optimizer maintains state matrices $\bm{S}$ for each parameter tensor (for AdamW, this includes the first-moment estimate $\bm{m}$ and the second-moment estimate $\bm{v}$; for Muon, it is the momentum matrix). The expanded parameter tensor $\bm{W}'$ is formed by concatenating the original weights $\bm{W}$ and the new weights $\tilde{\bm{W}}$. The optimizer state for the expanded tensor is set as:
where $\bm{S}$ is the pre-expansion state (preserving the training history of the original parameters) and $\mathbf{0}$ is a zero-initialized state for the new parameters.
What this does operationally: the original parameters retain their full optimizer history—their accumulated momentum, their per-parameter learning rate scaling from Adam's second moment, all the information that encodes their optimization trajectory up to $t_e$. The new parameters start with a blank slate: zero momentum, zero second-moment estimate. This means that even though the gradients for the two parameter blocks are initially identical, the updates are different because the optimizer applies different transformations to those identical gradients.
For AdamW specifically, the update for a parameter with gradient $g$, first-moment $m$, and second-moment $v$ at step $t$ (ignoring weight decay for simplicity) is:
When $m = 0$ and $v = 0$ initially (the new parameters' state), the first few updates are dominated by the raw gradient. When $m$ and $v$ have accumulated over billions of tokens (the original parameters' state), the updates are shaped by long-term momentum direction and adaptive scaling. Identical gradients thus produce different parameter updates, and the symmetry breaks.
Comparison to Alternative Treatments
Figure 2 evaluates four optimizer-state handling strategies under copy-copy expansion with RMS-preserved scaling:
- Drop Optimizer (
$\bm{S}' = \mathbf{0}$globally): resets all states. This is symmetric (both parameter blocks start from zero) and discards the original parameters' training history. Converges to a substantially higher loss. - Copy Optimizer (
$\bm{S}' = [\bm{S}, \bm{S}]$): duplicates the existing states. This is symmetric (both blocks start identical) and preserves the history but perpetuates the lock. Also converges to a substantially higher loss. - Asymmetric Reset (
$\bm{S}' = [\bm{S}, \mathbf{0}]$): the proposed approach. Achieves faster recovery and a lower final loss than either symmetric baseline. - Asymmetric Reset + Scaled Optimizer (
$\bm{S}' = [\bm{S}, \alpha \cdot \mathbf{0}]$with$\alpha$matching the RMS-preserving parameter scaling): additionally applies the parameter scaling factor to the new optimizer states. Yields no improvement over the simple reset, indicating that strict state-parameter scale alignment is unnecessary—any initial mis-scaling is quickly corrected by gradient updates.
Key design justification: the asymmetric reset is the minimal intervention that breaks symmetry. It preserves all useful training signal (the original parameters' optimizer states) while inserting exactly enough asymmetry (the new parameters' zero states) to prevent lock. Dropping all states loses useful information; copying all states perpetuates the problem.
3.4.4 Asymmetric Learning Rate Re-Warmup
Optimizer-state resetting breaks the symmetry lock but leaves the new parameters at a disadvantage: they start with zero momentum and must integrate gradient signal from scratch to catch up to the well-adapted original parameters. The asymmetric re-warmup schedule addresses this by giving the new parameters a transient learning rate boost.
The Baseline Schedule
The paper uses a standard cosine decay schedule with linear warmup for training from scratch. Let $T_w$ be the number of warmup steps, $T$ the total steps, and $\eta_0$, $\eta_{\max}$, $\eta_{\min}$ the initial, peak, and final learning rates respectively:
where $\psi(x) = \frac{1}{2}(1 + \cos(\pi x))$ is the cosine decay function.
What this computes: a learning rate that linearly warms up from $\eta_0$ to $\eta_{\max}$ over $T_w$ steps, then decays via a cosine curve to $\eta_{\min}$ over the remaining steps. The decay is shaped so that the learning rate changes most rapidly near the middle of the decay phase and flattens near the endpoints.
The Asymmetric Re-Warmup Schedule
At the expansion point $t_e$, the current learning rate is $\eta_e = \eta(t_e)$. The paper's design keeps the original parameters on the unmodified baseline schedule for continuity—their learning rate continues along the cosine decay as if expansion never happened. For the newly introduced parameters, a separate schedule is defined:
where $\rho$ is the rewarmup ratio, $\tau_w$ is the number of re-warmup steps, and $f(\cdot)$ is the same functional form as the baseline schedule but with shifted time origin.
What this computes operationally, step by step:
- At step
$t_e$(the expansion moment), the new parameters' learning rate starts at$\eta_e$—the current value from the baseline schedule, not zero. This ensures no discontinuity: at$t_e$, both old and new parameters have the same learning rate. - Over the next
$\tau_w$steps, the new parameters' learning rate linearly increases from$\eta_e$to$\hat{\eta}_{\max} = \rho \cdot \eta_e$. This is a "re-warmup": a deliberate increase above the current rate. - After
$\tau_w$steps, the new parameters' learning rate follows the same cosine decay tail as the baseline schedule, decreasing toward$\eta_{\min}$. - After both schedules reach
$\eta_{\min}$, they remain equal through the end of training.
Figure 6 (Appendix 11) visualizes this: the original parameters' curve (blue) continues its smooth cosine descent, while the new parameters' curve (orange) creates a small "bump" right after $t_e$ before rejoining the cosine tail.
Why this form—re-warmup rather than constant boost or global re-warmup:
- Starting at
$\eta_e$rather than zero prevents an initial period where the new parameters learn at an inappropriately low rate. Since the model is deep in training,$\eta_e$is substantially below$\eta_{\max}$(the cosine schedule has decayed significantly by mid-training). Starting the new parameters at zero would waste many steps climbing to a reasonable rate. - Increasing above
$\eta_e$(via$\rho > 1$) provides a transient boost that helps the new parameters integrate gradient signal faster and catch up to the original parameters' effective learning rate (which benefits from accumulated momentum). Without this boost, the new parameters would lag because they lack the optimizer state history. - Short duration (
$\tau_w$small, e.g., 250 steps) prevents the boosted rate from destabilizing the already-trained features. The boost is a targeted stimulus, not a sustained regime. - Asymmetric application (only to new parameters) preserves the original parameters' careful optimization schedule. The paper shows in Appendix 14 (Figure 9) that globally re-warming all parameters ("Re-warmup All") underperforms the asymmetric approach: perturbing the original parameters' learning rate disrupts their well-adapted trajectory.
Hyperparameter Selection
The paper sweeps $\rho$ and $\tau_w$ under expert-inner $2\times$ expansion (Appendix 12, Figure 7). The results show:
"
$\rho \approx 1.25$–1.3 and$\tau_w \approx 0$–250 steps achieve the lowest final loss, indicating that newly introduced parameters benefit from a modest, short-lived learning rate boost rather than a prolonged or overly strong re-warmup."
The chosen defaults are $\rho = 1.3$ and $\tau_w = 250$. The paper notes these also transfer to hidden-dimension expansion without re-tuning. The fact that $\tau_w = 0$ (no re-warmup, just continuing from $\eta_e$) is within the good region suggests that the boost is helpful but not extremely sensitive to the exact duration.
Why these values: a too-large $\rho$ (e.g., $\rho = 2$) would destabilize the new parameters by taking overly large steps before they have accumulated reliable gradient signal. A too-long $\tau_w$ (e.g., thousands of steps) would keep the new parameters at an elevated learning rate while the rest of the model is decaying, potentially causing the new capacity to overfit to late-training gradients or disrupt the original parameter dynamics through the shared forward pass.
Empirical Validation (Figure 3)
Figure 3 evaluates asymmetric re-warmup across three width-growth settings (Inner 2×, Hidden 2×, and joint Hidden 2× & Inner 2×) and two initialization strategies (copy-copy and zero-copy). The consistent finding: enabling re-warmup lowers the final loss for all configurations. The benefit is largest for copy-copy, where re-warmup closes the gap with zero-copy and achieves the best final loss overall. This is consistent with the framework logic: copy-copy has the strongest forward continuity but the most severe symmetry problem; asymmetric state reset breaks the symmetry, and re-warmup accelerates the new capacity's diversification into effective features.
3.4.5 The Complete SPARKLING Procedure
Putting the four components together, the expansion procedure at step $t_e$ (when the model has been trained on $D_e$ tokens) is:
Step 1: Expand the model architecture. Instantiate the target-width model with all width dimensions scaled by the desired factor (the paper primarily uses $2\times$). This includes:
- Hidden dimension
$d_{\text{model}}$: doubled from 1024 to 2048 for hidden expansion. - Expert intermediate dimension
$d_{\text{ffn}}$: doubled from 512 to 1024 for inner expansion. - RMSNorm parameter
$\bm{\gamma}$: expanded correspondingly. - Embedding matrix (tied with output projection): expanded in the vocabulary dimension, with special handling since it acts as fan-out on the embedding side but fan-in on the output side—the fan-in rescaling factor is applied to the output projection's logits rather than the shared weight matrix itself.
Step 2: Initialize new parameters by copying, then rescale all parameters for RMS preservation.
- For each weight matrix that underwent fan-out expansion (up_proj, gate_proj, vhead_proj): new rows are copied from existing rows. No rescaling is needed because fan-out is automatically RMS-preserving under copy initialization.
- For each weight matrix that underwent fan-in expansion (down_proj, output projection): new columns are copied from existing columns, and then all elements of the matrix are multiplied by the appropriate
$\alpha$factor derived in Section 3.2.3:- For
$2\times$expansion (the$c=1$case), both-sides copied fan-in uses$\alpha = 1/2$. - For one-side-copied or random fan-in,
$\alpha = 1/\sqrt{2}$.
- For
- For each RMSNorm parameter: new coordinates are copied from existing ones. No rescaling is needed.
- The paper empirically treats zero-initialized sides as random for RMS-preservation purposes (Appendix 9).
Step 3: Asymmetric optimizer-state reset.
For every parameter tensor that was expanded:
- The optimizer state slice corresponding to the original parameter dimensions is retained unchanged.
- The optimizer state slice corresponding to the newly added dimensions is set to zero (for AdamW: both
$\bm{m}$and$\bm{v}$are zeroed; for Muon: momentum is zeroed). - For non-expanded parameters (e.g., attention query/key projections if only inner dimension was expanded), optimizer states are left entirely unchanged.
Step 4: Asymmetric learning rate re-warmup.
From step $t_e$ onward:
- The learning rate scheduler for the original parameters continues on the unmodified cosine schedule
$\eta(t)$. - The learning rate scheduler for the newly introduced parameters starts at
$\eta_e = \eta(t_e)$, linearly warms to$\rho \cdot \eta_e$over$\tau_w$steps, then follows the cosine tail to$\eta_{\min}$. The default hyperparameters are$\rho = 1.3$and$\tau_w = 250$steps. - All other training hyperparameters (batch size, weight decay, gradient clipping, etc.) remain unchanged from the pre-expansion recipe.
Step 5: Resume training.
Standard training resumes immediately with the expanded model, modified optimizer states, and modified learning rate schedules in place. No special warmup period, no gradual unfreezing, no architectural changes beyond the width dimensions.
Design Choices and Their Justifications (Consolidated)
- Copy initialization over random/zero: provides the strongest forward continuity, and achieves the best final loss when the symmetry problem is properly handled by the backward-pass interventions. The paper shows that copy-copy + SPARKLING's backward components achieves the lowest final loss among all configurations (Figure 3).
- RMS-preserving rescaling over naive (no scaling): the paper's central reframing—function preservation is not enough; the activation statistics that govern long-term optimization must be preserved. Empirically validated in Figure 1 across multiple initialization strategies.
- Asymmetric optimizer-state reset over drop/copy: the minimal intervention that breaks symmetry while preserving the original parameters' training signal. Copying states perpetuates the lock; dropping all states discards useful history (Figure 2).
- Asymmetric re-warmup over global re-warmup: selectively stimulates new parameters without perturbing the original parameters' well-adapted schedule. Global re-warmup (Appendix 14, Figure 9) underperforms because it disrupts the original parameters' optimization.
- No optimizer-state scaling: the paper finds that scaling the new optimizer states to match the RMS-preserving scaling factor provides no benefit (Figure 2), suggesting the optimizer quickly adapts to any initial state-parameter scale mismatch.
What SPARKLING Does NOT Do
It is equally important to understand what the framework explicitly avoids, because these omissions are principled:
- No gradual expansion: unlike masking-based approaches (Yao et al., 2024), SPARKLING expands instantaneously at a single step. There is no schedule of gradual unfreezing or incremental capacity addition.
- No learned growth operators: the expansion is rule-based (copy + rescale), not learned from data.
- No architecture search: which dimensions to expand and by how much is pre-specified, not discovered during training.
- No modification to the data distribution or training objective: the same data, same loss function, same batch composition continue after expansion.
- No special treatment of the expansion point in the forward pass: the model forward pass at step
$t_e + 1$is exactly the standard forward pass of the expanded architecture; there is no "transition" forward function.
The framework's simplicity—it is four deterministic operations applied once—is a deliberate design choice for practical large-scale training, where complex multi-phase procedures introduce engineering risk and hyperparameter sensitivity.
4. Key Insights and Innovations
Innovation 1: Reframing Width Expansion from Forward-Preservation to Forward-Statistical + Backward-Asymmetric
The dominant paradigm in progressive width expansion since Net2Net (Chen et al., 2016) has been function preservation (FP): design the expanded model so that it computes exactly the same output as the original at the moment of expansion, thereby avoiding any instantaneous loss discontinuity. This principle motivated copy-based initialization, masking-based gradual unfreezing (Yao et al., 2024), and various parameter-mapping schemes (Chen et al., 2022; Zhang et al., 2024). FP is intuitively appealing because it provides a clean mathematical guarantee of loss continuity.
SPARKLING fundamentally reframes this. The key diagnostic move is that the mechanism behind successful width expansion is not forward-function matching at a single point, but rather the preservation of the optimization regime's statistical properties across the expansion point. The paper identifies two independent requirements that FP conflates:
- Signal Preservation: keeping activation RMS magnitudes stable so that downstream layers receive well-scaled inputs and residual streams maintain their calibrated mixing ratios.
- Symmetry Breaking: ensuring that newly copied parameters receive asymmetric gradient updates so they can diversify into distinct features rather than remaining permanently redundant.
The significance of this reframing is that FP addresses only the forward pass at $t_e$, while signal preservation addresses the ongoing forward statistics, and symmetry breaking addresses the backward pass's optimization dynamics—two failure modes that FP is blind to. The paper's counter-intuitive finding that RMS-preserved variants (which may cause a larger instantaneous loss spike than naive unscaled initialization) ultimately converge to a lower final loss (Figure 1) is the cleanest evidence that FP is measuring the wrong thing. Loss continuity at the expansion point is neither necessary nor sufficient for successful width expansion; activation-statistic stability and gradient asymmetry are what matter.
This is a fundamental conceptual shift, not an incremental improvement. It explains why prior works that expanded only at early training stages (where the statistical regime is less settled and symmetry locking matters less) could succeed with FP alone, and why those same methods fail at mid-stage (where the model's internal statistics are deeply entrenched and optimization dynamics are more fragile). The paper doesn't reject FP—it still uses copy initialization—but it demotes FP from the organizing principle to one component within a broader framework that explicitly manages forward statistics and backward asymmetry.
Innovation 2: Diagnosing, Proving, and Breaking the Symmetry Lock as a Universal Backward-Pass Failure Mode
Prior width-expansion work recognized that simple copying creates redundant parameters and proposed various heuristic interventions: uneven splitting with different scaling factors (Chen et al., 2016; Wang et al., 2024; Du et al., 2024), symmetric perturbations that cancel in the forward pass (Yuan et al., 2023; Wu et al., 2021; Liu et al., 2019), and weight-norm rebalancing (Yuan et al., 2023). These heuristics share an implicit assumption: injecting asymmetry at initialization is sufficient to enable diversification, because subsequent gradient updates will naturally amplify any initial differences.
SPARKLING's critical diagnostic contribution is proving that this assumption is false. The paper provides a formal analysis of the symmetry lock: under copy-based expansion, duplicated parameter blocks receive identical gradients at every step, and if the optimizer states are also symmetric, the updates are identical, keeping the blocks permanently coupled. This is not an initialization problem that can be solved by one-shot asymmetry; it is a persistent dynamical constraint that requires ongoing asymmetric forces to overcome.
Three aspects of this analysis are intellectually distinctive:
First, the universality proof. The paper shows that the symmetry lock holds not just for element-wise optimizers like AdamW, but also for spectral-style optimizers like Muon. The Newton-Schulz orthogonalization in Muon is often assumed to decorrelate features, but the paper proves formally that it cannot break the symmetry because it operates as a polynomial map of the Gram matrix, and duplicated columns produce a block-constant Gram matrix. This is a non-obvious theoretical result that explains why Muon's orthogonalization, despite its decorrelation properties, cannot spontaneously diversify copied parameters.
Second, the identification of optimizer-state symmetry as the bottleneck. The symmetry lock persists not just because gradients are identical, but because the optimizer transformation of those gradients is identical. This shifts the locus of intervention from the parameter initialization (where prior heuristics operate) to the optimizer state handling. The proposed asymmetric reset—retaining states for original parameters, zeroing states for new ones—is the minimal intervention that addresses the root cause.
Third, the empirical demonstration that one-shot initialization heuristics are insufficient. Figure 9 (Appendix 14) evaluates uneven splitting and symmetric perturbations against SPARKLING, finding that these prior approaches consistently underperform. The paper's interpretation is that initial asymmetry is quickly washed out by identical subsequent updates; the new parameters need persistent asymmetry in the optimizer dynamics—not just a different starting point—to develop distinct features.
This is a fundamental diagnostic contribution with practical implications. It tells the field that the symmetry problem in width expansion cannot be solved by better initialization tricks alone; it requires modifications to how the optimizer treats old versus new parameters. This has downstream implications for any progressive learning method that involves capacity duplication, not just width expansion.
Innovation 3: Establishing Mid-Stage Width Expansion as Practically Viable—and Characterizing Its Cost-Benefit Boundary
Prior width-progressive learning work expanded models during the initial 10–30% of training (Du et al., 2024; Shen et al., 2022), where the savings in FLOPs are negligible because the small model trains for too few tokens to amortize its lower cost. The implicit message from this body of work is pessimistic: width expansion is too unstable for mid-training, so progressive learning's computational benefits can only be realized through depth expansion.
SPARKLING overturns this message by providing the first empirical demonstration that mid-stage width expansion (at 50% of total training tokens) can be both stable and cost-effective. Table 2 reports 20–35% FLOP savings and up to 1.49× wall-clock speedup across three width-growth axes, while Table 1 shows that downstream performance matches or exceeds the from-scratch expanded baseline on most tasks. This is not an incremental extension of prior methods to a new expansion point; it is a qualitative regime change—mid-stage is where prior approaches break down because both activation-statistic drift and symmetry locking become severe, and SPARKLING's dual-principle design specifically targets those failure modes.
The significance goes beyond the raw numbers in two ways. First, it establishes width as a viable axis for progressive learning at scale, closing a major gap relative to depth-oriented methods. Modern architectures (MoE models in particular) scale primarily through width dimensions—expert count, expert hidden dimension, attention head dimension—and a PL framework that can only grow depth is incomplete. Second, the paper implicitly defines a boundary condition: mid-stage width expansion succeeds when signal preservation and symmetry breaking are simultaneously managed, and fails when either is neglected. The ablation structure (RMS scaling vs. no scaling; asymmetric reset vs. copy/drop; re-warmup vs. no re-warmup) provides a clear map of which failure modes matter and how they interact.
The paper is candid about the remaining gap: expanded models show a slightly higher final pre-training loss than from-scratch training (Table 1), even though downstream performance matches or exceeds. This acknowledges that SPARKLING does not perfectly recover the from-scratch loss trajectory—it is a practical cost-saving technique, not a loss-equivalent transformation.
Innovation 4: Unifying Activation-Scale Theory with Practical Width-Expansion Recipes Through RMS-Preserving Rescaling
The idea that weight initialization should preserve activation variance across layers is a classical concern in neural network theory (Glorot and Bengio, 2010; He et al., 2015), and the specific connection to RMS statistics appears in initialization schemes like µP (Yang et al., 2022) and RMSNorm (Zhang and Sennrich, 2019). However, these theoretical tools were developed for initialization at the start of training, where all parameters are random and the goal is to start optimization in a well-conditioned regime.
SPARKLING's distinctive contribution is to repurpose RMS-scale analysis for the expansion problem, where the situation is qualitatively different: the model has been training for billions of tokens, its activations have settled into a characteristic scale manifold, and the goal is to preserve that specific, data-conditioned scale rather than a generic initialization target. This requires two conceptual moves.
First, the paper shows that RMS preservation is sufficient where function preservation is too strong. The derivation of rescaling factors for different initialization regimes (copy-copy with $\alpha = 1/2$; random or one-side-copied with $\alpha = 1/\sqrt{2}$) provides a principled way to expand width without exact forward-function matching, enabling non-copy initializations (random, zero-copy) that would violate FP to nonetheless remain within the stable optimization regime. This explains why RMS-preserved random initialization can outperform unscaled copy initialization (Figure 1f): it preserves the statistics FP ignores while avoiding the symmetry lock FP creates.
Second, the paper identifies a practically important corner case (Appendix 9): zero-initialized channels should be treated as random under RMS-preserving expansion, not as a special loss-preserving case. The rationale—that zero-initialized blocks become gradient-driven random distributions after the first update—is a nuanced empirical observation that could easily be missed, and it prevents a common failure mode where a zero-initialized side of an expansion causes persistent RMS drift.
This is an incremental theoretical contribution rather than a fundamental breakthrough in initialization theory, but its practical value is substantial: the RMS-preserving rescaling rules are simple to implement (multiply a weight matrix by a scalar), do not require any learned components, and apply uniformly across different width axes and initialization strategies. The paper provides a self-contained derivation that any practitioner can apply to their specific expansion scenario, making this contribution directly actionable.
Innovation 5: Proposing Optimizer-State Asymmetry and Learning-Rate Re-Warmup as a Unified Backward-Pass Intervention Strategy
The paper's final distinctive contribution is the combination of asymmetric optimizer-state resetting and asymmetric learning rate re-warmup as a coordinated strategy for backward-pass management during capacity expansion. While each component has precedents in isolation (optimizer-state resetting appears in transfer learning and continual learning; learning-rate re-warmup is standard after architecture changes), the paper's contribution is showing that they serve distinct, complementary roles in the symmetry-breaking problem and must be combined for full effectiveness.
The logic is layered:
- Optimizer-state resetting (Section 4.2.1) provides direction asymmetry: even when gradients are initially identical, the zero-initialized states for new parameters cause the optimizer to produce different update directions than for the original parameters with their accumulated state history. This is what breaks the symmetry lock.
- Asymmetric re-warmup (Section 4.2.2) provides magnitude asymmetry: by giving the new parameters a transient learning rate boost, it accelerates their integration of gradient signal and helps them catch up to the original parameters' effective step size. This converts the broken symmetry into productive feature diversification rather than slow drift.
Neither component alone is sufficient. Optimizer-state resetting without re-warmup means the new parameters differentiate their update direction but move at the same (or slower, since Adam's second moment starts at zero and takes time to accumulate) step size, slowing recovery. Re-warmup without state resetting means the new parameters get a learning-rate boost but move in the same direction as the original parameters (due to identical momentum), perpetuating the coupling. Figure 3 shows that the combination achieves the lowest loss, and the ablation in Appendix 14 (Figure 9) confirms that global re-warmup (which is symmetric across all parameters) underperforms the asymmetric variant.
The significance is that this identifies the backward pass as the primary locus of intervention for symmetry breaking, rather than the parameter initialization or the forward function. Prior work's focus on initialization-time perturbations (uneven splitting, symmetric noise) implicitly assumed that backward dynamics would naturally amplify initial differences. SPARKLING shows this assumption is wrong and provides explicit backward-pass mechanisms that are individually necessary and collectively sufficient. This has implications beyond width expansion: any progressive learning method that involves capacity duplication (including depth expansion via layer copying) may need similar backward-pass asymmetry to fully utilize the added capacity.
5. Experimental Analysis
Evaluation Methodology
-
Dataset. All experiments use the OLMoE training corpus (Muennighoff et al., 2025), with pre-training conducted for 200B tokens total. Downstream evaluation is performed on 11 standard benchmarks: ARC-C, ARC-E (Clark et al., 2018), Arithmetic (Brown et al., 2020), BoolQ (Clark et al., 2019), CommonsenseQA (Talmor et al., 2019), HellaSwag (Zellers et al., 2019), MMLU (Hendrycks et al., 2021), OpenBookQA (Mihaylov et al., 2018), PIQA (Bisk et al., 2020), SciQ (Welbl et al., 2017), SocialIQA (Sap et al., 2019), and Winogrande (Sakaguchi et al., 2020). The paper reports both final pre-training loss (as a reference loss relative to the pre-expansion baseline) and downstream accuracy.
-
Base model. All experiments build on OLMoE (Muennighoff et al., 2025), an open-source Mixture-of-Experts language model. The pre-expansion "small" configuration uses: 24 layers, hidden size 1024, expert intermediate size 512, 16 attention heads with 4 key-value heads, head dimension 96, 64 experts with top-8 routing, SwiGLU activation, RMSNorm with pre-norm, RoPE positional embeddings, tied word embeddings, and no bias terms (Table 3). The total pre-expansion parameter count is approximately 450M active / 2.56B total. This model family is chosen because its MoE architecture makes width expansion particularly relevant—MoE models scale primarily through width dimensions (expert count, expert hidden size, attention dimensions)—and because its moderate scale (0.5B active) enables systematic ablation experiments at a meaningful training budget.
-
Metrics. Three quantitative metrics are tracked. (1) Reference loss: the pre-training cross-entropy loss, plotted as reference-loss relative to the pre-expansion baseline's loss curve—this isolates the effect of expansion by showing whether the expanded model recovers toward or diverges from the original training trajectory. (2) Final pre-training loss: the cross-entropy loss at the end of the 200B-token training budget, reported in Table 1. (3) Downstream accuracy: exact-match or multiple-choice accuracy on each of the 11 evaluation benchmarks, reported as the average across tasks. For the computational cost analysis, FLOPs saved is computed as
$1 - C^* / C_{\text{scratch}}$where$C^* \approx 6(N_{\text{small}} D_e + N_{\text{large}} (D - D_e))$and$C_{\text{scratch}} \approx 6 N_{\text{large}} D$, following the standard$C \approx 6ND$approximation (Kaplan et al., 2020). Wall-clock speed-up is measured as the ratio of total training time for the from-scratch model vs. the progressively trained model, reported in Table 2. -
Baselines. The paper compares against four categories of baselines. (1) Baseline (small): the pre-expansion model at its final state before expansion (trained on 100B tokens), evaluated to show the lower bound. (2) Baseline (expand) / From Scratch: the target-width model trained from scratch for the full 200B tokens under the same training recipe—this is the performance target that progressive learning aims to match or exceed. (3) Naive Expansion: the expanded model with copy-based initialization but without any of SPARKLING's interventions (no RMS-preserving rescaling, no asymmetric optimizer reset, no re-warmup)—this represents the default "function-preserving" approach from prior work. (4) Prior heuristics (Appendix 14, Figure 9): Uneven Splitting (Chen et al., 2016; assigning different scaling factors like fixed 1:2 or randomized r:(1−r) to copied channel pairs), symmetric ± perturbation (Yuan et al., 2023; Wu et al., 2021; adding equal-and-opposite noise that cancels in the forward pass), and global re-warmup (applying the re-warmup schedule to all parameters, not just newly introduced ones).
-
Generation budget / compute accounting. The paper does not use generation budgets in the usual sense (there is no inference-time sampling). Instead, the compute accounting is in terms of pre-training tokens and FLOPs. The standard training budget is 200B tokens total, with expansion at 100B tokens (50% of total). FLOPs are estimated using the
$C \approx 6ND$formula where$N$is active parameters and$D$is tokens. Wall-clock measurements use a cluster of 64× NVIDIA A100 80GB GPUs with global batch size 768 and per-device microbatch size 3 (Table 4). All expanded models and from-scratch baselines are trained under identical token budgets and hardware configurations for fair comparison. -
Cross-validation / statistical protocol. There is no cross-validation or statistical significance testing reported. The paper conducts sweep-based hyperparameter selection for the re-warmup parameters (
$\rho$and$\tau_w$, Appendix 12, Figure 7) on the expert-inner 2× expansion setting and then transfers the chosen values ($\rho = 1.3$,$\tau_w = 250$) to the hidden-dimension and joint expansion settings without re-tuning. All training runs appear to be single-seed; no error bars, confidence intervals, or multi-seed variance estimates are reported. This is a limitation—the robustness of the reported loss differences and downstream accuracy comparisons to random seed variation is unknown.
Main Quantitative Results
RMS-Preserving Scaling vs. Naive Initialization (Section 3.3)
The paper's first major experimental block establishes that RMS-preserved rescaling improves late-stage convergence across initialization strategies, and that the benefit is not captured by looking at the immediate loss discontinuity at the expansion moment.
Figure 1 (panels a–e) reports expert-inner 2× expansion (512 → 1024 expert intermediate dimension, expansion at 100B tokens) under five initialization pairs for the up_proj – down_proj weight matrices: copy-copy, copy-random, random-copy, random-random, and zero-copy. For each pair, two curves are shown: "Naive Init, No Scaled" (no rescaling, $\alpha = 1$) and "RMS-Preserved Scaled" (applying the derived scaling factors from Section 3.2). The key finding, stated in the paper:
"Across all initializations, Naive Init, No Scaled consistently yields a smaller immediate loss gap but worse late-stage convergence, whereas RMS-Preserved Scaled recovers steadily and converges to a lower final loss."
The quantitative pattern is consistent: the unscaled variant shows a smaller spike or discontinuity at the expansion point (the "immediate loss gap" is lower), but its loss curve remains elevated and converges higher. The RMS-preserved variant may show a larger instantaneous increase (because the scaling factor modifies the original weights, breaking exact forward-function matching), but its loss curve recovers and descends to a lower final value.
Figure 1f aggregates the final loss values across initialization pairs under RMS-Preserved Scaled only, revealing a notable hierarchy: both-sides-copied (copy-copy) significantly underperforms all other RMS-preserved variants despite providing the strongest forward-function preservation. The paper interprets this as evidence of the symmetry lock—even with correct RMS scaling, copy-copy's duplicated parameter subspaces fail to diversify, wasting the added capacity. The other variants (copy-random, random-copy, random-random, zero-copy) all achieve lower final losses, demonstrating that breaking symmetry at initialization (by not copying on at least one side) provides a benefit that RMS scaling alone cannot deliver.
Appendix 10 (Figure 5) replicates this analysis for hidden-dimension 2× expansion (1024 → 2048 at 100B tokens) and finds the same qualitative pattern: RMS-preserved rescaling consistently improves late-stage convergence relative to naive unscaled expansion across initialization pairs. This confirms that the RMS-preservation principle is not specific to a particular width axis.
Optimizer-State Reset: Symmetric vs. Asymmetric (Section 4.2.1)
Figure 2 isolates the effect of optimizer-state handling under copy-copy initialization with RMS-preserved scaling applied. Four treatments are compared:
- Drop Optimizer (reset all optimizer states to zero): converges to the highest final loss among the four.
- Copy Optimizer (duplicate existing states for new parameters): converges to a similarly high loss.
- Asymmetric Reset (retain states for original parameters, zero states for new parameters): achieves substantially faster post-expansion recovery and a lower final loss.
- Asymmetric Reset + Scaled Optimizer (additionally apply RMS-preserving scaling to the new optimizer states): performs nearly identically to the simple asymmetric reset.
The paper's interpretation: the first two treatments are symmetric—drop gives both parameter blocks zero states (so they receive identical updates), and copy gives both blocks identical non-zero states (also producing identical updates). Both exhibit the "symmetry lock" where duplicated parameters evolve identically. The asymmetric reset breaks this lock by making the optimizer transformation different for old vs. new parameters even when gradients are initially identical. The scaling variant's lack of improvement suggests that "strict state-parameter scale alignment is unnecessary and any initial mis-scaling can be quickly corrected by subsequent gradient updates."
Asymmetric Learning Rate Re-Warmup (Section 4.3)
Figure 3 evaluates the effect of asymmetric re-warmup across three width-growth settings (Inner 2×, Hidden 2×, Inner 2× & Hidden 2× jointly) and two initialization strategies (copy-copy and zero-copy). In all configurations, RMS-preserved scaling and asymmetric optimizer-state resetting are applied as the base configuration, and re-warmup is toggled on vs. off.
The finding: enabling re-warmup consistently lowers the final loss across all settings and initialization pairs. The benefit is present for both zero-copy (where new channels have near-zero forward contribution and need to integrate gradient signal from scratch) and copy-copy (where the symmetry lock has been broken by state resetting but the new parameters lack optimizer history). The gain is largest for copy-copy: with re-warmup, copy-copy achieves the lowest final loss among all configurations in all three expansion settings, closing the gap that Figure 1f showed between copy-copy and other initializations. This confirms the paper's layered logic: RMS scaling provides statistical stability, asymmetric state reset breaks the symmetry lock, and re-warmup accelerates the now-decoupled new parameters into productive feature diversification.
The re-warmup hyperparameter sweep (Appendix 12, Figure 7) under expert-inner 2× expansion shows a broad stable region: $\rho \approx 1.25$–1.3 and $\tau_w \approx 0$–250 steps achieve the lowest final loss. The chosen defaults ($\rho = 1.3$, $\tau_w = 250$) lie in this region and transfer to hidden-dimension expansion without re-tuning. The fact that $\tau_w = 0$ (no explicit re-warmup, just continuing from $\eta_e$) is within the good region indicates that the boost is helpful but not extremely sensitive.
Overall Downstream Performance (Section 5.1)
Table 1 reports the final pre-training loss and downstream accuracy across 11 benchmarks for three 2× width-growth settings: Inner 2×, Hidden 2×, and joint Hidden 2× & Inner 2×. Four models are compared per setting:
- Baseline (small): the pre-expansion model after 100B tokens (lower bound).
- Baseline (expand) / From Scratch: the target-width model trained from scratch for 200B tokens (the performance target).
- Naive Expansion: copy-based initialization with no SPARKLING interventions.
- SPARKLING: the full framework (RMS-preserved scaling + copy initialization + asymmetric state reset + asymmetric re-warmup).
Key findings from Table 1:
-
Final pre-training loss: SPARKLING consistently shows a slightly higher final loss than the from-scratch expanded baseline. This gap exists across all three expansion settings, indicating that mid-stage expansion—even with optimal interventions—does not perfectly recover the loss trajectory of training from scratch. The paper does not report the exact numerical loss values in the main text, but the pattern is visible in the Table 1 loss column.
-
Downstream average: Despite the slightly higher pre-training loss, SPARKLING matches or outperforms the from-scratch expanded baseline on most tasks and achieves the best downstream average in all three settings. The paper states: "SPARKLING matches or outperforms the from-scratch expanded baseline on most tasks and achieves the best average."
-
Naive Expansion vs. SPARKLING: The naive expansion variant consistently underperforms both the from-scratch baseline and SPARKLING across all three settings, confirming that the individual components of SPARKLING are necessary—copy initialization alone is insufficient.
-
Task-level variation: The paper reports per-task accuracies for all 11 benchmarks in Table 1. The pattern is not uniform—SPARKLING outperforms the from-scratch baseline on some tasks and slightly underperforms on others—but the average favors SPARKLING. This suggests that the benefit of progressive learning is robust across diverse evaluation domains (commonsense reasoning, science QA, arithmetic, reading comprehension).
The paper's interpretation of the loss-vs-downstream discrepancy is notable: mid-stage expansion leaves a small gap in pre-training loss, but the expanded model's downstream performance is nonetheless competitive or superior. This aligns with the broader observation in the PL literature that pre-training loss and downstream performance can decouple, and that progressive training sometimes acts as an implicit regularizer.
Computational Cost Analysis (Section 5.2)
Table 2 quantifies the computational savings of SPARKLING under a fixed 200B-token budget across all three expansion settings. The accounting:
- Inner 2× only (751M active / 5B total parameters): From-scratch FLOPs = 9.01 × 10²⁰, SPARKLING FLOPs = 7.21 × 10²⁰, FLOPs saved = 20%, wall-clock speed-up = 1.27×.
- Hidden 2× only (900M active / 5.13B total): From-scratch = 10.80 × 10²⁰, SPARKLING = 8.10 × 10²⁰, FLOPs saved = 25%, speed-up = 1.29×.
- Hidden 2× & Inner 2× jointly (1.5B active / 9.96B total): From-scratch = 18.00 × 10²⁰, SPARKLING = 11.70 × 10²⁰, FLOPs saved = 35%, speed-up = 1.49×.
The savings increase with the expansion scale: joint expansion (which grows the model more dramatically) yields the largest relative savings because the small model trains for a larger fraction of the total budget at lower cost. The wall-clock speed-ups (1.27–1.49×) are somewhat lower than the FLOPs savings would naively suggest, likely due to fixed overhead (data loading, communication) that does not scale linearly with model size.
The paper is explicit that these savings are computed at the same total token budget: both from-scratch and SPARKLING models are trained on 200B tokens. The savings come from the fact that the SPARKLING model spends the first 100B tokens training a smaller (cheaper) architecture. If the expanded model required more tokens to converge than the from-scratch model, these savings would be partially offset; the paper does not investigate whether the expanded model could benefit from a longer post-expansion training phase.
Effectiveness Under Muon Optimizer (Appendix 13)
Appendix 13 (Figure 8) validates that SPARKLING's core components generalize beyond AdamW to spectral-style optimizers. Under expert-inner 2× expansion with Muon:
- RMS-preserved scaling (Figure 8a) produces a stable and consistent improvement over naive unscaled initialization, converging to a lower final loss under the same training budget—the same qualitative finding as with AdamW.
- Asymmetric re-warmup (Figure 8b), applied on top of RMS-preserved scaling and asymmetric state reset, further lowers the final loss relative to the no-rewarmup variant.
This is significant because the paper's theoretical analysis (Section 4.1) proved that Muon's Newton-Schulz orthogonalization cannot spontaneously break the symmetry lock—it requires explicit intervention. The empirical results confirm this: even with Muon's decorrelating updates, the asymmetry must be injected via optimizer-state handling and learning-rate scheduling, not via the optimizer's internal mechanics.
Comparison to Prior Heuristics (Appendix 14)
Appendix 14 (Figure 9) evaluates three prior symmetry-breaking approaches against SPARKLING under expert-inner copy-copy expansion:
- Uneven Splitting (fixed 1:2 ratio or randomized r:(1−r) with r ∈ [0.1, 0.5]): assigns different scaling factors to the channel being copied and the copied one at initialization.
- Symmetric ± perturbation: adds equal-magnitude, opposite-sign noise to the two duplicated halves that cancels in the forward pass.
- Re-warmup All: applies the re-warmup schedule uniformly to all parameters.
All three alternatives converge to a higher final loss than SPARKLING. The paper's interpretation is that one-shot initialization heuristics (uneven splitting, perturbations) introduce initial asymmetry that is quickly washed out by identical subsequent gradient updates—they don't address the persistent symmetry in the backward dynamics. Global re-warmup fails because it perturbs the original parameters' well-adapted learning rate schedule, disrupting their optimization.
The zoomed-in inset in Figure 9 shows a transient loss up-shift immediately after expansion for SPARKLING, followed by rapid recovery. The paper characterizes this as "consistent with more effective symmetry breaking in the newly added capacity"—the brief destabilization is a sign that the new parameters are actively exploring rather than remaining locked in their initial state.
Ablation Studies and Robustness Checks
RMS-preserving scaling vs. Naive (no scaling) across five initialization pairs: RMS-preserved scaling consistently achieves lower final loss than naive unscaled initialization, despite naive initialization showing a smaller immediate loss discontinuity at expansion. Validated for expert-inner 2× expansion (Figure 1a–e) and hidden-dimension 2× expansion (Appendix 10, Figure 5). The effect is robust across copy-copy, copy-random, random-copy, random-random, and zero-copy initializations.
Copy-copy underperformance among RMS-preserved variants: Even with correct RMS scaling, both-sides-copied initialization significantly underperforms other RMS-preserved variants (Figure 1f). This identifies the symmetry lock as a failure mode that RMS preservation alone cannot address.
Optimizer-state handling (drop vs. copy vs. asymmetric reset vs. asymmetric reset + scaling): Asymmetric reset (retain original states, zero new states) substantially outperforms both symmetric baselines (drop all, copy all) in terms of recovery speed and final loss (Figure 2). Adding optimizer-state scaling provides no additional gain, suggesting that strict state-parameter scale alignment is unnecessary.
Asymmetric re-warmup vs. no re-warmup across expansion settings and initializations: Re-warmup consistently lowers final loss across Inner 2×, Hidden 2×, and joint expansion, for both copy-copy and zero-copy (Figure 3). The benefit is largest for copy-copy, where re-warmup closes the gap with other initializations and achieves the best final loss overall.
Re-warmup hyperparameters (ρ and τ_w): Sweeps under expert-inner 2× (Appendix 12, Figure 7) show a broad stable region around ρ ≈ 1.25–1.3 and τ_w ≈ 0–250 steps. The paper selects ρ = 1.3, τ_w = 250 as defaults and transfers them to hidden-dimension expansion without re-tuning, demonstrating reasonable robustness.
Zero-initialization treatment under RMS-preserving expansion (Appendix 9, Figure 4): When one side is zero-initialized, treating it as random under RMS-preserving scaling (rather than as a special loss-preserving case) yields activation RMS ratios that remain closer to the pre-expansion baseline after a few post-expansion steps. The unscaled zero variant shows persistent RMS drift without recovery.
Effectiveness under Muon (Appendix 13, Figure 8): Both RMS-preserved scaling and asymmetric re-warmup remain effective under the Muon optimizer, confirming that the framework's principles are not tied to AdamW's element-wise mechanics. This is an important robustness check given the paper's theoretical claim that Muon cannot spontaneously break the symmetry lock.
Prior heuristics vs. SPARKLING (Appendix 14, Figure 9): Uneven splitting, symmetric ± perturbation, and global re-warmup all underperform the full SPARKLING framework under copy-copy expansion. This ablation validates that the specific combination of asymmetric state reset + asymmetric re-warmup is more effective than heuristic alternatives that either inject only initialization-time asymmetry or apply symmetric post-expansion interventions.
Global vs. asymmetric re-warmup (Appendix 14, Figure 9): Global re-warmup (applying the boosted schedule to all parameters) underperforms asymmetric re-warmup (applied only to new parameters), confirming that selectively targeting new parameters without perturbing the original parameters' well-adapted schedule is critical.
Critical Assessment
Do the Experiments Support the Central Claims?
Claim: Mid-stage width expansion is viable and reduces training cost by up to 35%. The cost savings in Table 2 are computed correctly under the standard FLOPs approximation and the stated assumptions (expansion at 50% of tokens, same total token budget). The wall-clock speed-ups (1.27–1.49×) validate that the FLOPs savings translate to real time savings on the experimental hardware. However, three caveats apply. First, the savings are computed assuming no additional overhead from the expansion procedure itself; while SPARKLING's operations (copying weights, rescaling, resetting optimizer states) are cheap relative to training, this overhead is not quantified. Second, the savings assume that the 50% expansion point is optimal—the paper does not sweep expansion timing to find the best trade-off between savings and final performance. An earlier expansion would save less compute but might yield better final loss; a later expansion would save more but might degrade performance further. The optimal point is unknown. Third, the savings are reported relative to the specific training recipe used (200B tokens, specific learning rate schedule, etc.); whether the same 35% figure holds under different token budgets or recipes is untested.
Claim: SPARKLING matches or exceeds the from-scratch expanded baseline on downstream tasks. Table 1 supports this claim for the 11 benchmarks tested. The pattern of slightly higher pre-training loss but competitive or better downstream performance is consistent and interesting. However, the paper does not provide statistical significance tests or multi-seed variance estimates. With a single training run per configuration, it is impossible to determine whether the downstream performance differences (which appear to be within a few percentage points for most tasks) are reliably above noise. The claim that SPARKLING "achieves the best average" could be sensitive to the specific set of benchmarks chosen—adding or removing a single high-variance benchmark might shift the average. Additionally, the paper does not evaluate on generative tasks (e.g., summarization, translation, code generation) where the pre-training loss difference might translate more directly to output quality.
Claim: RMS-preserved scaling improves late-stage convergence compared to naive unscaled initialization.
This claim is strongly supported by Figure 1 and Appendix 10 (Figure 5) across two width axes and five initialization pairs. The effect is visually clear in the reference-loss curves: the RMS-preserved variants consistently end lower. The paper's interpretation—that the immediate loss discontinuity is misleading, and that activation-statistic stability matters more for long-term convergence—is a coherent explanation. A potential weakness: the paper compares only "RMS-preserved scaling" vs. "no scaling" as binary conditions. An intermediate comparison (e.g., scaling by a different factor, or scaling only fan-in layers but not fan-out layers) would clarify whether the specific derived scaling factors are necessary or whether any heuristic down-scaling of fan-in layers would suffice. The fact that the derived factors involve non-trivial calculations (the $1/\sqrt{1+3c}$ form for both-sides copied) suggests that getting the scaling right matters, but the paper does not ablate incorrect scaling factors to confirm this.
Claim: Copy-based expansion induces a symmetry lock that cannot be broken by initialization heuristics or orthogonalizing optimizers. The theoretical proof (Section 4.1) is rigorous for the stated assumptions (identical gradients under copy expansion, polynomial Gram map for Muon). The empirical validation in Figure 2 (Drop Opt. and Copy Opt. both underperform) and Figure 9 (uneven splitting and perturbations both underperform SPARKLING) provides supporting evidence. However, the claim that the symmetry lock is universal under copy expansion is tested only at one scale (0.5B active parameters) and one expansion point (100B tokens). At very early expansion points (e.g., 10B tokens), the pre-expansion model may not have settled into a regime where symmetry locking is as severe; at much larger scales, the stochasticity of large-batch training might provide more natural symmetry breaking through gradient noise. The paper's claim that "even advanced optimizers like Muon cannot break the symmetry" is demonstrated empirically for the specific Muon configuration used, but the paper does not explore whether different Muon hyperparameters or higher-order orthogonalization iterations might change this.
Claim: Asymmetric optimizer-state resetting and asymmetric learning rate re-warmup are each necessary and collectively sufficient for symmetry breaking. Figure 2 establishes that asymmetric state resetting outperforms symmetric alternatives. Figure 3 establishes that adding re-warmup on top of state resetting further improves convergence. However, the paper does not run the converse ablation: re-warmup without state resetting on copy-copy initialization. This would test whether re-warmup alone (without the state asymmetry) can break the symmetry lock. If re-warmup alone were sufficient, the need for state resetting would be less clear. The paper's theoretical argument predicts that re-warmup without state resetting would be ineffective (because the new parameters would move faster but in the same direction as the originals due to identical momentum), but this is untested.
Claim: SPARKLING generalizes across width axes and optimizer families. The claim is supported for the specific axes tested (expert-inner, hidden-dimension, and joint) and the two optimizer families (AdamW and Muon). However, several width axes that are common in practice are not tested: expanding the number of attention heads, expanding the number of MoE experts, or expanding the vocabulary size. Each of these would introduce unique challenges (head expansion requires care with QKV projection splitting; expert expansion requires routing-aware initialization). The generalization claim is therefore narrower than "any width expansion"—it applies specifically to the dimensional width axes tested. Similarly, only two optimizers are tested; whether SPARKLING's principles extend to other popular choices (Lion, Sophia, Adafactor) is unknown.
Genuine Weaknesses in Experimental Design
Single training run per configuration, no error bars. The paper reports single-seed results across all experiments. For 200B-token training runs, this is understandable (each run is expensive), but it means that the reported differences in loss and downstream accuracy cannot be statistically assessed. The final loss differences between SPARKLING and the from-scratch baseline (Table 1) appear small; without variance estimates, it's unclear whether they are reliably above noise.
Single model family and scale. All experiments use OLMoE at approximately 0.5B–1.5B active parameters. The paper argues this model is representative, but MoE architectures have specific properties (sparse activation, expert routing, load balancing) that might interact with width expansion in ways that dense models do not. In particular, MoE models' routing dynamics could amplify or mitigate the symmetry lock problem: if duplicated expert parameters receive different routing probabilities, natural asymmetry might arise. Conversely, if routing is load-balanced, duplicated experts might receive identical assignments and the lock could be even more severe. The paper does not analyze routing statistics before and after expansion.
No ablation of the expansion timing. All experiments expand at 100B tokens out of 200B total (50%). This is a single expansion point. The paper's motivation emphasizes that early expansion (10–30%) is not practically useful for cost savings, but it does not empirically verify that SPARKLING fails at early expansion (which would be fine) or that it succeeds at even later expansion (e.g., 75%, which would save more compute). The optimal expansion point is not characterized, which limits the practical guidance the paper provides.
No analysis of the expanded model's internal representations. The paper's theoretical framing centers on symmetry breaking and feature diversification, but it provides no direct evidence that the expanded capacity actually learns distinct features. Analyses that would strengthen this claim include: measuring the correlation between original and copied parameter vectors over training, computing the effective rank of the expanded weight matrices, or visualizing attention patterns from original vs. new heads. Without such evidence, the claim that the symmetry lock is "broken" rests entirely on loss curve improvement, which could be due to other factors (e.g., the additional parameters simply providing more capacity regardless of whether they diversify).
Downstream evaluation limited to multiple-choice benchmarks. The 11 tasks are all standard multiple-choice QA or classification benchmarks. These tasks are known to saturate for models in this parameter range and may not be sensitive enough to detect subtle differences in model quality. More challenging generative evaluations (e.g., perplexity on held-out text, few-shot reasoning, math problem solving, code generation) would provide a more demanding test of whether the expanded model truly matches the from-scratch model's capabilities.
No comparison to depth-progressive baselines. The paper motivates width expansion as an underexplored alternative to depth expansion, but it never compares SPARKLING against a depth-progressive baseline (e.g., layer stacking or block insertion at the same 100B-token point) under the same FLOPs budget. Such a comparison would directly test the paper's implicit claim that width-progressive learning deserves attention alongside depth-progressive methods.
No combination with depth expansion. The paper studies pure width expansion in isolation. In practice, a comprehensive progressive learning strategy would likely combine depth and width growth. The paper does not test whether SPARKLING's principles (RMS scaling, asymmetric optimizer reset, re-warmup) transfer to or interfere with simultaneous depth expansion, leaving the framework's generality to multi-axis growth unexamined.
Missing Experiments That Would Strengthen the Paper
- Multi-seed runs for the main results (Table 1, Figure 3) to estimate variance and confirm that the reported improvements are statistically reliable.
- Expansion timing sweep (e.g., 25%, 50%, 75% of total tokens) to characterize the trade-off between savings and final performance and identify the optimal expansion point for different width axes.
- Representational analysis (CKA similarity, effective rank, attention pattern diversity) of original vs. new parameters over post-expansion training to directly validate the symmetry-breaking mechanism.
- Evaluation on generative benchmarks (e.g., HumanEval, GSM8K, MMLU generative split) to provide a more demanding test of whether the expanded model's quality matches from-scratch training.
- Comparison to depth-progressive methods at matched FLOPs to establish width expansion as a competitive alternative.
- Scaling to larger models (e.g., 7B active parameters) to test whether SPARKLING's principles hold at scales closer to production LLM training.
- Transfer to dense architectures to verify that the findings are not specific to MoE routing dynamics.
6. Limitations and Trade-offs
The Difficulty Estimation Cost Is Unaccounted for and Dominates the Inference Budget
The assumption. SPARKLING is a one-time expansion procedure applied at a pre-specified midpoint of training (50% of total tokens in all experiments). The framework assumes that the expansion point—the fraction of total training after which to expand—is chosen correctly and that the width multiplier (always 2× in the experiments) is appropriate. The paper does not provide any mechanism or principle for choosing these values; they are fixed hyperparameters set by the experimenter. The paper is transparent about this scope limitation:
"a unified principle of simultaneous width and depth expansion has yet to be established" and "We view these as critical future work toward developing a more comprehensive, 'tuning-free' framework for progressive learning."
The consequence. This is not merely a missing feature—it is a fundamental deployment barrier. A practitioner wanting to use SPARKLING must choose: when to expand and by how much. These choices critically determine both the cost savings and the final model quality. Expanding too early (e.g., at 25% of tokens) reduces savings because the small model trains for too few tokens; expanding too late (e.g., at 75%) may degrade final performance because the expanded model has too few tokens to recover and diversify its new capacity. Expanding by less than 2× leaves savings on the table; expanding by more than 2× may exceed the regime where RMS-preserving rescaling formulas apply (the paper's derivation covers specific expansion regimes with piecewise scaling rules) or create optimization instabilities that SPARKLING does not address. Currently, these choices require expensive trial-and-error: running multiple full 200B-token training trajectories with different expansion points and width multipliers to find the best configuration, which partially defeats the cost-saving motivation.
What evidence exists. The paper performs no sweep over expansion timing (all experiments expand at exactly 100B tokens out of 200B) and no sweep over expansion ratios (all experiments use exactly 2× width growth). The computational cost analysis in Table 2 assumes the 50% expansion point without testing whether 40% or 60% would yield better cost-performance trade-offs. There is no experiment testing the 4× expansion that the RMS-preserving rescaling rules technically cover (the both-sides-copied formula handles c > 1 in Section 3.2.3). The re-warmup hyperparameter sweep (Appendix 12) is the only systematic hyperparameter study, and it is confined to the re-warmup parameters themselves under a fixed expansion configuration.
Mitigation status. The paper acknowledges this as future work (Section 6) but provides no partial solution. A practitioner currently has no principled way to select expansion timing or width multiplier beyond what the paper demonstrated (50%, 2×).
All Experiments Use a Single Model Family and Scale, with No Evidence of Transfer
The assumption. The entire experimental validation—every figure, every table, every ablation—uses the OLMoE architecture (Mixture-of-Experts, 24 layers, 64 experts with top-8 routing, SwiGLU activation) at active parameter counts ranging from 450M to 1.5B (total parameters 2.56B to 9.96B). The paper positions this choice as representative:
"We empirically validate the generality of SPARKLING across multiple width axes... and optimizer families"
But the model architecture, routing mechanism, training data distribution, and parameter scale are held fixed. The paper asserts but does not demonstrate generality beyond this specific configuration.
The consequence. Several aspects of SPARKLING's behavior could be architecture-dependent in ways that affect its applicability to other model families:
-
MoE routing dynamics. The paper experiments with expanding the expert intermediate dimension (the hidden size of each expert's MLP) and the model hidden dimension. MoE models route tokens to a subset of experts via learned gating. After expansion, the routing distribution might change—newly copied expert parameters might receive different token assignments from their originals simply because the gating network operates in the expanded hidden space, potentially introducing natural asymmetry that helps break the symmetry lock. In a dense model, there is no routing to provide this diversification signal, and the symmetry lock might be more severe. Conversely, if load-balancing keeps routing uniform, the copied expert parameters might receive identical assignments and the lock might be worse.
-
Pre-norm architecture specifics. The paper's RMS-preservation argument (Section 3.1) explicitly relies on the pre-norm residual structure (Equation 4). Many models use this (Qwen3, DeepSeek-V3, OLMoE), but some use post-norm or parallel residual formulations where the interaction between branch outputs and the residual stream differs. The paper's derived scaling factors might not directly apply to post-norm architectures because the balance between residual stream and branch output is sensitive to different quantities.
-
Parameter scale. At 0.5B–1.5B active parameters, the models are relatively small by contemporary LLM standards. The paper's symmetry lock analysis (Section 4.1) assumes deterministic gradient identities under exact copying, but at larger scales with larger batch sizes and higher learning rates, gradient stochasticity (from data sampling, dropout, or distributed training noise) might provide natural symmetry breaking that partially mitigates the lock. Conversely, at much larger scales where training is more expensive per step, the cost of delayed recovery from suboptimal expansion might be proportionally larger.
-
Optimizer interactions. The paper tests AdamW and Muon, but many production training pipelines use other optimizers (Lion, Sophia, Adafactor, distributed Shampoo variants). The proof that Muon cannot break the symmetry lock (Section 4.1) is specific to Newton-Schulz orthogonalization as a polynomial Gram map; whether other spectral optimizers with different orthogonalization schemes (e.g., Shampoo's preconditioner structure) exhibit the same property is untested.
What evidence exists. The paper provides two pieces of cross-setting validation: Muon optimizer results in Appendix 13 (Figure 8), confirming that RMS-preserved scaling and asymmetric re-warmup transfer to a non-AdamW optimizer, and multi-axis expansion (Inner, Hidden, Joint) in Figures 3 and 5 and Table 1, confirming that the principles hold across width axes within the same architecture. But these are within-family validations. There is no dense-model experiment, no post-norm experiment, no experiment at >1.5B active parameters, and no experiment with a different training data distribution.
Mitigation status. The paper does not claim to have tested dense models, other scales, or other training distributions. The title claim of "generality" is empirically supported only for the specific dimensions varied (width axes, optimizer families within the tested architecture). A practitioner using a dense Transformer architecture or a substantially different model scale cannot assume SPARKLING's specific hyperparameters (ρ = 1.3, τ_w = 250, expansion at 50%) will transfer without re-validation.
The Symmetry-Breaking Mechanism Is Demonstrated Through Loss Curves, Not Through Direct Representational Evidence
The assumption. The paper's central diagnostic claim is that copy-based initialization creates a "symmetry lock"—duplicated parameters receive identical gradients and evolve identically, rendering the expanded capacity functionally redundant. The paper further claims that asymmetric optimizer-state resetting and re-warmup "break" this lock, enabling the new parameters to diversify into distinct features. These claims are supported theoretically (Section 4.1 derives the gradient identities; Section 4.1 proves Muon's orthogonalization cannot break the lock) and through loss-curve evidence (Figures 2 and 3 show that asymmetric interventions improve final loss). However, the paper provides no direct measurement of parameter symmetry or feature diversity.
The consequence. Without direct representational evidence, alternative explanations for the loss improvement cannot be ruled out:
- The new parameters might not diversify meaningfully but might simply provide additional capacity that improves conditioning or reduces interference, with both copied and original parameters remaining highly correlated.
- The asymmetric learning rate re-warmup might be acting as a regularization mechanism (temporarily elevated learning rates can have annealing-like effects) rather than specifically enabling feature diversification.
- The loss improvement from asymmetric state resetting might come primarily from discarding stale optimizer momentum for the original parameters (which the "drop all" baseline also does, but less effectively because it discards too much), rather than from breaking symmetry between old and new parameters.
These alternative mechanisms are not mutually exclusive with symmetry breaking—they could all contribute—but without direct evidence, the paper's causal narrative (copying → symmetry lock → asymmetric interventions → diversification → better loss) remains a plausible interpretation of loss curves rather than a validated mechanism. This matters because if the mechanism is different from what the paper claims, the design principles might not generalize to other architectures, scales, or expansion types.
What evidence exists. The paper's evidence for symmetry and its breaking is entirely indirect:
- Theoretical: Section 4.1 proves that under exact copying and symmetric optimizer states, gradients and updates are identical—this establishes that the lock can occur, not that it does occur in the experimental setting.
- Loss curves: Figure 1f shows that copy-copy initialization underperforms other RMS-preserved variants (interpreted as evidence of the lock). Figure 2 shows that asymmetric resetting improves loss (interpreted as evidence that the lock was broken). Figure 3 shows that re-warmup further improves loss (interpreted as evidence that the now-broken symmetry enables productive diversification).
- Optimizer state comparison: Figure 2 compares drop/copy/asymmetric state handling, showing that the combination of retained old states + zeroed new states works best.
None of these measurements directly observes whether the copied parameters remain correlated or diverge, whether the new parameters develop distinct functional roles, or whether the effective rank of the expanded weight matrices increases. The paper does not report: cosine similarity between original and copied weight rows/columns over post-expansion training, the singular value spectrum of expanded weight matrices, CKA or CCA similarity between original and new representations, attention pattern diversity between original and new heads in hidden-dimension expansion, or expert routing distribution overlap for expanded expert MLPs.
Mitigation status. The paper does not acknowledge this as a limitation. The theoretical derivation of the symmetry lock is rigorous given its assumptions, and the loss-curve evidence is consistent with the proposed mechanism, but the gap between "consistent with" and "demonstrates" is significant. A skeptical reader could accept all the paper's empirical results while remaining agnostic about whether symmetry breaking is the operative mechanism.
No Robustness to Random Seed or Statistical Significance Reporting
The assumption. All experiments appear to be conducted with a single random seed per configuration. The paper reports precise numerical values for loss and downstream accuracy (Table 1, Figures 1–3) without error bars, confidence intervals, standard deviations, or any quantification of variance across training runs. The paper does not state the number of seeds used, and the absence of any variance reporting strongly implies single-seed results.
The consequence. Training LLMs at the scale of 200B tokens involves substantial randomness: weight initialization (for the from-scratch baselines), data order (even with fixed data, batch composition varies with distributed training dynamics), dropout, and floating-point non-determinism in distributed settings. The downstream evaluation benchmarks in Table 1 report accuracies that differ between configurations by single-digit percentage points for many tasks. For example, the difference between SPARKLING and the from-scratch baseline on individual tasks like MMLU or HellaSwag is often within 1–2 percentage points. Without variance estimates, it is impossible to determine whether these differences reflect a reliable improvement or are within the noise range of single-seed training runs. This directly undermines one of the paper's headline claims: that SPARKLING "matches or outperforms the from-scratch expanded baseline on most tasks and achieves the best average" (Section 5.1). If the "best average" is driven by small per-task differences that are within single-seed variance, the claim is not robust.
The loss curves in Figures 1–3 also lack any indication of variance. The reference-loss metric (relative to the pre-expansion baseline) helps control for some sources of run-to-run variation (both pre- and post-expansion phases share the same initial training trajectory), but the post-expansion phase involves different model architectures and optimizer configurations that could exhibit different sensitivities to data order and other random factors. Two runs of SPARKLING with identical hyperparameters but different seeds might produce visibly different loss curves.
What evidence exists. There is no statistical reporting anywhere in the paper. The number of training runs per data point is not stated. Appendix 12 (the re-warmup hyperparameter sweep) presents a heatmap of final loss values that appears smooth across neighboring hyperparameter settings, which is weakly suggestive of low variance, but this is not a substitute for explicit reporting.
Mitigation status. The paper does not address this limitation. The computational cost of running multiple seeds at 200B-token scale is acknowledged as substantial (64 A100 GPUs, multiple days per run), but single-seed reporting at this scale is a known challenge in the LLM training literature that the paper should at minimum acknowledge. The downstream evaluation section would be meaningfully strengthened by even a two-seed comparison for the main Table 1 results, or by reporting whether the relative ordering of methods is consistent across seeds for a subset of configurations.
Expansion Overhead and Implementation Complexity Are Not Quantified
The assumption. SPARKLING is presented as a cost-saving framework: by training a small model for half the tokens and then expanding, total FLOPs are reduced by 20–35% (Table 2). The paper's cost accounting (Section 5.2) compares C* ≈ 6(N_small D_e + N_large (D − D_e)) against C_scratch ≈ 6 N_large D and computes savings solely from this formula. The formula accounts only for forward and backward pass FLOPs during training iterations; it does not account for the one-time cost of the expansion procedure itself or any implementation overhead.
The consequence. The expansion procedure involves concrete operations that consume FLOPs, memory bandwidth, and engineering effort:
- Weight matrix manipulation: copying rows/columns, concatenating weight matrices, and applying element-wise scaling factors requires reading all expanded weight matrices from GPU memory, performing the operations, and writing back. For the joint expansion setting (1.5B active parameters), this involves manipulating billions of float16/float32 values.
- Optimizer state manipulation: the asymmetric reset requires selectively zeroing slices of optimizer state tensors (which are 2× the parameter count for AdamW, storing both m and v). This is a memory-intensive operation that must be carefully implemented to avoid fragmenting GPU memory or triggering expensive reallocations.
- Learning rate schedule modification: implementing per-parameter learning rates (where old and new parameters follow different schedules) requires either modifying the optimizer to support parameter groups with different schedulers or maintaining separate parameter tensors with different learning rate assignments. This is not supported by all training frameworks out of the box.
- Model architecture instantiation: the expanded architecture must be instantiated (either by modifying the model definition or by constructing a new model and copying weights), which adds code complexity and potential for bugs in weight-to-parameter mapping.
- Checkpointing: the checkpoint format changes at the expansion point (different tensor shapes, additional optimizer state slices), requiring custom checkpoint conversion logic or maintaining separate pre- and post-expansion checkpoint formats.
None of these costs is quantified in FLOPs, wall-clock time, or engineering effort. For the joint expansion setting, if the expansion procedure takes (hypothetically) 30 minutes of wall-clock time on the 64-GPU cluster, this is negligible compared to the multi-day training run. But if the expansion requires restarting the training job with a modified model configuration (common in cluster scheduling environments), the overhead includes job queuing time, which can be hours. The paper's wall-clock speed-up numbers (1.27–1.49×) do not indicate whether they include this overhead.
Additionally, the need for per-parameter learning rate schedules and selective optimizer state resetting imposes requirements on the training framework that might not be met by standard implementations. A practitioner using a managed training service or a less flexible codebase might find that implementing SPARKLING requires non-trivial modifications to the training loop, optimizer, and checkpointing logic.
What evidence exists. The paper provides no quantification of expansion overhead. Table 2 reports "Wall-clock (h)" for the full training runs and computes "Speed-up" as T_scratch / T_SPARKLING, but it does not specify whether these times include the expansion procedure or only the training iterations. Section 5.2 mentions the FLOPs approximation and relative reduction, but the one-time expansion cost is not added to C*.
Mitigation status. The paper does not discuss or acknowledge the overhead. It is likely that for 200B-token training runs measured in days, the one-time expansion cost is small relative to the savings (a few minutes of tensor manipulation vs. hours of saved training), but the lack of quantification means a practitioner cannot verify this. For shorter training runs or different hardware configurations, the relative overhead might be more significant. The implementation complexity is an unacknowledged practical barrier: SPARKLING is simple in principle but requires framework support that may not be universally available.
No Characterization of What Happens When Expansion Goes Wrong
The assumption. The paper's experiments demonstrate that SPARKLING's specific combination of components (RMS-preserved scaling, copy initialization, asymmetric state reset, asymmetric re-warmup) produces successful mid-stage width expansion. The ablation structure (Figures 1–3, Appendices 13–14) shows what happens when individual components are removed—performance degrades in predictable ways. However, the paper does not characterize whether SPARKLING has failure modes of its own that a practitioner might encounter at larger scales, different expansion points, or under different training conditions.
The consequence. A practitioner considering SPARKLING for a production training run faces an asymmetric information problem: the paper shows that the method works in its specific configuration, but provides no guidance on what to monitor, what failure looks like, or how to detect whether the expansion is proceeding correctly vs. heading toward a degradation. Concrete concerns include:
-
Re-warmup magnitude sensitivity at scale. The paper's hyperparameter sweep (Appendix 12) shows that ρ too large (e.g., ρ = 2.0) causes higher final loss. At larger model scales where training dynamics are more sensitive to learning rate perturbations, the acceptable range of ρ might narrow further, and the optimal value might shift. A practitioner has no way to estimate the correct ρ without running their own expensive sweep.
-
Interaction with curriculum learning or data schedule changes. The paper's training recipe uses a fixed data distribution throughout. In many production training pipelines, the data mixture or sequence length changes during training (e.g., extending context length partway through). If such a change coincides with or immediately follows width expansion, the combined perturbation might exceed what either change in isolation would cause, but the paper provides no evidence on this.
-
Divergence or loss spikes that recover vs. those that do not. Figure 9 (inset) shows that SPARKLING causes a transient loss up-shift at the expansion point that recovers within a few hundred steps under the paper's configuration. A practitioner seeing a similar (or larger) spike in their own run cannot know whether it is a normal SPARKLING signature that will recover (as in the paper) or a sign of misconfiguration that will lead to divergence. The paper does not report gradient norm statistics, activation RMS drift, or other diagnostic signals that could help distinguish healthy from unhealthy post-expansion dynamics.
-
Worst-case behavior under misconfiguration. What happens if the RMS-preserving scaling factor is computed incorrectly (e.g., applying the both-sides-copied formula to a one-side-copied initialization)? What happens if the re-warmup is applied to the wrong parameter subset? The paper does not test "adversarial" configurations to establish guardrails.
What evidence exists. The paper's ablation structure provides information about what happens when components are removed, but not when components are misconfigured. The only systematic hyperparameter study is the re-warmup ρ and τ_w sweep (Appendix 12), which shows a relatively broad stable region for the specific model and expansion setting tested. This is encouraging but does not address the broader question of failure characterization.
Mitigation status. The paper does not provide diagnostic tools, monitoring recommendations, or failure-mode characterization. Section 6 acknowledges that "a unified principle of simultaneous width and depth expansion has yet to be established" as future work, but does not address the more immediate practical question of how to safely deploy the existing width-only framework in new settings. This leaves the practitioner with a method that is empirically validated in one context and theoretically motivated, but without the operational knowledge needed to adapt it confidently.
7. Implications and Future Directions
How This Work Changes the Landscape
This paper causes a conceptual reframing of progressive width expansion, not an incremental refinement of existing heuristics. The dominant paradigm since Net2Net (Chen et al., 2016) has been that successful width expansion means preserving the forward function at the expansion moment—loss continuity buys you a smooth transition, and any symmetry problems can be patched with initialization-time perturbations. SPARKLING demonstrates that this paradigm is both insufficient and, in its narrow focus on the expansion instant, actively misleading.
The core intellectual move is to decompose the problem into two independent, equally necessary constraints that the function-preservation (FP) orthodoxy conflates:
Signal Preservation is not the same as function preservation. FP ensures the model computes the same output at step $t_e$. SPARKLING shows that what actually matters for long-term convergence is whether the model's internal activation statistics—specifically, the RMS magnitudes of hidden states at every layer—remain in the pre-expansion operating regime. The clearest evidence is Figure 1: RMS-preserved variants produce a larger instantaneous loss spike than naive unscaled initialization (because rescaling the original weights breaks exact FP), yet converge to a lower final loss. The metric that FP optimizes—loss continuity at $t_e$—is anticorrelated with the outcome that matters. This finding alone should cause the field to stop reporting "loss gap at expansion" as a meaningful quality metric for width-growth methods.
Symmetry Breaking is a backward-pass problem that persists beyond initialization. Prior work's heuristics—uneven splitting, symmetric perturbations, weight-norm rebalancing—all inject asymmetry at initialization and assume gradients will amplify it. SPARKLING's theoretical contribution is proving this assumption wrong: under exact copying, duplicated parameter blocks receive identical gradients at every step, and if the optimizer states are also symmetric, the blocks remain permanently coupled regardless of initial perturbation. The proof extends to spectral optimizers like Muon (Section 4.1), closing the plausible escape hatch that orthogonalizing updates might decorrelate naturally. The empirical confirmation (Appendix 14, Figure 9) that uneven splitting and symmetric perturbations underperform SPARKLING closes the loop: one-shot initialization asymmetry is insufficient. The symmetry problem requires persistent asymmetry in the optimizer dynamics.
The combination of these two insights produces a new design template for progressive learning: expand width by copying (best forward continuity), rescale weights to preserve activation RMS (statistical stability), reset optimizer states asymmetrically (direction asymmetry in updates), and re-warm learning rates asymmetrically (magnitude asymmetry to accelerate diversification). Each component addresses a specific, independently diagnosable failure mode. This template is more prescriptive and more mechanistic than prior "try these heuristics" approaches.
Two prior contradictions are resolved. First, the conflicting evidence on whether copy-based expansion works: it works at early training stages (where the statistical regime is fluid and symmetry locking matters less because the model hasn't converged to a narrow operating manifold) and fails at mid-stage (where both activation statistics are entrenched and optimizer states carry deep history). Prior work expanding at 10–30% of training saw success with FP alone; SPARKLING shows this was an artifact of the early expansion regime, not a property of FP. Second, the intuition that "more forward continuity is better" is partially inverted: RMS-preserved random initialization, which sacrifices perfect forward-function matching, outperforms unscaled copy initialization (Figure 1f) by avoiding the symmetry lock. Function preservation is beneficial only when paired with mechanisms that compensate for the symmetry it creates.
The paper also redirects research investment in concrete ways. The finding that verifier-analogous mechanisms are not the bottleneck (unlike in test-time compute scaling, where verifier over-optimization is the ceiling) shifts attention to optimizer-level interventions for capacity diversification. The discovery that even Muon—designed to decorrelate features—cannot break the symmetry lock implies that better optimizers alone won't solve the width-expansion problem; the intervention must be at the level of how the optimizer treats different parameter subsets. This makes research into parameter-group-specific training dynamics (already present in techniques like Layerwise Learning Rate Decay and differential learning rates for transfer learning) newly relevant to progressive learning.
Finally, the paper establishes width as a viable axis for cost-saving progressive learning at scale, which had not been demonstrated previously at mid-training. The 20–35% FLOP reduction (Table 2) is large enough to matter in practice, and the downstream parity with from-scratch training (Table 1) means the savings don't come at the cost of model quality. This opens a path for width-progressive training to complement depth-progressive methods in production training pipelines, rather than being the neglected axis it has been.
Follow-Up Research This Work Enables
1. Direct measurement of the symmetry lock and its breaking through representational analysis. SPARKLING's central claim—that copy-based initialization creates a "symmetry lock" that asymmetric optimizer-state resetting breaks—is supported through loss curves and theoretical derivation, but never directly observed. A strong follow-up would instrument the training run to track: (a) cosine similarity between original and copied weight rows/columns across post-expansion steps (the lock predicts similarity remains near 1.0 indefinitely without interventions; breaking predicts decay toward random baseline), (b) the effective rank or singular value entropy of expanded weight matrices over time (the lock predicts no increase; breaking predicts growth as new directions emerge), (c) for hidden-dimension expansion, CKA similarity between representations from original vs. new attention heads or MLP neurons, and (d) for expert-inner expansion, overlap in token routing distributions between original and copied expert parameters. This would convert the paper's plausible mechanism into a validated causal chain, and would reveal whether the symmetry actually breaks partially (the new parameters decorrelate somewhat but remain more similar than chance) or fully (they diversify to the same degree as independently trained parameters). A negative result—e.g., finding that original and copied parameters remain highly correlated even when loss improves—would force reinterpretation of the mechanism and redirect attention toward alternative explanations like improved conditioning.
2. Joint width-and-depth progressive learning with SPARKLING's principles applied to both axes. The paper studies pure width expansion and explicitly leaves simultaneous depth expansion to future work (Section 6). A natural follow-up would combine SPARKLING's width expansion at mid-training with a depth-expansion method (layer stacking, block insertion) at either the same or a different point. The key question is whether the two expansions interfere: width expansion changes the dimensionality of representations that depth-expanded layers inherit; depth expansion changes the computational graph through which RMS-preserved statistics propagate. SPARKLING's RMS-preserving rescaling rules were derived for width-only expansion; testing whether they hold under simultaneous depth growth would stress-test the framework's generality. A concrete experiment: train a baseline model at fixed width and depth, then compare (a) width-only expansion at 50% of tokens, (b) depth-only expansion at 50%, (c) both at 50% with SPARKLING's components applied to each, and (d) width at 33% + depth at 67% to test staged multi-axis growth. The metric would be final loss and downstream performance at matched total FLOPs, plus any interaction effects (does width expansion before depth produce different results than depth before width?).
3. Characterizing the optimal expansion point as a function of width multiplier and total token budget. All SPARKLING experiments expand at exactly 50% of total tokens with exactly 2× width growth. A systematic sweep of expansion timing (e.g., 25%, 40%, 50%, 60%, 75% of total tokens) at fixed 2× expansion, combined with a sweep of expansion ratios (1.5×, 2×, 3×, 4×) at fixed 50% timing, would produce a pareto frontier of cost savings vs. final performance. This would answer practical questions the paper currently leaves open: Is 50% optimal, or would 60% save more at minimal quality cost? Does 4× expansion at 50% succeed with SPARKLING, or does the increased scale of perturbation exceed the framework's stabilization capacity? The RMS-preserving rescaling rules in Section 3.2.3 already cover $d'_{\text{in}} > 2 d_{\text{in}}$ (the $c > 1$ regime), so the mathematical machinery exists—but the empirical behavior at larger expansion ratios is unknown. The experiment would also test whether the re-warmup hyperparameters ($\rho = 1.3$, $\tau_w = 250$) transfer across expansion ratios, or whether larger expansions require larger or longer boosts.
4. Stress-testing SPARKLING on dense Transformer architectures at larger scale. All experiments use OLMoE (MoE architecture, 0.5B–1.5B active parameters). A direct replication on a dense architecture (e.g., a Llama-style or GPT-style Transformer) at comparable total parameters and training tokens would test whether MoE routing dynamics are a hidden confound. The specific concern: in MoE models, tokens are routed to a subset of experts, and after expansion, the routing distribution might provide natural asymmetry (copied experts might receive different token assignments from their originals due to the expanded hidden dimension changing the gating network's behavior). In a dense model, there is no such routing—every token activates every parameter—and the symmetry lock might be more severe. A negative result (SPARKLING fails to break symmetry in dense models) would reveal that MoE routing was an unacknowledged enabling factor. A positive result (SPARKLING transfers) would substantially strengthen the generality claim. Scaling to larger models (e.g., 7B active parameters, 1T+ tokens) would also test whether the re-warmup hyperparameters and the 50% expansion point remain appropriate, or whether larger models have different sensitivity to post-expansion learning rate perturbations.
5. Developing a µP-compatible version of SPARKLING for hyperparameter transfer. The paper speculates in Section 6 about whether RMS-preserving expansion could satisfy the µP condition (Yang et al., 2022), where optimal hyperparameters transfer automatically across model scales. A concrete research direction: derive µP-compatible width-expansion rules that ensure the expanded model's optimal learning rate, initialization scale, and other hyperparameters are predictable from the small model's optimal values without re-tuning. The current RMS-preserving rescaling rules (Section 3.2) operate on weight variance ($\sigma_w$) and fan-in dimensionality ($d_{\text{in}}$), which are precisely the quantities that µP parameterizes. The research question is whether there exists a scaling of the expansion-time interventions (rescaling factor, optimizer reset fraction, re-warmup ratio) that depends only on the width multiplier and not on any other model-specific properties, such that a practitioner could run SPARKLING on a small proxy model, find optimal expansion hyperparameters, and transfer them directly to a production-scale model. This would address the current deployment barrier identified in the Limitations section (Section 6 of this analysis): practitioners must currently re-tune SPARKLING's hyperparameters for each new model configuration.
6. Investigating whether SPARKLING enables progressive learning with data distribution shifts. The paper's experiments use a fixed data distribution throughout all 200B tokens of training. In many production pipelines, the data mixture, sequence length, or domain emphasis changes during training (e.g., annealing on high-quality data in the final phase). A natural question is whether width expansion can be strategically aligned with these data shifts—e.g., expand width just before switching to a higher-quality data mix, so the new capacity is trained primarily on the better data. This would convert SPARKLING from a purely cost-saving technique into a capacity-scheduling technique where model growth is timed to coincide with data curriculum changes. The experiment would compare: (a) fixed data + expansion at 50%, (b) data shift at 50% + no expansion, (c) data shift at 50% + expansion at 50%, and (d) data shift at 50% + expansion at 50% with SPARKLING. The key question is whether SPARKLING's stability mechanisms prevent the compound perturbation (architecture change + data distribution change) from causing irrecoverable degradation, and whether the new capacity trained primarily on high-quality data yields better downstream performance than new capacity trained on the original data mix.
Practical Applications and Downstream Use Cases
Cost-efficient pre-training of MoE models at scale. The most direct application is to organizations training large Mixture-of-Experts language models. MoE architectures scale primarily through width dimensions—expert count, expert hidden dimension, and attention dimensions—making width-progressive learning particularly relevant. A team planning to train a model with, say, 10B active parameters and 100B total parameters across 1T tokens could: (1) train a smaller model (e.g., 5B active parameters) for the first 500B tokens, (2) apply SPARKLING to expand key width dimensions at the midpoint, and (3) train the expanded model for the remaining 500B tokens. Under SPARKLING's demonstrated 20–35% FLOP reduction (Table 2), this could save millions of GPU-hours without sacrificing downstream performance (Table 1). The practical requirements are modest: a training framework that supports per-parameter learning rate groups (common in modern distributed training libraries), the ability to checkpoint and reload with different tensor shapes (already necessary for architecture iteration), and a small hyperparameter sweep on a proxy model to determine the re-warmup ratio and timing. The paper's finding that the re-warmup hyperparameters ($\rho = 1.3$, $\tau_w = 250$) transfer between width axes within the same architecture (Section 4.3) suggests that a sweep on one expansion setting may generalize, reducing the tuning burden.
Budget-constrained academic or startup LLM training. For research labs or startups that cannot afford to train a target-scale model from scratch but can afford a smaller model's full training, SPARKLING provides a path to reach larger architectures within a fixed compute budget. The specific scenario: a team has a fixed GPU budget (e.g., 64 A100s for 30 days). Training a 1.5B-active-parameter model from scratch for 200B tokens might consume the entire budget. Using SPARKLING's joint expansion approach, the team could instead train a 450M-active-parameter model for 100B tokens, expand to 1.5B-active at the midpoint, and train the expanded model for the remaining 100B tokens—using 35% fewer FLOPs (Table 2) and finishing the training 1.49× faster in wall-clock time. The saved compute could be reinvested in: training for more tokens (potentially closing the small pre-training loss gap noted in Table 1), running more extensive downstream evaluations, performing architecture ablations, or simply reducing the project's cost. The critical caveat is that the team must have the engineering capability to implement per-parameter learning rate schedules and asymmetric optimizer state resetting—these are not default features of all training frameworks and may require non-trivial code modifications.
Iterative model development with progressive scaling. In industrial settings where models are continuously improved across generations, SPARKLING enables a workflow where the small model from the current generation becomes the seed for the next generation's larger model, rather than training each generation from scratch. Concretely: Generation 1 trains a 1B-active-parameter model. Generation 2 wants a 2B-active model. Instead of training from scratch, the team takes the Generation 1 checkpoint, applies SPARKLING with 2× width expansion, and continues training on new data for the remaining budget. This amortizes the Generation 1 training cost across two model versions, and provides a natural warm-start where the expanded model inherits not just the architecture but the learned representations and optimizer state. The risk—which the paper does not evaluate—is that the pre-expansion model might be overtrained on the previous generation's data distribution, and the expanded model's new capacity might need more than the standard token budget to adapt to new data. SPARKLING's asymmetric re-warmup partially addresses this by giving the new parameters a learning rate boost, but the interaction between progressive expansion and data distribution shift is unstudied.
When to Prefer This Method
The paper explicitly positions SPARKLING against training the target-width model from scratch (the "Baseline (expand)" in Table 1, "From Scratch" in Table 2). It does not compare against depth-progressive alternatives or other width-progressive methods at the same compute budget, so a broader decision matrix would be speculative. The paper's empirical boundary conditions support the following specific recommendations:
Prefer SPARKLING (mid-stage width expansion) when:
- The total training token budget is fixed and the expansion point can be chosen to be at least 40–50% of total tokens (to realize meaningful FLOPs savings; expanding earlier provides diminishing returns).
- The target architecture differs from the initial architecture primarily in width dimensions (hidden size, expert intermediate size) rather than depth or vocabulary size.
- The training framework supports per-parameter learning rate groups and selective optimizer state manipulation, or the engineering effort to add this support is justified by the compute savings.
- The downstream evaluation priorities are broad-coverage benchmarks (commonsense reasoning, QA, reading comprehension) where SPARKLING demonstrated parity with from-scratch training (Table 1).
- The optimizer is AdamW or Muon (tested), or a sufficiently similar element-wise or spectral-style optimizer (untested but plausible given the theoretical analysis).
Prefer training from scratch (no progressive expansion) when:
- The model scale is small enough that the absolute FLOPs savings from expansion are outweighed by the engineering complexity of implementing SPARKLING's per-parameter schedules.
- The training framework lacks support for differential learning rates or selective optimizer state manipulation, and adding it is impractical.
- The expansion timing cannot be chosen freely (e.g., the training pipeline has fixed-phase data schedules that don't align with a midpoint expansion).
- The architecture uses post-norm residual connections (untested—the RMS-preservation argument relies on pre-norm dynamics) or other structural features that SPARKLING's principles weren't validated on.
- The primary evaluation metric is exact pre-training loss parity, where SPARKLING shows a small but consistent gap relative to from-scratch training (Table 1).
Open question (insufficient evidence to decide): SPARKLING vs. depth-progressive methods at matched FLOPs. The paper does not run this comparison. A practitioner choosing between width-progressive SPARKLING and depth-progressive alternatives (layer stacking, block insertion) at the same compute budget currently has no basis for decision from this paper's results. The paper's theoretical framework (RMS preservation, symmetry breaking) is specific to width expansion, but the principles (maintaining activation statistics, preventing gradient coupling) might extend to depth expansion with appropriate modifications. This comparison is identified as a key follow-up direction above.
