Scaling Smart: Accelerating Large Language Model Pre-training with Small Model Initialization

🎯 Pitch

Initializing a large language model by mathematically expanding a smaller pre-trained model can slash pre-training time by 2–4× while also boosting final accuracy. This works because the larger model inherits the smaller model’s full predictive power from the first training step, avoiding the slow start of random initialization. Surprisingly, the benefit holds even when the two models are trained on completely different data, making it a practical shortcut for scaling experiments.

1. Executive Summary

This paper introduces HyperCloning, a function-preserving initialization method that expands the hidden dimensions of a pretrained small language model to initialize a larger target model, ensuring the larger model replicates the smaller model's output logits before any training begins. The approach is evaluated across three open-source model families—OPT (350M → 1.3B), Pythia (460M → 1.4B), and OLMO (1B → 2.9B)—with convergence measured on 10 downstream tasks using the Harness evaluation framework. HyperCloning achieves 2.2× to 4× faster convergence to the final accuracy of randomly initialized baselines while also improving final accuracy after training, with the gains attributed to direct weight transfer from the source model's linear layers, attention layers, normalization layers, and positional embeddings (e.g., stacking cloned weight blocks normalized by the expansion factor). The method proves effective even when the base model is trained on different data than the target model—establishing that function-preserving width expansion can accelerate pre-training convergence across model families and dataset regimes, though initial catastrophic forgetting occurs at the start of training before being overcome with sufficient token processing.

2. Context and Motivation

The Core Problem: Pre-training Large Language Models from Scratch Is Prohibitively Expensive

The central problem HyperCloning addresses is starkly economic: the cost of training large language models from randomly initialized parameters is enormous and growing. The paper opens with a concrete anchor point—"training a 12-billion-parameter model requires approximately 72,000 GPU hours" (citing Biderman et al., 2023)—and notes that public cloud pricing makes this financially burdensome. As models continue to scale following the trajectories documented by Kaplan et al. (2020) and Hoffmann et al. (2022), these costs compound, creating a situation where only well-resourced organizations can afford to experiment with and deploy state-of-the-art LLMs.

But this isn't simply a story about money. The paper identifies several structural risks built into the current paradigm of training from scratch:

Training fragility: Pre-training runs can fail for reasons ranging from improper learning rate tuning to hardware failures to loss divergence (Narayanan et al., 2021; Dubey et al., 2024). When you've invested tens of thousands of GPU hours, a late-stage failure isn't just an inconvenience—it's a catastrophic waste of compute and time.
Experimentation barriers: The high cost of training from scratch means researchers and smaller organizations cannot afford to iterate rapidly on architecture design, hyperparameter selection, or training data mixtures. Each experiment requires the full training budget, stifling the pace of progress.
Environmental impact: The energy consumption of large-scale training runs carries a substantial carbon footprint, and reducing unnecessary compute has direct environmental benefits.

Small language models, by contrast, are dramatically cheaper to train and impose lower financial and environmental burdens. But the paper acknowledges the obvious catch: "they often cannot achieve the accuracy of large models" (Section 1). This creates a fundamental tension—accuracy requires scale, but scale requires resources. Industries and businesses that prioritize performance feel they have "no choice but to scale up and utilize larger models" (Section 1).

The Gap: We Know How to Train Small Models Efficiently, but Not How to Leverage That Investment

The critical insight motivating HyperCloning is that the current pre-training paradigm treats each model size as an independent endeavor. When you train a 1.3B-parameter model, you start from random weights. When you train a 7B-parameter model, you start from random weights again. The compute spent training the smaller model is not leveraged in training the larger one—it is essentially discarded. Every large model must rediscover basic language understanding from scratch.

This is wasteful in a way that has no analogue in other engineering disciplines. If you build a bridge capable of supporting a certain load, and later need a bridge with higher capacity, you don't demolish the first bridge and start over—you extend or reinforce it. The paper asks whether a similar principle can apply to neural network training: can we develop a method to initialize large language models using smaller pre-trained models, and will such initialization bring benefits in training time and final accuracy?

This framing—which the paper explicitly states as its motivating question in Section 1—is deceptively simple but has profound implications. If the answer is yes, the economics of LLM development fundamentally change: organizations could amortize the cost of training small models across many larger descendants, experimental iteration could happen on small models with results transferred upward, and the wasteful "start from scratch" paradigm could be replaced with incremental scaling.

Prior Approaches: Model Growth and Width Expansion

The idea of growing networks rather than training them from scratch is not new. The paper acknowledges a substantial body of prior work under the umbrella term model growth (Section 1; reviewed comprehensively in Section 4), and it is worth understanding what this prior work achieved—and where it fell short.

Depth-based growth. The dominant paradigm in the model growth literature has been increasing model depth—adding more layers to an existing architecture. Methods include progressively stacking transformer blocks (Gong et al., 2019; Yang et al., 2020), duplicating blocks while accounting for residual connections (Samragh et al., 2023), and landscape-aware growing strategies (Karp et al., 2024). These approaches are effective: Du et al. (2024) conducted a comprehensive study of growth strategies and concluded that "depth growth achieves the best accuracies" among available methods.

Width-based growth (prior attempts). Expanding hidden dimensions rather than layer count has a longer history—Chen et al. (2015) introduced Net2Net for convolutional networks—but its application to modern decoder-style transformers has been limited and problematic. The key prior attempts and their shortcomings:

Non-function-preserving width expansion: Du et al. (2024) explored several strategies for width growth, including directly copying weights, projecting weights to a larger dimension, initializing new weights to zero, and randomly initializing expanded portions. Critically, none of these methods preserve the model's function—the larger model's outputs diverge from the smaller model's outputs at initialization. Their conclusion was stark: "non-function-preserving width growth results in poorer performance" compared to depth growth. This creates a gap: width expansion is architecturally desirable, but existing methods don't work well.
Diagonal initialization (Shen et al., 2022): This method initializes the non-diagonal blocks of expanded weight matrices to zero, which does preserve function in a limited sense but creates sparse weight structures that, as HyperCloning's ablation studies show, lead to slower convergence compared to symmetric approaches.
The symmetry concern (Wang et al., 2023b): When expanding width by duplicating neurons, a theoretical concern arises: duplicated neurons receive identical gradients and might remain identical throughout training, never developing independent specializations. Wang et al. (2023b) explicitly raised this as a limitation of neuron-duplication approaches, suggesting that symmetry could prevent the expanded model from utilizing its full parameter capacity. This concern had not been empirically resolved for large-scale transformer training.
BERT-focused width expansion (Chen et al., 2021): Width expansion was explored for encoder-only BERT-style transformers, but these methods did not generalize to the decoder-style architectures (GPT, OPT, Pythia, OLMO) that dominate modern LLM development. Specifically, Chen et al. (2021) did not handle the attention mechanism's dimensionality requirements, positional embedding scaling, or the scale of models and datasets now in use.

The fundamental gap in prior width expansion. Reading across this literature, the paper identifies a clear pattern: prior width expansion methods for transformers either (a) failed to preserve function, causing an initial accuracy collapse that undercut any benefits of warm-starting, or (b) preserved function only partially or with architectural compromises that slowed subsequent training. No prior method simultaneously achieved function preservation, width scaling for decoder transformers, and empirical validation at modern scales (billions of parameters, hundreds of billions of training tokens).

Why Width Expansion Matters Alongside Depth

The paper's focus on width expansion is not arbitrary—it is motivated by specific architectural and practical considerations that depth-only growth cannot address:

Complementary to depth growth: The paper explicitly positions HyperCloning as addressing a technique that "can be accompanied by any of these [depth growth] methods to provide a full recipe for model scaling" (Section 2). Width and depth are orthogonal scaling axes, and a complete model growth toolkit needs both.
Accuracy, robustness, and inference efficiency: The paper states that "width scaling can be beneficial for increased model accuracy, robustness, and inference efficiency, compared to solely increasing depth" (Section 2). Wider models have different computational properties than deeper ones—they can exploit more parallelism at inference time (since width increases don't add sequential computation the way deeper layers do) and may learn different types of representations.
The design space: Modern LLM architectures explore trade-offs between width and depth (e.g., OLMO uses wider but shallower configurations). A method that only supports depth expansion constrains the design space of target architectures to those that are simply deeper versions of the source model. HyperCloning enables width scaling within a fixed depth, allowing practitioners to target a broader range of architectural configurations.

How HyperCloning Positions Itself

The paper positions HyperCloning through four design goals that collectively differentiate it from prior work (Section 2):

Expansion dimension: The target network must have larger hidden dimensions than the source network while maintaining the same number of layers—pure width scaling.
Function preservation: "After converting the smaller model to its equivalent larger model, the logits in the final layers of both networks should match." This is the key differentiator from Du et al. (2024)'s width methods, which lost functional equivalence at initialization. Function preservation gives the larger model a genuine warm start—it inherits the smaller model's accuracy before any training tokens are processed on the larger architecture.
Low compute overhead: The conversion from small to large model should be "straightforward, avoiding heavy computations or iterative updates." This distinguishes HyperCloning from knowledge distillation approaches (Xu et al., 2024; Zhong et al., 2023), which require training the larger model to match the smaller model's outputs—a process that itself consumes compute. HyperCloning is a deterministic, one-shot weight transformation with negligible cost.
Unchanged training loop: "For ease of deployment, the training loop should remain unchanged. The only modification should be in the network initialization." This is a practical engineering constraint: if a method requires modifying the optimizer, loss function, or training schedule, adoption becomes much harder. HyperCloning operates purely as an initialization strategy, leaving the standard pre-training pipeline intact.

The paper frames these criteria as a response to what prior approaches got wrong. Du et al. (2024) showed that non-function-preserving width expansion underperforms; HyperCloning addresses this with function preservation. Shen et al. (2022) produced slow convergence with diagonal initialization; HyperCloning's symmetric approach avoids zero-initialized blocks. Wang et al. (2023b) raised concerns about neuron symmetry; HyperCloning empirically investigates whether symmetry breaks during training (showing it does, likely due to dropout). Chen et al. (2021) focused on BERT and didn't scale to modern decoder-only LLMs; HyperCloning explicitly handles attention layers, positional embeddings, layer normalization, and large-scale training regimes.

The paper also notes a practical motivation that distinguishes it from much prior work: the base models are freely available. The authors downloaded pre-trained weights from HuggingFace—they didn't need to train the small models themselves. In a world where HuggingFace hosts thousands of pre-trained models across scales, a method that can repurpose these existing checkpoints to accelerate larger training runs has immediate practical value. The OLMO experiment is particularly illustrative: the OLMO-1B base model was already trained on 2.4T tokens, and HyperCloning transfers this knowledge to initialize OLMO-2.9B before any training begins. This isn't just an algorithmic curiosity—it's a direct path to amortizing the immense compute already invested in publicly available models.

3. Technical Approach

3.1 Reader Orientation

The HyperCloning system is a weight transformation procedure that takes a fully-trained small language model and directly produces the initial weights for a larger target model—without any training, distillation, or optimization—such that the larger model, before seeing a single training token, computes exactly the same outputs as the smaller model. It solves the problem of leveraging existing pre-trained models to accelerate training of larger ones by ensuring that the expensive knowledge already encoded in the small model's parameters is transferred intact to initialize the larger architecture rather than being discarded in favor of random weights.

3.2 Big-Picture Architecture (Diagram in Words)

The system has two stages connected by a deterministic cloning procedure:

Source Model (pre-trained, frozen): A smaller language model that has already been fully pre-trained on some corpus. This model contains linear layers (feed-forward, attention projections, embedding, unembedding), attention mechanisms with multi-head structure, layer normalization operations, and positional embeddings. All of its parameters are fixed—HyperCloning reads them, never modifies them.
HyperCloning Transformation (zero-compute, one-shot): A set of mathematical rules that take each weight matrix, bias vector, and parameter tensor from the source model and produce corresponding—but dimensionally expanded—equivalents for the target model. The key operations are: (a) tiling or stacking copies of source weight blocks into larger target weight matrices, (b) dividing by the expansion factor to preserve activation magnitudes, and (c) optionally adding small random noise tensors to break symmetry. There are distinct rules for linear layers (three cases depending on which dimensions expand), attention layers (with careful scaling for the query-key dot product), layer normalization, and positional embeddings.
Target Model (initialized but untrained): The destination model with larger hidden dimensions (and potentially more attention heads) whose weights are set entirely by the HyperCloning transformation. Immediately after initialization, this model produces output logits identical to the source model's. It then enters standard language model pre-training with no modifications to the training loop, optimizer, or hyperparameters.

Information flows strictly forward: source model parameters → HyperCloning rules → target model initialization → standard pre-training. There is no feedback loop, no iterative refinement, and no change to the training pipeline beyond weight initialization.

3.3 Roadmap for the Deep Dive

First, the core mathematical operation—vector cloning—since every subsequent mechanism depends on understanding how repetition creates expanded representations while preserving information.
Second, the three cases of linear layer expansion, because linear layers are the fundamental building block and their treatment (which dimensions expand, how weights are constructed, where noise is added) establishes the pattern for everything else.
Third, attention layer cloning, which is the most subtle component—it requires handling both head dimension expansion and head count expansion, plus a critical scaling factor to preserve attention scores.
Fourth, layer normalization and positional embedding cloning, which are structurally simpler but essential for complete function preservation.
Fifth, the noise addition mechanism and symmetry-breaking considerations, which addresses the theoretical concern that duplicated neurons might remain identical throughout training.
Sixth, the overall algorithmic summary and design rationale, bringing together why the method satisfies its four design goals (expansion, function preservation, low overhead, unchanged training loop) and how it differs from prior width-expansion approaches.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a methodology paper whose core idea is that large language models can be deterministically initialized from smaller pre-trained ones by applying simple algebraic transformations—tiling weight blocks and scaling by the expansion factor—that guarantee mathematical equivalence of the forward pass at initialization.

Vector Cloning: The Fundamental Building Block

The entire method rests on a single operation: vector cloning. Let $\mathbf{x}_S \in \mathbb{R}^d$ be a hidden representation vector in the source (small) network—this could be a token embedding, an intermediate activation after a layer norm, the output of a feed-forward block, or any other vector of dimension $d$ . To expand to dimension $nd$ for an $n$ -fold expansion (the paper primarily uses $n=2$ , but notes the method generalizes to arbitrary $n$ ), the cloned vector $\mathbf{x}_D \in \mathbb{R}^{nd}$ is formed by:

$\mathbf{x}_D = \begin{bmatrix} \mathbf{x}_S \\ \vdots \\ \mathbf{x}_S \end{bmatrix}$

where the operation stacks $n$ identical copies of $\mathbf{x}_S$ vertically to form a vector $n$ times as long.

What it computes: The operation takes a vector from the small model and produces a vector for the large model where the first $d$ components equal $\mathbf{x}_S$ , the next $d$ components also equal $\mathbf{x}_S$ , and so on through $n$ blocks. Every block of $d$ consecutive components in the expanded vector is identical to the original representation.

Why this form: This is the simplest possible expansion that preserves all information—no computation is required beyond memory copying, and any function of the source vector can be recovered from the expanded vector by examining any single block. The redundancy is deliberate: it gives the target model $n$ independent "channels" through which information can flow, but at initialization, all channels carry identical signals, forcing the model to start from the same functional state as the source. An alternative—like padding zeros to expand dimension—would lose information (the zero components would destroy the representational content when processed by subsequent layers). The repetition approach ensures that every subsequent operation on $\mathbf{x}_D$ can be designed to replicate the source model's computations.

Linear Layer Cloning: Three Cases

Linear (fully-connected) layers are the building blocks of transformers—they appear in feed-forward networks, attention projections (query, key, value, output), embedding layers, and the final unembedding layer. A linear layer computes $\mathbf{y} = W\mathbf{x} + \mathbf{b}$ , where $W$ is a weight matrix, $\mathbf{b}$ is a bias vector, $\mathbf{x}$ is the input, and $\mathbf{y}$ is the output. When we expand hidden dimensions, the linear layers connecting these expanded dimensions must also expand. The paper identifies three distinct cases depending on which of the input and output dimensions are expanded (Figure 3).

Case 1: Only the input is expanded (output dimension unchanged).

This occurs at the unembedding layer—the final linear layer that maps from the model's hidden dimension to the vocabulary size. The input comes from the expanded hidden state, so it is an $n$ -fold cloned vector of dimension $nd$ , but the output must remain the original vocabulary size $v$ (since the vocabulary doesn't change when we scale model width). The source layer has $W_S \in \mathbb{R}^{v \times d}$ and $\mathbf{b}_S \in \mathbb{R}^{v}$ .

The expanded weight matrix is:

$W_D = \begin{bmatrix} \frac{W_S}{2} + \eta_1 & \frac{W_S}{2} - \eta_1 \end{bmatrix}$

where $\eta_1$ is a random tensor of the same shape as $W_S$ (so $\eta_1 \in \mathbb{R}^{v \times d}$ ). The bias is unchanged: $\mathbf{b}_D = \mathbf{b}_S$ .

What it computes: The output is:

$\mathbf{y}_D = W_D \mathbf{x}_D + \mathbf{b}_D = \begin{bmatrix} \frac{W_S}{2} + \eta_1 & \frac{W_S}{2} - \eta_1 \end{bmatrix} \begin{bmatrix} \mathbf{x}_S \\ \mathbf{x}_S \end{bmatrix} + \mathbf{b}_S$

$= \left( \frac{W_S}{2} + \eta_1 \right) \mathbf{x}_S + \left( \frac{W_S}{2} - \eta_1 \right) \mathbf{x}_S + \mathbf{b}_S$

$= W_S \mathbf{x}_S + \mathbf{b}_S = \mathbf{y}_S$

The noise terms cancel exactly ( $+\eta_1 \mathbf{x}_S - \eta_1 \mathbf{x}_S = 0$ ), and the weight contributions sum to the full $W_S$ because each half-weight block processes one copy of the input. The expanded model produces exactly the source model's logits.

Why this form: The weight matrix is split into two horizontal blocks ( $\frac{W_S}{2} \pm \eta_1$ ), each of shape $\mathbb{R}^{v \times d}$ , which together span the expanded input dimension $2d$ . The factor of $\frac{1}{2}$ is essential—without it, the output would be $2W_S\mathbf{x}_S$ , doubling the logit magnitudes and destroying calibration. The $\eta_1$ term adds symmetry-breaking noise (discussed in detail below) that does not affect the function preservation property but gives the duplicated weight blocks slightly different initial values so they can develop distinct specializations during training. The bias is simply copied because it has no dependence on the input dimension.

Case 2: Only the output is expanded (input dimension unchanged).

This occurs at the embedding layer—the first layer that maps from token indices to hidden representations. The input is a one-hot (or index) of dimension $v$ (the vocabulary size, unchanged by width scaling), but the output must be an expanded vector of dimension $nd$ for the rest of the network. The source layer has $W_S \in \mathbb{R}^{d \times v}$ and $\mathbf{b}_S \in \mathbb{R}^{d}$ .

The expanded weight matrix is:

$W_D = \begin{bmatrix} W_S \\ W_S \end{bmatrix}$

and the expanded bias is:

$\mathbf{b}_D = \begin{bmatrix} \mathbf{b}_S \\ \mathbf{b}_S \end{bmatrix}$

What it computes: For an input $\mathbf{x}_S$ (a one-hot vector of dimension $v$ ), the output is:

$\mathbf{y}_D = W_D \mathbf{x}_S + \mathbf{b}_D = \begin{bmatrix} W_S \\ W_S \end{bmatrix} \mathbf{x}_S + \begin{bmatrix} \mathbf{b}_S \\ \mathbf{b}_S \end{bmatrix} = \begin{bmatrix} W_S \mathbf{x}_S + \mathbf{b}_S \\ W_S \mathbf{x}_S + \mathbf{b}_S \end{bmatrix} = \begin{bmatrix} \mathbf{y}_S \\ \mathbf{y}_S \end{bmatrix}$

The output is simply two (or $n$ ) copies of the source embedding stacked vertically, producing a cloned hidden representation.

Why this form: The weight matrix is formed by stacking two copies of $W_S$ vertically, so each token index maps to a vector that is the cloned version of what it would have mapped to in the source model. No scaling is needed because the input dimension is unchanged—the same token index retrieves the same embedding values, just replicated to fill the expanded dimension. Noise is not added in this case (the paper's ablation studies in Section 3.4 show that symmetric initialization works well for Case 2). The bias is similarly stacked.

Case 3: Both input and output are expanded.

This is the most common case—it covers all hidden linear layers in the feed-forward networks and attention projections (except those at the boundaries). Both the input to the layer and the output from the layer must be expanded, because the hidden dimension is increased uniformly throughout the interior of the network. The source layer has $W_S \in \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}$ and $\mathbf{b}_S \in \mathbb{R}^{d_{\text{out}}}$ .

The expanded weight matrix is:

$W_D = \begin{bmatrix} \frac{W_S}{2} + \eta_1 & \frac{W_S}{2} - \eta_1 \\ \frac{W_S}{2} + \eta_2 & \frac{W_S}{2} - \eta_2 \end{bmatrix}$

where $\eta_1$ and $\eta_2$ are independent random tensors, each of the same shape as $W_S$ . The bias is:

$\mathbf{b}_D = \begin{bmatrix} \mathbf{b}_S \\ \mathbf{b}_S \end{bmatrix}$

What it computes: Given a cloned input $\mathbf{x}_D = \begin{bmatrix} \mathbf{x}_S \\ \mathbf{x}_S \end{bmatrix}$ , the output is:

$\mathbf{y}_D = \begin{bmatrix} \frac{W_S}{2} + \eta_1 & \frac{W_S}{2} - \eta_1 \\ \frac{W_S}{2} + \eta_2 & \frac{W_S}{2} - \eta_2 \end{bmatrix} \begin{bmatrix} \mathbf{x}_S \\ \mathbf{x}_S \end{bmatrix} + \begin{bmatrix} \mathbf{b}_S \\ \mathbf{b}_S \end{bmatrix}$

$= \begin{bmatrix} (\frac{W_S}{2} + \eta_1) \mathbf{x}_S + (\frac{W_S}{2} - \eta_1) \mathbf{x}_S + \mathbf{b}_S \\ (\frac{W_S}{2} + \eta_2) \mathbf{x}_S + (\frac{W_S}{2} - \eta_2) \mathbf{x}_S + \mathbf{b}_S \end{bmatrix}$

$= \begin{bmatrix} W_S \mathbf{x}_S + \mathbf{b}_S \\ W_S \mathbf{x}_S + \mathbf{b}_S \end{bmatrix} = \begin{bmatrix} \mathbf{y}_S \\ \mathbf{y}_S \end{bmatrix}$

The output is a cloned version of the source layer's output. The noise terms cancel within each row independently— $+\eta_1 \mathbf{x}_S - \eta_1 \mathbf{x}_S = 0$ and $+\eta_2 \mathbf{x}_S - \eta_2 \mathbf{x}_S = 0$ —so the function preservation property holds regardless of the noise magnitude.

Why this form: The weight matrix is partitioned into a 2×2 block structure (for 2-fold expansion; $n \times n$ for general $n$ ). Each block has shape $\mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}$ , matching the source weight matrix shape, so the overall matrix has shape $\mathbb{R}^{2d_{\text{out}} \times 2d_{\text{in}}}$ . The diagonal blocks ( $\frac{W_S}{2} \pm \eta_1$ in the first row, $\frac{W_S}{2} \pm \eta_2$ in the second) process the two copies of the input differently due to the noise, while the anti-diagonal blocks cancel the noise. The $\frac{1}{2}$ scaling is essential because each output component receives contributions from both input copies—without it, outputs would be doubled. The structure ensures that when the input is perfectly cloned (which it will be at initialization, since all prior layers were also cloned), the output is perfectly cloned. The independent noise tensors $\eta_1$ and $\eta_2$ give the two output "channels" different starting points for the weight optimization landscape.

The normalization by expansion factor. A unifying principle across all cases: wherever an output receives contributions from multiple expanded copies of the same source signal, the weight contributions are divided by the expansion factor $n$ (e.g., $\frac{1}{2}$ for 2-fold). This maintains the standard deviation of weights at $\frac{1}{n}$ of the source model's weight standard deviation, which the paper notes "aligns with the standard deviation requirement proposed by Glorot and Bengio (2010)"—the Xavier initialization principle that weight variance should scale inversely with fan-in to keep activation variances stable across layers.

Attention Layer Cloning

The attention mechanism requires special treatment because the dot-product operation between queries and keys creates a quadratic interaction between expanded dimensions that would otherwise distort the attention scores. The paper describes two approaches to expanding multi-head attention: expanding the dimension of each head, and expanding the number of heads.

Approach 1: Expanding the attention head dimension.

When each head's internal dimension is expanded, the query, key, and value projection matrices are each treated as linear layers undergoing Case 3 expansion (both input and output expand). However, a critical scaling correction is required for the query projection.

Let $\mathbf{q}_S$ and $\mathbf{k}_S$ be query and key vectors for a single head in the source model, each of dimension $d$ (the original head dimension). After cloning, the expanded query and key are:

$\mathbf{q}_D = \begin{bmatrix} \mathbf{q}_S \\ \mathbf{q}_S \end{bmatrix}, \quad \mathbf{k}_D = \begin{bmatrix} \mathbf{k}_S \\ \mathbf{k}_S \end{bmatrix}$

The attention score in the source model (pre-softmax) is:

$a_S = \frac{\mathbf{q}_S \cdot \mathbf{k}_S}{\sqrt{d}}$

In the expanded model, if we compute it naively:

$a_D^{\text{naive}} = \frac{\mathbf{q}_D \cdot \mathbf{k}_D}{\sqrt{2d}} = \frac{\mathbf{q}_S \cdot \mathbf{k}_S + \mathbf{q}_S \cdot \mathbf{k}_S}{\sqrt{2d}} = \frac{2(\mathbf{q}_S \cdot \mathbf{k}_S)}{\sqrt{2d}} = \sqrt{2} \cdot \frac{\mathbf{q}_S \cdot \mathbf{k}_S}{\sqrt{d}} = \sqrt{2} \cdot a_S$

The doubled dot product numerator (because each copy of $\mathbf{q}_S$ interacts with both copies of $\mathbf{k}_S$ ) does not cancel with the $\sqrt{2d}$ denominator—we get a factor of $\sqrt{2}$ inflation. The paper's solution: scale the expanded query weights by $\sqrt{\frac{d_S}{d_D}}$ , where $d_S$ is the source head dimension and $d_D$ is the expanded head dimension. For 2-fold expansion, this is $\sqrt{\frac{1}{2}}$ , which cancels the unwanted $\sqrt{2}$ factor.

Concretely, after applying Case 3 expansion to the query projection weights, multiply all query weight values by $\sqrt{\frac{d_S}{d_D}}$ . This ensures that when the cloned query interacts with the cloned key, the dot product magnitude remains exactly the source model's dot product, and therefore the attention distribution (after softmax) is identical.

Why this additional scaling is necessary: The linear layer expansion formulas handle the matrix-vector products correctly in isolation, but the attention mechanism involves a product of two expanded vectors (query and key), which introduces a cross-term that doesn't appear in single-argument linear operations. The standard Case 3 expansion would give $\mathbf{q}_D = \begin{bmatrix} \mathbf{q}_S \\ \mathbf{q}_S \end{bmatrix}$ , which is correct for the vector's dimensionality but leads to inflated dot products. The $\sqrt{\frac{d_S}{d_D}}$ factor compensates for this structural property of the attention operation.

Approach 2: Expanding the number of attention heads.

This case is structurally simpler. If the source model has $h$ heads and the target model should have $nh$ heads (e.g., doubling from 16 to 32 heads as in OPT-350M → OPT-1.3B), the paper states: "we can simply duplicate the attention heads." Each source head's full set of query, key, value, and output projection weights is copied $n$ times, and the heads operate independently as they do in standard multi-head attention. The concatenation of head outputs naturally produces the expanded hidden dimension.

The output projection after attention. After the multi-head attention operation (whether heads are expanded in dimension or count), the concatenated head outputs pass through a final linear projection (often called $W_O$ ). This projection undergoes Case 3 expansion if the head dimension was expanded, or Case 2 expansion (output only) if the head count was expanded and the per-head dimension stayed the same. The paper handles both scenarios within the Cases framework.

Layer Normalization Cloning

Layer normalization applies an affine transformation to normalized activations:

$\ell(\mathbf{x}_S) = \frac{\mathbf{x}_S - \mathbb{E}[\mathbf{x}_S]}{\sqrt{\text{var}(\mathbf{x}_S) + \epsilon}} \cdot \gamma_S + \beta_S$

where $\gamma_S$ and $\beta_S$ are learnable scale and shift parameters (each of dimension $d$ ), $\mathbb{E}$ and $\text{var}$ are computed over the feature dimension, and $\epsilon$ is a small constant for numerical stability.

The cloned layer norm parameters are:

$\gamma_D = \begin{bmatrix} \gamma_S \\ \gamma_S \end{bmatrix}, \quad \beta_D = \begin{bmatrix} \beta_S \\ \beta_S \end{bmatrix}$

What it computes: For a cloned input $\mathbf{x}_D = \begin{bmatrix} \mathbf{x}_S \\ \mathbf{x}_S \end{bmatrix}$ , the cloned layer norm produces:

$\ell(\mathbf{x}_D) = \frac{\begin{bmatrix} \mathbf{x}_S \\ \mathbf{x}_S \end{bmatrix} - \mathbb{E}\left[\begin{bmatrix} \mathbf{x}_S \\ \mathbf{x}_S \end{bmatrix}\right]}{\sqrt{\text{var}\left(\begin{bmatrix} \mathbf{x}_S \\ \mathbf{x}_S \end{bmatrix}\right) + \epsilon}} \cdot \begin{bmatrix} \gamma_S \\ \gamma_S \end{bmatrix} + \begin{bmatrix} \beta_S \\ \beta_S \end{bmatrix}$

The key property: $\mathbb{E}\left[\begin{bmatrix} \mathbf{x}_S \\ \mathbf{x}_S \end{bmatrix}\right] = \mathbb{E}[\mathbf{x}_S]$ (the mean of the cloned vector equals the mean of the original, since both halves have the same mean). Similarly, $\text{var}\left(\begin{bmatrix} \mathbf{x}_S \\ \mathbf{x}_S \end{bmatrix}\right) = \text{var}(\mathbf{x}_S)$ (the variance is unchanged, since the set of values across both halves is the same as the set in one half, just duplicated). Therefore:

$\ell(\mathbf{x}_D) = \begin{bmatrix} \frac{\mathbf{x}_S - \mathbb{E}[\mathbf{x}_S]}{\sqrt{\text{var}(\mathbf{x}_S) + \epsilon}} \cdot \gamma_S + \beta_S \\ \frac{\mathbf{x}_S - \mathbb{E}[\mathbf{x}_S]}{\sqrt{\text{var}(\mathbf{x}_S) + \epsilon}} \cdot \gamma_S + \beta_S \end{bmatrix} = \begin{bmatrix} \ell(\mathbf{x}_S) \\ \ell(\mathbf{x}_S) \end{bmatrix}$

The normalized output is a cloned vector.

Why this works: Layer normalization's statistics (mean and variance) are invariant under duplication—adding more copies of the same values doesn't change the mean or variance of the set. The affine parameters are simply stacked because they operate element-wise on each normalized component. The paper notes that "similar argument is true for batch normalization, RMS normalization, and group normalization"—any normalization scheme whose statistics are invariant under value duplication will work with straightforward parameter stacking.

Positional Embedding Cloning

Positional embeddings encode position information for each token in the sequence. In the source model, the positional embedding for position $i$ is a vector $\mathbf{p}_S(i) \in \mathbb{R}^d$ . The expanded positional embedding is defined as:

$\mathbf{p}_D(i) = \begin{bmatrix} \mathbf{p}_S(i) \\ \vdots \\ \mathbf{p}_S(i) \end{bmatrix}$

What it computes: For each position $i$ , the expanded positional embedding is simply $n$ copies of the source model's positional embedding for that position, stacked vertically. No scaling is needed because positional embeddings are added to token embeddings (not multiplied), and addition distributes over the stacking operation.

Why this form: Positional embeddings must have the same dimension as the token embeddings they are added to. Since token embeddings are expanded by Case 2 (stacking vertically, producing cloned vectors), the positional embeddings must be expanded in exactly the same way to maintain compatibility. The paper notes that in their codebase, they "define PyTorch equivalents of the expanded positional embedding layers when necessary," suggesting that some position encoding schemes (such as sinusoidal encodings) can be implemented more efficiently by simply evaluating the base encoding function at the expanded dimensionality rather than literally copying vectors, though the result is mathematically equivalent.

Noise Addition and Symmetry Breaking

A theoretical concern from prior work (Wang et al., 2023b) is that when neurons are duplicated, they receive identical gradients during training and might remain functionally identical forever, effectively wasting the additional parameters. The duplicated neurons would learn in lockstep, never developing distinct specializations, and the expanded model would have the representational capacity of the smaller model despite having more parameters.

HyperCloning addresses this through two mechanisms:

Explicit noise in Case 1 and Case 3 weight expansions. In Case 1, the weight matrix is $[\frac{W_S}{2} + \eta_1, \frac{W_S}{2} - \eta_1]$ ; in Case 3, it is a 2×2 block matrix with $\pm\eta_1$ and $\pm\eta_2$ offsets. The noise tensors $\eta$ are random, and the paper specifies a signal-to-noise ratio of 10 dB in their experiments for the noisy variants—meaning the noise power is 1/10 of the signal power, small enough to not meaningfully disrupt the function preservation property but large enough to create distinct initialization for duplicated blocks.
Implicit symmetry breaking from training stochasticity. The paper observes empirically that even without explicit noise (the "symmetric" initialization without $\eta$ terms), symmetry breaks during training. The authors attribute this to "random operations such as dropout"—dropout randomly zeroes different subsets of neurons in different forward passes, meaning duplicated neurons see different effective inputs and receive different gradient signals, gradually developing independent specializations.

The paper's ablation study (Section 3.4, Figure 7) compares four variants:

Symmetric: no noise, pure duplication with $\frac{1}{2}$ scaling.
Diagonal: weights initialized as $\begin{bmatrix} W_S & 0 \\ 0 & W_S \end{bmatrix}$ (no cross-talk between clones, following Shen et al., 2022).
Noisy symmetric: symmetric plus noise with 10 dB SNR.
Noisy diagonal: diagonal plus noise with 10 dB SNR.

The symmetric and noisy symmetric variants perform nearly identically and best overall, while diagonal variants underperform, likely "due to the presence of zero values in the expanded weight matrices" which reduce effective parameter utilization early in training. The paper's practical recommendation: use the noise-free symmetric variant "to avoid having to tune an extra hyper-parameter, the signal-to-noise ratio."

The Complete Algorithm: Step-by-Step

For a 2-fold expansion (source hidden dimension $d$ → target hidden dimension $2d$ ), the HyperCloning procedure processes each component of the transformer architecture in a specific order:

Token embedding layer (Case 2): Let $W_{\text{emb}} \in \mathbb{R}^{d \times v}$ be the source embedding matrix. The target embedding is $W_{\text{emb}}^D = \begin{bmatrix} W_{\text{emb}} \\ W_{\text{emb}} \end{bmatrix} \in \mathbb{R}^{2d \times v}$ . Bias is stacked similarly.
Positional embeddings: For learned positional embeddings with matrix $P_S \in \mathbb{R}^{L \times d}$ where $L$ is maximum sequence length, the target is $P_D \in \mathbb{R}^{L \times 2d}$ where each row $i$ is $\begin{bmatrix} \mathbf{p}_S(i) \\ \mathbf{p}_S(i) \end{bmatrix}$ . For sinusoidal encodings, evaluate the encoding functions at dimension $2d$ to produce the same pattern replicated across the expanded dimension.
For each transformer layer, repeat operations 4–7:
Layer normalization (pre-attention): Stack $\gamma$ and $\beta$ vectors. The normalization statistics are invariant under cloning, so the output is a cloned version of the source's normalized representation.
Multi-head attention:
- If expanding head dimension: Apply Case 3 to query, key, value projections, then multiply query weights by $\sqrt{d_S / d_D}$ . Apply Case 3 to the output projection.
- If expanding head count: Duplicate each head's projection weights (Case 3 for each copy, since both input and output expand across heads). The output projection handles concatenated heads—Case 3 if per-head dimension changed, Case 2 if only head count increased.
Residual connection: The attention output plus the pre-attention input. If the hidden dimension was expanded and input/output dimensions match (they do, by construction), the residual addition is straightforward—both operands are cloned vectors of the same size.
Feed-forward network (typically two linear layers with activation):
- First linear layer (up-projection): Case 3, expanding both input and output.
- Activation function (e.g., GELU): applied element-wise, identical behavior on cloned activations.
- Second linear layer (down-projection): Case 3, expanding both input and output (mapping back to hidden dimension).
- Second residual connection and second layer norm, processed as above.
Final layer norm: Same as intermediate layer norms—stack $\gamma$ and $\beta$ .
Unembedding layer (Case 1): The final linear projection to vocabulary logits. Target weight $W_{\text{unemb}}^D = [\frac{W_S}{2} + \eta, \frac{W_S}{2} - \eta] \in \mathbb{R}^{v \times 2d}$ , bias $\mathbf{b}_D = \mathbf{b}_S$ .

Result: After this transformation, if you feed the same token sequence to both the source and target models, every intermediate activation in the target model is a cloned version of the corresponding activation in the source model, and the final logits are identical. No training has occurred—this is purely a deterministic weight initialization.

Design Rationale: Why This Approach Satisfies the Four Goals

Goal 1 (Expansion dimension): The method produces a model with strictly larger hidden dimensions and the same number of layers. The paper's experiments use 2-fold width expansion (OPT-350M → OPT-1.3B, Pythia-410M → Pythia-1.4B, OLMO-1B → OLMO-2.9B), with attention heads either expanded in dimension or duplicated to double the count.

Goal 2 (Function preservation): Every layer type is treated so that the forward pass through the expanded model exactly replicates the forward pass through the source model. The paper verifies this empirically: at initialization (before any training), HyperCloning-initialized models achieve the same benchmark accuracy as their source models (visible in Figure 2—the curves start at non-zero accuracy equal to the base model's performance). The careful handling of attention scaling (the $\sqrt{d_S/d_D}$ factor) and the $\frac{1}{n}$ normalization in all Case 1/3 weights are what make this possible.

Goal 3 (Low compute overhead): The entire transformation is a series of tensor stacking and scaling operations—no optimization, no iterative refinement, no training. The computational cost is negligible compared to pre-training (essentially just memory copy operations). This is a key differentiator from knowledge distillation, which requires a full training process to transfer knowledge.

Goal 4 (Unchanged training loop): The method changes only the initial weight values—the architecture definition is a standard transformer with the target dimensions, the optimizer is standard AdamW, the data pipeline is unchanged, and the training hyperparameters are identical to those used for random initialization. This is verified by the experiments where "all other hyperparameters were kept identical, including learning rate, optimizer type, number of GPU nodes, batch size, context size, and order of training data" (Section 3.1.1).

Contrast with Prior Width Expansion Failures

The paper's approach can be understood as solving the specific failure modes of prior width expansion methods:

Du et al. (2024)'s non-function-preserving methods: By directly copying, projecting, zero-initializing, or randomly initializing expanded weights without normalization and without structural guarantees, these methods produce a model whose forward pass diverges from the source model's at initialization. The result is effectively a random initialization with some structured bias, which loses the source model's knowledge. HyperCloning avoids this through the Cases framework and explicit cancellation of cross-terms.
Shen et al. (2022)'s diagonal initialization: Initializing non-diagonal blocks to zero ( $\begin{bmatrix} W_S & 0 \\ 0 & W_S \end{bmatrix}$ ) does preserve function in a limited sense (the first half of the output depends only on the first half of the input, and similarly for the second half), but it creates a block-sparse weight structure with zero gradients in the off-diagonal blocks at initialization. This slows convergence because the model must learn cross-channel communication from scratch. HyperCloning's symmetric initialization gives all blocks non-zero weights from the start, enabling immediate information flow across all parts of the expanded dimension.
Wang et al. (2023b)'s symmetry concern: The concern that duplicated neurons don't learn independently is real, but the paper provides empirical evidence that it doesn't materialize in practice—cosine similarity between duplicated weight blocks decays to ~0.3–0.6 during training (Figure 5), and the singular value spectrum after training shows rank recovery comparable to random initialization (Figure 6). The combination of dropout stochasticity and the inherent asymmetry introduced by different positions in the expanded dimension (even without explicit noise) appears sufficient to break symmetry.
Chen et al. (2021)'s BERT focus: That work handled encoder-only architectures but didn't address decoder-specific components (causal attention masking, autoregressive generation) or scale to the model sizes and training token counts now standard. HyperCloning's validation on OPT, Pythia, and OLMO at scales up to 2.9B parameters with training on hundreds of billions of tokens demonstrates applicability to modern LLM pre-training regimes.

4. Key Insights and Innovations

Innovation 1: Function Preservation as the Missing Ingredient in Width Expansion

The dominant assumption in prior width-scaling literature was that copying or approximately transferring weights was "good enough" — that a rough structural correspondence between the small and large model would provide a useful initialization, even if the forward pass didn't perfectly match. Du et al. (2024) tested several width-expansion strategies (direct copying, projecting weights, zero-initializing new parameters, random initialization for expanded portions) and concluded that "non-function-preserving width growth results in poorer performance" compared to depth growth. This created a received wisdom: if you want to grow a model, increase depth, not width — width expansion doesn't work well.

HyperCloning's central intellectual move is to identify why prior width expansion failed and to demonstrate that the failure is not inherent to width scaling but is an artifact of non-function-preserving initialization. The insight is deceptively simple but has a non-obvious implication: if the larger model's forward pass at initialization diverges from the smaller model's, even slightly, the larger model starts training from a point that is worse than the smaller model's converged state. The warm-start benefit evaporates because the model must first unlearn the initialization errors before it can benefit from the transferred structure. This is fundamentally different from depth growth, where duplicating layers naturally preserves function in residual networks (as shown by Samragh et al., 2023) — width expansion has no such natural guarantee.

What makes this a genuine conceptual contribution rather than an incremental engineering fix is that the paper isolates function preservation as the necessary condition for effective width scaling, and then provides a complete constructive proof that it's achievable across every component of a modern decoder-only transformer. The Cases framework (Figure 3, Appendix A) is not just a set of formulas — it's a demonstration that a carefully designed linear algebra can make the expanded model a mathematical identity with respect to the source model's forward pass. The attention scaling factor (√(d_S/d_D)) is especially revealing: without it, the expanded attention scores are inflated by √2, meaning the model's internal attention distribution would be distorted at initialization — it would pay different attention to tokens than the smaller model did. Prior width-expansion methods either missed this subtlety entirely or handled it incompletely, producing models whose outputs differed from the source model's by margins that, while perhaps small in absolute terms, represented a loss of the very knowledge the initialization was supposed to transfer.

The empirical validation of this insight is stark: HyperCloning-initialized models begin training at the exact accuracy level of their pre-trained source models (visible in Figure 2's starting points for HyperCloning curves versus random initialization curves at zero tokens). This is not an improvement over random initialization at step zero — it's a completely different starting regime. The model already possesses the source model's language understanding, benchmark performance, and internal representations. Any further training is accretive, building on existing knowledge rather than reconstructing it from scratch. The paper's showing that this function-preserving initialization leads to 2.2× to 4× training speedups (Figure 2) and improved final accuracy (Figure 4) is a direct consequence of this property — the model never needs to recover from an initialization that has already discarded the source model's knowledge.

This reframes the model growth problem as one of knowledge preservation rather than architecture transformation. The question isn't "how do we expand the architecture?" — that's mechanically straightforward — but "how do we expand the architecture without destroying what the smaller model already knows?" HyperCloning's answer — that you can, through careful linear algebra, make the expansion a no-op from the perspective of the model's functional behavior — is the conceptual advance.

Innovation 2: Empirical Resolution of the Neuron Symmetry Concern

A foundational worry in the neuron-duplication literature, articulated most directly by Wang et al. (2023b), was that duplicating neurons to expand width would produce training-time symmetry — copied neurons receive identical gradients, update identically, and never develop independent functional specializations. The expanded model would have more parameters but effectively the same representational capacity as the smaller model, because the duplicated parameters would remain redundant throughout training. This was not a speculative concern: in theory, if two neurons have identical weights, identical inputs, and identical gradient contributions, gradient descent preserves the symmetry forever. If true, width expansion via duplication would be fundamentally limited — you could initialize a larger model from a smaller one, but you couldn't actually use the additional parameters.

HyperCloning provides the first systematic empirical evidence that this symmetry breaks spontaneously during large-scale transformer training, and the paper develops the analytical tools to track how it breaks. This is more than a "it works in practice" observation — it's a diagnostic contribution that changes how we think about the symmetry problem.

The key analytical move is tracking cosine similarity between duplicated weight blocks over the course of training (Figure 5). At initialization, this similarity is 1.0 by construction — the duplicated blocks are identical. By the end of training, cosine similarities have decayed to roughly 0.3–0.6 across most layers in all three model families (OPT, Pythia, OLMO). This is a large effect: the duplicated blocks end up substantially decorrelated, indicating they have developed distinct functional roles. The paper also examines the singular value spectrum before and after training (Figure 6). At initialization, HyperCloning produces weight matrices where half the singular values are zero — the matrix rank is at most the rank of the source model's matrix, which is at most half the maximum possible rank for the expanded matrix. After training, the singular value spectra of HyperCloning-initialized and randomly-initialized models are qualitatively similar, with both showing full-rank weight matrices. The model has recovered the additional representational capacity.

The paper attributes symmetry breaking to dropout — a standard regularization technique that randomly zeroes different subsets of activations in different forward passes, meaning duplicated neurons see different effective inputs and therefore receive different gradient signals. This is a compelling mechanism: it explains why the symmetry concern, while theoretically valid for deterministic training, doesn't materialize in practice when standard regularization is applied. The finding also explains why the noise-free symmetric variant works as well as the noisy variant in the ablation study (Figure 7): dropout provides sufficient stochasticity to break symmetry, making explicit noise addition (and its associated signal-to-noise ratio hyperparameter) unnecessary.

This finding is significant beyond HyperCloning because it addresses a theoretical objection that has lingered in the model growth literature. It suggests that neuron duplication is not just a convenient initialization trick but a viable scaling strategy — the duplicated capacity does eventually get utilized. The negative result on the diagonal initialization variant (Figure 7, where diagonal initialization underperforms symmetric across benchmark accuracies) further reinforces this: the off-diagonal blocks that symmetric initialization fills with non-zero weights are important for enabling cross-channel communication during training. Zero-initialized off-diagonal blocks (the diagonal strategy from Shen et al., 2022) slow convergence because the model must learn cross-channel interactions from scratch.

Innovation 3: Width Expansion as an Amortization Strategy for Pre-training Investment

The paper's framing of HyperCloning as a mechanism to amortize the cost of pre-training small models across larger descendants represents a shift in how to think about the economics of model development. The current paradigm treats each model scale as an independent capital investment: you pay the full pre-training cost for the 350M model, then separately pay the full pre-training cost for the 1.3B model, then the 7B model, and so on. Each model must rediscover basic language competence from random weights. The total cost of developing a model family is the sum of its parts.

HyperCloning changes this equation. If you've already trained a 350M model — whether you trained it yourself or downloaded it from HuggingFace — that investment can be reused to accelerate training of a 1.3B model. The 1.3B model doesn't start from zero; it inherits whatever the 350M model learned about syntax, semantics, factual knowledge, and reasoning patterns encoded in its weights. The cost of the 350M model is amortized across the 1.3B training run (and potentially across 2.6B, 5.3B, and larger descendants via multi-fold cloning, as demonstrated in the OPT-350M → OPT-1.3B → OPT-5.3B chain in Figure 9).

The OLMO experiment (Figure 2c) makes this point dramatically. The OLMO-1B base model was trained on 2.4 trillion tokens — an immense pre-training investment by the Groeneveld et al. (2024) team. HyperCloning transfers this knowledge to initialize OLMO-2.9B, which then achieves final accuracy (on 10 benchmark tasks) better than random initialization while training on only ~250 billion additional tokens — roughly one-tenth of the original 2.4T token budget. The 2.4T token investment in the 1B model was not discarded when moving to the larger scale; it was capitalized into the initialization of the 2.9B model.

This has implications beyond the specific numbers. In a world where organizations and research groups routinely release pre-trained checkpoints (OPT, Pythia, OLMO, LLaMA, Mistral, Gemma, etc.), HyperCloning transforms these public releases from static artifacts into reusable building blocks. A team wanting to train a model at a novel scale or with a novel architecture can bootstrap from any available smaller checkpoint in the same model family, rather than starting over. The Pythia experiment (Figure 2b) demonstrates that this amortization works even when the source and target models are trained on different datasets — Pythia-410M was trained on the Pile, while the target Pythia-1.4B was trained on DOLMA. The transferred knowledge generalizes across data distributions, suggesting that the abstract linguistic and reasoning capabilities encoded in the weights are the transferable asset, not dataset-specific memorization.

This is not a technical innovation in the method itself — the cloning formulas don't change based on how the source model was trained — but it is a conceptual reframing of what the method enables. HyperCloning converts model scaling from a sequence of independent capital expenditures into an incremental investment process where each scale inherits the accumulated knowledge of all prior scales. The paper's finding that a more accurate base model produces a better target model (Figure 8), and that a larger base model provides a better initialization for a given target scale (Figure 9), means the amortization benefits compound: better small models produce better large models, creating a virtuous cycle where investment in any scale benefits all larger scales.

Innovation 4: Catastrophic Forgetting at Initialization as a Diagnostic for Knowledge Transfer Quality

The paper documents a phenomenon that is easy to overlook but has significant implications: HyperCloning-initialized models exhibit catastrophic forgetting at the beginning of training (Figure 2, most visible in the OLMO panel where accuracy dips before recovering). The model starts at the source model's accuracy level, then loses accuracy in the first ~10–20 billion training tokens, before eventually recovering and surpassing the random-initialization baseline.

The paper treats this as an observation — it does not claim to fully explain or solve it — but the observation itself is an important diagnostic contribution. It reveals something about the relationship between the source model's knowledge and the target model's training dynamics: the initialization is function-preserving at step zero, but the gradients the model receives on the larger architecture immediately push it away from the source model's parameter configuration, even though that configuration was a good one. This suggests that the optimization landscape of the larger model has different local geometry than the smaller model's landscape, and the source model's parameters — while a good starting point functionally — are not a stable point in the larger model's optimization.

This has a direct connection to the over-optimization / reward hacking phenomenon documented in the test-time compute scaling literature (see the reference paper's Innovation 4), but in a different context. Here, the "optimization" is standard gradient descent on the language modeling objective, and the "over-optimization" is the tendency for training to degrade a good solution before improving upon it. Understanding why this happens — and whether it can be mitigated through initialization-aware learning rate schedules, gradual unfreezing, or other techniques — is an open research question that the paper explicitly flags.

The practical implication is that HyperCloning's benefits are not automatic — they require sufficient training budget to overcome the initial forgetting. If you only have budget for a small number of training tokens, the random initialization baseline might actually outperform because it doesn't suffer the initial accuracy dip. The paper's experiments use training budgets of hundreds of billions of tokens (250B for OLMO-2.9B, for example), which is more than enough to recover and surpass random initialization. But the forgetting phenomenon establishes a minimum viable training budget below which HyperCloning might not be beneficial — a boundary condition on the method's applicability that the field needs to characterize.

The fact that HyperCloning still outperforms random initialization by large margins despite this forgetting (Figure 4 shows final accuracy improvements of 2–5 percentage points across model families on 10-task averages) is actually stronger evidence for the method's effectiveness than if forgetting didn't occur. It means the beneficial effects of the transferred knowledge are robust to the initial optimization disruption — the model eventually recovers the source model's knowledge and builds beyond it. The forgetting is a transient effect, not a permanent loss.

5. Experimental Analysis

Evaluation Methodology

Dataset. All experiments use the DOLMA dataset provided by Groeneveld et al. (2024), which aggregates several open-source corpora and totals approximately 2.4 trillion tokens (Appendix B). The authors note they do not process the full 2.4T tokens due to cost constraints, and they shuffle the data shards with a fixed random seed across all experiments to eliminate ordering effects as a confound. For the Pythia experiments specifically, the base model (Pythia-410M) was originally trained on the Pile dataset, while the target model (Pythia-1.4B) is trained on DOLMA—meaning the source and target models see different training data, providing a test of whether HyperCloning's benefits transfer across dataset distributions.
Base model(s). The paper studies three model families, each with a source → target pair: OPT (350M → 1.3B, with the source model trained by the authors on 30B tokens because the HuggingFace OPT-350M checkpoint contains extra linear layers not present in OPT-1.3B), Pythia (410M → 1.4B, with the source model downloaded from HuggingFace, pre-trained on ~250B tokens of the Pile), and OLMO (1B → 2.9B, with the source model downloaded from HuggingFace, pre-trained on 2.4T tokens of DOLMA). These three families are chosen to span different pre-training regimes: OPT tests same-dataset transfer with limited source pre-training (30B tokens), Pythia tests cross-dataset transfer with moderate source pre-training (~250B tokens), and OLMO tests same-dataset transfer with extensive source pre-training (2.4T tokens). Architecture details for all models are provided in Table 1: OPT-350M has 24 layers, 16 heads, d_model=1024, d_FFN=4098; OPT-1.3B has 24 layers, 32 heads, d_model=2048, d_FFN=8192; Pythia-410M and Pythia-1.4B follow the same pattern; OLMO-1B uses 16 layers with 16 heads, d_model=2048, d_FFN=16384, while OLMO-2.9B uses 16 layers with 32 heads, d_model=4096, d_FFN=16384 (not a simple 2× expansion—the FFN dimension stays the same but the hidden dimension and head count double).
Metrics. The primary metric is benchmark accuracy averaged over 10 downstream tasks, evaluated using the Harness framework (Gao et al., 2023). The individual 10 tasks are not enumerated in detail by name; the paper presents both per-task accuracies (Figure 4 shows 10 separate bars per model configuration) and the task-averaged mean (presented as training curves in Figure 2 and as bar charts in Figures 4, 7, 8, 9). The paper also reports training speedup, defined as the ratio of training tokens required by random initialization versus HyperCloning to reach the same accuracy level (specifically, reaching the "final accuracy of the random initialization baseline," quoted as 2.2× to 4× across model families in Section 1).
Baselines. The primary baseline is random initialization—the standard practice of initializing all weights from the default distribution used in the respective model's training code (the paper states all other hyperparameters are kept identical between HyperCloning and random initialization runs, including learning rate, optimizer, GPU count, batch size, context size, and data ordering; Section 3.1.1). The paper also compares against alternative expansion strategies as ablations rather than external baselines: diagonal initialization (Shen et al., 2022, where non-diagonal blocks are zero-initialized), noisy symmetric (HyperCloning with explicit noise addition at 10 dB SNR), and noisy diagonal (diagonal initialization with noise). For the base model accuracy study (Figure 8), the baselines include random initialization plus HyperCloning from three different OPT-350M checkpoints (trained on 16B, 32B, and 64B tokens). For the base model size study (Figure 9), the baselines are random initialization, HyperCloning from OPT-350M (4-fold cloning), and HyperCloning from OPT-1.3B (2-fold cloning), all targeting OPT-5.3B.
Generation budget / compute accounting. The paper does not use "generation budget" or "test-time compute" as its unit—this is a pre-training paper, so the relevant resource measure is training tokens processed. All training curves in Figure 2 plot accuracy against training tokens, enabling direct comparison of how many tokens each initialization method requires to reach a given accuracy. The paper reports speedup in terms of tokens: "2.2× to 4×" means HyperCloning requires 2.2× to 4× fewer training tokens to reach the random baseline's final accuracy (Section 3.1.1). Training hyperparameters are provided in Appendix B (Table 2): all runs use AdamW with weight decay 0.05, β1=0.9, β2=0.999, gradient accumulation over 16 steps, ZeRO-2 gradient partitioning, linear warmup over 25K iterations to max LR, then cosine decay to 1/10 of max LR over 2.5M iterations, then constant LR. Models are trained on 64 GPUs. Specifics per model: OPT-1.3B and Pythia-1.4B use batch size 2, context size 1024, max LR 1.5E-4, ~65K tokens/iteration; OLMO-2.9B uses batch size 2, context size 2048, max LR 3E-4, ~82K tokens/iteration.
Cross-validation / statistical protocol. The paper does not report formal cross-validation, standard errors, confidence intervals, or multiple training runs with different random seeds. The training curves (Figure 2) show single-run trajectories for each configuration, evaluated at intervals on the 10-task benchmark suite. The random seed for data shuffling is kept the same across experiments to eliminate ordering effects, but the paper does not report whether results are robust to different random seeds for weight initialization or data ordering. This is a notable limitation—without error bars or replication, the reported 2.2×–4× speedup figures and accuracy improvements cannot be assessed for statistical significance.

Main Quantitative Results

Aggregate Training Speedup and Final Accuracy

Headline result (Figure 2, Figure 4): Across all three model families, HyperCloning achieves both faster convergence and higher final accuracy compared to random initialization. The speedup in reaching the baseline's final accuracy ranges from 2.2× to 4× depending on model family, and the final accuracy advantage (after training on hundreds of billions of tokens) is visible across individual benchmark tasks.

OPT (350M → 1.3B) results (Figure 2a, Figure 4a): The base OPT-350M model was trained by the authors on 30B tokens—a relatively small pre-training budget. HyperCloning initializes OPT-1.3B from this checkpoint. The training curve (Figure 2a) shows that HyperCloning starts at a non-zero accuracy equal to the OPT-350M base model's performance, then climbs steadily. The random initialization baseline starts near zero and rises more slowly. The curves cross somewhere in the first ~10–20B tokens (the HyperCloning curve shows a small initial dip, consistent with the catastrophic forgetting observation), after which HyperCloning maintains a consistent lead. By the end of training, HyperCloning achieves higher accuracy than random initialization on essentially all 10 benchmark tasks (Figure 4a), with per-task improvements ranging from roughly 1 to 8 percentage points depending on the task (exact per-task numbers not quoted in the text but visible in the bar chart). The speedup is estimated at roughly 2.2×–3× based on the horizontal distance between the curves at the random baseline's final accuracy level (Section 1, Figure 2a).

Pythia (410M → 1.4B) results (Figure 2b, Figure 4b): This is the cross-dataset transfer test: Pythia-410M was trained on the Pile, while Pythia-1.4B is trained on DOLMA. Despite the distribution shift, HyperCloning still shows substantial benefits. The training curve (Figure 2b) shows HyperCloning starting at the Pythia-410M's accuracy level and improving, while random initialization starts at zero. The gap is consistent throughout training—there's no visible catastrophic forgetting dip in this case, unlike OPT and OLMO. The final accuracy on the 10-task average is higher for HyperCloning than random initialization, with per-task improvements visible across most tasks in Figure 4b. The speedup is reported in the ~3× range (Section 1). This result is significant because it demonstrates that the transferred knowledge is not tied to the specific training data distribution—the abstract capabilities encoded in the weights generalize across corpora.

OLMO (1B → 2.9B) results (Figure 2c, Figure 4c): This tests the regime where the base model has been extensively pre-trained (2.4T tokens on DOLMA). HyperCloning initializes OLMO-2.9B from the fully-trained OLMO-1B checkpoint. The training curve (Figure 2c) shows a dramatic initial accuracy advantage—HyperCloning starts at the OLMO-1B's accuracy, roughly 35–40% on the 10-task average, while random initialization starts at zero. However, there is a severe catastrophic forgetting dip in the first ~10–20B tokens, where accuracy drops by roughly 5–8 percentage points before recovering. By ~50B tokens, the curve has recovered to near its starting level, and by ~150B tokens it has surpassed the initial accuracy. The random baseline rises steadily but more slowly. By the end of training (~250B tokens), HyperCloning achieves the highest accuracy among all configurations. The per-task breakdown (Figure 4c) shows HyperCloning outperforming random initialization on nearly all tasks, with particularly large gains on some tasks (visible as ~5–10 percentage point differences in certain bars). The speedup is estimated at ~4× (Section 1), the highest among the three model families. This is attributed to the "transfer of knowledge from the base model" which "was already trained on 2.4T tokens" (Section 3.1.1).

Key quantitative detail on speedup calculation: The paper describes the speedup as follows: "HyperCloning enables the network to reach the final accuracy of the random initialization baseline much faster, with a speedup ranging from 2.2× to 4× across different model types" (Section 3.1.1). This means: identify the accuracy that random initialization achieves at the end of its training run (the final data point on the random curve in each Figure 2 panel), then find the point on the HyperCloning curve that reaches that same accuracy, and compute the ratio of training tokens. The speedup is measured in training tokens, not wall-clock time, GPU hours, or FLOPs.

Per-Task Accuracy Analysis

Figure 4 provides the per-task breakdown across 10 benchmark tasks for all three model families. Exact task names are not enumerated in the paper text, but the Harness evaluation framework (Gao et al., 2023) typically includes tasks such as HellaSwag, PIQA, ARC-Easy, ARC-Challenge, WinoGrande, OpenBookQA, BoolQ, etc. The key patterns:

HyperCloning achieves higher accuracy than random initialization on the vast majority of individual tasks across all three model families. There appear to be very few tasks where random initialization outperforms HyperCloning.
The magnitude of improvement varies by task. Some tasks show a 1–3 percentage point advantage for HyperCloning, while others show 5–10+ percentage point gaps.
There is no visible systematic pattern suggesting that HyperCloning helps more on certain task types (e.g., reasoning vs. knowledge recall) versus others—the improvement appears broad.
The OLMO results (Figure 4c) show the largest gaps between HyperCloning and random initialization on average, consistent with the aggregate curve showing the largest speedup for OLMO.

Effect of Base Model Accuracy

Experiment (Figure 8, described in Section 3.5): The authors train OPT-350M checkpoints on 16B, 32B, and 64B tokens respectively, producing three base models with increasing accuracy. Each is used to initialize OPT-1.3B via HyperCloning, and all four configurations (three HyperCloning variants plus random initialization) are trained on the same data and compared.

Headline finding: "Initializing with the base model improves accuracy compared to random initialization when any of the base checkpoints are used for cloning. Among the cloned networks, those initialized with a more accurate base network achieve better accuracy, especially at the beginning of the training. However, as training continues, the differences between the curves become smaller" (Section 3.5).

Training curve analysis (Figure 8a): All three HyperCloning curves start at different accuracy levels corresponding to their base model's performance—the 64B-token base model starts highest, followed by 32B, then 16B. All three start above random initialization (which starts near zero). In the early training phase (~0–10B tokens), the ordering is strictly preserved: 64B base > 32B base > 16B base > random. As training progresses, the curves converge somewhat—by the end of training, all three HyperCloning variants perform similarly and all substantially outperform random initialization. The gap between the best (64B base) and worst (16B base) HyperCloning variant narrows from a large initial gap to a small final gap.

Per-task analysis (Figure 8b): The bar chart shows the same pattern across individual tasks: HyperCloning with 64B base > 32B base > 16B base > random for most tasks, but the differences among HyperCloning variants are modest compared to the gap between any HyperCloning variant and random initialization.

Interpretation: More accurate base models provide better initializations—the knowledge transfer is more valuable when the source model knows more. However, given enough training, even a relatively weak base model (trained on only 16B tokens) provides significant benefits over random initialization, and the advantage of using a stronger base model partly diminishes with extended training. This is practically important: it means the method is beneficial even if you only have a partially-trained or relatively weak small model available, though a fully-trained one provides additional gains early in training.

Effect of Base Model Size

Experiment (Figure 9, described in Section 3.6): The target model is OPT-5.3B, created by doubling the hidden dimension of OPT-1.3B. This model can be initialized via HyperCloning in two ways: (1) from OPT-350M using 4-fold cloning (since OPT-350M's hidden dimension is 1/4 of OPT-5.3B's), or (2) from OPT-1.3B using 2-fold cloning (since OPT-1.3B's hidden dimension is 1/2 of OPT-5.3B's). The comparison tests whether it's better to clone from a smaller but more distant model (4-fold from 350M) or a larger, closer model (2-fold from 1.3B).

Headline finding: "Initializing with either OPT-350M or OPT-1.3B achieves faster convergence compared to random initialization, with OPT-1.3B providing better convergence than OPT-350M. This is because OPT-1.3B is larger and more accurate than OPT-350M, thereby offering a superior initialization" (Section 3.6).

Training curve analysis (Figure 9a): OPT-1.3B → OPT-5.3B (2-fold) starts at a higher accuracy than OPT-350M → OPT-5.3B (4-fold), and both start above random initialization. The OPT-1.3B-initialized curve maintains a lead over the OPT-350M-initialized curve throughout training, though the gap narrows over time. Both HyperCloning variants substantially outperform random initialization at all training durations.

Per-task analysis (Figure 9b): The bar chart shows the same ordering on essentially all tasks: OPT-1.3B base > OPT-350M base > random. The gap between the two HyperCloning variants varies by task but is generally smaller than the gap between either HyperCloning variant and random initialization.

Interpretation: This experiment cleanly separates two confounded factors in the base model accuracy study (Figure 8): base model size (parameter count) and base model accuracy (which are correlated but not identical). A larger base model provides a better initialization because it is both more capable (higher accuracy) and architecturally closer to the target (requiring fewer-fold expansion). The fact that even 4-fold cloning from a much smaller model (350M → 5.3B, a more than 15× parameter increase) still significantly outperforms random initialization demonstrates the robustness of the approach to large expansion ratios.

Analysis of Weight Evolution During Training

Cosine similarity decay (Figure 5, described in Section 3.2): To test whether the duplicated weight blocks remain symmetric during training (which would indicate the model isn't utilizing its expanded capacity), the authors track the average cosine similarity between duplicated weight vectors within the expanded matrices. Specifically, for Case 3 weights (the 2×2 block structure), each row contains two horizontal weight vectors that are identical at initialization. Cosine similarity between these pairs is computed for each row and averaged within each layer.

At initialization, cosine similarity = 1.0 for all layers (complete symmetry). During training, cosine similarity decays in most layers, dropping to roughly 0.3–0.6 by the end of training across all three model families (OPT, Pythia, OLMO). The decay is not uniform across layers—some layers decorrelate more than others—but the overall trend is clear: the model breaks symmetry and develops independent weight specializations. The paper does not report the exact final values per layer but shows the full trajectories in Figure 5.

Singular value spectrum recovery (Figure 6, described in Section 3.3): The authors examine rank deficiency in HyperCloning-initialized weight matrices before and after training by plotting singular values. At initialization, the weight matrices from HyperCloning have approximately half their singular values at zero—the matrix rank is at most the source matrix's rank, which is at most half the expanded matrix's maximum possible rank. This means the model is not utilizing its full representational capacity at initialization, even though its functional output matches the source model's.

After training, the singular value spectra of HyperCloning-initialized weights closely resemble those of randomly initialized weights—both show full-rank matrices with smoothly decaying singular values. The authors show this for selected layers: Block 0 up-project weights, Block 3 QKV weights, and Block 12 down-project weights from the OLMO-2.9B model (Figure 6a, b, c). In all cases, the "before training" HyperCloning matrices show the characteristic half-zero singular value pattern, and the "after training" matrices show continuous spectra comparable to the randomly initialized "after training" spectra.

Key quantitative finding: The model recovers full rank during training. The additional parameters that were initially redundant (producing zero singular values) become meaningfully utilized. This complements the cosine similarity analysis: not only do the duplicated weight blocks become decorrelated (Figure 5), but the overall weight matrices achieve full representational rank (Figure 6).

Ablation Studies and Robustness Checks

Expansion strategy variants (Figure 7, Section 3.4): The paper compares four initialization strategies for the expanded weight matrices in a Pythia-1.4B training run: symmetric (HyperCloning's default, W_D = [[W_S/2, W_S/2], [W_S/2, W_S/2]]), diagonal (W_D = [[W_S, 0], [0, W_S]]), noisy symmetric (symmetric plus noise with 10 dB SNR), and noisy diagonal (diagonal plus noise with 10 dB SNR). All four strategies are function-preserving at initialization. The symmetric and noisy symmetric variants achieve the highest accuracy, performing nearly identically to each other—suggesting explicit noise addition provides negligible benefit when dropout is already present. The diagonal variant underperforms the symmetric by a visible margin on the average accuracy curve (Figure 7a) and on most individual benchmark tasks (Figure 7b). The noisy diagonal variant performs slightly better than pure diagonal but still worse than symmetric. The paper attributes the diagonal variant's underperformance to "the presence of zero values in the expanded weight matrices," which create blocks where gradients are initially zero and cross-channel communication must be learned from scratch. The practical conclusion: symmetric initialization without noise is the recommended approach, as it "avoid[s] having to tune an extra hyper-parameter, the signal-to-noise ratio" (Section 3.4).
Catastrophic forgetting at training onset (Figure 2, Section 3.1.1): The paper observes that models initialized with HyperCloning "tend to exhibit catastrophic forgetting at the beginning of training," most prominently in the OLMO experiment (Figure 2c). The OLMO-2.9B model starts at the OLMO-1B's accuracy (~35–40% on the 10-task average), then drops by roughly 5–8 percentage points over the first ~10–20B training tokens before recovering. The OPT model (Figure 2a) shows a smaller dip; the Pythia model (Figure 2b) shows essentially no dip. The paper does not conduct systematic ablations to diagnose the cause of forgetting (e.g., varying learning rate, gradual unfreezing, different optimizers) or to determine why it varies in severity across model families. The paper flags this as future work: "Understanding the underlying causes of catastrophic forgetting, identifying strategies to mitigate it, and exploring why HyperCloning continues to outperform random initialization despite its occurrence are valuable avenues for future research" (Section 3.1.1).
Cross-dataset transfer (Pythia experiment, Figure 2b and Figure 4b): The Pythia-410M base model was trained on the Pile, while the Pythia-1.4B target model is trained on DOLMA—a different dataset with different distributional properties. HyperCloning still provides substantial speedup and final accuracy gains compared to random initialization, indicating that the transferred knowledge is not tied to the specific pre-training corpus. This serves as an implicit robustness check: the method does not require the source and target models to be trained on the same data. The paper does not ablate this dimension further by testing additional dataset pairs or quantifying how dataset similarity affects transfer quality.
Multi-fold expansion (Figure 9, implied by Section 3.6): The OPT-350M → OPT-5.3B experiment uses 4-fold cloning, demonstrating that the method works beyond 2-fold expansion. The paper does not systematically sweep expansion ratios (e.g., 1.5×, 2×, 3×, 4×, 8×) or identify an upper bound beyond which cloning fails to provide benefits. The 4-fold result from OPT-350M still substantially outperforms random initialization but underperforms 2-fold cloning from OPT-1.3B—this confounds base model capability (OPT-1.3B is better than OPT-350M) with cloning ratio (2-fold vs. 4-fold), so the pure effect of expansion ratio on transfer quality cannot be isolated from this experiment.
Base model training duration (Figure 8, Section 3.5): The paper tests OPT-350M checkpoints at 16B, 32B, and 64B tokens to assess sensitivity to source model quality. All provide benefits over random initialization, with more-trained bases providing larger gains, particularly early in training. The paper does not test whether an overtrained small model (beyond compute-optimal) or an undertrained small model at a specific point in the loss curve transfers differently, which would inform optimal checkpoint selection strategies.
Training hyperparameters (Section 3.1.1, Appendix B): The paper emphasizes that "all other hyperparameters were kept identical" between HyperCloning and random initialization runs, including learning rate, optimizer type, GPU count, batch size, context size, and data order. This is a deliberate choice to isolate the effect of initialization alone. The paper does not ablate whether HyperCloning would benefit from a different learning rate schedule (e.g., lower initial learning rate to reduce catastrophic forgetting, or a different warmup period), which is a potentially important design choice given that the initialization starts the model in a very different region of parameter space than random initialization.
No comparison to continued training of the small model: The paper does not include a baseline where the small source model is simply trained for longer (rather than expanded and then trained). This would address the question: given a fixed total compute budget, is it better to (a) train a small model longer, or (b) expand to a larger model via HyperCloning and train that? The paper's experiments train target models from HyperCloning-initialized checkpoints and target models from random initialization, but never trains the source model for an equivalent number of additional tokens to assess the marginal value of expansion versus continued small-model training.
No depth expansion baseline: The paper positions HyperCloning as complementary to depth growth methods (Section 2, Section 4), but does not empirically compare HyperCloning against depth-based expansion (e.g., duplicating transformer blocks) at matched parameter counts. Such a comparison would help practitioners decide whether to scale width, depth, or both.

Critical Assessment

Central Claim 1: "HyperCloning achieves 2.2× to 4× faster convergence to the final accuracy of randomly initialized baselines" (Section 1)

What the experiments demonstrate: The training curves in Figure 2 clearly show that HyperCloning-initialized models start at non-zero accuracy equal to their base models' performance and reach any given accuracy level in fewer training tokens than randomly initialized models. The horizontal gap between the curves at the random baseline's final accuracy level supports the 2.2×–4× range (OPT ≈2.2–3×, Pythia ≈3×, OLMO ≈4×).

Limitations:

Single-run results: The paper reports one training run per configuration. Without replication across random seeds (for weight initialization noise, data ordering, dropout), it's impossible to assess the variance of the speedup estimate. If the random baseline's final accuracy varies by ±2 percentage points across seeds, the corresponding token count to reach that accuracy under HyperCloning could shift substantially, changing the speedup factor. The claim would be stronger with error bars or confidence intervals.
Speedup measured in tokens, not wall-clock time or FLOPs: The speedup metric compares training tokens processed, not actual compute cost. The target models trained with HyperCloning have more parameters than the base models (e.g., OPT-1.3B vs. OPT-350M), meaning each training token costs more FLOPs in the larger model. If the speedup is 4× in tokens but each token costs ~4× more FLOPs (since the model is ~4× larger), the wall-clock speedup may be modest. The paper does not compute FLOP-matched comparisons between HyperCloning-initialized large models and hypothetical continued training of small models, which would be needed to assess whether the total compute (not just tokens) is reduced. This is a significant omission—it's possible that simply training the small model for 4× more tokens would achieve comparable accuracy to the HyperCloning-initialized large model trained for 1× tokens, at lower total FLOPs.
Dependence on the "final accuracy" definition: The speedup is defined relative to the "final accuracy of the random initialization baseline." This means the speedup number is specific to the particular training budget used in the experiments. If the random baseline had been trained longer (more tokens), its final accuracy would be higher, and the speedup to reach that higher threshold might be different (potentially larger, if HyperCloning climbs faster; potentially smaller, if HyperCloning plateaus while random continues improving). The paper does not explore how speedup varies as a function of the target accuracy threshold.
No speedup analysis for per-task accuracy: The speedup is computed on the 10-task average (Figure 2). Per-task speedups might differ substantially—HyperCloning might provide 8× speedup on some tasks and only 1.2× on others—but the paper doesn't provide this breakdown.

Central Claim 2: "HyperCloning improves final accuracy compared to random initialization" (Section 1, Figure 4)

What the experiments demonstrate: Figure 4 shows per-task accuracies where HyperCloning consistently matches or exceeds random initialization at the end of training. The 10-task average (Figure 2) shows HyperCloning finishing higher in all three model families. The improvements appear to be roughly 1–5 percentage points on the average, depending on the model family, with larger gains for OLMO.

Limitations:

Training budget is finite and arbitrary: The "final" accuracy is the accuracy at whatever token count the authors stopped training (roughly 250B tokens for OLMO-2.9B, and whatever budget was used for OPT and Pythia—exact final token counts are not stated in the paper text). There's no guarantee that random initialization wouldn't eventually catch up or surpass HyperCloning if both were trained for substantially more tokens. The claim of "better final accuracy" is more accurately "better accuracy at the chosen training horizon." The paper doesn't provide evidence that HyperCloning changes the asymptotic performance ceiling rather than just accelerating convergence.
Effect size varies by model family: OLMO-2.9B shows the largest margin over random initialization (Figure 4c), which the paper attributes to the base model's extensive 2.4T-token pre-training. But this could also be influenced by (a) the OLMO architecture's specific width-to-depth ratio, (b) the fact that OLMO-2.9B's FFN dimension is not expanded relative to OLMO-1B (Table 1), meaning the expansion is not uniform, or (c) the specific training hyperparameters used. Without experiments across multiple base model training budgets within each family, it's hard to isolate whether the benefit comes from base model quality or architectural specifics.
No comparison to training the small model longer: Could the accuracy improvement from HyperCloning be achieved more cheaply by simply continuing to train the small source model? If OLMO-1B on 2.4T tokens achieves accuracy X, and HyperCloning to OLMO-2.9B + 250B tokens achieves accuracy Y > X, the relevant comparison for a practitioner is: is training OLMO-2.9B for 250B tokens from the HyperCloning initialization better than training OLMO-1B for an additional 250B tokens (total 2.65T)? The paper doesn't run this experiment, so the marginal benefit of width expansion versus continued small-model training is unknown.

Central Claim 3: "Function preservation is achieved—the larger model retains the functionality of the smaller model before training starts" (Section 1)

What the experiments demonstrate: The paper provides a mathematical proof (Appendix A) that the weight transformation formulas produce identical forward passes. Empirically, this is supported by two observations: (a) HyperCloning-initialized models' training curves start at accuracy levels consistent with their base models' performance (Figure 2), and (b) the paper states that "the larger model already inherits the predictive power and accuracy of the smaller model before the training starts" (Section 1). No explicit verification experiment is reported (e.g., comparing logit outputs between source and HyperCloning-initialized target on a sample of inputs to confirm they match exactly).

Limitations:

No direct forward-pass equivalence verification: A simple experiment would be: take 1000 prompts, run them through the source model and the HyperCloning-initialized target model, and verify that the output logits match to within floating-point precision. The paper does not report this. The accuracy curves showing the target model starts at non-zero accuracy are consistent with function preservation but don't rule out small discrepancies—for example, if the target model were 2% less accurate than the source at initialization, it would still appear far above random initialization on the training curves. The paper's mathematical derivation is convincing, but an empirical sanity check would strengthen the claim.
Attention scaling verification: The attention query scaling factor (√(d_S/d_D)) is a subtle detail that, if implemented incorrectly, would distort attention scores and potentially degrade the initialization without being visually obvious in aggregate accuracy. The paper doesn't ablate this scaling factor to show its importance.

Central Claim 4: "The benefits of HyperCloning transfer across datasets—source and target models can be trained on different data" (Pythia experiment, Section 3.1.1)

What the experiments demonstrate: Pythia-410M (Pile) → Pythia-1.4B (DOLMA) shows substantial speedup and accuracy gains (Figure 2b, Figure 4b), supporting cross-dataset transfer.

Limitations:

Single cross-dataset pair: Only one pair of datasets (Pile → DOLMA) is tested. The generalizability to other dataset shifts (e.g., English → multilingual, general web → code, text → scientific papers) is unknown. Both Pile and DOLMA are large, general-domain English web text corpora, so the distribution shift may be relatively mild. A stronger test would involve source and target models trained on significantly different domains.
No quantification of dataset similarity effect: The paper doesn't vary the degree of dataset mismatch to see how transfer degrades as source and target data diverge—for example, by training source models on subsets of DOLMA with varying degrees of overlap with the Pile, and measuring how target model convergence depends on that overlap.

Central Claim 5: "The model breaks weight symmetry during training and achieves full rank, meaning the expanded capacity is utilized" (Section 3.2, 3.3)

What the experiments demonstrate: Cosine similarity between duplicated weight blocks decays from 1.0 to ~0.3–0.6 (Figure 5). Singular value spectra recover from half-zero to full-rank distributions comparable to randomly initialized models after training (Figure 6).

Limitations:

Selected layers shown: Figure 5 shows cosine similarity for "several selected layers" (not all layers), and Figure 6 shows singular values for three specific weight matrices (Block 0 up-project, Block 3 QKV, Block 12 down-project in OLMO-2.9B). The paper doesn't report whether any layers fail to break symmetry—i.e., whether some duplicated weight blocks remain highly correlated throughout training, suggesting wasted capacity.
Correlation doesn't prove independent function: Low cosine similarity between weight vectors indicates they are numerically different, but doesn't guarantee they compute functionally distinct transformations. Two different weight vectors can produce similar outputs on the data distribution. Direct measures of functional specialization (e.g., clustering of activation patterns, mutual information between duplicated neuron outputs, ablation of one copy's effect on task performance) would provide stronger evidence that the expanded capacity is meaningfully utilized.
No comparison of utilization metrics across expansion strategies: The symmetric vs. diagonal ablation (Figure 7) shows that symmetric initialization achieves better accuracy, but the paper doesn't show whether this corresponds to different utilization patterns (e.g., do diagonal-initialized models fail to break symmetry, or do they break symmetry but recover more slowly?). This would connect the utilization analysis to the performance results.

Overall Assessment

The experiments convincingly demonstrate that HyperCloning provides a better initialization than random weights for training larger language models—the training curves are unambiguous, the per-task bar charts are consistent, and the effect appears across three model families with different base model training regimes. The method works, and it works robustly across the configurations tested.

However, the paper's claims outrun its experimental validation in several important ways:

The speedup claim (2.2×–4×) is measured in training tokens, not compute cost. A practitioner who reads "4× faster" might reasonably assume 4× fewer GPU hours, but the larger target model processes each token more expensively than the base model would. A FLOPs-matched or GPU-hour-matched analysis is needed to convert the token speedup into a real-world cost savings claim. The paper's Introduction mentions GPU hour costs ("72,000 GPU hours for a 12B model") but never computes GPU-hour savings for HyperCloning.
The "final accuracy" claim is relative to a fixed, finite training budget. Without evidence that the accuracy ordering persists at longer training horizons, the claim should be qualified as "better accuracy at the chosen training budget" rather than unconditionally "better final accuracy."
The amortization argument is conceptually compelling but not financially quantified. The paper argues that HyperCloning "leverages previously trained models, thus offering a cost-saving advantage" (Section 3.1.1). But if the base model costs 72K GPU hours to train and HyperCloning saves 50% of the target model's training cost, the net savings depend on how many times the base model is reused. The paper doesn't provide a break-even analysis: how many descendant models must be initialized from a given base model to amortize its training cost?
Missing baselines limit the force of the conclusions. The most important missing baseline is continued training of the small model: if OLMO-1B trained on 2.4T tokens achieves accuracy X, and HyperCloning to OLMO-2.9B + 250B tokens achieves accuracy Y, is the improvement from width expansion or from additional total FLOPs? A small-model baseline trained for an equivalent FLOP budget would answer this. Also missing: comparison to depth-based growth methods at matched parameter counts, which would help practitioners choose between width-scaling (HyperCloning) and depth-scaling (the dominant method in prior work).
Statistical rigor is limited. The paper reports single training runs, no error bars, no confidence intervals, and no multi-seed replication. For a paper making claims about training speedup—where variance across runs can be substantial due to data order, dropout, and initialization noise—this is a notable gap. The 2.2×–4× range might shift considerably if even modest variance exists in the random baseline's convergence curve.
The catastrophic forgetting observation raises questions the paper doesn't answer. The OLMO experiment shows a severe accuracy dip (5–8 percentage points over ~10–20B tokens) that consumes a non-trivial fraction of the total training budget to recover from. If this forgetting could be mitigated—through a lower initial learning rate, gradual unfreezing of cloned layers, or a specialized warmup schedule—the effective speedup could be substantially larger. Conversely, the forgetting might indicate a fundamental misalignment between the source model's parameter configuration and the larger model's optimization landscape, in which case HyperCloning might underperform random initialization at very small training budgets. Neither hypothesis is tested.
The three model families cover a relatively narrow scale range. All target models are in the 1.3B–2.9B parameter range. It's unknown whether HyperCloning's benefits scale to larger models (7B, 13B, 70B parameters) where pre-training costs are most acute and where the economic argument for initialization reuse is strongest. The paper's argument that the method "can be generalized to n-fold expansion" (Appendix A) is mathematically straightforward, but whether the training dynamics (symmetry breaking, forgetting, convergence rate) remain favorable at 8× or 16× expansion is untested.

These limitations don't undermine the paper's core contribution—HyperCloning is clearly a useful method that outperforms random initialization across all experiments—but they bound the strength of the claims that can be made. The paper demonstrates that HyperCloning is beneficial; it does not convincingly demonstrate how beneficial in practical cost terms, nor does it establish the boundaries of applicability. The speedup numbers (2.2×–4×) and the final accuracy improvements should be understood as estimates from single runs at moderate scale, not as precisely calibrated predictions for arbitrary model families and training regimes.

6. Limitations and Trade-offs

6.1 Speedup Is Measured in Training Tokens, Not Total Compute Cost

The assumption or constraint: The paper's headline speedup metric—2.2× to 4× faster convergence—is measured in training tokens processed, not in FLOPs, GPU hours, or wall-clock time. The larger target model has more parameters than the source model (e.g., OPT-1.3B has roughly 4× the parameters of OPT-350M), meaning each token processed by the target costs more FLOPs than a token processed by the source. The paper acknowledges training cost only in its motivational framing (Section 1): "training a 12-billion-parameter model requires approximately 72,000 GPU hours"—but never computes GPU-hour savings for HyperCloning itself.

The consequence: A practitioner who reads "4× faster" might reasonably assume a 4× reduction in GPU hours, but this is not what the paper measures. If the target model has 4× more parameters than the source, and HyperCloning provides a 4× token speedup, the FLOP-matched speedup could be closer to 1× (breakeven) because each token costs 4× more compute. The actual cost savings depend on the ratio of target-to-source model size, the training token reduction, and the overhead of the small model's pre-training (which HyperCloning amortizes but which must be paid upfront). Without a FLOPs-matched or GPU-hour-matched analysis, the economic argument for HyperCloning—which is the paper's primary motivation (Section 1, paragraph 1)—remains qualitatively compelling but quantitatively unvalidated.

What evidence exists in the paper: The paper provides full training hyperparameters (Appendix B, Table 2) including batch sizes, context sizes, GPU counts, and tokens per iteration, which would in principle allow a reader to compute approximate FLOPs if they had accurate FLOP-per-token estimates for each architecture. However, the paper itself never performs this calculation. The training curves (Figure 2) all plot accuracy against training tokens, not against compute. The paper does not report total GPU hours consumed by any training run. There is no table or figure comparing total compute cost across initialization methods.

Mitigation status: Not addressed. The paper does not acknowledge this gap, does not compute FLOPs or GPU hours, and does not discuss the relationship between token speedup and total cost speedup. A FLOPs-matched analysis—analogous to what the reference paper on test-time compute scaling provides in its Section 7—would directly address this limitation but is absent.

6.2 "Final" Accuracy Is Measured at a Fixed, Finite Training Budget—Asymptotic Behavior Is Unknown

The assumption or constraint: All experiments train the target models for a finite number of tokens (roughly 250B for OLMO-2.9B, and unreported exact counts for OPT and Pythia). The paper's claim that HyperCloning improves "final accuracy" (Section 1, Figure 4) refers to accuracy at the chosen stopping point, not at convergence or at any well-defined asymptotic limit. The paper does not demonstrate that the accuracy ordering between HyperCloning and random initialization would persist if both models were trained for substantially more tokens.

The consequence: This limitation matters because it distinguishes between two fundamentally different claims: (a) "HyperCloning accelerates convergence to a given accuracy level" (which the training curves in Figure 2 clearly support), and (b) "HyperCloning raises the asymptotic performance ceiling" (which is not established). If random initialization eventually catches up to HyperCloning given enough training tokens, then HyperCloning's benefit is purely one of training efficiency, and a sufficiently patient practitioner could achieve the same result with random initialization. If HyperCloning genuinely improves the final performance achievable at any budget, that's a stronger and more valuable claim—but the paper doesn't provide evidence to distinguish these scenarios. The catastrophic forgetting observed in OLMO (Figure 2c) adds further uncertainty: the initial accuracy dip consumes a non-trivial fraction of the training budget to recover from, and it's unclear whether this dip represents lost knowledge that is fully recovered or partial permanent damage.

What evidence exists in the paper: The training curves in Figure 2 show HyperCloning above random initialization at the rightmost point of each plot. However, the HyperCloning and random curves have not visibly plateaued or converged—they are still rising at the final data point, meaning the reported "final" accuracy is not a converged value. The paper does not discuss whether the accuracy gap is widening, narrowing, or stable as training progresses. Visual inspection of Figure 2 suggests the gap may be stable or slightly narrowing in OPT (Figure 2a) and Pythia (Figure 2b), and stable or slightly widening in OLMO (Figure 2c), but this is not quantified.

Mitigation status: Not addressed. The paper uses the term "final accuracy" throughout without qualification. There is no discussion of convergence criteria, no training of any model to saturation, and no extrapolation or scaling analysis of how the accuracy gap evolves with additional tokens.

6.3 Catastrophic Forgetting at Training Onset Is Observed but Not Diagnosed or Mitigated

The assumption or constraint: The paper documents that HyperCloning-initialized models "tend to exhibit catastrophic forgetting at the beginning of training" (Section 3.1.1), most dramatically in the OLMO experiment where accuracy drops by roughly 5–8 percentage points over the first ~10–20B tokens before recovering (Figure 2c). The paper does not identify the cause, does not propose mitigation strategies, and does not characterize the conditions under which forgetting is most severe. It explicitly states: "Understanding the underlying causes of catastrophic forgetting, identifying strategies to mitigate it, and exploring why HyperCloning continues to outperform random initialization despite its occurrence are valuable avenues for future research" (Section 3.1.1).

The consequence: This is a practical limitation for two reasons. First, the forgetting represents wasted training tokens: the model must spend the first ~10–20B tokens (in OLMO's case) simply recovering ground lost due to the initialization, which reduces the effective speedup. If forgetting could be eliminated or reduced—through a lower initial learning rate, gradual unfreezing of cloned layers, or a specialized warmup schedule—the speedup could be substantially larger than the reported 2.2×–4×. Second, forgetting creates a minimum viable training budget below which HyperCloning might actually underperform random initialization. If a practitioner only has budget for 5B training tokens, and HyperCloning spends the first 10B tokens in a forgetting dip, random initialization (which starts at zero but only rises) could be the better choice. The paper does not characterize this threshold for any model family.

What evidence exists in the paper: The forgetting is visible in the OLMO training curve (Figure 2c) as a clear dip in the HyperCloning curve before ~20B tokens. The OPT curve (Figure 2a) shows a smaller dip. The Pythia curve (Figure 2b) shows essentially no dip. The paper notes the phenomenon in text (Section 3.1.1) but conducts no ablations to diagnose its cause—no learning rate sweeps, no experiments with different warmup durations, no analysis of which layers or parameters change most during the dip, and no comparison of the forgetting magnitude across base model training budgets (Figure 8, where all three base model checkpoints show qualitatively similar early-training behavior but the dip magnitude is not quantified).

Mitigation status: The paper flags this as future work (Section 3.1.1) but makes no attempt to address it within the current study. No learning rate ablation, gradual unfreezing schedule, or any other forgetting mitigation is tested. The paper's practical recommendation—symmetric initialization without noise (Section 3.4)—is chosen for simplicity, not for its effect on forgetting.

6.4 All Experiments Use a Single Model Scale Range (1.3B–2.9B); Scaling to Larger Models Is Untested

The assumption or constraint: The paper's experiments span target models from 1.3B to 2.9B parameters, with expansion ratios of 2× (OPT-1.3B, Pythia-1.4B, OLMO-2.9B) and one experiment at 4× (OPT-5.3B in Section 3.6, but this is a smaller-scale experiment with incomplete convergence data). The paper's motivating examples (Section 1) reference models at much larger scales—"training a 12-billion-parameter model requires approximately 72,000 GPU hours"—and the economic argument for HyperCloning is strongest at these larger scales where pre-training costs are most burdensome. The paper states the method "can be generalized to a generalized n-fold expansion" (Appendix A), but this is a mathematical claim about the weight transformation, not an empirical claim about training dynamics at large scale.

The consequence: The paper cannot tell us whether HyperCloning's benefits persist, amplify, or diminish at the scales where they would matter most. Several training dynamics could shift unfavorably at larger scale:

Symmetry breaking (Section 3.2): At larger model sizes and larger expansion ratios, dropout might be less effective at decorrelating duplicated neurons, or the decorrelation might take disproportionately longer.
Catastrophic forgetting (Section 3.1.1): The forgetting dip might scale with model size or expansion ratio, consuming a larger fraction of the training budget at larger scales.
Convergence rate: The token speedup (2.2×–4× at 1.3B–2.9B) might not hold at 13B or 70B—it could be larger (making HyperCloning even more valuable) or smaller (making it less impactful).
Numerical precision: The 1/n weight scaling in Case 1 and Case 3 (Appendix A) means weights in the expanded model have standard deviation 1/n of the source model's weights. At large expansion ratios (e.g., 8× or 16×), this could produce very small initial weights that interact poorly with optimizer hyperparameters tuned for standard initialization scales.

What evidence exists in the paper: The OPT-5.3B experiment (Section 3.6, Figure 9) provides one data point for a slightly larger scale and a 4× expansion ratio. HyperCloning from both OPT-350M and OPT-1.3B outperforms random initialization, suggesting the method is not brittle at 4× expansion. However, OPT-5.3B is still a medium-scale model by current standards, and the training curve (Figure 9a) shows incomplete convergence—it's unclear whether the gap between HyperCloning and random initialization would persist at higher token counts. There are no experiments at 7B, 13B, 70B, or larger scales. There are no sweeps of expansion ratio beyond 4×.

Mitigation status: Not addressed. The paper does not discuss scale limitations, does not extrapolate results to larger models, and does not identify scale as an open question. The mathematical generalization to n-fold cloning is noted but not tested.

6.5 No FLOP-Matched Comparison Against Training the Small Model Longer

The assumption or constraint: The paper compares HyperCloning-initialized large models against randomly initialized large models, but never against the obvious alternative: continuing to train the small source model for an equivalent compute budget. This missing baseline addresses the question: given a fixed total compute budget, is it better to expand the model via HyperCloning and train the larger model, or to simply keep training the small model? The paper's amortization argument (Section 3.1.1) implicitly assumes that the small model has reached a point of diminishing returns where further training would be less valuable than scaling up—but this assumption is never tested.

The consequence: The practical recommendation implied by the paper—train a small model, then use HyperCloning to scale up rather than continuing to train the small model—is not empirically supported. It is entirely possible that training OLMO-1B for an additional 250B tokens (total 2.65T) would achieve comparable or better accuracy than HyperCloning to OLMO-2.9B and training that for 250B tokens, at lower total FLOPs (since OLMO-1B is smaller and each training token costs fewer FLOPs). If true, this would undermine the paper's core claim that HyperCloning provides a cost-saving advantage over the status quo. The relevant decision for a practitioner is not "HyperCloning vs. random initialization of the large model" but "HyperCloning (small model + expansion + large model training) vs. continued small model training vs. training the large model from scratch"—and only two of these three are compared.

What evidence exists in the paper: None. There is no experiment where the source model is trained for the same total FLOPs or GPU hours as the HyperCloning-initialized target model. The base model accuracy study (Figure 8) trains different OPT-350M checkpoints but then expands them; it never continues training the OPT-350M checkpoints themselves for comparison. The base model size study (Figure 9) compares different source models (OPT-350M vs. OPT-1.3B) for initializing the same target but never trains the source models further.

Mitigation status: Not addressed. The paper does not acknowledge this missing baseline, does not discuss the tradeoff between continued small-model training and expansion, and does not provide guidance on when a practitioner should expand versus continue training the current model.

6.6 Results Are Single-Run; Statistical Significance and Variance Are Unreported

The assumption or constraint: All training curves (Figure 2), per-task accuracy bars (Figure 4), and ablation comparisons (Figures 7–9) are presented as single-run results. The paper states that "the seed for random shuffling is kept the same across all our experiments to eliminate the impact of data ordering on our conclusions" (Appendix B), which controls for data order effects but does not address variance from other stochastic sources: weight initialization noise (for the random baseline), dropout masks, or GPU nondeterminism. The paper reports no error bars, confidence intervals, standard deviations, or multi-seed replications for any quantitative result.

The consequence: The reported speedup numbers (2.2×–4×) and accuracy improvements are point estimates from single training runs. Training large language models involves substantial run-to-run variance due to the compounding effects of stochastic gradient noise, dropout, and data sampling. A single random-initialization training run might be an unlucky seed that converges slower than typical, inflating the apparent speedup from HyperCloning. Conversely, the HyperCloning run could be a lucky seed. Without replication, the reader cannot assess whether the reported improvements are reliably larger than run-to-run noise. This is particularly important for the speedup metric, which is sensitive to the exact horizontal positioning of training curves—small shifts in either curve's trajectory could meaningfully change the speedup factor. The per-task accuracy comparisons (Figure 4), where some bars differ by only 1–3 percentage points, are especially vulnerable to being within the range of cross-run variance.

What evidence exists in the paper: The paper provides no variance estimates. It mentions controlling the data shuffling seed (Appendix B) but does not discuss other sources of stochasticity or report multiple runs. The training curves appear smooth (Figure 2), which is expected for aggregate accuracy metrics averaged over thousands of evaluation examples, but smoothness of a single trajectory does not imply low variance across trajectories started from different random seeds.

Mitigation status: Not addressed. The paper does not discuss statistical significance, does not report confidence intervals, and does not run multiple seeds. The authors do not flag this as a limitation or suggest that multi-seed replication is needed to validate the reported speedup and accuracy improvements.

7. Implications and Future Directions

How This Work Changes the Landscape

HyperCloning shifts the conversation around LLM pre-training from a paradigm of independent capital investments at each scale toward one where model scaling is an incremental, knowledge-cumulative process. This is less a paradigm shift than a reframing of existing practice—the technical ingredients (weight duplication, function preservation) have existed in various forms since Net2Net (Chen et al., 2015)—but the paper's contribution is demonstrating that these ingredients, when carefully assembled for modern decoder-only transformers, produce substantial training acceleration at scales where the economic stakes are real.

The conceptual shift is this: prior to HyperCloning, the dominant assumption in the model growth literature was that width expansion doesn't work well. Du et al. (2024) had concluded that "non-function-preserving width growth results in poorer performance" compared to depth growth, effectively steering the field toward depth-scaling as the only viable growth axis. HyperCloning reopens width as a legitimate scaling dimension by showing that the earlier negative results were an artifact of non-function-preserving initialization, not an inherent limitation of width expansion. The paper's four design goals—expansion dimension, function preservation, low compute overhead, and unchanged training loop—constitute a specification for what width expansion must achieve to be practically useful, and the cases framework provides a constructive proof that this specification is achievable. This has the effect of rehabilitating width expansion as a research direction that had been prematurely abandoned.

The paper also provides a partial resolution to the contradiction between the theoretical concern about neuron symmetry (Wang et al., 2023b—duplicated neurons might remain identical and waste capacity) and the empirical observation that neuron duplication methods sometimes work. The cosine similarity analysis (Figure 5) and singular value recovery (Figure 6) demonstrate that symmetry does break during training, and that the mechanism is likely stochastic regularization (dropout) rather than any special architectural intervention. This doesn't fully close the theoretical question—correlation ≠ functional independence—but it shifts the burden of proof: the symmetry concern, while theoretically valid for deterministic training, does not appear to materialize as a practical limitation in standard LLM training regimes. Researchers no longer need to treat symmetry as a disqualifying objection to duplication-based width expansion.

Perhaps most importantly, HyperCloning reframes publicly available pre-trained checkpoints as reusable infrastructure. The HuggingFace model hub contains thousands of pre-trained models at various scales, trained at immense collective cost. Prior to this work, those checkpoints were primarily useful for fine-tuning, distillation, or analysis—but not for initializing larger models. HyperCloning provides a mechanism to capitalize that existing investment into new, larger training runs. The OLMO experiment (1B checkpoint trained on 2.4T tokens → 2.9B model trained on only 250B additional tokens) is a concrete demonstration: the 2.4T-token investment by Groeneveld et al. (2024) is not discarded when a researcher wants a larger model in the same family. This changes the economics of model development for organizations that maintain model families across scales—the cost of each scale is partially amortized over all larger descendants.

However, the paper's impact is bounded by what it does not demonstrate. It does not show that HyperCloning changes the asymptotic performance ceiling rather than just accelerating convergence (the "final" accuracy is measured at a finite, arbitrary training budget). It does not provide a FLOP-matched comparison against continued training of the small model—meaning the core practical claim ("use HyperCloning instead of training from scratch to save money") lacks a crucial baseline. And it does not validate the method at the largest scales (7B, 13B, 70B+) where the economic argument is strongest. These gaps mean HyperCloning's contribution is currently a promising demonstration with clear practical implications, but not yet a fully validated scaling strategy. The paper opens a direction; it does not close it.

Several research directions become more attractive in light of this work:

Joint depth-and-width growth strategies become a natural next step—prior work established depth growth as effective, and HyperCloning establishes width growth as effective, so combining them (e.g., depth-expand via block duplication, then width-expand via HyperCloning) could provide compound benefits.
Adaptive expansion schedules—expanding in stages rather than in one shot—become worth exploring, since HyperCloning provides a cheap, function-preserving expansion operator that could be applied repeatedly during training.
Verifier-based model selection for expansion decisions becomes thinkable: if a PRM or similar verifier can assess model quality cheaply, one could decide when to expand based on estimated saturation of the current scale's learning, rather than using a fixed schedule.

Directions that become relatively less attractive:

Non-function-preserving width expansion methods (random initialization of expanded dimensions, zero-initialization, projection-based expansion) are now harder to justify. HyperCloning's ablation (Figure 7) shows that symmetric function-preserving initialization outperforms diagonal (zero-initialized off-diagonals) substantially, and prior work (Du et al., 2024) already showed that non-function-preserving width expansion underperforms depth growth. The combination of these results suggests that function preservation is not a nice-to-have but a necessary condition for effective width expansion. Future work on width growth should either adopt function-preserving methods or provide strong evidence for why a particular non-preserving approach is justified.
Training every model scale from scratch as a default practice becomes harder to defend economically, at least within a model family. If a 350M checkpoint exists (whether trained in-house or downloaded), and HyperCloning provides 2.2×–4× token speedup for the 1.3B training run, the cost of the 350M model is effectively amortized. The burden of proof shifts to proponents of from-scratch training: they must demonstrate that the accuracy gain from random initialization (if any exists at convergence) justifies the additional compute cost.

Follow-Up Research This Work Enables

FLOP-matched comparison: HyperCloning vs. continued small-model training. The single most important missing experiment. Train a small model (e.g., OPT-350M) to completion. Then, given a fixed additional FLOP budget (not token budget), compare three strategies: (a) HyperCloning-expand to OPT-1.3B and train, (b) continue training OPT-350M for the same total FLOPs, and (c) train OPT-1.3B from random initialization for the same total FLOPs. The key metric is accuracy per FLOP, not per token. This experiment directly tests the paper's central economic claim: that expansion plus large-model training is more compute-efficient than simply training the small model longer. It would also calibrate the real-world speedup factor—the paper's 2.2×–4× token speedup may translate to a much smaller FLOP speedup (or none at all) because the larger model processes each token more expensively. The OLMO setting is particularly interesting here: OLMO-1B on 2.4T tokens + 250B tokens of continued training vs. OLMO-2.9B via HyperCloning + 250B tokens of training. If continued small-model training matches or exceeds the expanded model's accuracy at lower total FLOPs, the practical case for HyperCloning weakens considerably.

Catastrophic forgetting ablation: learning rate and warmup schedules. The OLMO experiment (Figure 2c) shows a severe accuracy dip of ~5–8 percentage points over the first ~10–20B training tokens. This represents wasted compute—tokens spent recovering lost ground. A systematic ablation should test: (a) starting learning rate reduced by factors of 2×, 5×, 10× relative to the default schedule, (b) extended warmup periods (50K, 100K, 200K iterations instead of the default 25K), (c) gradual unfreezing—train only LayerNorm parameters and biases for the first N tokens before unfreezing full weights, and (d) layer-dependent learning rates, where earlier layers (which may have learned more general, transferable features) receive lower learning rates than later layers. The diagnostic metrics should include: magnitude of the accuracy dip, number of tokens to recover to the initial accuracy, and final accuracy at the training budget. If a simple learning rate schedule can eliminate or substantially reduce the forgetting dip, the effective speedup of HyperCloning could be significantly larger than the reported 2.2×–4×. Conversely, if the forgetting is robust to learning rate interventions, that tells us something about the relationship between the source model's parameter configuration and the target model's optimization landscape—perhaps the source parameters are genuinely suboptimal for the larger architecture and gradient descent must move away from them before finding a better basin.

Scaling laws for HyperCloning: how does benefit vary with model scale and expansion ratio? The paper tests three target model scales (1.3B, 1.4B, 2.9B) with expansion ratios of 2× (and one partial result at 4× for OPT-5.3B in Figure 9). A systematic scaling study would train HyperCloning-initialized models at multiple scales (e.g., 350M→1.3B, 1.3B→5.3B, 5.3B→13B for OPT-family models) and multiple expansion ratios (2×, 3×, 4×, 8×) and measure: (a) token speedup to reach a fixed accuracy threshold, (b) FLOP-matched accuracy vs. from-scratch training, and (c) whether the benefit saturates or diminishes at larger scales or higher expansion ratios. The hypothesis to test is whether HyperCloning's advantage is scale-invariant (e.g., always ~3× token speedup regardless of scale) or whether it grows or shrinks. The weight standard deviation after HyperCloning is 1/n of the source model's standard deviation (Section 3.2), which at large expansion ratios (e.g., 8× or 16×) could produce very small initial weights that interact poorly with optimizer hyperparameters—this sets a plausible upper bound on practical expansion ratios that should be empirically characterized. Ideally, such a study would also model the cost of training the base model, to produce a total-cost scaling law: given a target model size, what is the optimal base model size to clone from, accounting for the base model's training cost plus the target model's accelerated training cost?

Cross-domain transfer: how does source-target dataset mismatch affect HyperCloning? The Pythia experiment (Pile → DOLMA) is the paper's only cross-dataset test, and both datasets are general-domain English web text—the distribution shift may be modest. A systematic study would vary the degree of domain mismatch: source model trained on (a) general English web text, (b) code-heavy corpora, (c) scientific papers, (d) multilingual text, and target model trained on a held-out domain (e.g., all target models trained on DOLMA). Metrics would include: initial accuracy (how much of the source model's domain-specific knowledge transfers?), forgetting magnitude (does domain mismatch exacerbate the initial accuracy dip?), convergence speedup (does out-of-domain pre-training still provide a useful initialization?), and final accuracy. This would establish boundary conditions: if HyperCloning's benefit degrades gracefully with domain mismatch, it's a robust general-purpose tool; if it requires near-identical data distributions, its applicability is limited to within-family scaling on the same (or very similar) corpora. The experiment would also help distinguish whether HyperCloning transfers mainly linguistic competence (syntax, basic semantics, reasoning patterns—which should transfer across domains) or factual knowledge (which might be domain-specific and not transfer). If linguistic competence is the primary transferred asset, HyperCloning should work well across domains; if factual knowledge dominates, cross-domain transfer should be weaker.

Diagnosing and monitoring functional specialization after cloning. The paper's weight analyses (Figures 5 and 6) show that duplicated weight blocks decorrelate numerically and that matrices recover full rank. But these are structural metrics—they don't demonstrate that the duplicated parameters develop functionally distinct roles. A deeper diagnostic would measure: (a) activation-level specialization—for a given input, do the two copies of a duplicated neuron produce different activation values, and do those differences correlate with task-relevant features? (b) ablation impact—if one copy of a duplicated weight block is zeroed out, does the model's accuracy drop, and is the drop symmetric across copies or concentrated in one copy? If both copies are equally important, that suggests functional redundancy; if one copy matters more on certain inputs or tasks, that suggests specialization. (c) gradient alignment—during training, do the gradients for duplicated weight blocks point in different directions? High cosine similarity between gradients would indicate the copies are being pushed toward similar updates; low similarity would indicate distinct optimization trajectories. These diagnostics would strengthen (or qualify) the paper's claim that the expanded capacity is meaningfully utilized, and could reveal whether some layers or model families are more prone to persistent symmetry than others.

Combined depth-and-width expansion with HyperCloning + block duplication. The paper positions HyperCloning as complementary to depth growth methods (Section 2: "can be accompanied by any of these [depth growth] methods to provide a full recipe for model scaling"). This should be tested directly: take a small model, expand its depth via block duplication (Samragh et al., 2023) or progressive stacking (Gong et al., 2019), expand its width via HyperCloning, and train the jointly-expanded model. Compare against depth-only expansion, width-only expansion, and random initialization of the full target model—all at matched parameter counts. The hypothesis would be that depth and width expansion provide complementary benefits (e.g., depth helps with hierarchical abstraction, width helps with representational capacity per layer), and combining them yields better initialization than either alone. The key metric is accuracy after a fixed training budget relative to from-scratch training. This experiment would move the field toward a unified "model growth cookbook" where practitioners can specify a target architecture and a source checkpoint, and a combined expansion procedure produces the initialization. It would also help resolve the depth-vs-width question that Du et al. (2024) investigated but HyperCloning did not directly address.

Practical Applications and Downstream Use Cases

Efficient model family development for LLM providers. An organization developing a family of models at multiple scales (e.g., 350M, 1.3B, 7B, 13B) currently trains each from scratch at full cost. With HyperCloning, the workflow becomes: (1) train the smallest model (350M) to completion, (2) HyperCloning-expand to initialize 1.3B, train to completion at a fraction of the from-scratch token budget (2.2×–4× fewer tokens per the paper's estimates), (3) HyperCloning-expand 1.3B to initialize 7B, train, and so on. Each scale's training cost is reduced because it inherits the accumulated knowledge of all smaller scales. The OLMO result (Figure 2c) suggests this is particularly powerful when the base model is extensively pre-trained—the 2.4T-token training of OLMO-1B is leveraged to reduce OLMO-2.9B's training to ~250B tokens. In a production setting where a 70B model might cost millions of dollars to train from scratch, even a 2× token reduction represents substantial savings. The primary risk is the unvalidated assumption that the token speedup translates to cost speedup after accounting for the larger model's higher per-token FLOP cost—this should be benchmarked internally before committing to HyperCloning as the primary scaling strategy.

Bootstrapping large models from public checkpoints. A research lab or startup with limited compute can download a fully-trained public checkpoint (e.g., OLMO-1B, Pythia-410M, OPT-350M), apply HyperCloning to initialize a larger architecture, and train the larger model on their own data or for their own domain. The Pythia experiment (Figure 2b) demonstrates that this works even when the source and target training data differ—Pythia-410M (trained on Pile by EleutherAI) successfully bootstrapped Pythia-1.4B training on DOLMA. This means organizations don't need to train a small model themselves to benefit from HyperCloning; they can leverage the massive public investment in pre-trained checkpoints. For a team targeting a 7B-parameter model, starting from a downloaded 1.4B checkpoint via HyperCloning and training on their domain-specific corpus could be dramatically cheaper than training 7B from scratch, while still achieving better final accuracy (per the paper's consistent finding that HyperCloning improves final accuracy at the chosen training budgets). The key practical consideration is that the source and target architectures must be compatible—the number of layers must match, and the attention head structure must be expandable via one of the two strategies (head dimension expansion or head count duplication). This constrains the choice of target architecture but still leaves substantial design flexibility.

Iterative model improvement during active development. In a research setting where models are frequently re-trained with architectural tweaks, hyperparameter changes, or data updates, HyperCloning enables a checkpoint-reuse workflow: (1) train a small model quickly to validate a new training recipe, data mixture, or architectural variant, (2) if the results are promising, HyperCloning-expand to the target scale rather than re-running the experiment at full scale from scratch. The small-scale experiment serves as both a validation step and the initialization for the production-scale run. This reduces the risk of expensive large-scale training failures (the paper's motivation: "training can fail for reasons such as improper learning rate tuning, hardware failures, or loss divergence"—Section 1) by front-loading experimentation to the cheap small-scale regime, where failures cost a fraction as much. The paper's finding that even a partially-trained base model (16B-token OPT-350M in Figure 8) provides significant initialization benefits means the small-scale validation run doesn't need to be trained to convergence to be useful as an initializer—it just needs to be good enough to beat random initialization, which Figure 8 suggests is a low bar.

Pre-training on a compute budget with pre-trained checkpoints as starting assets. For academic groups, smaller companies, or researchers in compute-limited regions who cannot afford to train even a 1B model from scratch, the combination of a freely-downloaded checkpoint + HyperCloning + limited additional training provides a path to models that would otherwise be completely out of reach. If a group downloads a well-trained 1B checkpoint and HyperCloning-expands it to 2.9B (as in the OLMO experiment), then trains on their available compute budget (say, 50B tokens instead of the 250B used in the paper), they will get a model whose accuracy is somewhere on the HyperCloning curve in Figure 2c at 50B tokens. Even if that accuracy is below the paper's final reported accuracy (because training was cut short), it is likely substantially above what they could achieve by training a 2.9B model from scratch on the same 50B-token budget (which would be somewhere on the random initialization curve at 50B tokens, with no warm-start advantage). The paper doesn't characterize the minimum viable training budget for HyperCloning to outperform random initialization—the forgetting dip in OLMO suggests there is a crossover point—but for any reasonably-resourced training run (tens of billions of tokens), the initialization advantage appears to dominate. This use case is not speculative; it follows directly from the paper's training curves, which show HyperCloning above random initialization at every token count beyond the very earliest training steps (and even at step zero, HyperCloning is at the base model's accuracy while random is at chance).

🎯 Pitch

1. Executive Summary

2. Context and Motivation

The Core Problem: Pre-training Large Language Models from Scratch Is Prohibitively Expensive

The Gap: We Know How to Train Small Models Efficiently, but Not How to Leverage That Investment

Prior Approaches: Model Growth and Width Expansion

Why Width Expansion Matters Alongside Depth

How HyperCloning Positions Itself

3. Technical Approach

3.1 Reader Orientation

3.2 Big-Picture Architecture (Diagram in Words)

3.3 Roadmap for the Deep Dive

3.4 Detailed, Sentence-Based Technical Breakdown

Vector Cloning: The Fundamental Building Block

Linear Layer Cloning: Three Cases

Attention Layer Cloning

Layer Normalization Cloning

Positional Embedding Cloning

Noise Addition and Symmetry Breaking

The Complete Algorithm: Step-by-Step

Design Rationale: Why This Approach Satisfies the Four Goals

Contrast with Prior Width Expansion Failures

4. Key Insights and Innovations

Innovation 1: Function Preservation as the Missing Ingredient in Width Expansion

Innovation 2: Empirical Resolution of the Neuron Symmetry Concern

Innovation 3: Width Expansion as an Amortization Strategy for Pre-training Investment

Innovation 4: Catastrophic Forgetting at Initialization as a Diagnostic for Knowledge Transfer Quality

5. Experimental Analysis

Evaluation Methodology

Main Quantitative Results

Aggregate Training Speedup and Final Accuracy

Per-Task Accuracy Analysis

Effect of Base Model Accuracy

Effect of Base Model Size

Analysis of Weight Evolution During Training

Ablation Studies and Robustness Checks

Critical Assessment

Central Claim 1: "HyperCloning achieves 2.2× to 4× faster convergence to the final accuracy of randomly initialized baselines" (Section 1)

Central Claim 2: "HyperCloning improves final accuracy compared to random initialization" (Section 1, Figure 4)

Central Claim 3: "Function preservation is achieved—the larger model retains the functionality of the smaller model before training starts" (Section 1)

Central Claim 4: "The benefits of HyperCloning transfer across datasets—source and target models can be trained on different data" (Pythia experiment, Section 3.1.1)

Central Claim 5: "The model breaks weight symmetry during training and achieves full rank, meaning the expanded capacity is utilized" (Section 3.2, 3.3)

Overall Assessment

6. Limitations and Trade-offs

6.1 Speedup Is Measured in Training Tokens, Not Total Compute Cost

6.2 "Final" Accuracy Is Measured at a Fixed, Finite Training Budget—Asymptotic Behavior Is Unknown

6.3 Catastrophic Forgetting at Training Onset Is Observed but Not Diagnosed or Mitigated

6.4 All Experiments Use a Single Model Scale Range (1.3B–2.9B); Scaling to Larger Models Is Untested

6.5 No FLOP-Matched Comparison Against Training the Small Model Longer

6.6 Results Are Single-Run; Statistical Significance and Variance Are Unreported

7. Implications and Future Directions

How This Work Changes the Landscape

Follow-Up Research This Work Enables

Practical Applications and Downstream Use Cases

Adjacent in domain

Emu3: Next-Token Prediction is All You Need

Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model