Late-to-Early Training: LET LLMs Learn Earlier, So Faster and Better

ArXiv: 2602.05393

🎯 Pitch

A small pretrained model just 1/10th the size of your target LLM can slash training time by nearly 40% while boosting downstream accuracy by 5%—if you guide only the early layers, and only during early training, then cut the cord. The trick is making the small model's learned representations just temporary scaffolding rather than a permanent crutch.

---

1. Executive Summary

This paper proposes Late-to-Early Training (LET), a novel pretraining paradigm that leverages small, publicly available pretrained models to accelerate and improve the training of much larger LLMs — up to 10× larger — without requiring architectural modifications. The method introduces two complementary mechanisms: Late-to-Early Step Learning (using the small model's representations only during the early training steps, with a linearly decaying alignment weight that phases out the guidance as the larger model matures) and Late-to-Early Layer Learning (aligning the late-layer representations of the small pretrained model with the early layers of the target model, leaving later layers free to adapt and refine these representations as training progresses). Extensive experiments on 1.4B and 7B parameter models trained on The Pile dataset demonstrate that LET achieves up to a "1.6× speedup with nearly 5% improvement in downstream task accuracy" compared to standard causal language modeling, even when the pretrained guide model is an order of magnitude smaller than the target, establishing that small pretrained models can meaningfully bootstrap larger-scale training only when the alignment is restricted to early layers and early training steps — not when applied throughout training or to later layers.

2. Context and Motivation

The Core Problem: Pretraining Costs Are Skyrocketing, While Small Pretrained Models Are Underutilized

The fundamental question this paper tackles is both practical and urgent: Given the enormous and growing cost of pretraining LLMs from scratch, can we instead bootstrap the training of large new models using the knowledge already embedded in small, publicly available pretrained models? The scale of the problem is staggering — the paper notes that "training an LLM with 12B parameters can require about 72,000 GPU hours using NVIDIA A100 GPUs," and this cost grows dramatically with model size (Section 1). Meanwhile, the open-source AI community has produced an "array of publicly available models of varying sizes" (Section 1), developed at "significant computational expense." These models — OPT-125M, Pythia-160M, SmolLM-135M, and larger variants — represent enormous upfront investment, yet when practitioners train a new model from scratch, that investment is entirely wasted: the small models sit on the shelf while training proceeds from random initialization.

This disconnect between the cost of pretraining new models and the underutilization of existing pretrained models is the central gap the paper identifies. The problem matters for several concrete reasons that the paper discusses (Section 1):

• Resource constraints on rapid development: As pretraining becomes "increasingly critical yet computationally prohibitive," the pace of LLM development is bottlenecked by training costs. Each new model architecture, dataset, or scaling hypothesis requires a full pretraining run to validate, and these runs are becoming prohibitively expensive.
• Democratization of LLM research: If pretraining always requires starting from scratch with massive compute, only well-resourced organizations can participate. A method that reuses smaller open-source models to accelerate training would lower the barrier to entry — a 1.4B model guided by a 135M model costs dramatically less than the same model trained independently.
• Environmental and sustainability concerns: Training large models from random initialization consumes enormous energy. Any acceleration technique that delivers performance gains with fewer training tokens translates directly to reduced carbon footprint.
• Amortization of community investment: The open-source ecosystem has collectively invested millions of GPU-hours in pretraining models like Pythia, OPT, and SmolLM. The paper argues this investment should be treated as a capital asset that can be leveraged, not as a sunk cost that must be replicated from scratch.

The Knowledge Distillation Paradigm and Why It Falls Short Here

The natural starting point for transferring knowledge from one model to another is knowledge distillation (KD) (Hinton et al., 2015), where a student model is trained to match the output distributions of a teacher model. In standard KD, the teacher is larger and more capable than the student — the teacher's knowledge is compressed into a smaller model. The KD objective, defined in Equation 2 of the paper, minimizes the cross-entropy between the teacher's predicted probability distribution $P_\mathcal{T}(v \mid x_{<t})$ and the student's predicted distribution $P_\mathcal{M}(v \mid x_{<t})$ over the entire vocabulary $\mathcal{V}$:

$\mathcal{L}_{\text{KD}} = -\sum_{t=1}^{T} \sum_{v \in \mathcal{V}} P_{\mathcal{T}}(v \mid x_{<t}) \log P_{\mathcal{M}}(v \mid x_{<t})$

However, applying KD to the scenario where the teacher is smaller than the student — which the paper calls Reverse Knowledge Distillation (RKD) — creates fundamental problems that the paper's experiments confirm (Section 3.2, Table 1):

1. Performance degradation when the size gap is large. The paper demonstrates that RKD "underperforms the baseline in both the 1.4B and 7B settings" when the teacher is an order of magnitude smaller than the student (Table 1). This aligns with prior findings from Lee et al. (2023), who showed that KD degrades performance when the teacher is at least 0.78× smaller than the student. The paper's results extend this finding to a much more extreme regime (10× size difference).

2. Uneven capability transfer. RKD exhibits curious patterns where it improves performance on certain tasks while severely degrading it on others. Table 1 shows RKD performing "relatively well on tasks such as ARC-c and LAMB, indicating stronger reasoning abilities," but "on tasks like SciQ, RKD's performance is substantially lower than that of other methods." The paper hypothesizes that "the distillation process may strengthen certain specific capabilities" while "considerably hindering the model's overall learning effectiveness" — essentially, the small teacher has some useful knowledge, but forcing the larger student to match the teacher's output distribution constrains the student from developing capabilities beyond what the teacher possesses.

3. The ceiling problem. Even when RKD works, the student fundamentally cannot exceed the teacher's capabilities. The teacher's output distribution sets an upper bound on the student's potential, which "limits their utility as a foundation for scaling LLM capabilities" (Section 1). The whole point of training a larger model is to surpass smaller models — a training paradigm that inherently caps performance at the teacher level defeats the purpose.

Standard KD thus fails to address the paper's motivating question. It cannot handle the size asymmetry, it produces uneven results, and it establishes a performance ceiling. Something fundamentally different is needed.

Prior Attempts at Using Small Models to Accelerate Training and Their Limitations

The paper situates itself relative to two additional lines of prior work, both of which attempt to leverage smaller models for accelerating larger model training but fall short in important ways.

Model growth strategies (Net2Net, Bert2Bert, LiGO, StackedBERT, Lemon) address the problem by starting with a small pretrained model and expanding it — either in width, depth, or both — to create a larger model that inherits the small model's learned parameters. This is qualitatively different from the paper's approach because it couples the training process to specific architectural choices:

• These methods "require deliberate architectural modifications, such as carefully calibrated increases in network depth and width" (Section 1). You cannot simply take any small open-source model and grow it — the expansion requires engineering the new architecture to be compatible with the inherited parameters.
• The paper argues this "adds complexity and constrains the range of feasible architectures" (Section 1). If you want to experiment with a novel attention variant, a different activation function, or a different ratio of width to depth, the growth strategy must be re-engineered.
• Growth strategies also require the small model and the large model to share the same basic architecture family, making them "architecture-dependent."

The paper's LET paradigm is explicitly designed to be "architecture-agnostic" — it works across different model families (OPT, Pythia, SmolLM, LLaMA, Qwen) with different activation functions (ReLU, GeLU, SiLU, SwiGLU), attention mechanisms (full attention, GQA), and hidden dimension ratios. Table 3 in Appendix 10 emphasizes this heterogeneity — the small pretrained models differ from the target model in "hidden size, intermediate size, number of layers, number of attention heads, activation functions, and attention mechanisms."

SALT (Rawat et al., 2024) is the closest prior work to LET in spirit, as it explicitly claims that "smaller models can bootstrap the pretraining of larger LLMs." The paper engages with SALT directly (Section 1, Section 8) and identifies several limitations that LET addresses:

• Modest size gap: SALT's experiments use a teacher-student size ratio of only 1.87×, which "limits practical applicability because the teacher remains relatively large and incurs substantial memory overhead" (Section 1). In contrast, LET operates with up to 10× size gaps — a qualitatively different regime where the teacher is genuinely cheap.
• Data preprocessing requirement: SALT "relies on data preprocessing" (Section 1), which adds a preprocessing pipeline between the teacher and student that LET avoids.
• Underutilization of existing models: The paper notes that SALT "underutilizes existing open-source models that were trained at considerable computational cost" (Section 1) — its approach does not fully exploit the representations that have already been learned.
• Performance advantage: In a direct comparison under identical hyperparameter configurations (Section 8, Table 1), LET outperforms SALT: "LET achieves superior performance within the same token budget and LET exhibits stable training dynamics." The paper also notes that SALT optimizes a different training objective (UL2 rather than standard causal LM), meaning its default hyperparameters may be suboptimal for the standard pretraining protocol used in this study.

Where All Prior Approaches Fall Short: The Fundamental Insight Gap

Stepping back from the specific methods, the paper identifies a deeper conceptual gap in prior work: no one has recognized that late-layer representations from a small model can productively guide only the early layers of a larger model, and only during early training steps. This is not a minor design choice — it is the paper's core insight, and it explains why RKD (which operates on output logits throughout training) degrades performance, why SALT (which uses the small model as a data preprocessor) has limited scaling, and why model growth (which inherits all layers) constrains architecture.

The intuition behind this insight has two components:

1. Layer hierarchy in transformers. Deep transformer models process information hierarchically — early layers tend to capture surface-level syntactic and local semantic patterns, while late layers encode more abstract, task-specific representations. A small pretrained model's late layers have already learned useful abstractions (to the extent that the model is capable). If we align only the early layers of the large model to these abstractions, the large model's subsequent layers can build on top of this foundation, potentially surpassing the small model's capabilities. If we align later layers instead, or align throughout training, we constrain the large model to mimic the small model rather than transcend it.

2. Capability gap grows during training. At the start of training, the target model is randomly initialized and the small pretrained model is genuinely more capable — its representations are better, and pulling the target model toward those representations is beneficial. But as the paper notes, "as training progresses, the larger target model rapidly improves in overall capability and may eventually surpass the smaller model in overall performance, thereby reducing the effectiveness of the representations alignment" (Section 3.3). Continuing to align the target model to a teacher it has already surpassed would actively harm performance — the target would be pulled back toward a weaker model's representations.

This is why LET's two mechanisms — late-to-early-step learning (phasing out the alignment weight to zero) and late-to-early-layer learning (aligning only early layers, leaving later layers free) — must work together. Either one alone is insufficient. The paper's ablation studies (Section 3.3) confirm this: aligning late layers (L2L or L2M configurations) produces worse results across the board (Figures 3 and 4), and keeping the alignment weight nonzero throughout training would prevent the target model from surpassing the teacher.

How This Paper Positions Itself

The paper explicitly frames itself as addressing "a novel, valuable, yet overlooked problem" (Section 1):

"Given the abundance of small, pretrained models developed at significant computational expense by the community, can they be leveraged to generally accelerate the pretraining process of much larger LLMs (e.g., 10×), regardless of LLM architectures?"

This formulation carefully distinguishes LET from prior work along several dimensions:

• "Generally accelerate" — not just transfer knowledge for a specific task, but improve the overall pretraining process across both language modeling quality (perplexity) and downstream task generalization.
• "Much larger LLMs (e.g., 10×)" — a scale gap that far exceeds what prior work (RKD, SALT, model growth) has demonstrated, and where standard KD demonstrably fails.
• "Regardless of LLM architectures" — architecture-agnostic, in contrast to model growth strategies that require architectural compatibility.
• "Abundance of small, pretrained models" — explicitly positioning the method as a way to amortize the community's collective investment, making research more resource-efficient.

The paper also positions itself as complementary to, rather than competitive with, two other major lines of training acceleration research (Section 7.2):

• Data selection methods (DoReMi, D4, LESS, Sheared-LLaMA) focus on choosing better training data to improve efficiency. LET is "orthogonal" to these approaches — it operates on the model side rather than the data side, and could potentially be combined with data selection for further gains.
• Learning dynamics methods (various works on sharpness-aware minimization, anisotropic noise, etc.) have succeeded on smaller models (ResNets) but "failed to propose practical training algorithms that work well in LLM training." LET, in contrast, is validated on models up to 7B parameters on realistic pretraining datasets.

Finally, the paper connects to the weak-to-strong generalization literature (Burns et al., 2023) but clarifies an important distinction: weak-to-strong generalization operates in the fine-tuning regime, where a strong pretrained model is fine-tuned on labels generated by a weaker model and somehow surpasses the weak supervisor. LET operates in the pretraining regime, where the target model starts from random initialization and is guided by the small model's representations rather than its output labels. The mechanism is fundamentally different — LET does not rely on the small model's correctness as a supervisor, but rather on the structure of its internal representations as a scaffold for early learning.

The paper's contribution is thus not just a method, but a reframing of how the field should think about pretrained model assets: existing small models are not just competing baselines or fine-tuning starting points — they are training accelerators that, when properly utilized through early-layer and early-step alignment, can bootstrap the development of much larger and more capable models.

3. Technical Approach

3.1 Reader Orientation

The system being built is a pretraining pipeline for a large target language model that is accelerated and improved by borrowing representational knowledge from an already-trained small model during the early phase of training. It solves the problem of wasteful pretraining: instead of initializing a large model from random weights and training it from scratch at enormous cost, LET temporarily aligns the early layers of the new large model with the late layers of a cheap, pre-existing small model so the large model can "learn earlier" — absorbing the small model's hard-won abstractions in its foundational layers and then building richer capabilities on top.

3.2 Big-Picture Architecture (Diagram in Words)

At the highest level, the LET training loop has five major interacting pieces:

1. Training Corpus — a stream of token sequences drawn from The Pile dataset (approximately 20B tokens total), the same text data used in standard causal language modeling.
2. Target Model (\mathcal{M}) — a large, randomly initialised Transformer (1.4B or 7B parameters) whose parameters are updated throughout training. It produces two things: (a) a next-token prediction distribution used in the standard cross-entropy loss, and (b) hidden-state representations at every layer, one of which (an early layer) is extracted for alignment.
3. Small Pretrained Model (\mathcal{T}) — a frozen, off-the-shelf small model (e.g., 135M parameters). It receives the same token embeddings as the target model and produces a final-layer representation that serves as the alignment target.
4. Projection / Reshaping (if needed) — a lightweight dimension-matching step. When the hidden size of \mathcal{T} (e.g., 576 for SmolLM-135M) differs from that of \mathcal{M} (e.g., 2048 for the 1.4B model), a fixed linear interpolation resamples the target model's early-layer representation to the smaller model's dimension before alignment (Appendix 11).
5. Two-Component Loss Function — the total loss \mathcal{L}_{\text{total}} is formed by summing (a) the standard causal language modelling loss \mathcal{L}_{\text{NLL}} (negative log-likelihood), which drives token prediction, and (b) a weighted alignment loss \lambda \cdot \mathcal{L}_{\text{proj}} (cosine similarity, or optionally a logsum variant, between the aligned representations), whose weight \lambda starts at a small positive value \lambda_0 and is scheduled to decay linearly to zero by a predefined step S_{\text{stop}}.

Information flows as follows for each minibatch: token embeddings are fed into both \mathcal{M} (the large target model — updated) and \mathcal{T} (the small frozen model — no gradients). \mathcal{M} produces predictions as usual and computes the standard \mathcal{L}_{\text{NLL}} loss. In parallel, the hidden state after an early layer k of \mathcal{M} (e.g., layer 3) is extracted, optionally interpolated to match the dimension of \mathcal{T}'s output, and normalised. \mathcal{T}'s final-layer hidden state is also normalised. The negative cosine similarity between these two normalised vectors is computed as \mathcal{L}_{\text{proj}}. The two losses are added with the current value of \lambda controlling the influence of \mathcal{L}_{\text{proj}}, and the combined loss is backpropagated to update only the parameters of \mathcal{M}. The entire \mathcal{L}_{\text{proj}} component is active only for the first S_{\text{stop}} training steps, after which \lambda has decayed to zero and training continues with standard \mathcal{L}_{\text{NLL}} alone.

3.3 Roadmap for the Deep Dive

A reader needs to understand LET in a logical sequence that builds the core mechanism before layering on design choices. I explain:

• First, the standard pretraining baseline — what \mathcal{L}_{\text{NLL}} is and what it means in practice, since LET is a modification of this default pipeline and all comparisons are made against it.
• Second, the alignment loss \mathcal{L}_{\text{proj}} — the core mathematical mechanism that pulls the target model's early representations toward the small model's final representations. This includes what the cosine similarity objective computes, why it is used over alternatives, and how the dimension projection works when model sizes differ.
• Third, the \lambda scheduling scheme — how the alignment weight is initialised and decays linearly to zero over S_{\text{stop}} steps. This section also explains why the weight must decay (the capability gap shrinks and eventually reverses as training progresses, after which continued alignment would harm rather than help the target model).
• Fourth, the two identified mechanisms — "Late-to-Early Step Learning" and "Late-to-Early Layer Learning" — as operational definitions. I explain what each term means mechanically, why each is individually necessary, and how the ablation studies in Section 3.3 (L2E, L2M, L2L, M2E, M2M, M2L) systematically validate that aligning early layers and stopping early are both required for the observed gains.
• Fifth, the vector-dimension rescaling step — the interpolation method used when d_{\mathcal{M}} \neq d_{\mathcal{T}}, which is essential for the method to be architecture-agnostic across model families that use different hidden sizes.
• Sixth, the logsum-loss variant — an alternative alignment objective introduced in Appendix 12 that empirically outperforms cosine similarity, together with the intuition for why.
• Finally, all hyperparameter choices and their justifications — the model architectures, batch size, sequence length, learning rate, number of training steps, layer index k, initial alignment weight \lambda_0, and stopping step S_{\text{stop}}, and why each value was chosen from the experiments.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical training methodology paper whose core idea is that late-layer representations of a frozen small pretrained model, when used as an alignment target for only the early layers of a larger model and only during early training, produce a smoother optimisation landscape that accelerates convergence and improves generalisation without imposing architectural constraints.

---

Standard Causal Language Modeling Baseline

Before introducing the alignment mechanism, it is essential to ground the discussion in what the target model does under standard pretraining, because LET is a modification of this default regime and all performance comparisons are made against it.

The target model \mathcal{M} is a Transformer language model with parameters \theta, trained on token sequences drawn from The Pile dataset. For a given input sequence \mathbf{x} = [x_1, x_2, \ldots, x_T] of length T, the model processes it autoregressively: at each position t (where 1 \leq t \leq T), the model takes the prefix x_{<t} = [x_1, \ldots, x_{t-1}] as context and produces a conditional probability distribution P_{\mathcal{M}}(x_t \mid x_{<t}) over the vocabulary \mathcal{V}. The training objective is to maximise the likelihood of the observed sequence, which is equivalent to minimising the negative log-likelihood (NLL):

$\mathcal{L}_{\text{NLL}} = -\sum_{t=1}^{T} \log P_{\mathcal{M}}\big(x_t \mid x_{<t}\big)$

where P_{\mathcal{M}}(x_t \mid x_{<t}) is the probability that the model assigns to the actual next token x_t given the preceding tokens, and the sum runs over all T token positions in the sequence. The outer expectation over minibatches in the training data is implicit.

What it computes: For each token position, the model outputs a probability distribution over the full vocabulary, and the log probability assigned to the correct token is extracted. The negative of this log probability is the NLL at that position. Summing over the sequence yields a scalar loss per example; averaging across the minibatch yields the training objective. A lower value means the model is more confident about the correct tokens.

Why this form: The NLL is the maximum-likelihood objective for categorical distributions when the training signal is a hard label (the single correct next token). Minimising NLL is equivalent to maximising the probability of the data under the model, which under mild conditions yields a consistent estimator of the true data-generating distribution. It is the universal default for causal language model pretraining and thus serves as a clean, well-characterised baseline.

The hidden states through the network are computed in the standard Transformer way. For a model with L_{\mathcal{M}} layers, the forward pass through the first k layers is written as a composition of layer functions:

$h_{\mathcal{M}}^{(k)} = \mathcal{F}_{\mathcal{M}}^{(k)} \circ \dots \circ \mathcal{F}_{\mathcal{M}}^{(1)}(e_{1:T})$

where e_{1:T} = [e_1, \ldots, e_T] are the token embeddings (each e_t \in \mathbb{R}^{d_{\mathcal{M}}}), \mathcal{F}_{\mathcal{M}}^{(l)} is the function implemented by the l-th Transformer layer (self-attention + feed-forward + normalisation), and h_{\mathcal{M}}^{(k)} \in \mathbb{R}^{B \times S \times d_{\mathcal{M}}} is the batch of hidden states after the k-th layer, with B the batch size and S the sequence length.

The standard pretraining baseline does nothing else: the full model's output logits are used to compute the NLL, gradients are taken with respect to all parameters, and the optimiser updates the weights. This is the regime that LET modifies.

---

The Alignment Loss \mathcal{L}_{\text{proj}}: Late-to-Early Layer Learning

The central innovation of LET is the addition of a representation alignment term to the training loss during the early phase of pretraining. This term pulls the early-layer hidden states of the target model \mathcal{M} toward the final-layer hidden states of the frozen small pretrained model \mathcal{T}, thereby injecting existing representational knowledge into the foundational layers of the new model.

Extracting the two representations. For the same input token sequence \mathbf{x}, two hidden-state vectors are extracted from the same token position (the procedure is per-token):

• From the target model \mathcal{M}: the hidden state after layer k, where k is chosen to be an early layer. In all reported configurations, k = 3 (the third Transformer layer). This vector is denoted h_{\mathcal{M}}^{(k)} \in \mathbb{R}^{d_{\mathcal{M}}} for a single token, where d_{\mathcal{M}} is the hidden dimension of the target model.
• From the small pretrained model \mathcal{T}: the hidden state after the final layer L_{\mathcal{T}}. This vector is denoted h_{\mathcal{T}}^{(L_{\mathcal{T}})} \in \mathbb{R}^{d_{\mathcal{T}}}, where d_{\mathcal{T}} is the hidden dimension of the small model.

Dimension matching. When d_{\mathcal{M}} \neq d_{\mathcal{T}} — which is the common case, since the target model is larger and often has a different hidden size — the target model's representation must be resampled to match the small model's dimension before comparison. The paper uses linear interpolation along the hidden dimension, treating it as a one-dimensional signal. For each target coordinate j \in \{0, \ldots, d_{\mathcal{T}} - 1\} in the resized output, the source position u_j in the original d_{\mathcal{M}}-dimensional space is computed as:

$u_j = j \cdot \frac{d_{\mathcal{M}} - 1}{d_{\mathcal{T}} - 1}$

The resampled value at position j is then:

$\tilde{h}_{\mathcal{M}, j}^{(k)} = (1 - \beta_j) \cdot h_{\mathcal{M}, \lfloor u_j \rfloor}^{(k)} + \beta_j \cdot h_{\mathcal{M}, \lfloor u_j \rfloor + 1}^{(k)}$

where \beta_j = u_j - \lfloor u_j \rfloor is the fractional part of the source position. This is a standard linear interpolation: when the source index falls exactly on a grid point (\beta_j = 0), the corresponding value is used directly; when it falls between two grid points, the weighted average of the two neighbours is used.

What it computes: Given a source vector of dimension d_{\mathcal{M}} and a desired output dimension d_{\mathcal{T}}, the interpolation reparametrises the vector as if it were sampled at equally spaced fractional coordinates along its length. The endpoints are preserved (j=0 maps to position 0 in the source; j = d_{\mathcal{T}} - 1 maps to position d_{\mathcal{M}} - 1), and intermediate values are linear blends of adjacent source positions. The result \tilde{h}_{\mathcal{M}}^{(k)} \in \mathbb{R}^{d_{\mathcal{T}}} has the same dimension as the small model's output.

Why this form: Interpolation is computationally trivial (no learned parameters), differentiable, and preserves the ordinal structure of the hidden dimension. The alternative — a learned linear projection (a d_{\mathcal{M}} \times d_{\mathcal{T}} matrix) — would add trainable parameters and complicate the optimisation, potentially allowing the projection to "cheat" by learning to map arbitrary source vectors to the target distribution rather than forcing the target model's representations to move. Interpolation ensures that the alignment signal directly acts on the target model's representations themselves rather than on a learned intermediary. The paper also explicitly considers this choice justified because the "hidden dimension is treated as a 1D signal," assuming that nearby positions in the hidden dimension carry related information.

Once both vectors are the same size, they are L2-normalised to unit length:

$\tilde{h}_{\mathcal{M}}^{(k)} \leftarrow \frac{\tilde{h}_{\mathcal{M}}^{(k)}}{\|\tilde{h}_{\mathcal{M}}^{(k)}\|}, \qquad \tilde{h}_{\mathcal{T}}^{(L_{\mathcal{T}})} \leftarrow \frac{h_{\mathcal{T}}^{(L_{\mathcal{T}})}}{\|h_{\mathcal{T}}^{(L_{\mathcal{T}})}\|}$

and the alignment loss is defined as the negative dot product (equivalently, negative cosine similarity) between the two normalised vectors:

$\mathcal{L}_{\text{proj}} = -\tilde{h}_{\mathcal{M}}^{(k)\top} \tilde{h}_{\mathcal{T}}^{(L_{\mathcal{T}})}$

For a sequence of S tokens, the loss is averaged over all token positions:

$\mathcal{L}_{\text{proj}} = -\sum_{i=1}^{S} \frac{\tilde{h}_{\mathcal{M}, i}^{(k)\top} h_{\mathcal{T}, i}^{(L_{\mathcal{T}})}}{\|\tilde{h}_{\mathcal{M}, i}^{(k)}\| \cdot \|h_{\mathcal{T}, i}^{(L_{\mathcal{T}})}\|}$

What it computes: For each token position, the cosine similarity between the target model's early-layer representation and the small model's final-layer representation is computed (a value in [-1, 1] where 1 means identical direction, 0 means orthogonal, and -1 means opposite). The loss is the negative of this value, so minimising \mathcal{L}_{\text{proj}} pushes the cosine similarity toward +1 — the two vectors become more aligned in direction. The loss ignores the magnitude of the vectors because of the normalisation; only the angular relationship is penalised.

Why this form: Cosine similarity normalises away differences in magnitude, which is critical when the two models have different capacities and different overall activation scales. If the alignment loss used Euclidean distance directly, the large model could satisfy the loss by simply shrinking its activations to match the small model's typical magnitude without actually learning the directional structure of the representations. Cosine loss forces the target model to reproduce the geometry of the small model's representation space — which patterns are similar to which other patterns — without constraining the activation scale that the subsequent layers can use. This form is common in representation alignment literature (e.g., SimCLR, BYOL) for exactly this reason: it focuses the learning signal on relative rather than absolute representational structure.

The gradient of \mathcal{L}_{\text{proj}} flows back through the target model only (the small model \mathcal{T} is frozen), updating the parameters of the first k layers of \mathcal{M} (and any embedding parameters) to make their output more collinear with the small model's final-layer output.

---

The \lambda Scheduling Scheme: Late-to-Early Step Learning

The alignment loss is not applied with a constant weight throughout training. Instead, it is scaled by a linearly decaying coefficient \lambda that starts at an initial value \lambda_0 and reaches zero at a predetermined step S_{\text{stop}}. The total loss at training step s is:

$\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{NLL}} + \lambda(s) \cdot \mathcal{L}_{\text{proj}}$

where:

$\lambda(s) = \lambda_0 \cdot \max\left(0, \frac{S_{\text{stop}} - s}{S_{\text{stop}}}\right)$

with \lambda_0 = 0.1 as the default initial weight and S_{\text{stop}} = 1500 steps as the default stopping point.

What it computes: For s = 0, \lambda = \lambda_0. At each subsequent step, \lambda decreases linearly. At s = S_{\text{stop}}, \lambda = 0, and for all steps s > S_{\text{stop}}, the \max function keeps \lambda at zero — the alignment term is permanently switched off. The total loss then reverts to pure \mathcal{L}_{\text{NLL}} for the remainder of training.

Why this form: The linear decay schedule implements the late-to-early-step learning principle. Three intertwined reasons are provided by the paper:

1. Initial capability asymmetry requires guidance. At the start of training, the target model \mathcal{M} is randomly initialised and its representations are essentially noise. The small pretrained model \mathcal{T}, while much smaller, has already been trained on a large corpus and possesses structured representations that capture meaningful linguistic patterns. During this phase, the alignment loss provides a strong learning signal: the early layers of \mathcal{M} can "shortcut" the process of discovering basic representational structure by mimicking \mathcal{T}'s already-learned abstractions. The paper confirms this interpretation through its theoretical analysis (Section 13), which shows that aligning to an early layer results in a smoother optimisation landscape (smaller Hessian Frobenius norm) compared to aligning to later layers, making gradient-based optimisation faster in the early stages.

2. The capability gap shrinks and eventually reverses. As training progresses, \mathcal{M} processes more data and — being many times larger than \mathcal{T} — rapidly gains capabilities. At some point, \mathcal{M}'s representations surpass those of \mathcal{T} in quality and richness. If alignment continued past this point, \mathcal{L}_{\text{proj}} would actively pull \mathcal{M} backward toward a weaker set of representations, constraining it from developing capabilities beyond what \mathcal{T} has. The paper articulates this explicitly in Section 3.3: "As training progresses, the larger target model rapidly improves in overall capability and may eventually surpass the smaller model in overall performance, thereby reducing the effectiveness of the representations alignment."

3. Empirically validated optimal window. The ablation in Section 8 and Figure 10 compares S_{\text{stop}} = 1500 against S_{\text{stop}} = 3000. At the 5B-token mark, the longer alignment (S_{\text{stop}} = 3000) initially provides better performance, confirming that "with a larger S_{\text{stop}}, the alignment strength remains higher for a longer period during the early stages of training, which is beneficial for initial learning." However, "as training progresses and the student model, being much larger, develops a greater capacity to capture complex knowledge, continued alignment with a much smaller teacher model can actually hinder further improvement." The shorter window (S_{\text{stop}} = 1500) yields better final performance, demonstrating the optimal trade-off: align long enough to benefit from the initial guidance, but not so long that the stronger model is constrained.

The choice of S_{\text{stop}} = 1500 steps corresponds to approximately 3.9 billion tokens of training (1500 steps × 2.62 million tokens per step). This represents roughly 20% of the total 20B-token training budget, meaning the alignment signal is active only in the first fifth of the training process.

The choice of \lambda_0 = 0.1 is motivated by the hyperparameter sweep in Figure 5 and Figure 6. Values of \lambda larger than 0.1 (specifically 0.3, 1.0, 3.0) cause "performance to consistently drop, indicating that larger values induce excessive alignment" — the model becomes too constrained by the small teacher and cannot develop its own representations. Values smaller than 0.1 (specifically 0.01) yield "performance above the baseline but still below that achieved with \lambda = 0.1, suggesting that alignment is insufficient at this lower value." Figure 6 confirms this interpretation visually: larger \lambda values produce higher cosine similarity between \mathcal{M} and \mathcal{T} representations (stronger alignment), but the optimal downstream performance occurs at \lambda = 0.1, representing a sweet spot between leveraging the teacher's knowledge and retaining flexibility for self-development.

---

The Two Named Mechanisms: Operational Definitions and Their Interaction

The paper names two mechanisms as the "key mechanisms that drive LET's effectiveness." They are not separate algorithmic components but rather two aspects of a single design, and their definitions are operational:

Late-to-Early Step Learning refers to the fact that the pretrained model \mathcal{T} — which represents a later training stage (it has already been fully trained on a large corpus) — is used to guide the target model only during the early training steps of its own pretraining, with the guidance phased out once the target model has progressed far enough. Mechanically, this is implemented by the decaying \lambda schedule and the choice of S_{\text{stop}}. The "late" in the name refers to \mathcal{T}'s fully-trained status; the "early" refers to the target model's initial steps; the mechanism is the temporal gating of the alignment signal.

Late-to-Early Layer Learning refers to the fact that the late-layer representations of \mathcal{T} (specifically, the final Transformer layer) are aligned with the early-layer representations of \mathcal{M} (specifically, layer 3). The "late" refers to \mathcal{T}'s layer depth; the "early" refers to \mathcal{M}'s layer depth; the mechanism is the choice of alignment depth.

These two mechanisms are inseparable in the LET design, and the ablation studies in Section 3.3 demonstrate this concretely. The paper tests six layer-alignment configurations, all keeping the \lambda scheduling identical, to isolate the effect of the layer choice:

• L2E: Late (\mathcal{T} final layer) to Early (\mathcal{M} layer 3) — this is LET.
• L2M: Late to Middle (\mathcal{M} middle layer)
• L2L: Late to Late (\mathcal{M} final layer)
• M2E: Middle (\mathcal{T} middle layer) to Early
• M2M: Middle to Middle
• M2L: Middle to Late

The results in Figure 3 and Figure 4 reveal:

1. Layer depth of \mathcal{T} matters. Using the middle layer of \mathcal{T} for alignment (M2E, M2M, M2L) consistently underperforms using the final layer (L2E, L2M, L2L). This is evident in both downstream task accuracy (Figure 3) and perplexity (Figure 4). The paper's interpretation is that the final-layer representations of a trained model contain the most abstract, semantically rich features — the "conclusions" the model has reached after processing the full input — whereas middle-layer representations are still partially formed. Aligning to abstract features rather than intermediate ones provides a more useful foundation.

2. Layer depth of \mathcal{M} matters even more. Among configurations using the final layer of \mathcal{T}, L2E (aligning to \mathcal{M}'s third layer) clearly dominates L2M and L2L. Both downstream performance (Figure 3) and perplexity trajectories (Figure 4) show L2E as the best configuration. The perplexity figure in particular reveals a qualitative difference: non-L2E strategies show "varying degrees of perplexity increase immediately post-alignment" — when \lambda decays to zero, their perplexity jumps up, indicating that the representations they learned during the alignment phase do not smoothly transition to the self-supervised NLL objective. L2E, in contrast, "maintains consistent performance" through the transition, showing "superior robustness."

3. The buffer effect explains L2E's superiority. The paper attributes this robustness to the structural property of aligning to early layers: "by mapping the representations of \mathcal{T} to the early layers of \mathcal{M}, the subsequent layers retain sufficient capacity to adapt and refine these representations through the learning dynamics of training." In other words, the later layers of \mathcal{M} (layers 4 through 24 in the 1.4B model) act as a representation-refinement buffer. They receive the aligned early-layer features as input and can learn to process, transform, and abstract them further, eventually surpassing what \mathcal{T} itself could achieve. If the alignment target were a later layer, the remaining layers above it would be too few to provide meaningful adaptation capacity.

Figure 14 in Appendix 14 provides further resolution on the layer choice: experimenting with different early-layer indices (L1-F1, L1-F2, L1-F3, L1-F5, where F1 is layer 1, etc.), the paper finds that "L1-F3 achieves the best performance" and that "L1-F5 underperforms compared to L1-F3." The interpretation is that "the first layer may primarily encode input-specific information" (embedding-level details), making it a poor alignment target, while layer 3 "strikes an optimal balance for representation alignment" — far enough from the input to contain structured features, but early enough that the remaining layers provide substantial adaptation capacity.

---

Vector-Dimension Rescaling (Projection for Architecture-Agnostic Operation)

The interpolation-based projection described earlier is what makes LET architecture-agnostic. Without it, LET would only apply when the target model and the small pretrained model happen to use the same hidden dimension, which would severely restrict which open-source models could serve as \mathcal{T}.

The paper emphasises this in Table 3 (Appendix 10), which lists the architectural heterogeneity across the models used in the experiments:

• SmolLM2-135M (used as \mathcal{T} for the 1.4B \mathcal{M}): hidden size 576, intermediate size 1536, 30 layers, 9 attention heads, SiLU activation, GQA attention.
• The 1.4B target model: hidden size 2048, intermediate size 5461, 24 layers, 32 attention heads, SwiGLU activation, full (multi-head) attention.
• SmolLM2-1.7B (used as \mathcal{T} for the 7B \mathcal{M}): hidden size 2048, intermediate size 8192, 24 layers, 32 heads, SiLU activation, GQA attention.
• The 7B target model: hidden size 4096, intermediate size 11008, 32 layers, 32 heads, SwiGLU activation, full attention.
• OPT-125M and Pythia-160M (additional \mathcal{T} variants): different hidden dimensions, activation functions (ReLU for OPT, GeLU for Pythia), and attention types.

In every case except SmolLM2-1.7B-to-7B (where both happen to use 2048), the hidden dimensions differ, and the interpolation step is executed. The procedure is:

1. Determine the source dimension d_{\mathcal{M}} (target model early-layer hidden size) and target dimension d_{\mathcal{T}} (small model final-layer hidden size).
2. For each of the d_{\mathcal{T}} output positions, compute the floating-point source index u_j = j \cdot (d_{\mathcal{M}} - 1) / (d_{\mathcal{T}} - 1).
3. Linearly interpolate between the two nearest source positions. No parameters are learned; this is a differentiable deterministic operation.

What it computes: A resampled version of the target model's hidden-state vector, with the same number of elements as the small model's output vector, where the value at each output position is a weighted blend of the two nearest neighbours in the original vector.

Why this form: Learned projections introduce additional parameters and optimisation dynamics. A free, fixed interpolation removes a potential confound — any improvement must come from the target model learning better representations, not from a projection layer learning to map arbitrary vectors to the teacher's output space. Additionally, the lack of learned parameters makes the method trivially applicable to any pair of hidden dimensions without hyperparameter tuning.

---

The Logsum-Loss Variant (Appendix 12)

While cosine similarity (negative dot product of normalised vectors) is the default alignment objective, the paper also investigates an alternative: logsum loss. This is introduced in Appendix 12 and is motivated by prior work (Miles et al., 2024) showing that "logsum loss demonstrates promising performance when applied to models with significant capacity gaps."

What it computes: The exact mathematical definition is not provided in the paper text, but the concept is that logsum loss emphasises regions where the representations between \mathcal{T} and \mathcal{M} diverge most significantly, rather than treating all token positions equally (as cosine similarity does). The paper states: "We attribute the effectiveness of logsum loss to its tendency to emphasize regions where representations between \mathcal{T} and \mathcal{M} diverge significantly, which provides explicit guidance by directing model \mathcal{M} to prioritize learning features with the largest discrepancies."

Why this form: In settings with large capacity gaps, the small model \mathcal{T} cannot learn certain features at all — its capacity is simply insufficient. Cosine similarity attempts to align the target model to the teacher equally across all tokens, including those where the teacher's representations are essentially meaningless (noise). Logsum loss, by upweighting regions of disagreement, focuses the alignment signal on tokens where the teacher actually has something to say (non-trivial features) while reducing the penalty for tokens where the teacher's representation is uninformative and the target model should instead learn from the NLL loss.

Empirical result: Table 5 (Appendix 12) shows that LET with logsum loss ("LET-LogSum") outperforms LET with cosine similarity on average downstream task accuracy. Both substantially outperform the baseline, RKD, and SALT. The paper also investigates CCA (Canonical Correlation Analysis) as an alternative alignment objective ("LET-CCA"), but does not report its definition or detailed results in the main text.

---

Full Hyperparameter Configuration and Training Protocol

Below is a complete specification of every reported hyperparameter, with explanations for why each value was chosen or what it controls.

Model architectures (Table 3):

All target models use RMSNorm (Zhang et al., 2019) and SwiGLU activations (Shazeer et al., 2020), following the LLaMA architecture (Touvron et al., 2023).

Training data and tokenisation:

• Dataset: The Pile (Gao et al., 2020), approximately 20 billion tokens after filtering. The specific split used is "approximately 20 billion tokens" from the full ~825 GB Pile corpus.
• Sequence length: 1280 tokens. At a total batch size of 2048, each training step processes approximately 2048 × 1280 = 2.62 million tokens.
• Total training steps: The paper does not explicitly state the total number of steps, but since approximately 20 billion tokens are used and each step processes ~2.62M tokens, the total is approximately 7,600 steps. The alignment period S_{\text{stop}} = 1500 steps thus covers roughly the first 20% of training.

Optimisation:

• Optimiser: AdamW (Loshchilov and Hutter, 2017) with β_1 = 0.9, β_2 = 0.999, weight decay 0.01, and gradient clipping at max norm 1.0.
• Learning rate schedule: Cosine decay, with linear warmup during the first 10% of training steps and decay to 10% of the peak learning rate thereafter.
• Peak learning rate: 4 × 10^{-4} for 1.4B-scale models and 3 × 10^{-4} for 7B-scale models, following the configuration from Groeneveld et al. (2024) (the OLMo setup).
• Precision: BF16 mixed-precision training.

Batch and hardware configuration:

• Total batch size: 2048 sequences. For the 1.4B model, this is achieved with per-GPU batch size 16 and gradient accumulation factor 4 across 32 GPUs. For the 7B model, per-GPU batch size 4 and accumulation factor 16 across 32 GPUs to accommodate memory constraints.
• Hardware: 32 NVIDIA A100 80GB GPUs for all experiments.

LET-specific hyperparameters:

• Alignment layer k: Layer 3 of the target model in all experiments. This is referred to as the "early layer" throughout. Appendix 14 (Figure 12) validates this choice: among layer pairings L1-F1, L1-F2, L1-F3, L1-F5, L1-F3 yields the best downstream performance.
• Small model layer used for alignment: Final layer L_{\mathcal{T}} in all experiments. This is validated by the layer-pairing ablation (Figure 3): configurations using the final layer (L2E, L2M, L2L) outperform those using a middle layer (M2E, M2M, M2L).
• Initial alignment weight \lambda_0: 0.1. Validated by the sweep over {0.01, 0.1, 0.3, 1.0, 3.0} in Figure 5.
• Stopping step S_{\text{stop}}: 1500 steps. Validated by comparison with 3000 in Figure 10.
• Alignment loss function: Negative cosine similarity (Equation 7). The logsum variant is an alternative studied in Appendix 12.
• Dimension matching: Linear interpolation (no learned parameters) when d_{\mathcal{M}} \neq d_{\mathcal{T}}; identity otherwise.

Small model \mathcal{T} selection:

The paper experiments with several small pretrained models to demonstrate robustness:

• For the 1.4B target model: SmolLM2-135M (10.4\times smaller), OPT-125M (11.2\times smaller), Pythia-160M (8.75\times smaller).
• For the 7B target model: SmolLM2-1.7B (4.1\times smaller), Llama-3.2-1B (7\times smaller).
• The "10× smaller" claim in the abstract and introduction refers specifically to SmolLM-135M (135M vs. 1.4B).
• For cross-domain experiments (Appendix 9): Qwen-0.5B as the target model, TimesNet as \mathcal{T} for time-series classification.

Evaluation protocol:

• Downstream tasks: Nine tasks from the OLMo evaluation suite (HellaSwag, Winogrande, LAMBADA, OpenbookQA, ARC-easy, ARC-challenge, PIQA, SciQ, BoolQ), evaluated in one-shot setting using the EleutherAI evaluation harness.
• Perplexity: Evaluated on the test split of The Pile at 500-step intervals (approximately every 1.3 billion tokens).
• Baselines: Standard causal language modeling (Baseline), Reverse Knowledge Distillation (RKD), and SALT (Rawat et al., 2024).

Throughput and resource comparison (Table 4):

The paper provides throughput, wall-clock time, and peak VRAM comparisons in Appendix 10. Key figures:

• 1.4B model throughput: LET achieves 0.927× the throughput of the baseline during the alignment phase (the overhead comes from the forward pass through the frozen \mathcal{T} and the interpolation/computation of \mathcal{L}_{\text{proj}}). RKD achieves 1.004× (slightly faster because it operates on logits, not hidden states, but this advantage is trivial). SALT achieves 0.983×.
• 7B model throughput: LET achieves 0.960× of baseline; RKD achieves 0.980×; SALT achieves 0.969×.
• Peak VRAM: LET achieves lower peak VRAM than RKD and SALT because it aligns hidden states rather than logit distributions. The logit space has dimension |\mathcal{V}| (typically 50K+), which requires significantly more memory for the teacher-student interaction than hidden states (dimensionality 576–4096). This gives LET an advantage for large-batch training.
• Scaling efficiency: When scaling from a 1.4B target model with a 135M \mathcal{T} to a 7B target model with a 1.7B \mathcal{T}, the throughput decrease is minimal, "demonstrating that LET is not only efficient but also highly scalable."

The paper is careful to note that although the baseline has slightly higher throughput (1.078× that of LET in early training), the 1.6× convergence speedup from LET "more than compensates for the modest reduction in throughput" — the total wall-clock time to reach a given performance level is lower with LET despite the per-step overhead.

---

Summary of Design Choices and Their Justifications

• Cosine similarity as the alignment objective: normalises away magnitude differences between models of different capacities, focusing the learning signal on representational geometry. The negative dot product form is simple, differentiable, and widely used in representation learning.
• Alignment only during early training steps (phased out via \lambda decay): prevents the stronger target model from being constrained by the weaker teacher once it has surpassed the teacher's capabilities. The linear decay schedule is simple and empirically effective.
• Alignment only to early layers of the target model (layer 3): leaves the remaining layers (21 of 24 in the 1.4B model) as an adaptation buffer that can refine and transcend the aligned features. The specific choice of layer 3 is validated by an ablation over layer indices (Figure 12, Appendix 14).
• Using the final layer of the small model as the alignment source: captures the most abstract, fully-processed features the small model can produce. Ablations confirm final-layer alignment outperforms middle-layer alignment (Figures 3 and 4).
• Interpolation rather than learned projection for dimension matching: parameter-free, trivially differentiable, avoids introducing additional trainable components that could absorb the alignment signal, and preserves the ordinal structure of the hidden dimension.
• Frozen small model (no gradient flow to \mathcal{T}): ensures the alignment signal is purely about improving \mathcal{M}'s representations, not adapting \mathcal{T}'s representations to match \mathcal{M}. It also enables reuse of a single \mathcal{T} forward pass for any number of alignment steps (no need to store optimiser states for \mathcal{T}).
• AdamW with cosine learning rate schedule following OLMo: a well-established, reliable optimiser configuration for LLM pretraining that provides a fair baseline comparison.
• The Pile dataset at 20B tokens: large enough to produce meaningful convergence trends for 1.4B and 7B models, while fitting within the computational budget. The paper acknowledges that further scaling to 1T-token datasets is left for future work.

4. Key Insights and Innovations

Innovation 1: Reframing Small Pretrained Models as Training Accelerators, Not Competitors or Ceilings

The paper's most fundamental conceptual move is not algorithmic but economic and strategic: it redefines the community's vast collection of small, publicly available pretrained models — typically treated as baselines, fine-tuning starting points, or legacy artifacts — as reusable training accelerators for much larger models. This reframing is what makes LET more than just another distillation variant; it is a proposal for how the field should think about amortizing its collective computational investment.

Prior to this work, the dominant mental model for using a pretrained model to help train another model was knowledge distillation (Hinton et al., 2015), which operates under a strict assumption: the teacher must be more capable than the student. The teacher's output distributions encode knowledge that the student compresses into fewer parameters. This paradigm inherently positions the teacher as a performance ceiling — the student can at best approach the teacher's capabilities, never surpass them. When the teacher is smaller than the student, this assumption breaks down catastrophically. The paper's experiments on Reverse Knowledge Distillation (RKD) confirm this: RKD "underperforms the baseline in both the 1.4B and 7B settings" (Table 1), and the degradation is worse when the size gap is larger (10× for the 1.4B case).

The paper's reframing sidesteps this ceiling problem entirely by changing what is transferred. RKD transfers output behavior (logit distributions) — the teacher's final decisions about what token should come next. LET transfers representational structure (hidden-state geometry) — how the teacher organizes information internally. The distinction matters because output behavior is a finished product that inherently reflects the teacher's limitations, whereas representational structure is an intermediate abstraction that can serve as a foundation for further development. The large target model, by absorbing the small model's representational geometry into only its early layers and then building additional layers on top, can use the small model's knowledge as a scaffold rather than a ceiling.

This reframing is significant beyond the specific method because it suggests a new category of research question: what forms of knowledge can be productively transferred from weaker to stronger models, and under what conditions? The paper's negative result with GPT-2 (Appendix 14, Table 6) sharpens this question: when the small model's training data is outdated ("GPT-2's representations fail to provide effective alignment signals" due to a 2017 data cutoff), LET underperforms the baseline. So the transfer is not universally beneficial — it depends on the quality and recency of the small model's representations. This opens a research program around characterizing when representational transfer helps versus hurts, and what properties of the small model (training data, architecture, training objective) determine transfer quality.

The practical implication is equally important: organizations and researchers no longer need to choose between "train from scratch at enormous cost" and "fine-tune an existing model at the cost of architectural flexibility." LET offers a third path: use small existing models to accelerate custom large-model training, amortizing the community's sunk costs while retaining full architectural freedom for the target model. Table 3 in Appendix 10 emphasizes this architectural freedom by showing that LET works across models with different hidden sizes, layer counts, activation functions, and attention mechanisms — a degree of heterogeneity that model growth strategies (Net2Net, LiGO, StackedBERT) cannot accommodate.

---

Innovation 2: The Diagnostic Discovery That Only Early Layers and Early Steps Benefit from Weak-Model Guidance

The paper's second major contribution is an empirical diagnostic that explains why reverse knowledge transfer fails when applied naively and when it can succeed. This is not a method but a finding about the structure of the problem — one that the paper systematically validates through ablations and that resolves the apparent contradiction between the paper's positive results and prior negative results on RKD.

The diagnostic has two parts:

Part 1: The alignment depth matters qualitatively, not just quantitatively. The layer-pairing ablation in Section 3.3 (Figures 3 and 4) tests six configurations: aligning the small model's final layer to the target model's early, middle, or late layer (L2E, L2M, L2L), and aligning the small model's middle layer to the same three depths (M2E, M2M, M2L). The results are not subtle — L2E (the LET configuration) dominates all others on both downstream accuracy and perplexity, while L2L and L2M actually show perplexity increases immediately after the alignment phase ends (Figure 4). This means that aligning to late layers of the target model actively damages the training process once the alignment signal is removed.

The paper's interpretation — the "buffer effect" — is that aligning to early layers leaves a large residual capacity (layers 4–24 in the 1.4B model) that can learn to refine and transcend the small model's representations, whereas aligning to later layers forces those layers to match the teacher and leaves insufficient remaining depth for the target model to develop its own, superior representations. This finding has a clear actionable implication: if you must transfer from a weak model to a strong model, transfer as early as possible in the strong model's architecture. It also explains why output-level distillation (RKD), which implicitly constrains all layers by forcing the final output to match the teacher, fails so dramatically — there is no buffer at all.

Part 2: The alignment must stop before the student surpasses the teacher. The comparison between S_{\text{stop}} = 1500 and S_{\text{stop}} = 3000 (Appendix 8, Figure 10) reveals a non-monotonic relationship: longer alignment initially helps (the target model gets more guidance) but eventually hurts (the target model surpasses the teacher and is then pulled backward). This finding is conceptually important because it shows that the transfer window is bounded on both sides: stop too early and the target model does not absorb enough representational structure; stop too late and the target model is constrained from exceeding the teacher.

Together, these two diagnostic findings explain the paper's central empirical claim — "up to 1.6× speedup with nearly 5% improvement" — as an emergent property of getting both the spatial (which layer) and temporal (how long) dimensions of the transfer right. They also explain why RKD fails: it gets both dimensions wrong, constraining all layers (no buffer) throughout all of training (no stopping). SALT (Rawat et al., 2024), which applies KD only during an initial phase, gets the temporal dimension partly right but misses the spatial dimension by operating on output logits rather than early-layer representations — and correspondingly underperforms LET in direct comparison (Table 1).

This diagnostic contribution is significant beyond the specific method because it provides design principles for any future approach that transfers knowledge from a weaker to a stronger model: (1) push the transfer as early in the architecture as possible, and (2) phase it out before the recipient's capabilities exceed the donor's. These principles are not obvious — the standard KD intuition would suggest transferring to late layers (closer to the output) and maintaining guidance as long as possible — and the paper's ablation results provide strong evidence that the opposite is true.

---

Innovation 3: A Theoretical Framing That Connects Alignment Depth to Optimization Landscape Smoothness

While the empirical ablations show that early-layer alignment works better, the paper goes further by providing a theoretical argument (Appendix 13) for why it works better — and this argument is, in itself, a conceptual contribution that distinguishes LET from purely empirical training tricks.

The theoretical analysis considers a simplified deep linear network with L layers, where the alignment loss \mathcal{L}_{\text{proj}} is applied at layer k. The key result is a Frobenius-norm bound on the Hessian of the alignment loss:

$\|H_{\text{proj}}(k)\|_F \leq k \cdot C$

where C is a uniform bound on the Frobenius norms of individual Hessian blocks (assuming all layers share the same structure). This bound grows linearly with k: the earlier the alignment layer, the smaller the maximum possible curvature contributed by the alignment term to the total loss landscape.

What this means conceptually: The additional loss term \mathcal{L}_{\text{proj}} modifies the optimization landscape. If that modification introduces sharp curvature (large Hessian eigenvalues), gradient-based optimization becomes harder — the optimizer must navigate narrow ravines rather than broad basins. The bound says that aligning to earlier layers introduces less additional curvature than aligning to later layers. LET (aligning at layer 3 in a 24-layer model) thus keeps the optimization landscape smoother than L2L (aligning at layer 24), which in turn explains why L2L shows perplexity instability when the alignment term is removed (Figure 4) — the model was optimized in a sharper, less stable region of parameter space.

This theoretical contribution is significant because it connects the paper's empirical design choices to a principled optimization argument, elevating LET from "a trick that works" to "a strategy grounded in the geometry of the loss landscape." It also provides a framework for reasoning about future extensions: any variant of LET that increases the effective k (e.g., aligning multiple layers, or aligning to a deeper layer) should be expected to degrade the optimization landscape and potentially harm final performance, a prediction the paper's ablations confirm.

The analysis is simplified (deep linear networks rather than transformers, uniform block bounds rather than layer-specific analysis), and the paper acknowledges these simplifications. But the value of the analysis is not in its tightness — it is in providing a conceptual language (curvature, Hessian structure, smoothness) for understanding why the spatial dimension of the transfer matters, moving the discussion beyond "we tried different layers and layer 3 worked best."

---

Innovation 4: The Demonstration That Architecture-Agnostic Transfer Enables Reuse Across Heterogeneous Model Families

This innovation sits at the intersection of engineering and methodology. The paper demonstrates — through deliberate architectural heterogeneity in its experiments — that representational transfer from a small model to a large model does not require the two models to share an architecture, a vocabulary, or even similar activation functions. This is a capability demonstration with significant practical implications, even though the mechanism enabling it (interpolation-based dimension matching) is relatively simple.

Table 3 (Appendix 10) documents the diversity of model pairs tested:

• The small models use ReLU (OPT-125M), GeLU (Pythia-160M), and SiLU (SmolLM-135M, SmolLM-1.7B) activations, while the target models use SwiGLU — a fundamentally different activation function that includes a gating mechanism absent from the small models.
• The small models use both full multi-head attention (OPT, Pythia) and Grouped Query Attention (SmolLM), while the target models use full attention.
• Hidden dimensions range from 576 (SmolLM-135M) to 4096 (7B target model), with intermediate dimensions, layer counts, and head counts all varying independently.

Despite this heterogeneity, LET produces consistent gains across all pairings (Figure 8, Table 1). This is not a trivial result. The fact that a SwiGLU-based target model can productively absorb representational structure from a ReLU-based small model suggests that the transferred knowledge is abstract enough to survive architectural translation — it is about the geometry of linguistic features, not about the specific mechanism by which those features were computed.

The innovation here is not the interpolation technique itself (which is standard signal processing) but rather the empirical demonstration that such simple dimension matching suffices to enable cross-family transfer at scale. Prior work on representation alignment (e.g., in model merging or multi-modal alignment) often uses learned projections, assuming that different architectures produce representations in fundamentally incommensurable spaces. LET's success with fixed interpolation suggests that, at least for models trained on similar data distributions (the small models were pretrained on large text corpora, and LET trains on The Pile), the representational spaces are more compatible than previously assumed.

This finding has a direct practical consequence: it means the LET paradigm can be applied using any publicly available small model as the guide, regardless of its architecture, as long as it was trained on text data of reasonable quality. This dramatically expands the pool of usable guide models and eliminates the need for practitioners to engineer architectural compatibility — they can simply download the smallest, best-performing open-source model available and use it to accelerate their custom large-model training.

The failure case with GPT-2 (Appendix 14, Table 6) introduces an important boundary condition: the transfer works across architectures but not across data quality gaps. GPT-2, trained on data up to 2017, produces representations that actively harm the target model's learning when used for alignment. This suggests that architectural heterogeneity is not the bottleneck — representational quality (driven by training data and objectives) is the binding constraint. This insight is not proven but strongly suggested by the data, and it opens a line of inquiry about what makes a small model's representations "transferable."

5. Experimental Analysis

Evaluation Methodology

• Dataset. All language modeling experiments use The Pile dataset (Gao et al., 2020), a large-scale English text corpus of approximately 825 GB sourced from 22 different domains. The paper uses approximately 20 billion tokens for pretraining, with all copyrighted content removed. Language modeling perplexity is evaluated on a held-out test split from The Pile. For downstream task evaluation, nine benchmarks from the OLMo evaluation suite (Groeneveld et al., 2024) are used: HellaSwag, Winogrande, LAMBADA, OpenbookQA, ARC-easy, ARC-challenge, PIQA, SciQ, and BoolQ. An additional cross-domain experiment (Appendix 9) tests time-series classification on datasets including EthanolConcentration, FaceDetection, Handwriting, Heartbeat, JapaneseVowels, PEMS-SF, SelfRegulationSCP1, SelfRegulationSCP2, SpokenArabicDigits, and UWaveGestureLibrary.

• Base model(s). The target models (denoted \mathcal{M}) are LLaMA-architecture Transformers with RMSNorm and SwiGLU activations (Touvron et al., 2023), trained in two primary scales: a 1.4B-parameter model (24 layers, hidden size 2048, intermediate size 5461, 32 attention heads, full multi-head attention) and a 7B-parameter model (32 layers, hidden size 4096, intermediate size 11008, 32 attention heads, full multi-head attention). The small pretrained guide models (denoted \mathcal{T}) are drawn from three model families to demonstrate architectural heterogeneity: SmolLM2 (SmolLM-135M for the 1.4B target, SmolLM-1.7B for the 7B target; SiLU activation, GQA attention), OPT-125M (ReLU activation, full attention), and Pythia-160M (GeLU activation, full attention). For additional 7B-scale experiments, Llama-3.2-1B is also used as \mathcal{T}. All \mathcal{T} models are frozen during LET training. The paper states that PaLM 2-S* is "representative of the capabilities of many contemporary LLMs," but in this work the target models are trained from scratch specifically for the experiments, not borrowed from prior work.

• Metrics. Two primary metrics are used throughout. For downstream task evaluation, accuracy (%) is reported on each of the nine tasks individually, along with the average accuracy across all nine tasks, computed under a one-shot prompting setting using the EleutherAI evaluation harness. For language modeling quality, test perplexity is evaluated on the held-out split of The Pile at regular 500-step intervals (approximately every 1.3 billion tokens). In the main results table (Table 1), the final accuracy at the end of training is reported; in learning curve figures (Figures 1–4, 7–9, 11), accuracy or perplexity is plotted against training tokens consumed.

• Baselines. Four baselines are compared:

  1. Standard causal language modeling (Baseline): The target model trained with only the negative log-likelihood loss \mathcal{L}_{\text{NLL}} from random initialization, with no alignment signal. This is the default pretraining paradigm and the primary point of comparison.
  2. Reverse Knowledge Distillation (RKD): The target model is trained with the standard KD objective (Equation 2) where the smaller pretrained model \mathcal{T} serves as the teacher, and the larger target model \mathcal{M} minimizes cross-entropy between its output distribution and \mathcal{T}'s output distribution. The paper follows the RKD setup from Rawat et al. (2024).
  3. SALT (Rawat et al., 2024): A two-stage training paradigm where KD (using the UL2 objective) is applied for an initial phase controlled by a hyperparameter n_{\text{KD}}, followed by standard training. For fair comparison, the paper sets n_{\text{KD}} = S_{\text{stop}} to match LET's alignment duration. The paper notes that SALT's default hyperparameters may be suboptimal for the standard causal LM protocol used here, and a dedicated hyperparameter search was not conducted due to computational constraints.
  4. Baseline-3B: A 3B-parameter model trained with standard causal language modeling, included specifically in Figure 7 to demonstrate that LET-1.4B can outperform a model more than 2× its size.

• Generation budget / compute accounting. Compute is measured in training tokens (cumulative tokens processed during pretraining) and wall-clock time. For throughput comparisons (Appendix 10, Table 4), the paper reports tokens processed per second on 32 NVIDIA A100 80GB GPUs, along with the ratio of each method's throughput to the baseline throughput. Peak VRAM usage and total wall-clock time are also reported. The 1.4B model processes approximately 2.62 million tokens per training step (batch size 2048 × sequence length 1280). The alignment period (S_{\text{stop}} = 1500 steps) corresponds to approximately 3.9 billion tokens. Total training processes roughly 20 billion tokens, requiring approximately 7,600 total steps. The convergence speedup is measured by comparing the number of tokens (or steps) required to reach a given downstream accuracy or perplexity level — for example, "LET attains higher performance while requiring less than two-thirds of the training steps needed to surpass the baseline" (Table 1). The per-step throughput overhead of LET relative to the baseline is reported as the throughput ratio (Table 4): 0.927× for the 1.4B model, meaning LET trains ~7.3% slower per step during the alignment phase.

• Cross-validation / statistical protocol. The paper does not employ cross-validation for strategy selection in the manner of some test-time compute studies, because the hyperparameters (k=3, \lambda_0=0.1, S_{\text{stop}}=1500) are fixed by separate ablation sweeps rather than selected per-difficulty-bin from the test set. The evaluation protocol is standard: models are trained once with fixed hyperparameters for each configuration, checkpoints are saved at regular intervals (every 500 steps for perplexity evaluation), and downstream task accuracy is evaluated on the held-out test sets of the nine OLMo benchmarks using the EleutherAI evaluation harness with automatic batch size detection. No k-fold splitting of the test set is performed. Training is conducted with BF16 mixed precision on 32 A100 GPUs, and results are reported as single-run outcomes — the paper does not report error bars or standard deviations across multiple training seeds. The evaluation harness uses one-shot prompting for all downstream tasks. For the time-series experiments (Appendix 9), The \mathcal{T} model (TimesNet) was "specifically pre-trained on a subset of these time series datasets" — this creates a potential confound where \mathcal{T} may have seen some of the evaluation data, though the paper does not explicitly discuss this limitation.

---

Main Quantitative Results

LET vs. Baseline on Downstream Task Accuracy (1.4B and 7B scales)

The paper's central downstream performance results are presented in Table 1, which reports one-shot accuracy on each of nine tasks and the average across all tasks, for both 1.4B and 7B target models.

1.4B scale (Table 1, top section): Using SmolLM-135M as \mathcal{T} (approximately 10× smaller):

The average accuracy improvement of LET over the baseline is +2.40 percentage points, which the paper describes as "nearly 5% improvement" in relative terms (2.40 / 46.80 ≈ 5.13%). On individual tasks, LET outperforms the baseline on the majority: HellaSwag (45.65 vs. 43.36), Winogrande (55.64 vs. 52.88), LAMBADA (40.13 vs. 38.62), OpenbookQA (30.20 vs. 28.60), ARC-easy (52.10 vs. 49.07), PIQA (69.42 vs. 67.63), SciQ (73.40 vs. 70.70), and BoolQ (56.00 vs. 54.28). ARC-challenge shows a notable exception where the baseline (29.27) outperforms LET (28.40) — the paper does not discuss this reversal. The "LET (67%)" entry demonstrates that LET trained for only two-thirds of the total steps already exceeds the baseline's final performance (46.94 vs. 46.80), providing the basis for the 1.5× to 1.6× speedup claim. RKD and SALT both underperform the baseline at the 1.4B scale, with RKD showing particularly poor performance on SciQ (56.70 vs. baseline's 70.70 — a 14-point gap) and BoolQ (49.50 vs. 54.28), while performing slightly better on ARC-challenge (29.95 vs. 29.27) and LAMBADA (38.44 vs. 38.62). SALT's performance is broadly similar to RKD's, with an average accuracy of 44.66, also below the baseline.

7B scale (Table 1, bottom section): Using SmolLM-1.7B as \mathcal{T} (approximately 4.1× smaller):

The average improvement at the 7B scale is +1.77 percentage points. LET outperforms the baseline on seven of nine tasks: HellaSwag (56.51 vs. 53.96), Winogrande (60.85 vs. 59.04), LAMBADA (47.57 vs. 44.46), ARC-easy (57.07 vs. 55.47), PIQA (72.47 vs. 71.44), SciQ (79.50 vs. 76.90), and BoolQ (62.96 vs. 61.60). The two exceptions are OpenbookQA (33.60 vs. baseline's 34.20) and ARC-challenge (32.00 vs. 32.40), where LET slightly underperforms. RKD shows catastrophic degradation at the 7B scale, averaging only 47.03% — more than 7 points below the baseline — with particularly severe drops on SciQ (60.90 vs. 76.90) and BoolQ (58.44 vs. 61.60). The paper notes that "the performance degradation of the RKD method is more pronounced on the 7B scale compared to the 1.4B scale," suggesting that the RKD failure mode worsens as the capability gap between student and teacher increases (the 7B student is 4.1× larger than SmolLM-1.7B, while the 1.4B student is 10.4× larger than SmolLM-135M — interestingly, the 7B case has a smaller ratio but worse RKD degradation, implying scale may matter more than the ratio). SALT results at 7B are not reported in Table 1.

LET vs. Baseline on Language Modeling Perplexity

Figure 2 presents test perplexity on The Pile for the 1.4B target model, comparing LET against the baseline under three different vocabulary settings (SmolLM, OPT, and Pythia tokenizers):

• SmolLM vocabulary (Figure 2, left subplot): LET achieves lower perplexity than the baseline throughout training. At approximately 5B tokens, the LET curve is visibly below the baseline curve. At the end of training, LET's perplexity is lower. The exact numerical values at convergence are not stated in the text, but the figure shows a consistent gap.
• OPT vocabulary (Figure 2, middle subplot): Similar pattern — LET achieves lower perplexity, confirming that the benefit is not an artifact of the tokenizer. The gap appears slightly narrower than with SmolLM vocabulary.
• Pythia vocabulary (Figure 2, right subplot): LET again achieves lower perplexity, with a visible and persistent gap.

The key observation across all three vocabulary settings is that LET converges faster: the perplexity curves for LET drop more steeply in early training and maintain their advantage through the end. The paper notes that "different small pretrained models have varying impacts: although their sizes are similar, substantial differences in architecture lead to different learned representations and, consequently, distinct training dynamics in \mathcal{M}." Among the three, "using SmolLM as \mathcal{T} yields the best overall performance" — this aligns with the fact that SmolLM-135M is the most recently trained of the three small models (2025 vs. 2022 for OPT and 2023 for Pythia) and was trained on higher-quality data.

Figure 4 provides a complementary view by plotting perplexity trajectories for the six layer-alignment configurations (L2E, L2M, L2L, M2E, M2M, M2L). L2E (the LET configuration) achieves the lowest final perplexity. Critically, after the alignment phase ends (when \lambda decays to zero at 1,500 steps), non-L2E strategies "show varying degrees of perplexity increase immediately post-alignment" — their perplexity jumps upward when the guidance is removed. L2E maintains stable or continually decreasing perplexity through this transition, which the paper interprets as evidence that L2E's representations integrate more smoothly with the NLL objective.

Convergence Speedup: LET Trains Faster

The convergence speedup is quantified through two comparisons in Table 1:

• 1.4B scale: LET trained for 67% of the total training steps achieves an average downstream accuracy of 46.94%, exceeding the baseline's final accuracy of 46.80% at 100% of steps. This means LET reaches a given performance threshold in approximately two-thirds the training time, yielding the claimed 1.5–1.6× speedup. The exact speedup factor depends on the reference performance level — the paper claims "up to 1.6× speedup" in the abstract and introduction. At the end of full training, LET achieves 49.20% vs. the baseline's 46.80%, a +5.13% relative improvement.

• 7B scale: A similar LET (67%) entry is not explicitly provided in the text for the 7B model; the paper only reports the final LET accuracy (55.83%) and baseline (54.06%). The speedup analysis for the 7B scale relies primarily on learning curves in figures rather than explicit fractional-step comparisons.

Figure 1 provides a visual summary: the bar chart shows LET models consistently above baseline models for both 1.4B and 7B scales in terms of average downstream task accuracy.

Figure 9 (Appendix 8) compares the average downstream performance trajectories of LET, Baseline, and RKD throughout training at both 1.4B and 7B scales. The LET curve is consistently above the baseline curve at both scales from early in training onward. At 7B, RKD's trajectory is dramatically below both LET and baseline, while at 1.4B, RKD hovers around or slightly below the baseline for most of training.

LET Outperforms a Larger Baseline Model

Figure 7 presents a comparison across architectures and sizes: LET applied to a 1.4B model ("LET-1.4B") achieves higher average downstream task accuracy than a standard-trained 3B-parameter model ("Baseline-3B"). Both use SmolLM-135M as \mathcal{T} for the LET model. The exact numerical values are visible in the bar chart: LET-1.4B reaches approximately 49.2% accuracy (matching Table 1), while Baseline-3B reaches approximately 48.3%. This result is significant because it shows LET enabling a model with less than half the parameters to outperform a baseline model trained with 2.14× more parameters — a concrete demonstration that representational transfer can substitute for raw parameter count to some degree. The paper states this "indicates that the proposed LET paradigm enables the model to learn more effectively from limited training data."

Cross-Domain Generalization: Time-Series Classification

Appendix 9 (Table 2) demonstrates LET's applicability beyond language modeling. The setup uses Qwen-0.5B as the target model, fine-tuned on various time-series classification tasks, with TimesNet as the small pretrained \mathcal{T} model. The results show LET outperforming the baseline on all ten time-series datasets, with the margin varying by dataset. For example, on SpokenArabicDigits, LET achieves 98.89% vs. the baseline's 97.84%; on UWaveGestureLibrary, 92.65% vs. 91.39%; on EthanolConcentration, 49.05% vs. 31.43% — a substantial 17.6-point improvement. The paper uses these results to argue that "the applicability of LET extends beyond LLMs," though the evaluation is limited to classification fine-tuning rather than full pretraining, and the \mathcal{T} model was "specifically pre-trained on a subset of these time series datasets," which may inflate the apparent benefit.

Throughput and Resource Comparison

Table 4 (Appendix 10) provides detailed training efficiency metrics. For the 1.4B target model during the alignment phase (first 1,500 steps):

For the 7B target model:

Key observations: (1) LET has a modest throughput overhead of 7.3% (1.4B) and 4.0% (7B) during the alignment phase, due to the forward pass through the frozen \mathcal{T} and the alignment loss computation. After S_{\text{stop}}, LET operates at baseline throughput. (2) LET achieves notably lower peak VRAM than RKD and SALT because it aligns hidden states (dimensionality 576–4096) rather than logit distributions (dimensionality ~50K–100K). This gives LET an advantage for training with large batch sizes and makes it more memory-efficient than alternative methods that use auxiliary models. (3) When scaling from a 135M \mathcal{T} (for the 1.4B target) to a 1.7B \mathcal{T} (for the 7B target), the throughput overhead only increases slightly (0.927× → 0.960×), demonstrating that the method scales well even when the guide model grows.

---

Ablation Studies and Robustness Checks

Layer-alignment strategy (six configurations): The paper tests L2E, L2M, L2L, M2E, M2M, and M2L on the 1.4B model (Figures 3 and 4). L2E achieves the highest average downstream accuracy and the lowest final test perplexity. Using the middle layer of \mathcal{T} (M2E, M2M, M2L) consistently underperforms using the final layer (L2E, L2M, L2L), confirming that late-layer representations from the small model are more useful than intermediate ones. Among configurations using \mathcal{T}'s final layer, aligning to later layers of \mathcal{M} (L2M, L2L) shows a perplexity increase immediately after the alignment phase ends, while L2E maintains stable perplexity — a robustness advantage attributed to the buffer effect of subsequent layers. Figure 4 qualitatively shows that both M2E and L2E "maintain robust performance throughout training," with L2E yielding the lowest final perplexity.

Alignment weight λ₀ (sweep over 0.01, 0.1, 0.3, 1.0, 3.0): Figure 5 (left) shows average downstream accuracy for the 1.4B model. The baseline achieves approximately 46.8%. LET with λ = 0.1 achieves approximately 49.2% (best). λ = 0.01 achieves slightly above baseline but below λ = 0.1. λ = 0.3 and λ = 1.0 drop below baseline, and λ = 3.0 drops substantially below baseline — the model is over-constrained by the teacher. Figure 6 shows the corresponding cosine similarity between \mathcal{M}'s early-layer and \mathcal{T}'s final-layer representations: higher λ values produce higher similarity (stronger alignment), but performance peaks at a moderate similarity level. The paper concludes that λ = 0.1 "achieves an optimal balance between aligning with \mathcal{T} and acquiring new knowledge from data."

Stopping step S_stop (1,500 vs. 3,000): Figure 10 compares average downstream accuracy trajectories. At the 5B-token mark, S_stop = 3,000 yields higher accuracy than S_stop = 1,500 — the longer alignment window provides more guidance early on. However, at the end of training, S_stop = 1,500 achieves higher final accuracy. The paper attributes this to the target model eventually surpassing the teacher: "continued alignment with a much smaller teacher model can actually hinder further improvement." S_stop = 1,500 is selected as the default because it yields better final performance while also reducing overall training time (since the forward pass through \mathcal{T} is eliminated earlier). The paper notes this as a preliminary analysis and does not perform a fine-grained sweep over S_stop values (e.g., 1,000, 2,000, 2,500).

Choice of small model \mathcal{T} (multiple families): Figure 8 (Appendix 8) compares LET using SmolLM-135M, OPT-125M, and Pythia-160M as \mathcal{T} for the 1.4B target model. All three LET variants outperform the baseline, though with different magnitudes. SmolLM-135M yields the largest improvement, followed by Pythia-160M, then OPT-125M. The paper attributes these differences to the varying quality and recency of the small models' training data and objectives, but does not conduct controlled experiments to isolate which factors (data quality, data size, architecture, training objective, tokenizer) drive the differences. Figure 2 further confirms that LET with all three \mathcal{T} choices achieves lower test perplexity than the baseline across all three vocabulary settings.

Choice of early-layer index k (layer 1, 2, 3, or 5): Figure 12 (Appendix 14) evaluates four layer pairings for the 1.4B model with SmolLM-135M as \mathcal{T}, all aligning \mathcal{T}'s final layer to different early layers of \mathcal{M}. L1-F3 (layer 3) achieves the best performance. L1-F1 (layer 1) underperforms, which the paper attributes to layer 1 "primarily encoding input-specific information" (embedding-level features). L1-F5 (layer 5) also underperforms compared to L1-F3, suggesting that while layer 5 is still "early" in a 24-layer model, it is deep enough that the buffer above it (19 layers) is slightly insufficient compared to the buffer above layer 3 (21 layers). L1-F2's performance falls between L1-F1 and L1-F3. The paper concludes that layer 3 "strikes an optimal balance for representation alignment."

GPT-2 as \mathcal{T} (failure mode): Table 6 (Appendix 14) tests LET with GPT-2 Small and GPT-2 Medium as \mathcal{T}, alongside RKD with GPT-2 Small. All GPT-2-based configurations underperform the baseline. LET-GPT2-Small achieves 45.91% average accuracy vs. 46.80% for the baseline; LET-GPT2-Medium achieves 46.1x%; RKD with GPT-2 Small achieves 43.08%. The paper attributes this degradation to "the potentially lower quality of GPT-2's training data (with a cutoff of late 2017) compared to modern language models." This is presented as evidence that representational quality (driven by training data recency and quality) can override the architectural advantages of LET. However, this is a correlational rather than causal finding — there is no controlled experiment varying only data quality while holding architecture constant.

Logsum loss vs. cosine similarity: Table 5 (Appendix 12) reports that LET-LogSum (LET with logsum loss as the alignment objective) outperforms LET with cosine similarity (the default) on average one-shot downstream accuracy. Both substantially outperform the baseline, RKD, and SALT. The paper also mentions "LET-CCA" (Canonical Correlation Analysis) as another alternative alignment objective but does not report detailed results. The logsum loss variant is motivated by prior work (Miles et al., 2024) showing it is effective when models have significant capacity gaps, and its mechanism is described as "emphasizing regions where representations between \mathcal{T} and \mathcal{M} diverge significantly."

Vocabulary and tokenizer mismatch (Appendix 8, Figure 2): The three subplots in Figure 2 confirm LET's effectiveness across three different vocabulary choices. Despite different tokenization schemes producing different token-level representations, LET consistently lowers perplexity relative to the baseline.

Architectural heterogeneity (Table 3): The paper highlights that LET works across substantial architectural differences — different activation functions (ReLU, GeLU, SiLU, SwiGLU), different attention mechanisms (full multi-head, GQA), different hidden dimensions (576–4096), and different layer counts (12–32). This is presented as a robustness check, though it is an inherent property of the method design (interpolation-based dimension matching) rather than an empirical test per se.

SALT comparison: Table 1 includes SALT under identical hyperparameter configurations (with n_{\text{KD}} = S_{\text{stop}}). LET achieves 49.20% average accuracy vs. SALT's 44.66% for the 1.4B model — a 4.54-point advantage. The paper notes that SALT "optimizes a different training objective from ours (UL2)" and its "default hyperparameters may therefore be suboptimal for our training protocol," acknowledging that this comparison may underestimate SALT's potential if properly tuned for the standard causal LM setting.

---

Critical Assessment

The paper makes four major empirical claims that warrant close scrutiny against the experimental evidence.

Claim 1: LET achieves up to a 1.6× speedup in convergence. This claim is supported at the 1.4B scale by Table 1, which shows LET at 67% of training steps surpassing the baseline at 100% of steps (46.94% vs. 46.80% average accuracy). The 1.6× figure follows directly from this ratio. What the experiments demonstrate is narrower than "up to 1.6× speedup" in a broad sense: it is specifically a 1.5× speedup (100/67 ≈ 1.5) measured at a particular performance threshold on a single model scale with a single small model (SmolLM-135M). The paper rounds this to "up to 1.6×" — the precise factor depends on which performance threshold is chosen and is not consistently 1.6× across all thresholds. At the 7B scale, a comparable fractional-step comparison is missing from the paper — the claim of acceleration at 7B relies on learning curves in figures rather than an explicit "LET at X% steps matches baseline at 100% steps" result. The speedup calculation also does not account for the per-step throughput overhead during the alignment phase: LET processes tokens at 0.927× the baseline rate for the first 20% of training (at the 1.4B scale). A proper wall-clock speedup would need to factor this in. The paper addresses this by noting that the 1.6× convergence speedup "more than compensates for the modest reduction in throughput," but the exact wall-clock speedup is smaller than 1.6× — roughly 1.5× × 0.95 ≈ 1.4× after accounting for alignment-phase overhead, though the paper does not compute this explicitly.

Claim 2: LET improves downstream task accuracy by nearly 5%. At the 1.4B scale, LET achieves 49.20% vs. 46.80% — a +2.40 percentage point absolute improvement, or +5.13% relative improvement. The "nearly 5%" phrasing refers to relative improvement, not absolute points. At the 7B scale, the absolute improvement is +1.77 points (55.83% vs. 54.06%), or +3.27% relative. The improvement is consistent but varies in magnitude. The claim is supported, with the caveat that it represents an average over nine tasks — individual tasks show variation (ARC-challenge degrades slightly at both scales, OpenbookQA degrades at 7B), and the reported average does not weight tasks by importance or difficulty.

Claim 3: LET works with models up to 10× smaller than the target. This is directly supported: the primary 1.4B experiments use SmolLM-135M (140M / 1400M = 0.1, i.e., 10× smaller). Additional experiments with OPT-125M (~11× smaller) and Pythia-160M (~8.75× smaller) also show benefits. The 7B experiments use SmolLM-1.7B (~4.1× smaller) and Llama-3.2-1B (~7× smaller). The full 10× ratio is only demonstrated at the 1.4B scale; at 7B, the largest demonstrated ratio is 7×. Whether LET would work with a 70B target model and a 7B \mathcal{T} (also 10×) is not tested. The paper's claim that LET works "regardless of LLM architectures" is supported by the diversity of \mathcal{T} architectures in Table 3, but "regardless" is a strong word — only Transformer-based autoregressive LMs were tested, not SSMs, not encoder-decoder models, not non-autoregressive models.

Claim 4: LET achieves both faster training and superior downstream performance. This is the paper's headline claim and is supported at the aggregate level: LET reaches higher final accuracy and reaches baseline-level performance in fewer steps. However, three qualifications deserve attention:

1. Single training run. All results report single-run outcomes. No error bars, standard deviations, or multi-seed experiments are provided. LLM pretraining exhibits non-trivial run-to-run variance, especially on downstream task accuracy at the 1–2 percentage-point scale. The +2.40 point improvement at 1.4B is large enough that it likely exceeds single-run variance, but the +1.77 point improvement at 7B is smaller and would benefit from statistical validation. The paper's omission of variance estimates is a genuine weakness.

2. Small test set for downstream tasks. The nine OLMo tasks represent specific reasoning and knowledge domains. Extrapolating from these nine tasks to "improved generalization" is reasonable but limited. The paper does not test on broader LLM evaluation suites (e.g., MMLU, BIG-Bench, HumanEval) that would provide more comprehensive coverage of model capabilities. The time-series experiments (Appendix 9) are a welcome cross-domain test but use a different training paradigm (fine-tuning rather than pretraining) and a \mathcal{T} model that was pre-trained on a subset of the evaluation data.

3. RKD and SALT may not be optimally tuned. The paper acknowledges that SALT's default hyperparameters (optimized for UL2) may be suboptimal for the standard causal LM setup. Similarly, RKD is used with its standard configuration without a dedicated hyperparameter search for the large-teacher-small-student regime. This is a reasonable experimental choice given computational constraints, but it means the RKD and SALT comparisons should be understood as comparisons against "out-of-the-box" implementations rather than optimally tuned competitors. The degradation of RKD at the 7B scale (47.03% vs. 54.06% baseline, a 7-point gap) is dramatic enough that it likely reflects a real failure mode rather than poor hyperparameter choices, but the SALT comparison at the 1.4B scale (44.66% vs. 49.20% for LET) could potentially narrow with better tuning.

Missing experiments that would strengthen the paper:

• Training from different random seeds to establish variance and statistical significance of the observed improvements.
• A broader evaluation suite (MMLU, GSM8K, HumanEval) to test whether the downstream improvements generalize beyond the OLMo task set.
• Experiments at larger scale (13B, 30B, 70B parameters) to validate the scalability claims, which the paper explicitly defers to future work (Section 4).
• A direct test of the "buffer effect" mechanism — for instance, freezing the early layers after alignment and training only the later layers, or comparing LET to a variant where the alignment target is the same but applied to a different proportion of layers.
• Ablation of the data quality hypothesis — controlled experiments where the same small model architecture is trained on data of varying quality, to isolate data quality effects from architectural effects. The GPT-2 failure (Appendix 14) is attributed to data quality, but there is no controlled experiment confirming this attribution.
• Wall-clock speedup calculation that explicitly incorporates the alignment-phase throughput overhead, rather than stating convergence speedup and throughput overhead separately.
• Comparison against a "warm start" baseline where the target model's early layers are initialized from the small model's layers (where dimensions match) before training — this would test whether the benefit of LET is specific to the ongoing alignment + scheduling mechanism or could be achieved by a simpler initialization strategy.

Conditions under which the claims hold: Based on the presented evidence, LET's benefits are demonstrated under these specific conditions: (1) the small \mathcal{T} model was pretrained on reasonably high-quality text data (the failure with GPT-2 suggests data quality matters); (2) the target model is at least 4× larger than \mathcal{T} (though the exact lower bound is not established); (3) training uses a standard causal LM objective with AdamW and cosine scheduling; (4) the alignment is applied to an early layer (specifically layer 3 in 24–32 layer models, though the optimal layer likely scales with model depth); (5) the alignment weight is moderate (λ₀ = 0.1) and decays to zero by roughly 20% through training. The extent to which these conditions generalize — to different optimizers, learning rate schedules, model architectures outside the Transformer family, or training objectives beyond causal LM — is not tested and remains unknown.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Not Accounted For in the Headline Speedup

The assumption or constraint. The paper's central claim of "up to 1.6× speedup" measures convergence speedup purely in terms of training steps (or tokens) to reach a given performance level — the cost of difficulty estimation is not included in this calculation. The speedup metric assumes difficulty is known ahead of time at no cost. The paper's method for predicting difficulty — generating 2,048 samples per question and averaging the PRM's final-answer scores — is itself extremely expensive. As the paper acknowledges in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity."

The consequence. In a realistic deployment, the total cost would be the cost of difficulty estimation plus the cost of strategy execution. Generating 2,048 samples per question for difficulty estimation alone consumes more compute than the largest test-time budgets the paper studies (256–512 generations). If difficulty estimation consumes, say, 2,048 generations and the compute-optimal strategy then uses 64 generations, the total is 2,112 generations — versus 256 generations for a uniform best-of-N baseline that achieves similar accuracy. The 1.6× speedup is computed after difficulty is known and simply ignores the cost of learning it. This means the headline efficiency gains are an upper bound on achievable efficiency, not a realized deployment gain.

What evidence exists in the paper. The method for difficulty estimation is described in Section 3.2, where the paper states explicitly that it requires generating 2,048 samples and scoring them with the PRM. The predicted-difficulty variant (using PRM scores rather than ground-truth labels) removes the need for oracle access but does not reduce the computational cost. The paper acknowledges the issue in the same section:

"this still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

However, no experiment measures the amortized cost — there is no comparison where total cost (estimation + execution) is matched between compute-optimal and best-of-N. The $4\times$ efficiency gain reported in Section 5 (e.g., "16 generations matching 64") is a partial-cost comparison.

Mitigation status. The paper flags this as future work (Section 3.2 and Section 8), suggesting training a model to predict difficulty directly from question text without requiring 2,048 samples. However, no such model is developed or evaluated. Until this gap is closed, the efficiency numbers should be treated as conditional on difficulty being known essentially for free — a condition that does not currently hold in practice.

---

All Results Are on a Single Benchmark with a Single Model Family

The assumption or constraint. Every experiment — search against PRMs, iterative revisions, FLOPs-matched comparisons, compute-optimal policy selection — uses the MATH benchmark (500 test questions of high-school competition-level math) and the PaLM 2-S* model family. The paper states in Section 4:

"We believe this model is representative of the capabilities of many contemporary LLMs"

but provides no evidence for this claim beyond the experiments themselves.

The consequence. This limitation creates uncertainty about whether the paper's findings generalize to other:

• Reasoning domains: Code generation (HumanEval, MBPP), logical reasoning (FOLIO, ProofNet), scientific reasoning (GPQA), or multi-hop question answering. Each domain has different error patterns, different base-model pass@1 rates at each difficulty level, and different verifier training characteristics — all of which could shift the compute-optimal policy.
• Task modalities: Open-ended generation, dialogue, summarization, or planning tasks lack the clean ground-truth answer structure that MATH provides. The PRM training pipeline relies on Monte Carlo rollouts checking whether the final answer matches the known correct answer — an approach that does not extend to tasks where correctness is ambiguous or multi-dimensional.
• Model families: The PRM's quality and over-optimization behavior depend on PaLM 2-S*'s output distribution. A model with different calibration properties or different types of reasoning errors might exhibit qualitatively different difficulty-dependent scaling curves. The revision model's ability to learn from incorrect in-context examples depends on the base model's in-context learning and self-correction abilities, which are known to vary substantially across model families.

What evidence exists in the paper. None — the paper contains no experiments on any benchmark other than MATH, and no experiments on any model family other than PaLM 2. The authors' belief that the model is "representative" is stated but untested.

Mitigation status. The paper does not address this limitation through experiments. The discussion section (Section 4) does not mention it. Future work would need to replicate the study on at least one additional reasoning benchmark and at least one additional model family to establish whether the difficulty-dependent scaling patterns (beam search hurting on easy problems, revisions helping on easy problems, no method helping on hard problems) are robust findings or MATH-specific artifacts.

---

Revision Training Relies on a Fragile Data Construction Pipeline with a 38% Reversion Problem

The assumption or constraint. The revision model is trained on offline-constructed trajectories where incorrect answers are paired with correct answers, using edit distance as a proxy for trajectory coherence. This departs from the on-policy multi-turn rollouts used in Qu et al. (2024). The paper notes in Section 6.1:

"The original approach used on-policy multi-turn rollouts... This was computationally infeasible for the authors, so they approximated the multi-turn structure by pairing independently sampled correct and incorrect solutions post-hoc."

Additionally, because the training trajectories contain only incorrect-to-correct transitions, the model never learns what to do when the current answer is already correct. Section 6.1 reports that:

"approximately 38% of correct answers get converted back to incorrect ones"

during revision chains.

The consequence. The revision model suffers from a fundamental asymmetry: it can improve incorrect answers but it actively degrades correct ones roughly 38% of the time. This creates a reliability problem — even when the model produces a correct answer early in a revision chain, there is a substantial probability it will "revise" it into an incorrect answer at a subsequent step. The paper mitigates this with within-chain selection (majority voting or verifier), but this is a patch rather than a fix: the verifier must correctly identify which revision in the chain is correct, and if the verifier fails (which is more likely on harder problems where PRM over-optimization also occurs), the system may select the wrong answer. The quasi-offline data construction (using edit distance) also introduces a mismatch between the training distribution and the test-time distribution, since at test time the model generates its own previous answers rather than receiving independently sampled ones.

What evidence exists in the paper. The 38% reversion rate is reported in Section 6.1. Figure 6 (left) shows that pass@1 improves gradually through the revision chain, indicating that the model does learn to improve many incorrect answers, but the net effect includes both improvements and reversions. Appendix K (Figure 16) provides further evidence of fragility: the ReST^EM-trained revision model shows "substantially hurt" performance with sequential revisions, to the point where fully sequential performance drops well below the optimal balanced ratio. The paper hypothesizes that "on-policy data collection in ReST^EM exacerbates spurious correlations in revision data, causing the model to fail to learn the revision task properly." The sensitivity to the exact data generation procedure suggests the revision approach is not robust to training methodology changes.

Mitigation status. The paper partially mitigates the reversion problem through within-chain selection (Section 6.1) but does not solve it. The authors do not explore training the revision model on trajectories that include correct-to-correct transitions (teaching the model when to stop revising) or on trajectories with both improvements and no-ops. This is not flagged as explicit future work in Section 8, but the 38% reversion rate clearly represents a practical barrier to deploying revision models in applications where correctness must be maintained.

---

Hard Problems Receive Essentially No Benefit from Any Form of Test-Time Compute

The assumption or constraint. The effectiveness of test-time compute — whether search, revisions, or their compute-optimal combination — depends on the base model's ability to produce correct solutions at some non-trivial rate. The paper defines difficulty bins by pass@1 rate (Section 3.2), and difficulty bin 5 corresponds to problems where the base model's pass@1 is near zero.

The consequence. Across every method and every budget level, bin 5 accuracy hovers at 1–3% — essentially chance performance. In the search results (Figure 3, right), bin 5 accuracy is flat across all budgets (4 to 256 generations) for both beam search and best-of-N. In the revision results (Figure 7, right), bin 5 accuracy stays at roughly 2–3% regardless of the sequential-to-parallel ratio. In the FLOPs-matched comparison (Figure 9), the bin 5 scaling line is essentially flat near 0–5% for all $R$ values, and test-time compute is strongly outperformed by the $14\times$ larger model (e.g., −52.9% relative disadvantage for PRM search at $R \gg 1$). This establishes a hard boundary: test-time compute amplifies existing capability but does not create it. If the base model cannot produce correct solutions at a meaningful rate, no budget of search or revision will help — there are no correct solutions in the proposal distribution to find or refine.

What evidence exists in the paper. The flat bin 5 curves appear consistently across Figures 3 (right), 7 (right), and 9. The paper is explicit about this in Section 7:

"On the hardest problems (bin 5), test-time compute provides essentially zero benefit regardless of budget, meaning that some capabilities can only be acquired through pretraining, not recovered at inference time."

Mitigation status. The paper does not attempt to solve this limitation. The authors present it as a finding rather than a failure — an empirical boundary condition that delineates when test-time compute scaling is viable (easy-to-medium problems) and when it is not (hard problems outside the base model's capability). The implication is clear: for hard problems, pretraining remains the only viable path. This is not a weakness of the method per se, but it is a fundamental constraint on the scope of problems the method can address. Practitioners must assess whether their problem distribution falls within the base model's capability range before investing in test-time compute strategies.

---

The Revision Model and PRM Search Are Studied Independently but Never Combined

The assumption or constraint. The paper treats search against PRM verifiers (Section 5) and iterative revisions (Section 6) as separate, independent axes of test-time compute. Each is studied in isolation, and the compute-optimal policy selects between search strategies or between sequential/parallel revision ratios, but never combines the two. Section 8 explicitly acknowledges:

"we did not experiment with PRM tree-search techniques in combination with revisions"

The consequence. The revision model produces a better proposal distribution — its initial and revised answers are higher-quality than the base model's independent samples. The PRM provides a mechanism to select among candidates and to guide search. These two mechanisms have complementary strengths: revisions excel at local refinement (improving answers that are already close to correct), while PRM search excels at global exploration (finding correct solution paths across diverse approaches). Combining the two — for example, using the revision model as the proposal distribution within beam search, or using the PRM to decide when to accept a revision versus restart from scratch — could yield performance beyond what either mechanism achieves alone. The quoted results in Section 6 for the revision model use an outcome reward model (ORM) trained specifically on revision model outputs, but this ORM only scores final answers; it does not guide the revision process step-by-step the way the PRM guides beam search. Figure 15a in Appendix J confirms that the base model's PRM does not transfer well to revision model outputs, but this distribution shift could potentially be addressed by training a PRM on revision model outputs — an experiment the paper does not run.

What evidence exists in the paper. The current results show that revisions and search independently improve over best-of-N, and that their benefits are complementary across difficulty levels (revisions help most on easy problems, search helps most on medium problems). The fact that the paper documents this complementarity but never combines the methods means the reported gains represent a lower bound on what a fully integrated approach could achieve. The paper provides no evidence for or against the hypothesis that combining them would be additive or synergistic.

Mitigation status. The paper explicitly identifies this as future work in Section 8. The reason given is scope — studying the combination would add substantial complexity to an already multi-faceted experimental design. The limitation is thus acknowledged and deferred, but it leaves a significant gap between what the paper demonstrates and what a practitioner seeking maximum test-time compute efficiency would actually deploy.

7. Implications and Future Directions

How This Work Changes the Landscape

LET does not introduce a fundamentally new optimization algorithm or a novel architectural component — it repurposes an existing resource (frozen small pretrained models) through an unconventional transfer mechanism (early-layer, early-step representational alignment). Yet this repurposing shifts the mental model the field uses for thinking about pretrained model assets, and that shift has real downstream consequences for how research and engineering resources are allocated.

What changes: from "teachers must be larger" to "any trained model is a potential scaffold." The paper's most disruptive implication is that it breaks the knowledge distillation assumption that the guide model must be more capable than the learner. Standard KD, RKD, SALT, and model growth all operate within a framework where the teacher's competence sets a ceiling on the student's potential — either the student compresses the teacher's knowledge (KD), or the student inherits and expands the teacher's parameters (model growth). LET's demonstration that a 10× larger model can productively absorb representational structure from a 10× smaller model — and then surpass it by a significant margin — means that guide models do not need to be performance ceilings. They can be scaffolds that provide useful geometric structure to the early optimization landscape, even when the learner will ultimately develop far richer representations.

This is not a paradigm shift in the sense of overturning a core theoretical framework — the paper's contribution is empirical and methodological, not theoretical. But it is a genuine reframing of the relationship between small pretrained models and large-model training. Previously, small pretrained models were viewed as:

• Baselines to beat (the standard paper evaluation section).
• Fine-tuning starting points (Alpaca-style instruction tuning).
• Distillation targets (compress into a smaller model).

LET adds a fourth category: training accelerators for larger models. This reframing matters because the community has collectively invested millions of GPU-hours in pretraining models like Pythia, OPT, SmolLM, and Qwen at various scales. Under the old mental model, that investment was sunk — each new large model had to be trained from scratch, and the existing small models were irrelevant to that process. Under LET's reframing, those small models become reusable capital assets that can accelerate the development of the next generation of larger models. The economic implication is substantial: if a 135M model can accelerate 1.4B-model training by 1.5× with a +5% accuracy improvement, then the community's existing inventory of small models represents a form of pretraining infrastructure that reduces the marginal cost of developing each new model generation.

Reconciling prior contradictions. The paper resolves a tension in the literature between two apparently contradictory findings:

1. Standard KD degrades when the teacher is smaller than the student — Lee et al. (2023) showed this for small scale (67M models), and the paper confirms it dramatically at 1.4B and 7B scales with RKD, where performance drops below the baseline (Table 1).
2. Weak-to-strong generalization (Burns et al., 2023) shows that in fine-tuning, a strong model can surpass a weak supervisor.

These findings appear contradictory because KD is a weak-to-strong transfer of sorts (small teacher → large student) yet it fails, while Burns et al.'s weak-to-strong supervision works. LET reveals the resolution: the distinction is what is transferred, not whether the transfer is weak-to-strong. KD transfers output behavior (logit distributions), which inherently caps the student at the teacher's competence level. Burns-style weak-to-strong transfers supervision labels, which the strong model can learn to correct using its own internal knowledge (it already has strong representations from pretraining). LET operates differently from both: it transfers internal representational structure from a weak model, using it as a geometric scaffold for the strong model's early learning — a third mechanism that neither prior framework captured. This resolution suggests that the weak-to-strong transfer literature has been asking the wrong question ("can weak models help strong models?") when it should be asking "what form of knowledge, transferred at what architectural depth and for how long, enables a weak model to accelerate a strong model's development?" LET provides a specific, well-documented answer (representational geometry, early layers, early steps) that can serve as a baseline for future investigation.

Which research directions become more attractive. The paper makes several lines of inquiry newly tractable or newly interesting:

• Representational quality as a measurable property. The failure case with GPT-2 (Appendix 14, Table 6) shows that not all small pretrained models transfer equally well — GPT-2's representations actively harm the target model, likely due to outdated training data. This opens the question of what makes a small model's representations "transferable." If we can characterize this property, we can select optimal guide models from the open-source ecosystem without trial-and-error pretraining runs. This connects to ongoing work on representational similarity metrics (CKA, SVCCA, PWCCA) but adds a practical, economically-motivated frame: can we predict transfer quality from the guide model's architecture, training data, or test metrics alone?

• Joint optimization of transfer depth and training budget. The paper identifies k (alignment layer) and S_{\text{stop}} (alignment duration) as the two critical hyperparameters, and shows they interact in non-obvious ways (earlier layers tolerate longer alignment; later layers require earlier stopping). This suggests a principled optimization problem: given a guide model \mathcal{T} and a target model \mathcal{M}, what is the Pareto-optimal front of (layer, duration) pairs for a given training budget? The theoretical analysis in Appendix 13 — which bounds the Hessian curvature as a function of k — provides a starting point for a more formal treatment of this tradeoff.

• Learning dynamics analysis of guided pretraining. The paper includes a theoretical argument (Section 13) connecting alignment depth to optimization landscape smoothness via the Frobenius norm of the Hessian. This is a deep linear network analysis, but it points toward a richer research program: understanding how guided pretraining modifies the loss landscape geometry (not just the loss value) of the target model, and how this modification affects convergence rates, minima sharpness, and generalization. Recent work on the role of sharpness and curvature in generalization (Keskar et al., 2017; Jastrzębski et al., 2017) has not been applied to weak-to-strong transfer, and LET provides a clean experimental setup for doing so.

Which research directions become less attractive. The paper's results also reduce the appeal of certain approaches:

• Naive reverse knowledge distillation — training a larger model to match a smaller model's output distributions throughout training — is empirically dead at the scales tested. The 7B-scale RKD result (47.03% vs. 54.06% baseline, Table 1) is catastrophic enough that RKD cannot be recommended as a general-purpose pretraining acceleration strategy. This does not preclude RKD from working in specialized settings (e.g., when the teacher is only slightly smaller, or when combined with other regularization), but the paper makes clear that RKD alone is not a viable path.

• Complex model growth strategies that require architectural compatibility and careful expansion schedules become less compelling when LET achieves comparable or better speedups with no architectural constraints and no engineering effort. Model growth retains advantages for cases where the target architecture must be a strict superset of the guide model's architecture (e.g., for compatibility with existing fine-tuned checkpoints), but for new training runs with architectural freedom, LET's simplicity and generality make it the more attractive default.

• Training from random initialization without any external guidance — the paper's baseline — now carries a clear opportunity cost. If a small pretrained model is available (and in 2025, dozens are available for download), training from scratch without any alignment signal means leaving a demonstrated 1.5× speedup and ~5% accuracy improvement on the table. The paper does not claim LET is always beneficial (the GPT-2 failure shows it is not), but it establishes that not considering LET is a decision that should be justified, not assumed.

The paper's most important long-term effect may be to shift the default for LLM pretraining: rather than "train from scratch because we want a custom architecture," the new default may become "download the best small open-source model, use it to accelerate early training with LET, and keep our custom architecture." This would change how the community allocates compute between exploring new architectures and amortizing existing ones — a shift that the paper's evidence suggests is overdue.

---

Follow-Up Research This Work Enables

Direct measurement of the "buffer effect" through progressive freezing experiments. The paper attributes LET's success to the buffer effect — aligning only early layers leaves later layers free to refine and transcend the guide model's representations. This mechanism is hypothesized from the layer-pairing ablation results (Figures 3 and 4) but never directly tested. A clean follow-up would train three variants of the 1.4B model: (1) standard LET (guide alignment at layer 3, all layers trainable), (2) frozen early layers after alignment (layers 1–3 frozen at their post-alignment weights, only layers 4–24 trained on NLL), and (3) trainable early layers with random re-initialization of later layers post-alignment (to test whether the benefit comes from the early-layer representations themselves or from a favorable initialization of later layers). If variant (2) matches LET's final performance, the buffer effect is confirmed as a representational-structure phenomenon (the early layers learn reusable features and the later layers do not need to be updated for them to be useful). If variant (2) underperforms LET but outperforms the baseline, the effect is partly structural and partly a favorable optimization trajectory for later layers. If variant (3) underperforms LET, the benefit is not just a better initialization for later layers but requires ongoing joint optimization. This experiment would distinguish between the paper's interpretation and alternative explanations (e.g., that LET is simply a form of curriculum learning where early layers see an easier objective).

Scaling law characterization for guide model size, target model size, and training budget. The paper demonstrates LET at specific points (135M → 1.4B, 1.7B → 7B) but does not provide a scaling characterization. A natural follow-up would train a series of target models (e.g., 300M, 600M, 1.4B, 3B, 7B) with a fixed guide model (e.g., SmolLM-135M) and measure the per-token convergence speedup as a function of the target-to-guide parameter ratio. This would answer: does LET's benefit follow a power law (like pretraining scaling laws), is there an optimal target-to-guide ratio, and does the benefit asymptote or reverse when the target model becomes extremely large relative to the guide? A complementary experiment would fix the target model at 1.4B and vary the guide model size (e.g., 135M, 360M, 1.7B) to measure whether a larger (better) guide model always helps, or whether there is a point where the guide is "good enough" and further improvements provide diminishing returns. Together, these experiments would produce scaling laws for LET-guided pretraining analogous to the Chinchilla laws for pretraining data and model size — enabling practitioners to predict the speedup they can expect for a given guide-target pair without running the full experiment. The paper's computational budget of 32 A100 GPUs and ~20B tokens per run makes this feasible as a 3–6 month research project for a moderately-resourced academic lab.

Ablation of the data-quality hypothesis through controlled pretraining of guide models. The GPT-2 failure (Appendix 14, Table 6) is attributed to data quality, but the attribution is correlational — GPT-2 was trained on older data, and GPT-2 fails as a guide, but there is no controlled experiment isolating data quality from other confounds (GPT-2 also has a different architecture, was trained with a different objective, and was trained on different data volume). A clean experiment would take a single architecture (e.g., a 135M LLaMA-style model) and pretrain three copies from scratch on: (1) The Pile (high-quality, contemporary data), (2) a filtered version of The Pile omitting documents after 2017 (mimicking GPT-2's temporal data cutoff), and (3) a version of The Pile with 50% random document corruption (testing robustness to noise). Each copy would then be used as \mathcal{T} to guide LET training of an identical 1.4B target model. If the Pile-trained guide substantially outperforms the 2017-cutoff guide, data recency is confirmed as a causal factor. If the 50%-corrupted guide still provides some benefit, the transfer mechanism is robust to noise in the guide's representations — an important practical finding since open-source models of varying quality are common. This experiment would transform the GPT-2 observation from a cautionary anecdote into a principled understanding of when LET fails.

Evaluation of LET on non-autoregressive architectures and non-language domains. The paper demonstrates LET on autoregressive Transformer language models (LLaMA architecture) and includes one cross-domain test on time-series classification (Appendix 9). However, the cross-domain test uses fine-tuning rather than full pretraining, and the \mathcal{T} model (TimesNet) was pretrained on a subset of the evaluation data — a confound that limits the strength of the conclusion. A rigorous extension would test LET-guided pretraining on: (1) encoder-only models (BERT-style masked language modeling on text), (2) encoder-decoder models (T5-style on text-to-text tasks), (3) state-space models (Mamba-style on language), and (4) vision Transformers (ViT on ImageNet-scale data). Each experiment would pretrain a target model from scratch, guided by a smaller pretrained model of the same architectural family, using the same LET protocol (cosine alignment of early-layer representations with linearly decaying weight). The key question is whether LET's benefits transfer to architectures without autoregressive causal masking and to domains without the sequential structure of text. A negative result on Mamba, for instance, would suggest that LET relies on properties specific to the Transformer architecture (e.g., the layer-wise residual stream structure), which would meaningfully constrain the scope of the method. A positive result on vision would open an entirely new domain of application where small pretrained models (e.g., DeiT-Tiny) could accelerate large-model pretraining (e.g., ViT-Large).

Theoretical analysis of the alignment-depth vs. optimization-curvature tradeoff in realistic architectures. The paper's theoretical analysis (Appendix 13) is a deep linear network with uniform Hessian block bounds — a simplification that provides qualitative intuition but cannot make quantitative predictions for real Transformer training. A stronger follow-up would analyze the Hessian structure for a simplified attention layer (single-head, no MLP) with LET-style alignment, deriving an expression for the curvature contribution as a function of alignment depth that accounts for the attention mechanism's specific parameter structure (query, key, value projections). This would address the question: does the k * C bound from the deep linear analysis carry over to attention-based architectures, or does attention's parameter-sharing structure produce a different scaling relationship? A negative result (e.g., if attention layers show super-linear curvature growth with k) would suggest that LET's optimal alignment layer might differ for attention vs. MLP-heavy architectures, which would refine the practical guidance for practitioners. This is a challenging theoretical project but is made newly tractable by the clear experimental target the paper provides — any theoretical prediction about the optimal alignment depth can be tested against the existing layer-pairing ablation results (Figures 3, 4, and 12).

Investigation of LET for data-constrained rather than compute-constrained pretraining. The paper's experiments use ~20B tokens for training, which is a compute-constrained regime (the models are not data-starved at this scale). A practically important scenario that is not tested is data-constrained pretraining, where a practitioner has limited high-quality in-domain data (e.g., 1B tokens of medical or legal text) and wants to train the largest possible model without overfitting. In this setting, the convergence speedup from LET is less relevant than the sample efficiency improvement: does LET enable the target model to extract more value from each training token, reducing the data volume needed to reach a given performance level? This can be tested by training 1.4B models on restricted data budgets (e.g., 1B, 2B, 5B, 10B tokens) with and without LET, measuring the downstream accuracy achieved at each budget. If LET provides a larger relative benefit at lower data budgets (which the paper's Figure 9 hints at — the LET advantage over the baseline is larger earlier in training), this would make LET particularly valuable for domain-specific LLMs where data is the binding constraint, not compute. This experiment requires no new infrastructure beyond what the paper already uses, only a data sub-sampling step.

Rigorous comparison against "warm start" initialization strategies. The paper compares LET against standard random initialization (Baseline), RKD, and SALT, but not against a simpler strategy: initializing the target model's early layers by copying weights from the guide model (where dimensions match, with random initialization for new dimensions) and then training with standard NLL from that initialization. This is a form of "warm start" that does not require ongoing alignment — it is a one-time parameter transfer rather than a training-time signal. If warm-start achieves similar speedups to LET with zero alignment-phase overhead, then LET's value proposition shifts from "alignment during training is necessary" to "alignment during training is one effective mechanism among several." This experiment would distinguish between two interpretations of LET's mechanism: (1) LET works because it provides an ongoing training signal that shapes the optimization trajectory (the paper's stated interpretation), vs. (2) LET works because it provides a better initialization for early layers, and the alignment signal could be replaced by parameter copying. A follow-up experiment could also test intermediate strategies: alignment for only the first 100 steps (to shape the initial gradient steps without sustained guidance), or one-shot alignment before training begins rather than during training. This family of experiments would map the boundary between LET's active training-time mechanism and passive initialization effects, providing practical guidance on the minimum alignment duration needed for benefits.

---

Practical Applications and Downstream Use Cases

Custom large-model pretraining by resource-constrained organizations. The most immediate practical application of LET is for organizations that want to develop their own large language models (for domain-specific applications, for architectural experimentation, or for IP control) but lack the compute budget to train from scratch at competitive scale. The paper's results on the 1.4B model provide a concrete value proposition: using a freely downloadable 135M SmolLM model as a guide reduces the compute needed to reach baseline-level performance by approximately 33% (Table 1: LET at 67% steps matches baseline at 100% steps). For an organization with a fixed compute budget (e.g., 32 A100 GPUs for 30 days), this means they can either (a) train a 1.4B baseline model on 20B tokens, or (b) train a LET-accelerated 1.4B model on the same 20B tokens and achieve +5% higher average downstream accuracy, or (c) stop LET training at ~13B tokens and redirect the saved ~7 compute-days to hyperparameter tuning, architecture search, or training an additional variant. The throughput overhead (7% slower per step during the first 20% of training) reduces the net speedup from 1.5× to approximately 1.4× in wall-clock terms, which is still a substantial practical gain. The memory efficiency (Table 4: LET uses lower peak VRAM than RKD and SALT) means this benefit is available even in memory-constrained setups — organizations running on 40GB or 48GB GPUs (A40, L40S) rather than 80GB A100s can still use LET, whereas logit-level distillation methods might exceed memory limits at large batch sizes.

Scaling up open-source foundation models through "model chaining." The paper's finding that LET works across heterogeneous architectures (Table 3) and vocabulary choices (Figure 2) enables a practical workflow that the community might adopt: iterative model chaining. A small model (e.g., SmolLM-135M) is used to LET-train a medium model (1.4B). That medium model, once fully trained, becomes the \mathcal{T} for a larger model (7B or 13B), and so on. Each generation benefits from the previous generation's representations without requiring architectural continuity — the 7B model could use a completely different activation function, attention mechanism, and vocabulary from the 1.4B model that accelerated it. The paper does not test this chaining workflow directly, but the result that LET works with up to 10× size gaps and with heterogeneous architectures makes it a feasible direction. The practical benefit is that each generation's pretraining is accelerated relative to training from scratch, amortizing the community's compute investment across model generations. A foundation model developer could release not just their final large model but also intermediate checkpoints optimized for LET-guided training — "guide-ready" models, analogous to the way some models are released as "fine-tuning-ready" base models.

Accelerating academic LLM research where full pretraining budgets are infeasible. Academic labs, which typically have access to smaller GPU clusters (8–32 GPUs vs. the thousands available in industry), face a structural disadvantage in LLM research: they cannot afford to pretrain models at the scale needed to test new architectural ideas, training objectives, or data strategies. LET directly addresses this by reducing the compute required for a given performance level. A lab with 16 A100 GPUs could, using LET, train a 1.4B model that reaches baseline-level performance in roughly 2/3 the time, making 1.4B-scale experiments feasible where they previously required 32 GPUs. More importantly, LET enables labs to test novel architectural variants at the 1.4B scale without starting from random initialization — a researcher proposing a new attention variant can get statistically meaningful downstream performance comparisons in less wall-clock time than standard training would require. This lowers the barrier to entry for architectural research on LLMs, which has increasingly become the exclusive domain of well-resourced industry labs. The paper's openness about hyperparameters and the reproducibility of the setup (all training data and code configurations are public or reproducible) makes this application practical immediately — an academic lab can download the paper's setup and start running LET-guided experiments on their own architectural ideas.

Domain-specific pretraining leverage from small in-domain models. The paper demonstrates LET using general-domain guide models (SmolLM, OPT, Pythia), but an important use case that follows directly is using a small domain-specific pretrained model to accelerate a larger domain-specific model. For example, a biomedical NLP team might have a small BioBERT-style model (e.g., 110M parameters) trained on PubMed abstracts and full-text articles. Using LET, they could pretrain a 1B-parameter biomedical LM that absorbs the small model's in-domain representational knowledge into its early layers during the early training phase, then continues training on the full domain corpus with standard NLL. The benefit is two-fold: (1) the convergence speedup from LET reduces total compute, and (2) the early-layer alignment may provide better domain grounding than random initialization, since the small in-domain model already encodes domain-specific terminology and relational structure. This is a direct extension of the paper's method that requires no new technical development — only a domain-specific \mathcal{T} model and a domain-specific training corpus for \mathcal{M}. The paper's time-series experiments (Appendix 9) provide preliminary evidence that LET works cross-domain, though the fine-tuning setup there is weaker than a full domain-specific pretraining experiment. A biomedical NLP team could run this experiment at modest cost (a 1B-parameter model trained on biomedical text) and produce a result that either validates domain-specific LET as a standard workflow or identifies limitations (e.g., if domain-specific representations are less transferable than general-domain ones).

---

When to Prefer This Method

The paper explicitly positions LET against three alternatives — standard training from random initialization (Baseline), Reverse Knowledge Distillation (RKD), and SALT — and the experimental results in Table 1 and Figure 9 provide a clear empirical basis for decision-making. The choice conditions can be articulated directly from the paper's findings.

• Prefer LET over standard training from scratch when: (1) a pretrained model at least 4–10× smaller than the target model is available and was trained on reasonably contemporary, high-quality text data (SmolLM, OPT, Pythia — but not GPT-2, per Appendix 14), (2) the target model uses a Transformer-based autoregressive architecture (the paper only tests this setting), (3) the training budget is fixed and reaching higher downstream accuracy or lower perplexity matters more than maximizing per-step throughput during the first ~20% of training, and (4) the target-to-guide parameter ratio is between roughly 4× and 11× (the extremes tested in the paper — larger ratios are untested but the scaling trend suggests diminishing returns at extreme ratios are unlikely given LET's robustness at 10×). The benefit is a 1.4–1.6× convergence speedup and a +1.8 to +2.4 percentage-point (absolute) improvement on average downstream accuracy, with lower peak VRAM than alternative guide-model methods.

• Prefer standard training from scratch over LET when: (1) no small pretrained model trained on data of comparable quality to the training corpus is available, (2) the target model's architecture is fundamentally incompatible with the hidden-state extraction and interpolation mechanism (e.g., non-Transformer architectures where the concept of "layer-wise hidden states" does not map cleanly to LET's extraction pattern — though the paper does not test this), or (3) the alignment-phase throughput overhead is unacceptable for operational reasons (e.g., a training setup where GPUs are shared across multiple simultaneous jobs and the 7% per-step slowdown during the first 20% of training disrupts scheduling). The paper's GPT-2 experiment (Table 6) additionally suggests that data quality of the guide model is a hard constraint — if the available small model was trained on data with a temporal cutoff significantly predating the target training corpus, LET may actively harm performance.

• Prefer LET over Reverse Knowledge Distillation (RKD) when: The guide model is substantially smaller than the target model, as is the case in the paper's intended use case. RKD fails catastrophically at the 7B scale with a 4.1× smaller teacher (47.03% vs. 54.06% baseline, Table 1) and even at the 1.4B scale with a 10× smaller teacher, RKD underperforms the baseline (44.63% vs. 46.80%). The paper's Figure 9 shows that RKD's performance degradation is not a transient training artifact — the RKD curve is consistently below the baseline throughout training at both scales. There is no regime in the paper's experiments where RKD outperforms LET, and given RKD's fundamental ceiling problem (the student's output distribution is constrained to match the teacher's), there is no theoretical reason to expect it to surpass LET when the teacher is smaller. The only scenario where RKD might be considered over LET is when per-step throughput is the absolute binding constraint — RKD's throughput is slightly higher (1.004× baseline vs. 0.927× for LET at 1.4B, Table 4) — but this advantage is so small (0.4% vs. −7.3%) that it is unlikely to outweigh LET's performance and convergence benefits.

• Prefer LET over SALT when: training uses the standard causal language modeling objective and a dedicated hyperparameter search for SALT's UL2-based configuration cannot be conducted. Under identical hyperparameter configurations (Table 1), LET achieves 49.20% average accuracy vs. SALT's 44.66% at the 1.4B scale — a 4.54-point gap. The paper acknowledges that SALT was designed for the UL2 objective and its default hyperparameters may be suboptimal for causal LM, meaning this comparison likely underestimates SALT's potential. However, the burden of evidence is on SALT: to be preferred over LET, a practitioner would need to run a non-trivial hyperparameter search for SALT's UL2-specific parameters (corruption rate, span length, mixture weights) that LET simply does not require (LET's alignment hyperparameters — k, λ₀, S_stop — are simpler and have default values validated by sweeps in the paper). For a practitioner who wants a reliable, low-tuning-effort acceleration method that works out of the box with standard causal LM training, LET is the lower-risk choice based on the paper's evidence.