LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding

ArXiv: 2404.16710

🎯 Pitch

LLMs can skip over a quarter of their layers per token without hurting quality, but only if trained with a counterintuitive recipe: dropout rates that increase with depth. This paper shows that combining such dropout with a shared early exit loss lets a single model serve as both its own draft and verifier, achieving up to 2.16× speedups with no separate draft model and no accuracy loss.

---

1. Executive Summary

This paper introduces LayerSkip, an end-to-end approach for accelerating LLM inference through three integrated mechanisms—a training recipe combining layer dropout with early exit loss, inference via early exit at intermediate layers, and a novel self-speculative decoding algorithm where earlier layers draft tokens and later layers verify and correct them (reusing shared KV cache and activations to avoid the memory overhead of a separate draft model). Experiments across pretraining from scratch, continual pretraining, and finetuning on Llama-family models (7B, 13B, 1.5B) on tasks including summarization (CNN/DM, XSUM), coding (HumanEval), and semantic parsing (TOPv2) demonstrate speedups of up to 2.16× on CNN/DM summarization, 1.82× on coding, and 2.0× on TOPv2, while maintaining accuracy comparable to autoregressive decoding. The approach establishes that self-speculative decoding can match or exceed traditional two-model speculative decoding performance—but only when the base model has been trained with layer dropout that is exponentially higher for later layers and an early exit loss that directly supervises intermediate layers to produce meaningful predictions through the shared LM head.

2. Context and Motivation

The Core Problem: LLM Inference Is Too Expensive, and Existing Speedups Either Sacrifice Accuracy or Demand Extra Resources

The fundamental problem this paper tackles is the high computational cost of LLM inference, specifically the cost of executing all transformer layers for every token during autoregressive generation. When a decoder-only model like Llama generates text, each new token requires a forward pass through every layer of the model — even for tokens that are conceptually simple or predictable. This is wasteful. In Figure 2b, the authors demonstrate that for a typical HumanEval coding example, tokens require on average 23.45 out of 32 layers to converge to the correct prediction, meaning roughly 26% of layer computations are, in principle, unnecessary. But the actual waste is worse than this number suggests: even tokens that are straightforward (e.g., Token 02 starting a for loop) still consume all 32 layers, and intermediate layers frequently exhibit indecisive "mind-changing" behavior (e.g., Token 05 oscillating between "range" and other candidates across layers 7–26 before settling). The model has no incentive to route easy tokens through fewer layers — by default, deep learning models "spread their compute across all layers" (as the authors note, citing Voita et al. 2019, 2023).

This matters for several practical reasons the authors highlight (Section 1):

• Financial and energy costs: Deploying LLMs on GPU servers incurs substantial compute expenses and energy consumption (Samsi et al., 2023).
• Edge and mobile deployment: Current acceleration techniques either suffer significant accuracy drops when deployed to commodity laptop GPUs (Zhu et al., 2023) or remain an active, unsolved research area for mobile/edge devices (Çöplü et al., 2023; Liu et al., 2024).
• Hardware compatibility: Unlike quantization or unstructured sparsity — which require specialized hardware kernels for efficient execution — reducing the number of layers executed per token does not require custom hardware support, making it broadly deployable across existing GPU and CPU architectures.

The Landscape of Existing Approaches and Their Limitations

The paper positions itself at the intersection of three research areas: early exit inference, speculative decoding, and layer dropout / stochastic depth. Each area offers partial solutions but has significant shortcomings that LayerSkip directly addresses.

Early Exit: Can Reduce Compute, but Existing Methods Add Modules or Struggle with Accuracy

Early exit inference — where a model routes some tokens through only a subset of layers and skips to the output — has been explored in CNNs (Panda et al., 2016; Teerapittayanon et al., 2017) and later in language models. The central challenge is that intermediate layers in a standard transformer are not trained to produce meaningful outputs through the LM head. This manifests in the qualitative results of Figure 2b, where earlier layers in an untrained model produce essentially random or degenerate predictions.

Prior early exit work in language models took two broad approaches, both with drawbacks:

Adding auxiliary modules per exit point. BERxiT (Xin et al., 2021) for encoder models, CALM (Schuster et al., 2022) for encoder-decoder models, and BranchyNet-style approaches (Teerapittayanon et al., 2017) all introduce dedicated LM heads — or even more complex auxiliary modules (Zhang et al., 2019) — at each potential exit layer. The authors argue this increases memory consumption, training complexity, and deployment burden. Each additional LM head is essentially a separate linear projection matrix (the same size as the vocabulary, which can be 32K–128K entries), multiplying storage costs at each exit point.

Using heuristics or confidence-based exit criteria. SkipDecode (Corro et al., 2023) and work by Geva et al. (2022) developed predictors to estimate when prediction "saturates" and exit can occur without accuracy loss. But these heuristics are imperfect and do not address the root cause: the model's intermediate representations are not inherently suitable for early exit. As the authors demonstrate concretely (Figure 6, Table 1), baseline Llama models show catastrophic accuracy collapse on generation tasks when exiting at middle layers — for example, NaturalQuestions on Llama2 7B drops from 25.1% exact match to 0% when exiting at layer 16. On perplexity evaluations, middle-layer perplexity on Wikipedia jumps from ~4.3 (final layer) to ~1900 (middle layer) for the baseline Llama2 7B (Table 1).

A critical observation the authors surface is that early exit accuracy degrades with more pretraining, not less. In Section 7 (Figure 11), they show that when pretraining a Llama 1.5B model from scratch, the middle-layer perplexity on The Stack dataset drastically increases as the model trains on more tokens — from approximately 0.6 at early training steps to over 600 by the end — unless early exit loss is applied. This means that as LLMs scale to more pretraining data (e.g., Llama3's 8T tokens vs. Llama2's 2T), the problem of useless intermediate representations only worsens. This finding is one of the paper's key motivating insights: current pretraining recipes actively make models less suitable for early exit as they scale.

Speculative Decoding: Lossless but Doubles Memory and Model Maintenance

Speculative decoding (Leviathan et al., 2023; Chen et al., 2023) is the dominant lossless acceleration technique and directly motivates LayerSkip's self-speculative decoding component. The mechanism exploits an elegant asymmetry: auto-regressive generation is slow because it must generate one token at a time, but verifying a batch of candidate tokens can be done in a single parallel forward pass. The draft model generates speculative tokens quickly (but possibly incorrectly), and the main model verifies all of them simultaneously, correcting where the draft deviates.

The practical problem with standard speculative decoding is resource overhead:

• Two separate models must be stored and maintained. The draft model and the verification model have independent weights, independent KV caches, and independent activations — effectively doubling the GPU memory footprint during inference. For large models deployed on memory-constrained devices, this is often prohibitive.
• Operational complexity. Two models require two sets of weights to be distributed, versioned, and updated. In production systems, this doubles the model management surface area.

Zhang et al. (2023) proposed a self-speculative decoding approach where the draft model is not a separate architecture but rather the same model with certain intermediate attention and FFN layers skipped. This eliminates the dual-model storage problem. However, the authors identify a critical limitation: because the Zhang et al. approach skips intermediate layers (not contiguous early layers), the draft and verification stages cannot share KV cache. The skipped layers disrupt the sequential continuity of transformer states, meaning the verification stage must recompute from scratch rather than reusing the draft stage's cached activations. LayerSkip addresses this explicitly by designing the draft stage to use only the first E contiguous layers, enabling the verification stage to continue from layer E onward using the same KV cache and a novel exit query cache that stores only the query vector of layer E−1.

Layer Dropout: Has Been Used for Regularization and Pruning, Not for Early Exit

Layer dropout (or stochastic depth) — randomly skipping layers during training — was introduced by Huang et al. (2016) for ResNets. In language models, LayerDrop (Fan et al., 2020) applied dropout to every other transformer layer during training, which improved robustness to layer pruning at inference time (entire layers could be removed post-hoc with less accuracy degradation). Progressive layer dropping (Zhang and He, 2020) accelerated BERT pretraining with a schedule that increased dropout rates across training iterations.

The authors position their use of layer dropout as fundamentally different from these prior approaches in three ways:

First, prior work used uniform dropout across layers (or simple alternating patterns), while LayerSkip applies an exponentially increasing dropout rate: near-zero for early layers, high for later layers (Equation 3: $D(l) = e^{l \ln 2 / (L-1)} - 1$). The motivation is specific to early exit: the model should learn to rely less on later layers while preserving early-layer capabilities, creating an ensemble of sub-models of varying depths within a single set of weights (Figure 3).

Second, prior layer dropout work in language models was limited to encoder-only models (BERT) at moderate scales. The authors claim this is "the first to propose using layer dropout to improve early exit inference" specifically for decoder-only LLMs at scale.

Third, existing dropout applications were primarily for regularization or pruning robustness, not for making intermediate layer outputs directly suitable for prediction through the shared LM head. LayerSkip's combination of layer dropout with early exit loss is what distinguishes the approach — dropout alone improves robustness to skipping layers, but it doesn't teach the LM head to decode from intermediate representations.

A Critical Gap: No One Has Shown That a Single Model Can Serve as Both Drafter and Verifier with Shared Compute

The paper's synthesis of the above limitations identifies a clear gap in the literature:

• Early exit methods can reduce per-token compute but either introduce accuracy loss (if exiting aggressively) or add auxiliary modules (if maintaining accuracy at each exit point).
• Speculative decoding is lossless but requires dual models and dual KV caches, doubling memory.
• Self-speculative decoding (Zhang et al., 2023) removes the dual-model requirement but prevents KV cache reuse between draft and verification stages.
• Layer dropout training has been studied for pruning and regularization but never combined with early exit loss or repurposed for speculative decoding.

The missing piece — what this paper provides — is a unified training recipe that produces a model whose early layers can serve as a competent draft model (via early exit) and whose full layer stack can serve as the verification model, with shared KV cache and activations because both stages process layers in the same sequential order.

How This Paper Positions Itself

The authors frame LayerSkip as an end-to-end solution spanning training and inference (Figure 1). The contributions are not individual techniques in isolation but their specific combination and the empirical demonstration that this combination unlocks practical speedups:

• The training recipe (Section 4.1) uses exponentially increasing layer dropout rates to make the model less dependent on later layers, combined with an early exit loss that directly supervises intermediate layers to produce meaningful predictions through the single shared LM head. The curriculum schedules (rotational or gradual) are designed to make this training overhead manageable.

• The inference mechanism (Section 4.2 for early exit, Section 4.3 for self-speculative decoding) leverages this training to either exit early directly (when some accuracy loss is acceptable) or use early exit as a drafting stage within speculative decoding (when lossless acceleration is required).

• The cache reuse technique (Section 4.3.3) — the novel exit query cache and single KV cache architecture — is what makes the self-speculative approach practically competitive with dual-model speculative decoding, reducing both memory footprint and the latency of recomputing shared layers.

The paper's empirical scope is deliberately broad: it tests the recipe across pretraining from scratch, continual pretraining, domain-specific finetuning, and task-specific finetuning, on models ranging from 1.5B to 13B parameters. This breadth is intended to demonstrate that the approach is not narrowly tuned to a single training regime or model scale.

A subtle but important point in the paper's positioning: the authors are careful to acknowledge that the training recipe is not optional. The baseline models' early exit performance is so poor (0% exact match on multiple generation tasks at middle layers, Tables 1 and 2) that self-speculative decoding would provide virtually no speedup without LayerSkip training — the draft model would produce tokens the verifier almost never accepts, defeating the purpose of speculation. This distinguishes LayerSkip from Zhang et al. (2023), whose approach works on unmodified models (by skipping intermediate layers) but at the cost of cache reuse inefficiency.

3. Technical Approach

3.1 Reader Orientation

LayerSkip is an end-to-end system that trains a single language model to support fast inference using only a subset of its layers, then uses that capability to accelerate text generation — either by exiting early (accepting some accuracy trade-off) or by using the early layers as a fast draft model and the remaining layers as a verifier in a lossless speculative decoding loop. The core insight is that standard pretraining leaves intermediate layers useless for prediction (their outputs produce near-random tokens when passed through the LM head), and that a specific combination of exponentially-increasing layer dropout during training plus a direct supervision signal at every layer can transform the model into an ensemble of sub-models of varying depths — all sharing one set of weights and one LM head — which then enables self-speculative decoding where the draft and verification stages reuse the same KV cache and activations because they process layers in the same sequential order.

3.2 Big-Picture Architecture (Diagram in Words)

The system has three stages that form a pipeline — training enables capabilities that inference then exploits:

1. Training with Layer Dropout and Early Exit Loss (Section 4.1): takes a standard transformer language model and modifies its training to randomly skip later layers more often than earlier layers (layer dropout with exponential per-layer scaling), while simultaneously adding a loss term that forces the shared LM head to produce correct token predictions from every intermediate layer's output (early exit loss with curriculum scheduling). This produces a model where early and middle layers can generate coherent predictions, and later layers serve as refinement rather than essential computation.

2. Early Exit Inference (Section 4.2): at generation time, for each token, run only the first E transformer layers (where E < L) and feed the output of layer E directly into the LM head, skipping layers E+1 through L. This reduces per-token compute proportionally but incurs some accuracy loss since the model hasn't seen the full depth.

3. Self-Speculative Decoding (Section 4.3): an alternative inference mode that recovers lossless accuracy. The first E layers draft multiple tokens auto-regressively (using early exit at each step), then the remaining L-E layers verify all draft tokens in a single parallel forward pass by reusing the KV cache and a novel exit query cache from the draft stage. Tokens where draft and verification agree are accepted; at the first disagreement, the verified token is used and drafting restarts. This achieves speedup because verification of multiple tokens happens in one batched forward pass (exploiting the asymmetry that parallel verification is faster than sequential generation), while the shared cache eliminates redundant computation.

3.3 Roadmap for the Deep Dive

• First, the training recipe's two components — layer dropout (Section 4.1.1) and early exit loss (Section 4.1.2) — since these create the fundamental capability that all inference strategies depend on. I will explain the dropout probability function, the per-layer and per-time scaling choices, the early exit loss formulation, and the curriculum scheduling that makes training tractable.

• Second, the early exit inference mechanism (Section 4.2) — since this is the simplest inference mode and introduces the core concept of selecting an exit layer E.

• Third, the self-speculative decoding algorithm (Section 4.3) — which builds on early exit and adds the verification loop, the cache reuse technique, and the exit query cache optimization.

• Fourth, the cache reuse architecture (Section 4.3.3) — since this is the mechanism that makes self-speculative decoding practically competitive with dual-model approaches, and its design constraints explain why contiguous early layers must be used for drafting rather than skipping intermediate layers as in prior work.

3.4 Detailed, Sentence-Based Technical Breakdown

This is an empirical methods paper whose core idea is that a specific training recipe — exponentially-scaled layer dropout combined with curriculum-scheduled early exit loss applied to a single shared LM head — can transform a standard transformer into a model whose intermediate layers produce useful predictions, enabling both direct early exit inference and a novel form of self-speculative decoding where draft and verify stages share KV cache and activations due to contiguous layer processing.

---

4.1 Training Using Layer Dropout and Early Exit Loss

Notation and Baseline Architecture

The training recipe modifies a standard decoder-only transformer language model. I establish notation first because the equations in Sections 4.1.1 and 4.1.2 are dense and reference these symbols repeatedly.

Let the input tokens be denoted X and the target output tokens be Y. The embedding layer maps token indices to token embeddings x_0. The model has L transformer layers, where layer l (for l from 0 to L-1) transforms its input embeddings x_l into output embeddings x_{l+1} via a residual connection:

$x_{l+1} = x_l + f_l(x_l)$

where f_l is the transformer block at layer l (comprising self-attention and feed-forward network sublayers, plus normalization). The final layer norm and linear projection (the LM head) is denoted g, which maps the output of the last layer x_L to logits e_L = g(x_L). The standard training objective is the cross-entropy loss between these logits and the target tokens: J_CE(e_L, Y).

What this notation establishes: the model processes tokens sequentially through L identical-depth paths, with each layer adding its contribution via a residual connection, and only the final layer's output is ever fed to the LM head during standard training.

Why this matters for what follows: the early exit modifications will feed intermediate x_{l+1} values directly to g, and the layer dropout modifications will stochastically skip the f_l computation, making the model's effective depth vary per sample during training.

4.1.1 Layer Dropout

Layer dropout modifies the standard transformer forward pass by stochastically skipping entire layers. During training, at each iteration t and for each layer l, the computation becomes:

$x_{l+1,t} = x_{l,t} + M(p_{l,t}) \cdot f_l(x_{l,t})$

where M(p) is a Bernoulli function that returns 0 with probability p and returns 1 with probability 1-p. When M(p_{l,t}) = 0, the layer is skipped entirely — x_{l+1,t} = x_{l,t}, meaning the embeddings pass through unchanged. When M(p_{l,t}) = 1, the layer operates normally.

What it computes: a stochastic depth for each layer per sample, where the probability of skipping layer l at iteration t is p_{l,t}. The dropout is applied independently per sample within a batch: dropped samples are removed from the batch for that layer's computation, the transformer block f_l processes only the non-dropped samples, and the output is then concatenated with the dropped samples' unchanged embeddings. Random number generators are seeded identically across GPUs so that each layer drops the same number of samples per iteration, maximizing training throughput (since all GPUs perform the same amount of work per skipped/non-skipped split).

Why this specific form — the residual connection remains intact: even when a layer is dropped, the identity path x_{l,t} is preserved. This means dropped layers contribute nothing but don't destroy information. If the dropout instead zeroed out the output (as unstructured dropout does), a dropped layer would inject zeros that propagate forward. The residual formulation ensures that skipping a layer simply means that layer doesn't modify the representation — the representation from the previous layer passes through unchanged. This is critical for training stability because the model must learn to function with arbitrary subsets of its layers active.

Layer Dropout Rate Schedule

The dropout probability p_{l,t} is the product of three terms:

$p_{l,t} = S(t) \cdot D(l) \cdot p_{max}$

where p_max is a global hyperparameter setting the maximum dropout rate in the model during training (the rate applied at the last layer when the time curriculum is at its maximum), D(l) is a per-layer scaling function, and S(t) is a per-time-step scaling function.

Per-layer scaling D(l): the dropout rate increases exponentially from 0.0 at layer 0 to 1.0 at the last layer L-1:

$D(l) = e^{\frac{l \ln 2}{L-1}} - 1$

where e is Euler's number, ln is the natural logarithm, l is the layer index (0-indexed), and L is the total number of layers.

What it computes: at layer 0, D(0) = e^0 - 1 = 0, meaning the first layer is never dropped. At layer L-1, D(L-1) = e^{\ln 2} - 1 = 2 - 1 = 1, meaning the last layer receives the full dropout rate p_max. The exponential curve means dropout increases slowly in early layers and rapidly in late layers. For example, in a 32-layer model with p_max = 0.1, layer 16 (the middle) would have D(16) = e^{(16 \cdot \ln 2)/31} - 1 = e^{0.358} - 1 \approx 0.43, giving a dropout rate of approximately 0.043 (4.3%), while layer 24 would have D(24) = e^{(24 \cdot \ln 2)/31} - 1 \approx 0.71, giving approximately 7.1%.

Why exponential across layers: the motivation is to make the model progressively less reliant on later layers. If dropout were uniform, the model would learn to be equally robust to skipping any layer — but that doesn't specifically push computation into earlier layers, which is what early exit inference needs. By heavily dropping later layers, the model is forced to produce useful representations in early and middle layers because it cannot count on later layers being present. The first layer is never dropped because it processes raw token embeddings and the model needs at least one transformation before representations are meaningful. The authors compared exponential against constant dropout (same average rate but uniform across layers) and found exponential leads to lower training loss (Figure 12), confirming this intuition.

Per-time scaling S(t): the paper uses two different schedules depending on the training scenario:

• For finetuning or continual pretraining (starting from a pretrained model): S(t) = 1 for all t, meaning no temporal curriculum — dropout is applied at full strength from the first training step.

• For pretraining from scratch: an exponential curriculum that gradually increases dropout over the course of T total training steps:

$S_{exp}(t) = e^{\frac{t \ln 2}{T-1}} - 1$

where t is the current training iteration index and T is the total number of training steps.

Why a temporal curriculum for pretraining from scratch: when the model weights are randomly initialized, aggressive layer dropout at the start of training would be too destabilizing — the model hasn't yet learned basic representations, and forcing it to function without later layers would prevent it from learning anything. The exponential curriculum starts with near-zero dropout (at t=0, S(0) = 0), allowing the model to learn standard representations first, then gradually increases the dropout pressure. This is analogous to how other regularization techniques (like dropout rate annealing in some vision models) are often applied more aggressively later in training. For finetuning, the pretrained model already has well-formed representations, so dropout can be applied at full strength immediately — the model adapts its existing representations rather than building them from scratch.

Implementation Detail: Batch Efficiency

The paper specifies two implementation details that make layer dropout practical at scale:

Per-sample dropout: each sample within a batch independently draws from the Bernoulli distribution. When a layer is dropped for a particular sample, that sample is temporarily removed from the batch for the f_l computation, the transformer block processes only the non-dropped subset, and the output is concatenated with the unchanged embeddings of dropped samples. This means the effective batch size for each layer's computation varies per iteration and per layer, but the total number of samples flowing through the network remains constant (they just skip computation in some layers).

Identical GPU seeds: to ensure balanced workload across GPUs in distributed training, all GPUs use the same random seed for the Bernoulli draws, so at each layer and iteration, all GPUs drop the same fraction of samples (though which specific samples are dropped depends on the per-sample seeding within that fraction).

4.1.2 Early Exit Loss

Layer dropout alone makes the model robust to missing layers, but it does not teach the LM head g to produce meaningful predictions from intermediate layer outputs. During standard training, g only ever sees x_L (the final layer's output), so its weights are optimized exclusively for unembedding representations at the end of the network. The early exit loss addresses this by providing direct supervision at every layer.

The total loss at iteration t is a weighted sum of cross-entropy losses computed at each layer's output:

$J(X, Y, t) = \sum_{l=0}^{L-1} \tilde{e}(t, l) \cdot J_{CE}(g(x_{l+1}), Y)$

where J_CE(g(x_{l+1}), Y) is the standard cross-entropy loss between the target tokens Y and the logits obtained by applying the LM head g to layer l's output x_{l+1}, and \tilde{e}(t, l) is a normalized per-layer loss weight.

What it computes: instead of only computing loss at the final layer, the model computes loss at every layer — as if each layer had its own early exit. The \tilde{e}(t, l) weights determine how much each layer's loss contributes to the total. The sum of all \tilde{e}(t, l) across layers is 1, so the total loss magnitude is comparable to standard training (just redistributed across layers rather than concentrated at the end).

Why compute loss at every layer: this directly forces the LM head to learn to unembed intermediate representations. During backpropagation, gradients from early-layer losses flow through g and back to the early transformer layers, teaching those layers to produce representations that are useful for token prediction (not just useful as inputs to the next layer). Without this, early layers optimize only for being good inputs to later layers, which produces the degenerate behavior seen in Figure 2b where early-layer predictions are noise.

Per-Layer Loss Weights

The normalized weight \tilde{e}(t, l) is:

$\tilde{e}(t, l) = \frac{C(t, l) \cdot e(l)}{\sum_{i=0}^{L-1} C(t, i) \cdot e(i)}$

where e(l) is a per-layer importance scale (higher for later layers) and C(t, l) is a binary curriculum function that equals 1 if early exit loss is enabled for layer l at iteration t, and 0 otherwise.

The per-layer scale e(l): defines how much weight each layer's loss receives when enabled. The paper uses a quadratically-increasing scale, where the weight at layer l is proportional to the sum of weights of all previous layers:

$e(l) = \begin{cases} e_{scale} \cdot \sum_{i=0}^{l} i, & \text{if } 0 \leq l < L-1 \\ L-1 + e_{scale} \cdot \sum_{i=0}^{L-2} i, & \text{if } l = L-1 \end{cases}$

where e_scale is a hyperparameter (typically 0.1, 0.2, or 1.0 depending on the experiment) that controls how strongly early layers are penalized relative to later layers. The sum \sum_{i=0}^{l} i = l(l+1)/2 grows quadratically with l.

What it computes: for a model with 32 layers and e_scale = 0.2, the raw scale e(0) = 0, e(1) = 0.2, e(2) = 0.6, e(3) = 1.2, ..., e(16) = 0.2 \cdot 136 = 27.2, and e(31) = 31 + 0.2 \cdot 465 = 31 + 93 = 124. After normalization across enabled layers, early layers receive very small weights and late layers receive large weights.

Why quadratically increasing: the authors state that "predicting in later layers is easier" — since later layers have seen more transformations and are closer to the final output, they naturally have an easier time producing correct predictions. Without differential weighting, the model would allocate most of its learning capacity to making later layers slightly better (since that's the easiest way to reduce loss), and early layers would remain poor. The quadratic scaling counteracts this by making later layer losses much more expensive, forcing the model to invest learning signal in earlier layers. The final layer's special case (L-1 + e_scale \cdot \sum_{i=0}^{L-2} i) adds a base weight of L-1 to ensure the final layer always has the highest relative importance regardless of e_scale.

The hyperparameter e_scale: controls the trade-off. Higher values (up to 1.0 for task-specific finetuning) push more loss weight toward earlier layers, improving early exit accuracy but potentially reducing final-layer accuracy. Lower values (0.1 for larger models like Llama2 13B) prioritize maintaining final-layer quality. The paper uses e_scale = 0.2 for most continual pretraining and pretraining-from-scratch experiments, e_scale = 1.0 for finetuning on code data and task-specific data, and e_scale = 0.1 for Llama3 models.

Early Exit Loss Curriculum C(t, l)

Adding loss at all layers at every training iteration would be computationally expensive (requiring L separate LM head forward passes and loss computations per iteration) and was found to "slow down training and reduce the accuracy of the last layer." The curriculum C(t, l) addresses this by enabling early exit loss at only a subset of layers per iteration.

The paper explores two curricula:

Rotational curriculum C_rot,R: enables early exit loss at every R-th layer, rotating which layers are active at each iteration. At iteration t, the enabled layers are those where (l + t) mod R = 0 (my paraphrase of the circular rotation mechanism). This means at each training iteration, only ceil(L/R) unembedding operations are performed rather than L. A layer is enabled once every R iterations on average.

What the hyperparameter R controls: for continual pretraining of Llama2 7B (32 layers), R=8, meaning approximately 4 layers are enabled per iteration (32/8). For Llama2 13B (40 layers), R=39, meaning typically only 1–2 layers are enabled per iteration — a very sparse schedule that minimizes training overhead. For pretraining from scratch, R=23 for the 24-layer Llama 1.5B and R=31 for the 32-layer Llama2 7B.

Why rotational: it ensures all layers receive early exit supervision over time (just staggered across iterations) while keeping per-iteration computational cost bounded. The rotation is circular so no layer is permanently disabled.

Gradual curriculum C_grad: enables early exit loss starting from the last layer L-1 and progressively enabling earlier layers, one new layer every T/(2L) iterations, working backward from layer L-1 down to layer 0. By the end of training, all layers have early exit loss enabled.

What it computes: in the first T/(2L) iterations, only the last layer has early exit loss. After T/L iterations, the last two layers have it. After T/2 iterations, the second half of the layers have it. By iteration T, all layers have it.

Why gradual: this ensures the model first learns the task well at the final layer (standard training), then gradually extends that capability to earlier layers. It prevents early exit loss from interfering with the model's ability to learn the core task in early training. The paper uses this curriculum only for the task-specific finetuning experiment (TOPv2) rather than the rotational curriculum.

Choice between curricula: the rotational curriculum is used for most experiments (continual pretraining, pretraining from scratch, code finetuning) because it provides more uniform supervision across layers throughout training. The gradual curriculum is used for the TOPv2 task-specific finetuning, possibly because the small dataset size (only 5.8K steps) makes the gradual introduction more stable.

Training Hyperparameters Summary

The complete set of hyperparameters introduced by the training recipe:

• p_max: maximum dropout rate for the last layer. Values used: 0.1 (continual pretraining: Llama2 7B, 13B, Llama3 8B, Llama3.2 1B; pretraining from scratch: Llama 1.5B), 0.2 (pretraining from scratch: Llama2 7B; task-specific finetuning: Llama 1.5B).
• S(t): dropout time curriculum. Either S(t) = 1 (finetuning, continual pretraining) or S(t) = S_exp(t) (pretraining from scratch).
• e_scale: early exit loss scale. Values used: 0.1 (Llama2 13B continual pretraining, Llama3 models), 0.2 (Llama2 7B continual pretraining, pretraining from scratch), 1.0 (code finetuning, task-specific finetuning).
• C(t, l): early exit loss curriculum. Either rotational C_rot,R or gradual C_grad.
• R: dilation for rotational curriculum. Values used: 8 (Llama2 7B continual pretraining, Llama3 models), 39 (Llama2 13B continual pretraining), 23 (Llama 1.5B pretraining), 31 (Llama2 7B pretraining), 16 (code finetuning).
• Learning rate adjustment: when pretraining from scratch with layer dropout, the learning rate is increased (e.g., Llama 1.5B: 4e-4 without dropout → 8e-4 with dropout; Llama2 7B: 3e-4 without → 8e-4 with) following the insight from Srivastava et al. (2014) that dropout benefits from higher learning rates.

Design Choices: Why a Shared LM Head Instead of Per-Layer Heads

A key design decision is using a single shared LM head g for all layers rather than training separate LM heads per exit point (as in Elbayad et al., 2020; Schuster et al., 2022). The authors state this "makes training faster, require less memory consumption for both training and inference, and eases deployment and maintenance."

The trade-off: per-layer LM heads would allow each exit point to have its own specialized unembedding weights optimized for the specific representational properties of that layer's outputs. This could potentially achieve higher accuracy at each exit layer. However, the memory cost scales with L times the vocabulary size (typically 32K–128K entries), which for large models with many layers is substantial. The shared head forces all layers to produce representations in the same "space" — the space that the single g can decode — which acts as an additional regularization pressure toward layer-to-layer consistency.

The fact that the shared head approach works (achieving significant early exit accuracy improvements) suggests that the early exit loss combined with layer dropout is sufficient to align intermediate representations with the final layer's representational format, making separate heads unnecessary.

---

4.2 Inference Using Early Exit

Early exit inference is the simplest deployment mode enabled by the training recipe. During autoregressive generation, for each token to be generated, the model executes only the first E transformer layers (where E < L), then feeds the output of layer E (denoted x_E) directly into the LM head, producing logits g(x_E) and sampling the next token from these logits. Layers E+1 through L are not executed at all.

What it computes: a faster but potentially less accurate next-token prediction. The computational cost per token is reduced from executing L transformer layers to executing E layers — approximately a factor of E/L reduction in transformer FLOPs per token (plus fixed overhead from the embedding layer and LM head, which are executed regardless of E).

The accuracy-speed trade-off: since the model was trained with layer dropout and early exit loss, the intermediate layer outputs are meaningful — but they are still less accurate than the full model's output. The paper evaluates this trade-off by measuring accuracy metrics at various exit layers E (Figures 6, 8, 10 and Tables 1, 2). The key pattern: early layer accuracy is substantially better for LayerSkip-trained models than baselines, but there is always some degradation compared to the full model. For example, on the coding task HumanEval with Llama2 7B continually pretrained with LayerSkip, exiting at layer 8 gives pass@1 of ~4.9% vs. 15.9% for the full 32-layer model (Table 1).

Dynamic vs. static exit: the paper only experiments with a fixed exit layer E for all tokens and all inputs, noting that future work could explore "dynamic conditions to determine a different exit layer for each token" (Section 9). This means the current implementation is a static early exit — the speed-accuracy trade-off is chosen globally rather than optimized per-token.

---

4.3 Inference Using Self-Speculative Decoding

Self-speculative decoding is the paper's primary contribution for lossless acceleration. It combines the early exit capability (from the training recipe) with the speculative decoding framework to achieve speedup without accuracy degradation, while avoiding the dual-model memory overhead of traditional speculative decoding.

4.3.1 Self-Drafting

The first stage of self-speculative decoding generates d candidate tokens (referred to as "draft tokens" or "speculations") using early exit inference. The model runs the first E layers auto-regressively: for the first draft token, it processes the input prompt through layers 0 to E-1 and exits at layer E to produce a token; for the second draft token, it appends the first draft token to the input and again processes through layers 0 to E-1; and so on for d total draft tokens.

What it computes: a sequence of d tokens generated quickly (since each requires only E layers of computation), but with lower accuracy than the full model would produce. This is directly analogous to the draft model in traditional speculative decoding, except the draft model here is the first E layers of the same model rather than a separate smaller model.

The parameter d: the number of speculations (draft tokens generated before verification). The paper uses values of d=12 (CNN/DM, XSUM for Llama2 7B), d=4 (XSUM for Llama2 13B), d=6 (HumanEval for Llama2 7B), d=4 (HumanEval for Llama2 13B), and d=7 or d=8 (TOPv2 for Llama 1.5B), among others. Larger d means more tokens are attempted per draft-verify cycle, which increases potential speedup if acceptance rate is high, but reduces speedup if most draft tokens are rejected (since the verification pass costs the same regardless of how many are accepted).

Choice of exit layer E for drafting: the paper evaluates different E values and reports a trade-off: smaller E means faster drafting (fewer layers per draft token) but lower token acceptance rate during verification. For example, on TOPv2 with Llama 1.5B (Table 6): E=18 gives 98.9% acceptance but only 1.24× speedup; E=12 gives 97.6% acceptance and 1.64× speedup; E=6 gives 76.0% acceptance and 2.0× speedup. The sweet spot depends on the task and model.

4.3.2 Self-Verification

After drafting d tokens, the verification stage uses the full model (layers E through L-1) to verify all draft tokens in a single parallel forward pass. The key insight that makes this efficient is that only the remaining layers need to be computed — the draft stage already computed layers 0 through E-1, so the verification stage can start from the saved representations at layer E-1.

The verification procedure:

1. Single parallel forward pass: the full sequence (prompt + d draft tokens) is processed through layers E through L-1 in one forward pass. Since transformer verification is not autoregressive (it processes all positions simultaneously), this is much faster than generating d tokens sequentially with the full model.

2. Token-by-token comparison: for each draft token position, the verification model produces a probability distribution over the vocabulary. The draft token is compared to the verification model's greedy prediction (or sampled prediction). If they match, the draft token is "accepted." If they don't match, the draft token is rejected and the verification model's prediction is used instead.

3. Acceptance and continuation: all consecutive accepted draft tokens from the beginning of the draft sequence are appended to the output. At the first rejection point, the verification model's corrected token is appended, and the process restarts: the prompt is now the previous prompt plus all accepted tokens plus the correction token, and a new draft-verify cycle begins for the next d tokens.

What it computes: lossless token generation — the output is identical to what the full model would produce with standard autoregressive decoding (under greedy decoding; for sampling, rejection sampling ensures the output distribution matches the full model's distribution exactly, following the standard speculative decoding protocol). The speedup comes from the fact that when many draft tokens are accepted, most tokens require only the cheap draft computation, while verification cost is amortized across the accepted tokens.

Why the verification pass is faster than generating tokens sequentially: this is the fundamental asymmetry that all speculative decoding exploits. Verifying d tokens in a single forward pass costs approximately the same as generating one token autoregressively with the full model (both execute L-E layers once, operating on a sequence of length N+d rather than length N for one new token). If even 2 out of d draft tokens are accepted, the per-accepted-token verification cost is roughly halved. If acceptance rate is α, the verification cost per accepted token is approximately (L-E)/α layer computations, compared to L layers for standard autoregressive generation. The total cost per accepted token becomes E + (L-E)/α (draft cost plus amortized verification cost). When α is high and E is small relative to L, this is much less than L.

4.3.3 Reusing the Cache

This is the component that distinguishes LayerSkip's self-speculative decoding from prior self-speculative approaches (specifically Zhang et al., 2023) and from traditional dual-model speculative decoding. It eliminates redundant computation between draft and verification stages.

The standard transformer KV cache stores key and value projections from each attention layer for all previously processed tokens, so that when generating a new token auto-regressively, only the new token's query/key/value need to be computed and attention can be computed against the cached history. In traditional speculative decoding with two separate models, each model maintains its own KV cache — the draft model's cache is separate from the verification model's cache, even though both cache representations of the same tokens.

LayerSkip's cache reuse exploits the fact that the draft and verification stages share the same early layers (layers 0 through E-1) in the same order:

Single KV Cache: since both stages process layers 0 through E-1 identically, the KV cache entries for those layers are identical between draft and verification. The draft stage computes them; the verification stage reuses them directly without recomputation. This halves the KV cache storage for the shared layers compared to dual-model approaches (which would store two separate copies) and eliminates the latency of recomputing those layers during verification.

Exit Query Cache (KVQ cache): a novel optimization specific to self-speculative decoding with contiguous layer sharing. When the draft stage auto-regressively generates the i-th draft token, it computes the query vector at layer E-1 for that token as part of the normal self-attention computation. This query vector is needed for the verification stage because, when the verification pass processes the i-th draft token through layers E through L-1, layer E's self-attention needs to attend to all previous tokens — which requires the query vector at layer E-1 for the i-th token as an input to layer E's attention mechanism.

What the exit query cache stores: for each draft token, the query vector output from layer E-1 is saved. This is a single vector per draft token (not the full KV cache for all layers), making it small — dimension d_model (typically 4096 for 7B models) per draft token, compared to the full KV cache which stores L * 2 * d_model per token (key and value for all layers). The paper terms the combined KV cache plus exit query cache the "KVQ cache."

Why the exit query is sufficient but necessary: without it, the verification stage would need to recompute layers 0 through E-1 for the draft tokens just to produce the query vector at layer E-1 — which is exactly the computation the cache reuse was trying to avoid. With it, the verification stage can start directly at layer E for all tokens (including the newly added draft tokens), using the KV cache for layers 0 through E-1 to provide key/value pairs and the exit query cache to provide the query vectors for the draft tokens at the boundary layer. For the prompt tokens (which were processed before any drafting), their full KV cache and query vectors are already available from the initial processing.

The constraint this imposes — why contiguous early layers matter: this cache reuse mechanism only works because LayerSkip uses the first E contiguous layers for drafting and the remaining L-E contiguous layers for verification. If drafting skipped intermediate layers (as in Zhang et al., 2023, which skips some middle attention and FFN layers), the draft and verification stages would not share a contiguous prefix of layers, and the KV cache from the draft stage would not be directly reusable — the verification model's layer structure would differ from the draft model's, requiring separate caches or recomputation. This architectural constraint is what makes the training recipe essential: the model must be trained so that a contiguous prefix of its layers can serve as an effective draft model, which standard training does not provide.

Measured impact of cache reuse (Table 7): the paper ablates the cache reuse by comparing self-speculative decoding with and without KVQ reuse on CPU inference. For the TOPv2 task with E=18, cache reuse saves 143 − 134 = 9 ms per token. For CNN/DM with E=18, it saves 182 − 166 = 16 ms per token. With E=12, the savings are 110 − 104 = 6 ms/token (TOPv2) and 185 − 165 = 20 ms/token (CNN/DM). The savings increase with E (more layers shared means more computation saved by reuse) and vary by task (likely due to different sequence lengths affecting KV cache size).

---

Summary of Design Choices and Their Justifications

• Exponential layer dropout (vs. constant or uniform): forces the model to encode information in earlier layers by disproportionately dropping later layers, making early exit viable. Empirical validation: exponential outperforms constant dropout at same average rate (Figure 12).

• Shared LM head (vs. per-layer heads): reduces memory, simplifies deployment, and acts as an alignment pressure forcing intermediate layers into the same representational format as the final layer. Empirical validation: the approach works despite not having specialized per-layer unembedding weights.

• Rotational curriculum (vs. all-layers-all-the-time): makes training overhead manageable (only ceil(L/R) unembedding passes per iteration instead of L) and prevents early exit loss from degrading final-layer accuracy during training.

• Contiguous early layers for drafting (vs. skipped intermediate layers): enables KV cache and exit query cache reuse between draft and verification stages, reducing both memory footprint and redundant computation.

• Exponential time curriculum for pretraining from scratch (vs. constant dropout): prevents aggressive dropout from destabilizing early training when representations are unformed, gradually increasing pressure as the model stabilizes.

• Higher learning rates with layer dropout (vs. same learning rate): follows established dropout practice (Srivastava et al., 2014) that stochastic depth benefits from larger gradient steps to compensate for the effective reduction in model capacity per update.

4. Key Insights and Innovations

Innovation 1: Intermediate Layer Collapse Is a Training Artifact, Not an Architectural Inevitability

The paper's most intellectually significant contribution is not the training recipe itself, but the diagnostic insight that drives it: the catastrophic uselessness of intermediate layers in standard LLMs is not a fundamental property of transformer architectures, but a consequence of how we train them. This reframes the problem completely. Prior early exit work (Elbayad et al., 2020; Schuster et al., 2022; Geva et al., 2022) accepted the baseline state — that intermediate layers produce near-random token predictions — as a given, and built auxiliary modules or heuristic exit criteria to compensate. The underlying assumption was: transformers naturally distribute computation across all layers, and intermediate representations simply aren't meaningful until late in the network.

LayerSkip's motivation section (Figure 2b, and critically the scaling experiment in Figure 11) demolishes this premise. The authors show that when pretraining a Llama 1.5B model from scratch on increasing numbers of tokens, middle-layer perplexity dramatically increases as training progresses — from roughly 0.6 in early training steps to over 600 by the end — unless early exit loss is applied. Far from being an accidental side effect, standard language modeling training actively makes intermediate layers worse. The LM head is only ever trained on final-layer representations, so gradient signal flows backward through layers optimized solely to be good inputs to the next layer, not good direct predictors. As the model gets better at the overall task (lower final-layer perplexity), the representations at different depths diverge more sharply because there is no pressure to keep them aligned with the output space.

This is a genuine conceptual advance with implications beyond early exit. It suggests that transformer depth may be substantially underutilized in current LLMs — later layers are not just refining predictions but fundamentally repairing the representational misalignment that standard training creates in intermediate layers. If training could maintain representational alignment across depth (as LayerSkip's early exit loss does), the same model might need fewer layers to achieve the same quality, or might achieve better quality at the same depth by avoiding the need for later layers to undo earlier representational drift.

The evidence anchoring this claim is strong: Figure 11 shows the divergence happening monotonically across training tokens, and Tables 1–2 show that models pretrained on more data (Llama3, 8T tokens) have worse intermediate-layer perplexity than those pretrained on less data (Llama2, 2T tokens) — the baseline Wikipedia perplexity at the middle layer is ~1,900 for Llama2 7B vs. ~110,000 for Llama3 8B. The trend is clear and the diagnostic value is high: as the field scales to ever-larger pretraining runs, the intermediate layer problem gets worse, not better, making this insight practically urgent.

Innovation 2: Exponentially-Weighted Layer Dropout as a Mechanism for Capability Redistribution, Not Just Regularization

Layer dropout has existed since Huang et al. (2016) for ResNets and was applied to language models by Fan et al. (2020). Prior work used dropout either as a regularizer (to reduce overfitting by training an ensemble of sub-networks) or as a pruning preparation technique (to make the model robust to removing layers post-hoc). The dropout rates were typically uniform across layers or followed simple alternating patterns.

LayerSkip's use of layer dropout is conceptually different: it's a capability redistribution mechanism. The exponentially-increasing schedule — near-zero dropout for early layers, high dropout for later layers — is designed not to regularize the model as a whole but to shift computational reliance from later layers to earlier layers. The model learns that later layers are unreliable (they're frequently absent during training), so early and middle layers must carry more of the predictive burden. This is a targeted intervention in the model's internal computation allocation, not a generic robustness measure.

What makes this genuinely novel is the pairing of the dropout schedule with a specific goal (early exit capability) and the empirical validation that uniform dropout at the same average rate is strictly worse (Figure 12). If layer dropout were merely providing generic regularization, any reasonable schedule should help. The fact that exponential outperforms constant at equal average dropout rate is direct evidence that the specific distribution of dropout across layers matters for the res Allison effect.

This reframes layer dropout from a generic training trick into a control knob for where computation happens in a deep network. The paper doesn't just use dropout to make the model tolerate missing layers — it uses a carefully designed dropout profile to actively reshape which layers learn to do what. This is a new way of thinking about stochastic depth that could generalize: for any model where different layers have different computational costs or latency characteristics, asymmetric dropout profiles could be used to shift work toward cheaper layers during training.

Innovation 3: Contiguous Layer Reuse as an Architectural Constraint That Enables Cache-Sharing in Self-Speculative Decoding

Self-speculative decoding — where a single model drafts and verifies — existed before LayerSkip (Zhang et al., 2023). What LayerSkip contributes is the recognition that how layers are selected for drafting determines whether cache reuse is possible, and that this constraint has implications that ripple backward into training design.

Zhang et al. (2023) achieved self-speculation by skipping intermediate attention and FFN layers — effectively creating a draft model with "holes" in its layer structure. This works for creating a faster-but-less-accurate draft model from an existing trained model, but it introduces a fatal inefficiency: the draft and verification models have different layer topologies beyond the first skipped layer, so they cannot share KV caches. The verification stage must recompute from scratch because its attention layers see different key/value histories than the draft model's.

LayerSkip's insight is to invert this: use only the first E contiguous layers for drafting, which guarantees that draft and verification share layers 0 through E-1 in identical order. This simple constraint — draft from the prefix, verify with the suffix — is what enables the single KV cache and the exit query cache optimization. The innovation is not the cache reuse technique itself (reusing intermediate computation is an obvious optimization), but the recognition that the layer selection pattern is a first-class design parameter that determines whether efficient reuse is architecturally possible.

This has a secondary implication that the paper only partially develops: if you want efficient self-speculative decoding with cache reuse, you must train the model such that a contiguous prefix of its layers produces good predictions. You cannot simply take any pretrained model, select a subset of layers, and expect both accuracy and efficiency — the Zhang et al. approach can skip arbitrary layers but pays a cache-reuse penalty, while the LayerSkip approach achieves cache reuse but requires the training recipe to make contiguous prefix exit viable. This is a fundamental trade-off that future self-speculative methods will need to navigate.

The empirical evidence for this innovation is comparative rather than ablative within the paper: on the common model and task where both papers report results (CNN/DM with Llama2 7B), LayerSkip achieves 1.81× speedup vs. Draft & Verify's 1.56× (Table 3). The paper attributes this difference primarily to cache reuse, and the ablation in Table 7 quantifies cache reuse as saving 9–20 ms per token depending on configuration.

Innovation 4: The Training-Inference Co-Design Perspective: Inference Strategy Should Dictate Training Objectives

Underlying all three components is a meta-contribution about methodology: LayerSkip treats training and inference as a jointly optimized system rather than sequential, independent phases. The training recipe (layer dropout profile, early exit loss weight, curriculum schedule) is explicitly designed to create capabilities that specific inference strategies (early exit, self-speculative decoding) require. The inference strategy, in turn, imposes architectural constraints (contiguous layer sharing for cache reuse) that shape the training design.

This contrasts with the dominant paradigm in LLM research, where models are trained for a single objective (next-token prediction) and inference techniques are developed post-hoc to accelerate the fixed trained model. Quantization, pruning, and even most prior early exit work start from a pretrained model and attempt to work around its limitations. LayerSkip's approach is more like hardware-software co-design applied to model architecture: specify the inference-time operational constraints first, then design the training procedure to produce a model that satisfies those constraints.

The concrete manifestation of this philosophy is the decision to use a single shared LM head rather than per-layer exit heads. Per-layer heads would be the natural choice if one were approaching early exit from a pure accuracy-maximization perspective (each head specializes to its layer's representational properties). But they would add memory overhead at inference time and complicate the deployment surface — both concerns from the inference deployment perspective. The shared head is a training-time concession to inference-time practicality, and the fact that it works (producing meaningful improvements in early exit accuracy) validates the co-design approach.

This is not a claim that can be anchored to a single figure, but the entire paper's structure — training recipe first, then two inference modes that exploit the trained capabilities differently — embodies this design philosophy. The method would make little sense if presented as training-only (why add early exit loss if you don't plan to exit early?) or inference-only (the baseline models' early exit performance is too poor to provide useful drafting). Only as a co-designed system do the pieces fit together.

5. Experimental Analysis

Evaluation Methodology

• Dataset(s). The paper evaluates on multiple datasets depending on the training regime and task. For continual pretraining and pretraining from scratch, models are evaluated on a diverse suite of tasks including perplexity on held-out splits of Wikipedia, Books, and The Stack (Kocetkov et al., 2022); commonsense reasoning tasks (BoolQ, PIQA, SIQA, HellaSwag, WinoGrande, ARC Easy/Challenge, OBQA, COPA); reading comprehension (RACE Middle/High); MMLU; open-ended question answering (NaturalQuestions, TriviaQA); mathematics (GSM8K, MATH); and code generation (HumanEval, MBPP). For self-speculative decoding generation experiments, the primary tasks are CNN/DM (Nallapati et al., 2016) abstractive summarization (1-shot), XSUM (Narayan et al., 2018) abstractive summarization (0-shot), HumanEval (Chen et al., 2021) coding, and TOPv2 (Chen et al., 2020) semantic parsing. The CNN/DM and XSUM experiments follow the same setup as Zhang et al. (2023) for direct comparison. The TOPv2 dataset is post-processed into JSON format aligned with code pretraining. All self-speculative decoding experiments evaluate on the respective test sets or evaluation splits of each dataset. For generation experiments, a maximum of 512 tokens are generated per sample, and the first 100 samples from the TOPv2 test set are used for CPU inference experiments.

• Base model(s). The paper uses Llama-family decoder-only transformer models across four training regimes: (1) Continual pretraining: pretrained Llama2 7B (32 layers) and Llama2 13B (40 layers) from Touvron et al. (2023b), plus Llama3 8B and Llama3.2 1B; (2) Pretraining from scratch: a custom Llama-like 1.5B model (24 layers, dim 2048, 16 heads, context 4096) and Llama2 7B (32 layers), both randomly initialized; (3) Finetuning on code data: pretrained Llama1 7B (Touvron et al., 2023a) finetuned on 5.2B tokens of CodeLlama (Rozière et al., 2023) data mix; (4) Finetuning on task-specific data: a pretrained Llama 1.5B model finetuned on the TOPv2 training set. The choice of multiple model scales (1.5B, 7B, 13B) and both pretrained and randomly initialized starting points is designed to test whether the training recipe generalizes across model sizes and training scenarios. For self-speculative decoding speedup comparisons, the paper also compares against Draft & Verify (Zhang et al., 2023) on shared model/task configurations.

• Metrics. The paper uses a variety of task-specific metrics. For early exit evaluation (Section 6.1, Figures 6, 8, 10, Tables 1–2): perplexity (PPL, lower is better) on held-out text corpora; accuracy (Acc, %) for commonsense reasoning, reading comprehension, and MMLU (multiple-choice tasks); exact match (EM, %) for open-ended question answering, mathematics, and semantic parsing; pass@1 (%) for code generation tasks (HumanEval, MBPP). For self-speculative decoding generation evaluation (Section 6.2, Tables 3–6): ROUGE-2 (Ganesan, 2018) for summarization quality; EM (exact match) for semantic parsing; token acceptance rate (%), defined as the fraction of draft tokens that the verification stage accepts (how often verification agrees with the draft); throughput measured in tokens per second averaged over the sampled dataset; and speedup, calculated as the acceleration of average inference time per token compared to the autoregressive baseline on the same hardware setting. Speedup follows the same definition as Zhang et al. (2023), enabling direct comparison.

• Baselines. The paper employs several baselines depending on the experiment. For early exit evaluation, the primary baseline is the standard pretrained/finetuned model without LayerSkip — the identical architecture trained for the same number of tokens/steps but without layer dropout or early exit loss. This baseline is evaluated at the same exit layers to quantify the improvement from the training recipe. For the training recipe itself, the paper ablates three variants: LayerSkip-LD (layer dropout only, no early exit loss), LayerSkip-EE (early exit loss only, no layer dropout), and LayerSkip-LD+EE (both components combined). For self-speculative decoding comparisons, baselines include: Autoregressive decoding (the standard model without LayerSkip, generating one token at a time through all layers), Early Exit inference (Section 4.2, using only the first E layers without verification), and Draft & Verify (Zhang et al., 2023), a prior self-speculative decoding approach evaluated on shared model/task configurations (CNN/DM and XSUM with Llama2 7B). All self-speculative and early exit experiments use models trained with LayerSkip; the autoregressive baseline uses models trained without LayerSkip.

• Generation budget / compute accounting. For early exit evaluation, the computational cost per token is proportional to the number of layers executed: E layers for early exit, L layers for the full model. The paper sweeps different values of E (typically powers of 2 or intermediate values) and reports accuracy at each exit point. For self-speculative decoding, the paper reports the number of speculations d (draft tokens generated per cycle) and the exit layer E. The effective per-token cost is E + (L-E)/α where α is the token acceptance rate. Throughput and speedup are measured empirically on specific GPU hardware: NVIDIA H100 GPUs for continual pretraining, pretraining from scratch, and TOPv2 experiments; NVIDIA A100 GPUs for code finetuning experiments. All generation experiments use greedy decoding. The paper does not report FLOPs or detailed operation counts, relying instead on wall-clock throughput and speedup relative to autoregressive decoding on identical hardware.

• Cross-validation / statistical protocol. The paper does not describe a formal cross-validation or statistical significance protocol for the generation speedup experiments. For early exit evaluation, the test sets are standard benchmark evaluation splits used as-is (e.g., the 500-question MATH test set, the HumanEval benchmark). The self-speculative decoding experiments sample from the respective test sets (e.g., first 100 samples from TOPv2 test set for CPU experiments, full CNN/DM test set for GPU experiments). The lack of confidence intervals or multiple-run averaging on the speedup and throughput numbers is a notable absence — the paper reports point estimates for speedup without error bars.

Main Quantitative Results

Early Exit Inference: Accuracy vs. Exit Layer Trade-offs

The early exit results (Section 6.1) answer a focused question: after training with LayerSkip, how does each layer's unembedded prediction quality compare to the baseline model? The headline finding is that LayerSkip dramatically improves early and middle layer accuracy while maintaining approximate parity at the final layer, across all four training regimes.

Continual pretraining results (Figure 6, Table 1). For Llama2 7B continually pretrained on 52B tokens with LayerSkip-LD+EE:

• At the final layer (layer 32), LayerSkip achieves comparable or slightly lower accuracy than the baseline across most tasks. For example: BoolQ 77.8% vs. 77.4% baseline (slightly better); PIQA 77.9% vs. 78.0% (essentially equal); MMLU 43.1% vs. 46.0% (notable drop of ~3 points); HumanEval pass@1 15.9% vs. 13.4% (actually improved). The pattern is mixed — some tasks show slight improvement, others slight degradation — but the overall final-layer quality is maintained within a few percentage points.

• At the middle layer (layer 16), the difference is stark: LayerSkip provides coherent predictions where the baseline produces near-zero accuracy on generation tasks. For NaturalQuestions: 4.07% EM with LayerSkip vs. 0.0554% baseline — a 73× relative improvement, though both are far below the final-layer accuracy of ~23–25%. For TriviaQA: 11.8% vs. 0.619%. For HumanEval: 4.88% vs. 0% (the baseline never produces a correct program when exiting at layer 16). For GSM8K: 2.05% vs. 0%. The pattern holds across all open-ended generation tasks: the baseline collapses to ~0% at middle layers while LayerSkip preserves measurable (if low) accuracy.

• For "classification" tasks (multiple choice questions), the baseline already maintains non-trivial accuracy at middle layers — MMLU drops only from 55.2% to 49.2% on the Llama2 13B baseline — and LayerSkip provides smaller but consistent improvements. For Llama2 7B: ARC-Easy at layer 16 is 57.5% (LayerSkip) vs. 38.6% (baseline); HellaSwag is 43.8% vs. 31.5%.

• The perplexity results tell the same story: Wikipedia perplexity at layer 16 drops from ~1900 (baseline) to 8.12 (LayerSkip) for Llama2 7B — a reduction of over two orders of magnitude. For Llama2 13B: The Stack perplexity at layer 20 drops from 65.8 to 3.71. The perplexity curves in Figure 6 show LayerSkip maintaining low perplexity across essentially all layers, while baseline perplexity explodes in early and middle layers.

For Llama2 13B continually pretrained (Table 1, right columns), the pattern replicates at a larger scale, with final-layer metrics showing minimal change (e.g., MMLU 53.7% vs. 55.2% baseline; HumanEval 18.3% vs. 18.9%) and middle-layer metrics showing dramatic improvement (NaturalQuestions at layer 20: 4.43% vs. 0.609% baseline; TriviaQA: 11.4% vs. 4.36%).

For Llama3 8B and Llama3.2 1B continually pretrained (Table 2), a new phenomenon emerges: final-layer accuracy degrades more significantly than in Llama2 experiments. MMLU drops from 66.5% to 60.5% (Llama3 8B), HumanEval from 37.8% to 28.7%, GSM8K from 54.2% to 45.0%. The authors attribute this to the much higher perplexity of early layers in Llama3 baseline models (middle-layer Wikipedia perplexity of ~110,000 for Llama3 8B vs. ~1,900 for Llama2 7B), making it harder to improve early layers without sacrificing final-layer quality. Despite the final-layer regression, middle-layer accuracy still improves dramatically (e.g., NaturalQuestions at layer 16: 4.63% vs. 0.00% baseline; HumanEval: 7.32% vs. 0.00%).

Pretraining from scratch results (Figure 8). For the 1.5B model pretrained on 26B tokens with LayerSkip variants:

• LayerSkip-LD+EE provides the best or near-best accuracy at early layers across all tasks, while LayerSkip-LD (layer dropout only) and LayerSkip-EE (early exit loss only) each provide partial benefits. For example, on HumanEval at layer 12 (middle): baseline ~0%, LayerSkip-EE ~1%, LayerSkip-LD ~1%, LayerSkip-LD+EE ~2%. On TriviaQA at layer 12: baseline ~0%, LayerSkip-EE ~2% EM, LayerSkip-LD ~5%, LayerSkip-LD+EE ~8%.

• Final-layer accuracy for LayerSkip-LD+EE is competitive with baseline in most cases: Wikipedia perplexity ~2.8 for both; HumanEval pass@1 ~2% for both; TriviaQA EM ~10% for both. The small scale of pretraining (26B tokens is tiny compared to the 2T tokens of Llama2) means absolute accuracies are low across the board, limiting the strength of conclusions from this regime.

For Llama2 7B pretrained from scratch on 26B tokens (Figure 8b), similar patterns hold, with LayerSkip variants showing clear early-layer improvement over baseline.

Code finetuning results (Figure 10a). Finetuning Llama1 7B on 5.2B code tokens with LayerSkip-LD+EE shows:

• At the final layer: HumanEval pass@1 is nearly identical to baseline (both ~14–15%); MBPP shows slight improvement (~22% vs. ~21% baseline).
• At layer 8 (25% of depth): LayerSkip-LD+EE achieves ~5% pass@1 on HumanEval vs. ~0% for baseline. On MBPP at layer 8: ~14% vs. ~0%. LayerSkip-LD+EE consistently outperforms the LD-only and EE-only variants at early layers.
• The benefit of combining LD+EE is especially clear here: at layer 16 (middle), MBPP with LD+EE reaches ~19% vs. ~15% for EE-only and ~8% for baseline.

Task-specific finetuning results (Figure 10b). For TOPv2 semantic parsing with Llama 1.5B:

• The baseline model achieves 0% exact match at any layer except the final layer (24), where it achieves ~89% EM.
• LayerSkip-LD+EE achieves 77% EM at layer 12 (half depth), ~83% at layer 18, and ~86% at layer 24 — a 3 percentage point regression from the baseline's ~89% but with the ability to exit at half depth with high accuracy. LayerSkip-EE alone achieves ~72% at layer 12, and LayerSkip-LD alone achieves ~68%. The combination is synergistic: LD+EE is better than either component alone at all exit layers.

Key observation across all regimes: The ablation of LayerSkip-LD vs. LayerSkip-EE vs. LayerSkip-LD+EE consistently shows that the combination is better than either component alone, confirming the complementary nature of the two training modifications.

Self-Speculative Decoding: Speedup Results

The self-speculative decoding results (Section 6.2) measure the end-to-end inference acceleration that the training recipe enables when combined with the self-speculative decoding algorithm.

Continual pretraining results (Table 3). For Llama2 7B continually pretrained with LayerSkip:

• On CNN/DM (1-shot summarization): exiting at layer 8 with 12 speculations, self-speculative decoding achieves ROUGE-2 of 0.078 (matching autoregressive baseline of 0.079), token acceptance rate of 68.9%, throughput of 127.9 tokens/sec, and 1.86× speedup over autoregressive (62.7 tokens/sec). Early exit alone (E=8, no verification) achieves only 0.012 ROUGE-2 but 232.4 tokens/sec — fast but unacceptable quality. Self-speculation recovers the quality loss while preserving substantial speedup.

• On XSUM (abstractive summarization): E=8, d=12, ROUGE-2 0.073 (matching baseline 0.073), acceptance 54.6%, throughput 104.7 tokens/sec, 1.54× speedup.

• On HumanEval (coding): E=8, d=6, ROUGE-2 0.042 vs. 0.041 baseline, acceptance 67.1%, throughput 122.8 tokens/sec, 1.83× speedup.

• Comparison with Draft & Verify (Zhang et al., 2023): on CNN/DM, LayerSkip achieves 1.81× (they also test E=15, d=12 for Llama2 13B in the same table) vs. Draft & Verify's 1.56× — a 16% relative improvement. On XSUM, LayerSkip achieves 1.54× vs. Draft & Verify's 1.48× — a 4% relative improvement. The authors note that Draft & Verify achieves slightly higher ROUGE-2 on CNN/DM (0.079 reported vs. 0.078 for LayerSkip), so the speedup comparison includes a slight quality difference.

For Llama2 13B continually pretrained (Table 3, right columns):

• CNN/DM: E=15, d=12, ROUGE-2 0.098 (matching baseline), acceptance 74.5%, throughput 70.2 tokens/sec, 1.81× speedup over autoregressive (37.2 tokens/sec).
• XSUM: E=15, d=4, ROUGE-2 0.124 (matching baseline), acceptance 67.7%, throughput 60.5 tokens/sec, 1.34× speedup.
• HumanEval: E=7, d=4, ROUGE-2 0.055 (matching baseline), acceptance 57.0%, throughput 84.2 tokens/sec, 1.66× speedup.

An important trend: for the smaller model (Llama2 7B), self-speculative decoding achieves higher speedups than for the larger model (Llama2 13B) on the same tasks — e.g., XSUM 1.54× vs. 1.34×. This is likely because the ratio E/L is more favorable for the smaller model (exiting at layer 8/32 = 25% of depth for 7B vs. 15/40 = 37.5% for 13B on most tasks), meaning the draft stage is proportionally faster relative to the full model.

Pretraining from scratch results (Table 4). For models pretrained from scratch on only 26B tokens:

• Llama2 1.5B on CNN/DM (1-shot): E=8, ROUGE-2 0.063 (matching baseline 0.063), acceptance 77.4%, throughput 167.4 tokens/sec vs. 91.6 baseline, 1.76× speedup.
• Llama2 7B on CNN/DM: E=8, ROUGE-2 0.067 (actually slightly higher than baseline 0.060), acceptance 77.8%, throughput 145.6 tokens/sec vs. 64.5 baseline, 2.16× speedup — the highest speedup reported in the paper.

The trend here reverses from continual pretraining: the larger model (7B) achieves higher speedup than the smaller model (1.5B) — 2.16× vs. 1.76×. This is notable because these models were trained on the same number of tokens (26B), which is proportionally much less data for the larger model (the 7B model is severely undertrained). The high acceptance rate (77.8%) suggests the draft model is particularly accurate, perhaps because the full model itself hasn't converged to high accuracy.

Code finetuning results (Table 5). For Llama1 7B finetuned on code and evaluated on HumanEval:

• E=6, d=12 speculations (implied from the context; the table shows the self-speculative result with the same d as other experiments). ROUGE-2 0.0513 (matching baseline exactly), token acceptance 45%, throughput 62 tokens/sec vs. 34 baseline, 1.82× speedup.
• Early exit alone at E=6 achieves only 0.0035 ROUGE-2 (near-zero quality) but 170 tokens/sec. Self-speculation recovers the full quality while retaining about one-third of the early exit speedup.

The relatively low token acceptance rate (45%) compared to other experiments (where acceptance rates range from 55% to 78%) indicates that for code generation, exiting at layer 6 produces draft tokens that the full model disagrees with more than half the time. Despite this, the 1.82× speedup demonstrates that speculation can still be beneficial even with moderate rejection rates — the 12 speculations per cycle mean that even at 45% acceptance, ~5.4 tokens are accepted per verification pass on average, amortizing the verification cost.

Task-specific finetuning results (Table 6). For Llama 1.5B finetuned on TOPv2 and evaluated on the TOPv2 test set:

The paper reports results for three exit layers, showing the speed-accuracy trade-off explicitly:

• E=18: self-speculation EM 82.9% (vs. autoregressive 85.9% — note this is a slight degradation, not lossless), acceptance 98.9%, time per token 29 ms (vs. 36 ms baseline), 1.24× speedup.
• E=12: EM 82.9%, acceptance 97.6%, time per token 22 ms, 1.64× speedup.
• E=6: EM 82.9%, acceptance 76.0%, time per token 18 ms, 2.0× speedup.

Notably, the self-speculative decoding EM at all three exit layers (82.9%) is identical and slightly below the autoregressive baseline (85.9%), suggesting a small but consistent accuracy cost for the TOPv2 task that does not appear in other tasks (where ROUGE-2 typically matches exactly). The early exit-alone EM degrades substantially: 83.3% at E=18, 79.4% at E=12, 62.9% at E=6 — showing that self-speculation recovers most but not all of the accuracy loss. The paper does not explain this residual gap.

CPU inference results (Table 11). Using the first 100 samples from TOPv2 test set with 7 speculations and generating 50 tokens per sample:

• E=18: EM 82.9%, acceptance 99%, time per token 134 ms (vs. 165 ms autoregressive), ~1.23× speedup.
• E=12: EM 82.9%, acceptance 97%, time per token 104 ms, ~1.59× speedup.
• E=6: EM 82.9%, acceptance 76%, time per token 87 ms, ~1.90× speedup.

The CPU speedups are comparable to GPU speedups, suggesting the approach is hardware-agnostic (as the paper notes, reducing layer count does not require specialized kernels).

A key negative detail: the paper reports that for Llama2 13B on XSUM, despite using only d=4 speculations (vs. d=12 for the 7B model), the speedup is only 1.34×. The larger model's deeper layer count (40 vs. 32) means the draft stage (E=15 layers) is proportionally heavier, and verification (L-E=25 layers) dominates the cost, limiting speedup. This illustrates a fundamental scaling challenge: as models get deeper, the relative benefit of exiting at a fixed early layer diminishes because the verification cost (remaining layers) grows.

Ablation Studies and Robustness Checks

Layer dropout configuration: exponential vs. constant across layers (Figure 12). When pretraining, an exponentially increasing dropout schedule (0 at layer 0, scaling to p_max at layer L-1) achieves lower average training loss than a constant dropout rate (equal probability across all layers) at the same average dropout rate. Both configurations have equivalent average dropout across all layers (0.0889 constant vs. 0 to 0.2 exponentially, same average), but the exponential schedule produces training loss of ~1.0–1.5 vs. ~1.5–2.3 for the constant schedule across the 2,500 training steps shown. This validates the paper's claim that the specific dropout profile matters — the model benefits more from heavily penalizing later layers while preserving early layers than from uniformly random depth.

KV cache and exit query cache reuse (Table 7). Running self-speculative decoding without the KVQ reuse optimization increases per-token latency by 6–20 ms depending on configuration. On TOPv2 CPU inference: E=18 without reuse costs 143 ms/token vs. 134 ms/token with reuse (saving 9 ms); E=12 without reuse costs 110 ms/token vs. 104 ms/token (saving 6 ms). On CNN/DM GPU inference: E=18 without reuse costs 182 ms/token vs. 166 ms/token (saving 16 ms); E=12 without reuse costs 185 ms/token vs. 165 ms/token (saving 20 ms). The savings are larger for CNN/DM likely due to longer sequences (larger KV cache, more computation saved by reuse). This ablation directly quantifies the benefit of the contiguous-layer architectural constraint — without it, the cache cannot be shared, and the efficiency gain is lost.

Scaling with pretraining tokens (Figure 11). When pretraining Llama 1.5B from scratch with increasing numbers of tokens (32×, 64×, 128× GPU configurations, corresponding to more total tokens), the baseline model's middle-layer perplexity on The Stack increases dramatically — from approximately 0.6 at the smallest token count to over 600 at the largest — while the last-layer perplexity decreases slightly (from ~2.9 to ~2.6). LayerSkip-EE and LayerSkip-LD+EE both substantially mitigate this increase: middle-layer perplexity stays below ~10 for LayerSkip-EE and below ~5 for LayerSkip-LD+EE across all token counts. LayerSkip-LD (without early exit loss) shows a middle ground: perplexity increases but more slowly than baseline, staying around ~100 at the largest token count. This ablation provides direct evidence that standard training causes intermediate representations to diverge from the output space, and that early exit loss specifically addresses this — layer dropout alone slows the divergence but doesn't prevent it.

Training recipe components: LD only vs. EE only vs. LD+EE (Figures 6, 8, 10 throughout). Across all training regimes (continual pretraining, pretraining from scratch, code finetuning, task-specific finetuning), the combination of layer dropout and early exit loss (LD+EE) consistently outperforms either component alone at early and middle exit layers. The pattern is most clearly visible in Figure 10a (code finetuning) and Figure 10b (TOPv2 finetuning), where LD+EE is strictly above the LD-only and EE-only curves for all exit layers except the final one. For the final layer, the three variants are typically comparable, with LD+EE sometimes slightly below LD-only or EE-only (e.g., TOPv2 where LD+EE achieves ~86% at final layer vs. ~88% for EE-only and ~89% for baseline). The synergy between the two components is a central empirical claim of the paper.

Effect of e_scale hyperparameter. The paper implicitly ablates e_scale across different experiments: 0.1 for Llama2 13B and Llama3 models, 0.2 for Llama2 7B continual pretraining and pretraining from scratch, 1.0 for finetuning on code and task-specific data. The results suggest that higher e_scale (more weight on early layer losses) is tolerable when finetuning on specific data (where the model only needs to perform well on a narrow distribution) but causes final-layer degradation when applied to diverse pretraining data. However, there is no controlled experiment varying only e_scale while holding other hyperparameters constant, so this is an observation across experiments rather than a clean ablation.

Effect of R (rotational curriculum dilation). The paper uses values of R from 8 (Llama2 7B) to 39 (Llama2 13B), but does not present an ablation varying R. The choice of R=39 for Llama2 13B (meaning only ~1 layer enabled per iteration) is extreme and suggests that even minimal early exit supervision per iteration is sufficient, but this is not explicitly tested.

A notable negative result: Llama3 final-layer degradation (Table 2 vs. Table 1). When continually pretraining Llama3 8B with LayerSkip, the final-layer accuracy drops more substantially than for Llama2 models: MMLU drops 6.0 percentage points (66.5% → 60.5%), HumanEval drops 9.1 points (37.8% → 28.7%), GSM8K drops 9.2 points (54.2% → 45.0%). The authors attribute this to the much higher baseline middle-layer perplexity of Llama3 models, making the early exit loss gradient signal disproportionately large relative to final-layer signal during training. This is a significant limitation: the LayerSkip recipe, as configured, is less effective at preserving final-layer quality for models pretrained on very large datasets, suggesting the hyperparameters may need to be re-tuned per model family and pretraining scale.

Critical Assessment

Does the paper demonstrate that the training recipe improves early exit accuracy?

Yes, with strong evidence across multiple model scales and training regimes. The improvement is most dramatic on open-ended generation tasks where baseline models achieve 0% accuracy at middle layers — Tables 1 and 2 show exact match scores going from 0.00% to measurable non-zero values (e.g., 4.07%, 11.8%, 4.88%) after LayerSkip training. The perplexity reductions are even more striking: two orders of magnitude improvement at middle layers (e.g., Wikipedia PPL from ~1900 to 8.12 for Llama2 7B).

However, there is an important nuance about what "improves" means here. The absolute early exit accuracy remains low on most generation tasks. For HumanEval on Llama2 7B, exiting at layer 16 achieves 4.88% pass@1 — up from 0%, but still far below the 15.9% of the full model. For NaturalQuestions: 4.07% vs. 23.2% full model. The training recipe makes early exit possible where it was previously impossible, but the early exit predictions are still substantially degraded relative to the full model. This is not a shortcoming per se — the paper is clear that self-speculative decoding is the intended lossless deployment mode — but it means that standalone early exit inference (Section 4.2) is not practically useful for generation tasks requiring high accuracy. The paper's claim that the recipe "increases the accuracy of early exit at earlier layers" is valid, but the absolute accuracy levels should not be overstated.

Does the paper demonstrate that self-speculative decoding achieves speedups without accuracy loss?

Partially, with important caveats. The strongest evidence is in Tables 3, 4, and 5, where ROUGE-2 for self-speculative decoding matches or nearly matches the autoregressive baseline across CNN/DM, XSUM, HumanEval, and coding tasks. The speedups range from 1.34× to 2.16×.

However, the TOPv2 results (Table 6) show a consistent accuracy gap: self-speculative EM is 82.9% vs. 85.9% autoregressive across all exit layers — a 3-percentage-point degradation that is not "lossless." The paper neither explains nor addresses this gap. It may be due to the task's exact-match nature requiring perfect token-by-token reproduction, or a property of the specific model configuration. Regardless, it means the "lossless" claim does not hold universally across all tasks tested.

The comparison with Draft & Verify (Zhang et al., 2023) is limited to two tasks (CNN/DM and XSUM) on a single model (Llama2 7B) — see Table 3. LayerSkip achieves higher speedup (1.81× vs. 1.56× on CNN/DM) but at very slightly lower ROUGE-2 (0.078 vs. 0.079). The XSUM comparison (1.34× vs. 1.48×) actually favours Draft & Verify. These are close enough that no definitive superiority claim can be made from two datapoints. The comparison is also confounded: the draft model configuration (which layers, how many) differs between approaches, so the speedup difference reflects both algorithmic efficiency and draft model quality.

A more fundamental limitation: the paper does not compare against traditional two-model speculative decoding (a separate smaller model as drafter) on the same hardware. This is the most natural baseline — does self-speculative decoding with cache reuse actually outperform having a separate small draft model? The memory footprint argument (single model, single KV cache) is made qualitatively but not quantified against a two-model baseline at equivalent total parameter count or memory usage. This comparison would be especially informative for edge deployment scenarios where memory is the primary constraint.

Does the paper's evidence scale across model sizes?

The paper tests 1.5B, 7B, and 13B models, which is a reasonable range but does not approach the scale of frontier models (70B+). The trend for continual pretraining suggests that speedups are lower for larger models (Llama2 13B: 1.34–1.81× vs. Llama2 7B: 1.54–2.16×) because the draft stage (fixed at early layers) represents a smaller fraction of total depth. If this trend continues, a 70B model with 80 layers exiting at layer 20 would have a much smaller relative speedup, as the verification stage (60 layers) dominates the cost. The paper does not extrapolate or discuss this scaling behavior.

The pretraining-from-scratch results go in the opposite direction (7B: 2.16× vs. 1.5B: 1.76× on CNN/DM), but this is for severely undertrained models (26B tokens for a 7B model), so it's unclear whether this pattern would hold for properly trained models. The higher acceptance rate for the 7B model (77.8% vs. 77.4%) may reflect that the 7B model's full accuracy is so low (due to undertraining) that the draft model is proportionally closer to it in capability.

The lack of results on models larger than 13B is a significant gap. The paper's claims about applicability to LLM deployment at scale would be strengthened by at least one experiment on a 30B+ model, even if only for a single task.

Are the self-speculative decoding experiments representative of real deployment?

Several aspects of the experimental setup limit generalizability:

• Greedy decoding only. All experiments use greedy decoding. In practice, many applications use temperature sampling, which changes the token acceptance dynamics — sampled draft tokens may diverge from verification more frequently, reducing acceptance rates and therefore speedups. Standard speculative decoding typically requires rejection sampling to preserve the exact output distribution, which the paper does not discuss. Whether the acceptance rates reported (55–78%) would hold under sampling is unknown.

• Fixed exit layer and speculation count. Each task uses a single hand-chosen E and d without any adaptivity. The paper does not explore whether these values could be tuned per-sample or per-token to improve the speed-accuracy Pareto frontier. The acknowledgment in Section 9 that future work could "explore dynamic conditions to determine a different exit layer for each token" is an admission that the current static configuration is suboptimal for heterogeneous inputs.

• No batching experiments. The paper evaluates single-sequence generation, not batched inference. In batched settings, the relative benefit of speculative decoding can change because the verification pass processes multiple sequences simultaneously, potentially reducing the per-token speedup advantage. This is a standard caveat in speculative decoding research that goes unmentioned here.

• Small sample sizes for latency measurement. The CPU experiments use only 100 samples from TOPv2. The GPU experiments use unspecified sample counts (presumably the full test sets, but this is not stated). The speedup numbers are point estimates without confidence intervals, making it impossible to assess whether the reported differences between configurations (e.g., 1.81× vs. 1.56×) are statistically meaningful.

• No latency breakdown. The paper does not report where time is spent in the self-speculative loop — what fraction goes to drafting, to KV cache management, to the verification forward pass, to the LM head, etc. This makes it difficult to diagnose bottlenecks or predict how speedups would change with different model architectures or hardware.

Missing experiments that would strengthen the paper

• Comparison with standard two-model speculative decoding at matched memory. If LayerSkip's primary advantage is memory efficiency, this should be demonstrated directly: compare self-speculative decoding (one model, shared KV cache) against standard speculative decoding with a smaller draft model, where the total memory footprint (weights + KV cache) is matched. Without this, the memory argument remains qualitative.

• Ablation of the number of speculations d. The paper uses different values of d for different tasks and models without explaining how they were chosen or showing a sweep. A figure showing speedup vs. d at fixed E (analogous to the acceptance-rate analysis in speculative decoding literature) would clarify the trade-off.

• Training overhead quantification. The paper mentions that early exit loss "may introduce an overhead for pretraining" (Section 9) but never quantifies it. How much slower is training with LayerSkip compared to standard training, wall-clock time? The rotational curriculum is designed to minimize overhead, but the actual cost is never stated.

• Dynamic exit evaluation. The paper shows early exit accuracy at each layer (Figures 6, 8, 10) but never evaluates a dynamic policy that selects E per token. Even a simple baseline — exit when the predicted token at layer E matches the predicted token at layer E+1 for some number of consecutive layers — would provide a baseline for what speedups are possible with adaptivity.

Do the results support the paper's central framing as an "end-to-end solution"?

The paper positions LayerSkip as an end-to-end solution spanning training and inference (Section 1, Figure 1). The evidence partially supports this framing: the training recipe demonstrably improves early exit capability, and the self-speculative decoding algorithm demonstrates speedups. However, the "end-to-end" claim implies a degree of integration and automation that the current results fall short of:

• The hyperparameters (p_max, e_scale, R, E, d) require per-task, per-model tuning. There is no automated procedure for selecting them, and the paper's choices are justified post-hoc rather than algorithmically.
• The approach is evaluated on models the authors trained themselves with the recipe; there is no demonstration that the recipe can be applied to an existing pretrained model downloaded from a public repository without access to the original training pipeline. The continual pretraining experiments (Figures 6, Tables 1–2) require starting from a pretrained checkpoint and training on additional data with the LayerSkip modifications — this is not a post-hoc application to a frozen model.
• The self-speculative decoding implementation requires custom inference code (KVQ cache management, separate forward_early and forward_remainder functions), which is a non-trivial engineering effort beyond standard model serving frameworks.

These are not fundamental flaws — they are characteristic of research-stage systems — but the "end-to-end solution" language overstates the current level of maturity. The paper would be more accurately described as demonstrating the feasibility of an integrated training-inference approach, with practical deployment requiring additional engineering and automation.

The training recipe's scalability to large-scale pretraining is unproven

The most significant open question is whether the LayerSkip training recipe can be applied during the full pretraining of a frontier-scale model (70B+ parameters, 2T+ tokens) without unacceptable accuracy degradation. The Llama3 continual pretraining results (Table 2) are concerning: final-layer accuracy dropped substantially (MMLU -6 points, HumanEval -9 points, GSM8K -9 points) despite using a low e_scale of 0.1. If this degradation scales with model size or pretraining data volume, it would be unacceptable for production models where every point of benchmark accuracy matters. The pretraining-from-scratch experiments at 26B tokens are too small (by two orders of magnitude) to address this question.

The paper's response would likely be that the hyperparameters can be tuned to reduce final-layer degradation (lower p_max, lower e_scale, sparser curriculum), but this tuning itself requires running multiple expensive pretraining experiments — a cost barrier that may limit adoption. The paper does not provide guidance on how to select hyperparameters for a new model scale without extensive trial and error, which is a practical limitation for the "end-to-end" ambition.

6. Limitations and Trade-offs

Limitation 1: The Training Recipe Cannot Be Applied Post-Hoc to Existing Pretrained Models

The assumption or constraint. LayerSkip's training recipe — layer dropout with exponential per-layer scaling combined with curriculum-scheduled early exit loss — must be applied during training. The paper explicitly acknowledges in Section 5 that all models evaluated with LayerSkip were trained from scratch with the recipe, continually pretrained from a base checkpoint with the recipe, or finetuned with the recipe. For the baseline catastrophic collapse in early exit accuracy (Tables 1–2, Figure 6), standard pretrained models without LayerSkip produce near-zero exact match on generation tasks when exiting at middle layers — NaturalQuestions drops from 25.1% to 0% on Llama2 7B baseline at layer 16. The self-speculative decoding speedup depends entirely on the draft model (early exit) having non-trivial token acceptance; without the training recipe, the draft stage's predictions would be rejected so frequently as to eliminate any speedup.

The consequence. Any practitioner or organization using an off-the-shelf pretrained model (e.g., Llama2 7B from HuggingFace) cannot simply apply LayerSkip's inference algorithms to their existing model and expect meaningful acceleration. They must either (a) have access to the full pretraining pipeline to train a model from scratch with LayerSkip, (b) invest the compute to continually pretrain the off-the-shelf model on substantial additional data (52B tokens for Llama2 7B, 419B for Llama3 8B, 839B for Llama3.2 1B — Table 8) with the LayerSkip recipe, or (c) restrict themselves to narrow-domain finetuning where the recipe can be applied to a task-specific dataset. This is a fundamentally different adoption barrier than inference-only techniques like quantization or standard speculative decoding with a separate draft model, which operate on frozen pretrained weights.

What evidence exists in the paper. The paper never evaluates self-speculative decoding on a model that was not trained with LayerSkip. The comparison against Draft & Verify (Zhang et al., 2023) in Table 3 is confounded: Draft & Verify applies to unmodified models (by skipping intermediate layers), while LayerSkip requires model modification. The baseline early exit results (Figure 6, Figure 10a-b) implicitly demonstrate that unmodified models produce useless draft tokens, but the paper never runs the full self-speculative loop on an unmodified baseline to quantify just how low the speedup would be (presumably close to 1.0× if acceptance rates are near 0%). The continual pretraining data requirements themselves (Table 8) are substantial: 52B additional tokens for Llama2 7B is approximately equivalent to the entire pretraining budget of many mid-scale models, making "continual pretraining" a non-trivial training investment, not a lightweight adaptation step.

Mitigation status. The paper does not attempt to address this limitation. Section 9 acknowledges that LayerSkip's self-speculative decoding "requires finetuning a model or pretraining it with our recipe, while the self-speculative decoding approach proposed in Zhang et al. (2023) does not require changing a model's weights." This is presented as a straightforward trade-off: Zhang et al. sacrifices cache reuse for compatibility with arbitrary pretrained models, while LayerSkip requires training investment but achieves better per-token speedup through cache sharing. The paper leaves unresolved whether a hybrid approach — applying LayerSkip's training recipe but only during a lightweight finetuning phase rather than full (continual) pretraining — could achieve acceptable draft quality at lower training cost, though the task-specific finetuning results (TOPv2, Figure 10b) are encouraging in this direction for narrow domains.

Limitation 2: Final-Layer Accuracy Degradation Is Not Consistently Controlled, and the Degradation Grows with Pretraining Scale

The assumption or constraint. The LayerSkip training recipe modifies the loss landscape by redistributing gradient signal from the final layer across all intermediate layers. The paper assumes this can be done with minimal impact on final-layer quality through careful tuning of p_max, e_scale, and the curriculum schedule. Section 4.1.1 states that the rotational curriculum was designed to prevent early exit loss from "reducing the accuracy of the last layer," and the hyperparameter e_scale is explicitly positioned as controlling the trade-off between early-layer and final-layer accuracy.

The consequence. In practice, final-layer accuracy degradation appears consistently across experiments, and the severity varies unpredictably with model family and pretraining scale. The most concerning case is Llama3 8B continual pretraining (Table 2): MMLU drops from 66.5% to 60.5% (−6.0 percentage points), HumanEval from 37.8% to 28.7% (−9.1 points), GSM8K from 54.2% to 45.0% (−9.2 points), and MATH from 17.3% to 12.3% (−5.0 points). These are substantial regressions that would be considered unacceptable in a production model where benchmark scores directly translate to perceived capability. Even in the better-behaved Llama2 7B continual pretraining (Table 1), MMLU drops from 46.0% to 43.1% (−2.9 points) and GSM8K from 14.3% to 12.2% (−2.1 points). The task-specific finetuning on TOPv2 (Table 6) shows a consistent 3-percentage-point gap (85.9% → ~83% at multiple exit layers) that self-speculative decoding does not close.

The fundamental concern is that this degradation appears to scale with pretraining data volume. The authors themselves note in Section 6.1 that Llama3's much higher baseline middle-layer perplexity (110,000 vs. 1,900 for Llama2 at Wikipedia) makes "reducing such high perplexity of earlier layers while maintaining accuracy of last layer more challenging." If this trend continues to even larger models (70B, 405B) trained on trillions of tokens, the final-layer accuracy cost of applying LayerSkip during pretraining could become prohibitive. The paper provides no method for predicting or bounding this cost for a new model scale without running the full training experiment.

What evidence exists in the paper. Tables 1 and 2 provide the direct evidence for final-layer accuracy comparisons across multiple tasks and model scales. The Llama3 results in Table 2 are the most pronounced negative case. The paper's ablation in Section 7 (Figure 11) provides mechanistic insight — middle-layer perplexity diverges as pretraining progresses, and early exit loss can suppress this divergence — but does not provide a solution for the associated final-layer cost. The paper never ablates whether training for longer with LayerSkip (more tokens of continual pretraining) would recover the lost final-layer accuracy while maintaining early-layer gains, or whether the degradation is permanent under the modified loss function.

Mitigation status. The paper partially acknowledges the issue (Section 6.1: "This could be a motivation to consider our LayerSkip recipe in pretraining future LLMs from scratch"), implicitly suggesting that applying the recipe from the beginning of pretraining might avoid the conflict between early and late layer optimization. However, the pretraining-from-scratch experiments (Figure 8, 26B tokens) are at too small a scale to test this hypothesis — they cannot distinguish between "no degradation" and "degradation that would emerge at 2T tokens." The paper mentions that hyperparameters require tuning "to avoid a drop in last layer accuracy" (Section 8), but provides no systematic methodology for this tuning beyond the ad-hoc per-experiment choices in Appendix Table 9. The fundamental trade-off — that early exit supervision and final-layer optimality pull in opposite directions — remains unresolved.

Limitation 3: Speedup Comparisons Are Confounded by Missing Baselines and Training-Hardware Mismatches

The assumption or constraint. The paper's central speedup claims (1.34×–2.16×) are measured against an autoregressive baseline that uses the original pretrained model (without LayerSkip training), while the self-speculative decoding and early exit experiments use LayerSkip-trained models. This creates a confound: any accuracy differences between the baseline and LayerSkip models (whether improvements or degradations, as discussed in Limitation 2) mean the speedup is measured between models of different quality, not between equivalent-quality models with different inference strategies.

More critically, the paper omits the most natural comparative baselines for self-speculative decoding:

• Standard two-model speculative decoding with a separate smaller draft model (Leviathan et al., 2023; Chen et al., 2023) — the dominant lossless acceleration method. The paper argues qualitatively that LayerSkip reduces memory footprint (Section 1, Section 4.3), but never quantifies this advantage in a controlled comparison at equivalent total parameters, equivalent memory usage, or equivalent speedup.
• Naive early exit within self-speculation on the unmodified baseline model — what speedup would you get by using the baseline model for drafting (with its terrible middle-layer accuracy) and full model for verification? This would quantify the "cost" of not applying the training recipe and contextualize how much of the speedup comes from the recipe vs. from the self-speculative algorithm itself.

The consequence. The headline speedup numbers conflate three distinct effects: (1) the self-speculative decoding algorithm's efficiency, (2) the draft model's accuracy improvement from the training recipe, and (3) the final-layer accuracy change from the training recipe (which affects the "target" accuracy the self-speculative system is trying to match). A practitioner evaluating whether to adopt LayerSkip cannot separate these effects to estimate what speedup they would achieve on their own model. If the training recipe degrades their model's final-layer accuracy by several points (as in Llama3, Table 2), they face a choice between a slower-but-accurate baseline and a faster-but-degraded LayerSkip model — a very different value proposition than the paper's implicit framing of "same accuracy, more speed."

The Draft & Verify comparison (Zhang et al., 2023) in Table 3 is the closest the paper comes to a competitive baseline, but it is limited to two tasks (CNN/DM and XSUM) on one model size (Llama2 7B) and shows mixed results: LayerSkip is faster on CNN/DM (1.81× vs. 1.56×) but at slightly lower ROUGE-2 (0.078 vs. 0.079), and Draft & Verify is slightly faster on XSUM (1.48× vs. 1.34×). From two close data points, no general superiority can be established.

What evidence exists in the paper. Table 3 presents the speedup comparisons. The confound is visible in the numbers: the autoregressive baseline uses the original pretrained model, while the "Self Speculative" row uses the LayerSkip-trained model — these are different models with potentially different accuracies. The paper reports ROUGE-2 for all configurations, but speedup is always calculated relative to the original model's autoregressive throughput, not relative to the LayerSkip-trained model's full-depth throughput (which might differ due to the training modifications). Table 6 for TOPv2 makes this explicit: self-speculative decoding achieves only 82.9% EM vs. 85.9% for autoregressive, so the 2.0× speedup comes with a 3-point accuracy cost that the paper does not factor into the speedup calculation.

Mitigation status. Not addressed. The paper does not run a two-model speculative decoding baseline, does not report self-speculative decoding throughput against a LayerSkip-trained autoregressive baseline (which would isolate the inference-time algorithm's contribution from the training-time recipe's effect), and does not acknowledge the confound in its speedup calculation methodology. This is a significant methodological weakness that limits the strength of the paper's comparative claims.

Limitation 4: Hyperparameter Sensitivity and the Absence of a Selection Methodology Create a Deployment Barrier

The assumption or constraint. The LayerSkip training recipe introduces at least five new hyperparameters — p_max, S(t) choice, e_scale, C(t,l) curriculum type, and R (rotational dilation) — plus the inference-time choices of exit layer E and number of speculations d. The paper demonstrates that these interact in non-obvious ways: higher e_scale improves early exit accuracy but risks final-layer degradation (Tables 1–2 vs. Figure 10b, where e_scale = 1.0 works for narrow-domain finetuning but would likely be catastrophic for diverse pretraining); the dropout schedule choice (exponential vs. constant) significantly affects training loss (Figure 12, where constant dropout at the same average rate is substantially worse); and the optimal E for self-speculative decoding varies by task and model size (Tables 3–6, where E ranges from 6 to 18 for comparably-sized models depending on the training regime).

The consequence. A practitioner wanting to apply LayerSkip to a new model, dataset, or task must discover these hyperparameters through expensive trial and error. Each candidate configuration requires a full training run (continual pretraining or finetuning) followed by inference evaluation at multiple exit layers. For large-scale pretraining, this is cost-prohibitive: a single Llama2 7B continual pretraining run used 64 A100 GPUs for 50,000 steps (Table 8), and exploring even 5–10 hyperparameter combinations would require hundreds of thousands of GPU-hours. The paper provides no transfer learning insights — whether optimal hyperparameters for Llama2 7B transfer to Llama2 13B, or whether e_scale = 0.1 found for Llama3 8B would work for Llama3 70B. The authors' own hyperparameter choices vary substantially across experiments (Section 5, Table 9) without clear rules, suggesting that each new model scale required independent tuning.

Furthermore, the inference-time choice of E and d creates a combinatorial search problem: for each exit layer (which could be any integer from 1 to L−1) and each speculation count, a separate speedup measurement is needed. The paper reports point evaluations at a small number of hand-chosen combinations (typically one per task per model), but there is no evidence that these are Pareto-optimal or that a simpler selection heuristic could substitute for brute-force search.

What evidence exists in the paper. The hyperparameter variations across experiments are documented in Section 5 and Appendix Table 9: p_max ∈ {0.1, 0.2}, e_scale ∈ {0.1, 0.2, 1.0}, R ∈ {8, 16, 23, 31, 39}, and E varies from 6 to 18 across tasks. The paper never ablates any of these parameters systematically — no sweep over e_scale at fixed p_max, no comparison of two R values on the same model, no sensitivity analysis showing how final-layer accuracy degrades as p_max increases. The rotational vs. gradual curriculum choice is tested only across experiments (rotational for most, gradual for TOPv2), never on the same task. Section 8 explicitly acknowledges that hyperparameters "require tuning in order to avoid a drop in last layer accuracy" and that "tuning learning rate to get optimal accuracy could be tricky and time consuming."

Mitigation status. Minimally mitigated. The paper provides the hyperparameter values they used for each experiment (Tables 8–9), which serves as a starting point for replication, but offers no methodology for adapting these values to new settings. Section 9 gestures at future work on "dynamic conditions to determine a different exit layer for each token," which would eliminate the need to pre-select E, but does not develop this capability. The rotational curriculum is presented as reducing the sensitivity to the R parameter (since even very sparse rotation, e.g., R=39, worked for Llama2 13B), but this is an empirical observation rather than a principled solution.

For a practitioner, this limitation means that adopting LayerSkip is not a matter of "add these flags to your training config" but rather "budget for multiple training runs to discover viable hyperparameters for your specific model and task" — a substantially higher adoption cost than the paper's presentation suggests.

Limitation 5: The Approach Has Only Been Demonstrated on a Single Model Family (Llama) and No Evidence Supports Transfer to Architecturally Different Models

The assumption or constraint. All experiments in the paper use Llama-family decoder-only transformer models: Llama1 7B, Llama2 7B/13B, Llama3 8B, Llama3.2 1B, and custom Llama-like architectures (Table 10). These models share the same basic architectural pattern — pre-norm residual transformers with SwiGLU FFNs, RoPE positional embeddings, and grouped-query attention — and differ primarily in scale (depth, width) rather than structural design choices. The paper states (Section 5) that it tests "different Llama model sizes on different types of training" to demonstrate generality, but this generality is within a single architectural lineage.

The consequence. It is unknown whether the LayerSkip training recipe would work for models with fundamentally different architectures. Several design choices could interact with alternative architectures in ways that break the approach:

• Mixture-of-Experts (MoE) models: where different tokens route through different subsets of layers or experts. The layer dropout mechanism assumes a fixed sequential layer structure; if tokens already skip layers via routing, the dropout might either be redundant or destabilizing.
• Encoder-decoder models (T5, BART): where the encoder processes the full input in parallel and only the decoder is autoregressive. The paper's approach modifies only the autoregressive decoder layers; the interaction between early exit in the decoder and cross-attention to encoder states is unexplored.
• Models without residual connections around transformer blocks, or with different normalization placement (post-norm vs. pre-norm): the layer dropout formulation x_{l+1} = x_l + M(p)f_l(x_l) relies on the residual connection to pass embeddings through when a layer is dropped. Without residuals, dropped layers would zero out the representation entirely.
• Models with tied input-output embeddings: where the LM head shares weights with the embedding layer. The early exit loss forces intermediate representations into the space of the LM head, which is also the embedding space in tied-weight architectures — this could have different dynamics than in Llama where embeddings and LM head are separate.

Even within the Llama family, the results show significant variation in LayerSkip's effectiveness: Llama3 models (Table 2) show worse final-layer degradation than Llama2 models (Table 1), which the paper attributes to Llama3's larger pretraining data volume. This suggests that the approach may be sensitive to factors (pretraining scale, data mixture, training hyperparameters) that vary across model families even when the architecture is nominally similar.

What evidence exists in the paper. The paper's entire experimental corpus is Llama-family models (Table 10). The same training recipe is applied across these models, but the hyperparameters vary and the outcomes differ. The paper does not report any experiments on non-Llama architectures, nor does it cite or discuss work applying similar techniques to other model families. The related work section (Section 3) discusses early exit in encoder-only models (BERxiT: BERT), encoder-decoder models (Elbayad et al., 2020: translation models, CALM: encoder-decoder), and decoder-only models (SkipDecode), but LayerSkip's specific combination of layer dropout + early exit loss + shared LM head is only tested on Llama.

Mitigation status. Not addressed. The paper makes no claims about architectural generality, and the limitation is acknowledged only implicitly through the scope of experiments. The title and abstract describe the approach as a general solution for "large language models" without qualification, which overstates the evidence base. A cautious reader should treat the results as validated for Llama-family decoder-only transformers at 1.5B–13B scale, with unknown transfer to other architectures.

Limitation 6: The Approach Introduces a Tension Between Accuracy and Speedup That Cannot Be Resolved Within the Current Framework for Hard Problems

The assumption or constraint. The self-speculative decoding speedup depends on the draft model (early exit at layer E) having high token acceptance rates relative to the full model. The paper's speedup model (Section 4.3.2) implies that per-token cost is approximately E + (L−E)/α where α is the acceptance rate. For fixed E, as α decreases, speedup drops because verification cost is amortized over fewer accepted tokens. For difficult tasks or tokens where the early exit prediction is less reliable, α will be lower, reducing or eliminating the speedup.

The paper demonstrates this relationship implicitly across its experiments: tasks and exit layers with lower acceptance rates produce lower speedups. For coding (Table 5), α = 45% yields 1.82× speedup; for TOPv2 at E=6 (Table 6), α = 76% yields 2.0× speedup; for TOPv2 at E=18, α = 98.9% yields only 1.24× speedup. The trade-off is clear: exiting earlier (smaller E) reduces draft cost but also reduces α, creating an optimal E that depends on task difficulty.

The consequence. The LayerSkip framework provides no mechanism for dynamically adapting E or d to per-token difficulty. A token that is genuinely hard (requiring the full model's depth to predict correctly) will be drafted incorrectly by the early exit and rejected during verification, wasting the draft computation. A token that is easy will be drafted correctly but could have been drafted from an even earlier layer (with even lower compute cost). The static choice of E and d — one value for all tokens in all inputs for a given task — means the system operates at the average-case optimal point, leaving per-token efficiency on the table for both easy and hard tokens.

This is the direct analog of the "difficulty-dependent" limitation identified in the example paper's analysis (test-time compute scaling paper, Limitation: "Hard Problems Remain Essentially Unsolved"). Just as that paper's compute-optimal strategies provided no benefit on the hardest difficulty quintile because the base model couldn't produce correct answers regardless of compute allocation, LayerSkip's self-speculative decoding provides diminishing (or zero) speedup on tasks where early exit accuracy is so low that α approaches 0, because the verification cost (L−E)/α diverges. The paper does not identify what fraction of tokens or what types of tasks fall into this regime, nor does it provide a diagnostic for predicting when self-speculation will fail to accelerate.

What evidence exists in the paper. The acceptance rate variation across experiments is the primary evidence: α ranges from 45% (code, Table 5) to 98.9% (TOPv2 E=18, Table 6). The paper does not break down acceptance rates by token position within a sequence, by input difficulty, or by any other conditioning variable — the reported α is an average across all tokens in the test set. If α varies substantially across tokens (e.g., near 100% for common function words but near 0% for semantically critical content words), the average could mask bimodal behavior where some tokens are trivially accelerated and others provide no benefit, making the average speedup less representative of the per-token experience. The paper also never evaluates whether there exist tokens or tasks where α is so low that self-speculative decoding is slower than autoregressive (due to the overhead of rejected drafts), which would define a failure regime for the method.

Mitigation status. The paper explicitly identifies dynamic exit as future work in Section 9: "We can also explore dynamic conditions to determine a different exit layer for each token (like Schuster et al. (2022)) and hence improve token acceptance rate of self-speculative decoding." This acknowledgment confirms that the current static configuration leaves efficiency gains unrealized, particularly for heterogeneous inputs where token difficulty varies. However, no experiments or prototypes of dynamic exit are presented, and the challenge is non-trivial: a dynamic exit policy would need to predict, before generating a token, whether an early exit will produce the correct prediction — a meta-cognitive capability that may itself require additional model components or training objectives beyond those in the current recipe. The limitation is therefore acknowledged but entirely unresolved by the current work.

7. Implications and Future Directions

How This Work Changes the Landscape

LayerSkip introduces a training-inference co-design perspective to LLM acceleration that is qualitatively different from the dominant post-hoc optimization paradigm. The field currently treats model training and inference optimization as sequential, independent phases: train a model for the best possible accuracy on next-token prediction, then apply quantization, pruning, or speculative decoding to accelerate the frozen model at deployment. LayerSkip inverts this — it asks what training objective would produce a model whose inference-time operational characteristics are inherently favorable, then designs the training recipe to satisfy those constraints. This is not an incremental refinement of existing acceleration techniques; it is a methodological shift in how we think about the relationship between training and deployment.

The magnitude of this shift should not be overstated. The technical components — layer dropout, early exit loss, speculative decoding — are individually well-established. The contribution is their specific combination and the empirical demonstration that this combination can achieve practical speedups (1.34×–2.16×) while maintaining accuracy on par with autoregressive decoding. This is more akin to the introduction of compute-optimal pretraining scaling laws (Hoffmann et al., 2022) than to the invention of the transformer — it changes how we allocate effort in the model development pipeline rather than introducing a fundamentally new capability. The finding that intermediate layers in standard LLMs become catastrophically worse at token prediction as pretraining scales (Figure 11, where middle-layer perplexity increases from ~0.6 to over 600 across training) provides a concrete diagnostic: current pretraining recipes are actively hostile to efficient inference, and this hostility compounds with scale.

Perhaps the most consequential reframing the paper enables is a shift in how the field evaluates model quality. Today, models are judged almost exclusively by their final-layer output quality — perplexity, benchmark accuracy, generation quality. LayerSkip demonstrates that this metric is incomplete: two models with identical final-layer accuracy can have radically different computational properties at inference time, because one might require all its layers for every token while another can produce useful predictions from early layers. If the training recipe were adopted as a standard component of pretraining, early exit accuracy at multiple depths could become a supplementary evaluation metric alongside final-layer quality, analogous to how parameter count and FLOPs are reported alongside accuracy today. A model that achieves 95% of its final-layer accuracy at 50% depth would be strictly more valuable for deployment than one that collapses to near-zero at the same depth, even if both score identically on standard benchmarks. The paper's results on classification tasks (Table 1, where baseline MMLU drops from 55.2% to 49.2% at middle layers vs. LayerSkip maintaining much of the accuracy) suggest that this kind of depth-efficiency varies substantially across tasks and models, making it a useful diagnostic dimension.

The paper also partially resolves a tension in the speculative decoding literature. Prior work presented a binary choice: use two separate models and accept the memory overhead (Leviathan et al., 2023; Chen et al., 2023), or use a single model with skipped intermediate layers and accept the cache inefficiency (Zhang et al., 2023). LayerSkip demonstrates a third path — a single model where the draft and verification stages share compute via contiguous layer reuse — but reveals that this path requires a specific training investment. The resolution is therefore not "self-speculation is better" but "self-speculation with cache reuse is better conditional on training the model appropriately," which reframes the problem from inference-only algorithm design to training-inference system design. This makes the speculative decoding literature's training-aware branch more attractive (e.g., training dedicated draft heads, distillation-based draft models) while making inference-only approaches seem like an incomplete solution for memory-constrained deployment.

A negative reframing that the paper's evidence supports: standard LLM pretraining is wasteful in a way that scales with data. The scaling experiment in Figure 11 — where longer training makes intermediate layers worse at prediction — suggests that as the field pours ever-larger compute budgets into pretraining, we are simultaneously making models more accurate and more computationally inefficient at inference. LayerSkip offers one antidote, but the underlying dynamic (representations at different depths diverging as the model specializes) is a fundamental property of depth-constrained optimization that the paper diagnoses but does not fully explain. This opens a broader research question: what other hidden inefficiencies are being baked into models during standard pretraining that could be addressed through modified objectives?

Follow-Up Research This Work Enables

Dynamic per-token exit policies with lightweight meta-predictors. The paper's most obvious open problem is the static nature of the exit layer E. Every token in every sequence exits at the same layer, despite the motivating observation in Figure 2b that "most of the time, the final token prediction is predicted fewer layers before the end" — implying substantial variation in when predictions stabilize. The paper explicitly flags this as future work (Section 9), and the infrastructure to pursue it is now available: the training recipe produces models where exit quality at each layer is meaningful (Figures 6, 8, 10), so a meta-predictor trained to estimate confidence or difficulty at each layer could decide dynamically whether to exit or continue. A concrete experiment: train a lightweight linear classifier on top of each layer's hidden states (or on the PRM-style confidence of the top-1 prediction) to predict whether the current layer's argmax matches the final layer's argmax for that token. During inference, exit when the classifier's confidence exceeds a threshold. Measure the speed-accuracy Pareto frontier of this dynamic policy against the static-E baseline on a task like CNN/DM summarization. The key challenge is the predictor's overhead — if it requires a full forward pass through an auxiliary network per layer, it could negate the savings from exiting early. A successful system would need to amortize the predictor cost by using extremely lightweight signals (e.g., the entropy of the predicted token distribution at each layer, which is already computed).

Combining LayerSkip training with knowledge distillation for draft model quality. The paper's training recipe makes early exit predictions possible where they were previously impossible, but the absolute accuracy remains low — HumanEval pass@1 at layer 8 is 4.88% vs. 15.9% for the full Llama2 7B model (Table 1). This limits self-speculative speedup because low acceptance rates (45% on code, Table 5) mean many draft tokens are rejected. A natural extension is to add a distillation objective during training: the early exit predictions at each layer are trained to match not just the ground-truth tokens but also the full model's final-layer output distribution, via a KL divergence loss between g(x_E) and g(x_L). This would give intermediate layers a richer training signal than the one-hot cross-entropy used in the current early exit loss, potentially closing the gap between early exit and full-model accuracy. A concrete protocol: add a distillation term α · KL(g(x_L) || g(x_E)) to the loss in Equation 5, where α is a scaling coefficient swept in {0.1, 0.5, 1.0}. Evaluate whether this improves token acceptance rates in self-speculative decoding on the HumanEval and CNN/DM tasks where the paper provides baseline acceptance numbers (67–69% for summarization, 45% for coding). The risk is that distillation could homogenize layer representations, reducing the model's ability to use depth for genuinely hard tokens — an ablation showing per-token accuracy as a function of depth would be essential.

LayerSkip applied to mixture-of-experts architectures for compound efficiency gains. The paper evaluates only dense Llama models, but the exponential layer dropout mechanism has an intriguing synergy with mixture-of-experts (MoE) architectures: in an MoE model, each token already routes through a subset of available FFN experts within each layer, meaning compute is already unevenly distributed across tokens per layer. LayerSkip's layer-level skipping could compound this: easy tokens skip entire layers, hard tokens execute all layers but with sparse expert activation. The training recipe would need modification — layer dropout would skip entire MoE layers (including the gating mechanism), and the early exit loss would need to account for expert load balancing objectives already present in MoE training. A concrete experiment: take a pretrained Mixtral-style MoE model (e.g., 8×7B, 32 layers with 8 experts per FFN layer), apply continual pretraining with LayerSkip (following the Llama2 7B protocol: 52B tokens, p_max=0.1, e_scale=0.2, C_rot,R=8), and measure the compound speedup from expert sparsity + layer skipping in self-speculative decoding on the same CNN/DM and HumanEval tasks. If the speedups are multiplicative (e.g., 2× from MoE sparsity × 1.8× from LayerSkip = ~3.6× total), this would make a compelling case for deploying both techniques together on memory-constrained hardware.

Stress-test: does the training recipe scale to 70B+ models without unacceptable accuracy degradation? The most critical empirical question the paper leaves open is whether LayerSkip works at the scale of frontier models. The Llama3 8B results (Table 2, showing MMLU degradation of −6 points and HumanEval of −9 points) are concerning and suggest the recipe may not transfer unmodified to larger models with more pretraining data. A decisive experiment — though expensive — would be to apply continual pretraining with LayerSkip to a Llama2 70B model, varying e_scale in {0.05, 0.1, 0.2} and p_max in {0.05, 0.1}, and measuring both final-layer benchmark accuracy (MMLU, HumanEval, GSM8K, as in Table 1) and early exit accuracy at multiple depths (25%, 50%, 75% of total layers). If there exists a hyperparameter setting that preserves final-layer accuracy within 1–2 points while achieving non-trivial early exit accuracy (e.g., >50% of full-model accuracy at 50% depth), this would validate the approach for production-scale models. If no such setting exists — if the accuracy-speed trade-off is fundamentally worse at scale — this would establish an important boundary condition on the method's applicability. The paper's pretraining-from-scratch experiments at 26B tokens (Figure 8) are too small to distinguish these outcomes; only a large-scale experiment can resolve the question.

Reconciling LayerSkip with other early exit methodologies through a controlled comparison. The paper positions itself against prior early exit work that uses per-layer LM heads (Elbayad et al., 2020; Schuster et al., 2022) by arguing that the shared head reduces memory and deployment complexity. But it never empirically compares shared vs. per-layer heads under otherwise identical training conditions. A clean ablation would: take the same Llama2 7B base model, apply the same layer dropout schedule and early exit loss, but in one condition use the shared head as in the paper, and in another condition train separate LM heads for each layer (initialized from the shared head and fine-tuned). Measure early exit accuracy at each layer on the full benchmark suite from Table 1. The hypothesis: per-layer heads improve early exit accuracy (each head specializes) but at higher memory cost, while the shared head acts as a regularizer that forces representational alignment. Quantifying this trade-off — accuracy gain per KB of additional LM head memory — would help practitioners decide which approach to adopt for their deployment constraints.

Diagnosing why intermediate layer representations diverge during standard pretraining. The paper's most intriguing diagnostic (Figure 11 — middle-layer perplexity increases with training tokens) is presented as motivation but never explained. A dedicated mechanistic study could trace this phenomenon: use representational similarity metrics (CKA, centered kernel alignment) between layers at different training checkpoints to measure when and how layer representations diverge. The hypothesis: in early training, gradients from the final layer propagate backward and align all layers' representations toward the output space; as training progresses and the model specializes, later layers learn to transform earlier-layer representations in ways that are useful for the next layer but degrade direct unembedding quality. If this hypothesis is correct, it predicts that the divergence should be most pronounced in models with more layers (deeper networks have more opportunity for gradual representational drift) and in models trained with larger learning rates (faster specialization). Testing this with controlled pretraining runs at different depths and learning rates, measuring CKA between each layer's output and the final layer's output at multiple training checkpoints, would turn the motivational observation into an explained phenomenon and potentially inspire new training objectives that prevent representational collapse without the overhead of full early exit loss.

Practical Applications and Downstream Use Cases

On-device LLM deployment for interactive applications. The most directly actionable use case is deploying LLMs on edge devices (laptops, phones) where memory and compute are tightly constrained. The self-speculative decoding algorithm eliminates the need for a second draft model — a critical advantage when total model memory is the binding constraint. For a concrete scenario: deploying a Llama2 7B model quantized to 4 bits (~3.5 GB) on a laptop with 8 GB of GPU memory. Adding a separate draft model (even a small one at ~500M parameters) would consume additional memory, potentially exceeding the budget. LayerSkip's approach uses the same model weights and a single KV cache (plus the small exit query cache, ~4 KB per token at 4096 dimensions), staying within the memory envelope while achieving 1.54×–1.86× speedup on summarization tasks (Table 3). The practical benefit: on a laptop GPU, reducing CNN/DM summarization latency from ~1 second to ~550 ms per 512-token generation makes the difference between an assistant that feels responsive and one that feels sluggish. The trade-off is the need to continually pretrain the model with LayerSkip rather than using an off-the-shelf checkpoint — a one-time cost amortized over all deployments using that model.

Batch inference for cost-sensitive cloud deployments. For cloud inference serving multiple users, the primary cost driver is GPU-hours per query. LayerSkip's speedups translate directly to cost reduction: a 1.83× speedup on HumanEval coding tasks (Table 3) means that a cluster of H100 GPUs can serve 83% more coding completion requests per hour, proportionally reducing per-query infrastructure cost. This is particularly valuable for coding assistants (e.g., GitHub Copilot-style inline completion) where latency tolerance is moderate (users expect ~200–500 ms) but cost-per-completion directly impacts the service's economic viability. The self-speculative approach provides these speedups without the operational complexity of deploying and versioning a separate draft model, which reduces engineering overhead in production systems. The caveat: the paper evaluates only single-sequence throughput; batched inference with continuous batching may show different speedup characteristics, and a production deployment would need to benchmark the algorithm under realistic serving loads with variable sequence lengths and request arrival patterns.

Fine-tuning for specialized assistants with built-in acceleration. The task-specific finetuning results on TOPv2 (Table 6, 2.0× speedup at 82.9% EM) suggest a pattern where an organization with a narrow-domain task can finetune an existing model with LayerSkip to get both task specialization and inference acceleration in one step. For a company building a customer-support chatbot that needs to parse structured queries from natural language (the semantic parsing use case), applying LayerSkip during finetuning simultaneously adapts the model to the company's query patterns and makes it faster to serve. The 2.0× speedup reduces the GPU fleet needed to serve peak query volume by half, which for a mid-size deployment (e.g., 10 GPUs serving 1000 queries/minute) translates to ~$15K–30K/year in cloud savings at typical GPU rental rates. The trade-off is the 3-point accuracy regression (85.9% → 82.9% EM) relative to standard finetuning, which may or may not be acceptable depending on the downstream error tolerance.

Pretraining recipe for next-generation foundation models. At the largest scale, the paper's diagnostic (Figure 11 — intermediate layer collapse worsens with pretraining scale) and the pretraining-from-scratch results (Figure 8, despite being at only 26B tokens) suggest that incorporating LayerSkip's training components into the initial pretraining of foundation models could produce models that are inherently more efficient at inference without requiring a separate continual pretraining phase. If a future Llama4-scale model were pretrained from scratch with exponential layer dropout and curriculum-scheduled early exit loss, it would emerge from pretraining with two properties: (1) comparable final-layer accuracy to standard pretraining (if hyperparameters are tuned correctly, though the Llama3 results in Table 2 raise doubts), and (2) the ability to serve as its own draft model for self-speculative decoding with no additional training investment. For a model trained on 15T tokens at a cost of tens of millions of dollars, even a 1.5× inference speedup across all downstream deployments would recover substantial value. The risk — and the reason no one has done this — is that if the training recipe degrades final-layer accuracy in ways that cannot be recovered through hyperparameter tuning at scale, the entire pretraining investment is compromised. The paper does not de-risk this enough for production adoption, but it provides the experimental template for a team willing to run the necessary scaling experiments.