An Efficient Inference Framework for Early-exit Large Language Models

ArXiv: 2407.20272

🎯 Pitch

Skipping layers in early-exit LLMs actually breaks standard inference engines—but this paper’s trick of fabricating KV cache entries for skipped layers unlocks up to a 1.25× throughput gain and slashes per‑token latency by over 3×.

1. Executive Summary

This paper introduces an efficient inference framework for early-exit LLMs that extends iteration-level batch scheduling to accommodate models that skip decoder layers when confident. The system targets CALM — an early-exit variant of T5 — and addresses two mechanisms: batch inference at iteration-level granularity (processing all sequences in the batch until every sequence surpasses its early-exit confidence threshold, rather than processing a fixed number of layers) and KV cache management for skipped layers (filling the key-value cache of unexecuted layers using the saturated hidden state from the exit layer before the iteration terminates). Implemented on vLLM with three early-exit confidence techniques — softmax response, hidden states similarity, and a dedicated early-exit classifier — the framework achieves up to 1.25× token generation throughput improvement over full-layer vLLM (1081 to 1166 tokens/s vs. 1046 tokens/s for T5-v1.1-small) and up to a 3.39× reduction in inner-token latency, establishing that early-exit inference frameworks can substantially outperform standard full-layer serving infrastructure only when the KV cache of skipped layers is explicitly filled to enable correct attention computation in subsequent iterations.

2. Context and Motivation

The Core Gap: Inference Systems Are Designed for Static-Layer Models, Not Dynamic Early-Exit Models

The paper addresses a straightforward but underexplored mismatch: LLM inference serving systems are optimized for models that execute every layer for every token, while early-exit LLMs — which dynamically skip layers during decoding — lack any serving infrastructure that exploits their variable-depth computation. This gap exists not because early-exit models are hypothetical. They have been extensively studied at the model level for years, starting with depth-adaptive transformers (Elbayad et al., 2019), and have been shown to maintain output quality while substantially reducing the average number of layers executed per token. Yet, as the authors state explicitly:

"there is no work of LLM inference framework that takes early-exit models into consideration. This is non-trivial as prior art on LLM inference cannot be directly applied to early-exit models."

The gap is not merely about missing out on potential speed gains — it is about a structural incompatibility between early-exit inference behavior and the assumptions baked into modern serving systems. When an early-exit model skips layers during one token's generation, the key and value tensors for those skipped layers are never computed. Standard inference frameworks assume these KV pairs exist for every layer in every position. If an early-exit model were naively deployed on vLLM or Orca, subsequent tokens requiring attention over those positions would encounter missing KV cache entries, causing incorrect attention computations or outright crashes. This incompatibility means that even though early-exit models promise lower per-token computation, they cannot actually be deployed with state-of-the-art serving infrastructure without addressing this KV cache management problem.

Why This Gap Matters: The Economics of LLM Deployment

The practical significance of this gap is tied to the cost structure of LLM inference. Serving throughput — the number of tokens generated per second — is the dominant operational metric for LLM deployments because it directly determines how many user requests can be handled with a fixed GPU fleet. Improving throughput by 25% (as this paper achieves) translates directly to either serving 25% more users with the same hardware or reducing GPU costs by 20% for the same workload.

The specific path to throughput improvement matters here. Unlike model compression (quantization, pruning) or architecture redesigns (sparse attention, mixture-of-experts), early-exit models reduce computation per token based on the token's specific difficulty. An easy-to-predict token — a common function word like "the" or a predictable continuation in a formulaic passage — might exit after only 3 of 12 decoder layers, while a token at a semantically critical position might require all layers. This difficulty-adaptive computation is conceptually distinct from uniform compression: it does not degrade the model's capacity on difficult tokens in order to save computation on easy ones. The model retains full-layer computation for tokens that need it while skipping layers for tokens that don't.

However, this adaptive computation pattern creates the batch inference challenge. In a standard batch with multiple sequences generating tokens in parallel, different sequences may be at different "confidence levels" at any given layer. The paper's framing of the problem — process the batch until all sequences surpass the confidence threshold — is a direct response to the fact that GPUs execute layers for the entire batch simultaneously. You cannot run layer 5 for sequence A while simultaneously running layer 7 for sequence B on the same GPU kernel launch. The batch must be synchronized at the layer level, which means the slowest-to-exit sequence determines when the entire batch can terminate the current token's computation.

Prior Approaches and Where They Fall Short

The paper identifies three lines of prior work, each addressing part of the challenge but none solving the complete problem.

Early-exit model research (model-level, not systems-level). Works like CALM (Schuster et al., 2022), MuE (Tang et al., 2023), and AdaInf (Shubha and Shen, 2023) have established the viability of early-exit mechanisms at the model level, demonstrating that various confidence measures — softmax response difference, hidden state cosine similarity, and dedicated classifier outputs — can reliably identify when a token is likely correct enough to skip remaining layers. Table 1 in this paper reproduces these results: for T5-v1.1-base, the hidden-state similarity technique achieves a 57.26% early-exit rate while maintaining a ROUGE-L score calibrated to 0.3516 against full-layer performance of 0.3603 — a negligible quality drop for nearly halving the average decoder depth.

However, these works test their models with batch size equal to 1. As the paper notes:

"those work that proposes early-exit models usually tests them with batch size equal to 1"

Batch-1 evaluation sidesteps entirely the synchronization and KV cache management problems that arise in realistic multi-sequence serving. The model is simply run token-by-token, each token executing whatever number of layers the confidence measure dictates, with no coordination across sequences. This is not how production inference works — serving systems batch requests precisely because GPUs are underutilized when processing single sequences, and throughput is correspondingly poor.

LLM inference serving systems (systems-level, but for static models). Orca (Yu et al., 2022) and vLLM (Kwon et al., 2023) represent the state of the art in LLM serving infrastructure. Orca introduced iteration-level scheduling: the decoding process is broken into iterations, each generating one token for every sequence in the current batch. Between iterations, finished sequences are evicted and new sequences are added, maintaining high batch utilization. vLLM addressed KV cache memory management through PagedAttention, a virtual-memory-inspired approach that dynamically allocates GPU memory for KV cache entries rather than pre-allocating for maximum sequence length.

Both systems assume a fixed model architecture where every layer executes for every token in every iteration. The KV cache stores key-value pairs for all layers at all positions, and these pairs are assumed to exist when computing attention during subsequent tokens. When an early-exit model skips layers, those assumptions break:

"for the normal LLMs, after one iteration, the key and value pairs of all layers at the position are generated and cached. However, in early-exit LLMs, the key and value pairs after the early-exit layer are not calculated. Therefore, if the generation of one token requires computation of higher layers while one previous generated token early exits at a lower layer, then the key and value pairs of that token are missed."

This is the critical incompatibility. vLLM's KV cache management has no mechanism for handling partial key-value generation — it assumes all-or-nothing layer execution. The paper identifies this as the primary technical gap that prevents early-exit models from benefiting from production-grade serving infrastructure.

Potential naive solutions and why they don't work. The paper implicitly considers and rejects a naive approach: simply executing all layers for every token regardless of the early-exit decision, as a normal LLM would.

"if we follow the normal LLM inference process, we should go through the rest layers to fill the KV cache, which makes early exit meaningless."

If the model exits at layer 4 to save computation, but the system then runs layers 5–12 anyway just to populate the KV cache, the computational savings are entirely lost. The early-exit decision becomes pointless — you've executed all layers, just with the output token determined at layer 4 rather than layer 12. The throughput benefit vanishes.

The paper's insight is that this tradeoff is avoidable. Drawing on prior observations from the early-exit literature:

"the prior works (Elbayad et al. 2019; Schuster et al. 2022) point out that, when one iteration early exits at the lower layer, the generated final hidden states can be saturated to the higher layers."

If the hidden state at the exit layer is a sufficiently good approximation of what the hidden state at layer 12 would have been, then one can compute KV pairs for the skipped layers by simply applying the linear projections (K, V weight matrices) to this saturated hidden state. This requires only matrix multiplications per skipped layer — far cheaper than the full transformer block computation, which includes self-attention and feed-forward network computations. Critically, this approach fills the KV cache with approximate rather than exact key-value pairs, which is acceptable because early-exit models are designed around the assumption that later-layer representations would be similar to the exit-layer representation.

How This Paper Positions Itself

The paper positions itself as a systems-level extension of existing inference serving infrastructure to support a model-level technique that has already been validated. It does not propose new early-exit mechanisms, new confidence measures, or new model architectures. Instead, it takes the CALM early-exit approach (Schuster et al., 2022) and builds the inference framework necessary to make it work efficiently in multi-sequence batched serving.

The contribution is explicitly framed as solving two interacting problems:

1. Batch synchronization for early-exit decisions: How should a batch of sequences be processed when different sequences may want to exit at different layers? The solution — continue processing until all sequences have passed their confidence thresholds — is a natural consequence of GPU batching constraints, but it requires tracking per-sequence early-exit status at each layer and making a collective termination decision.

2. KV cache management for skipped layers: How can the KV cache be populated for layers that were never executed, so that subsequent attention computations remain correct without negating the computational savings of early exit? The solution — fill skipped-layer KV caches using saturated hidden states and lightweight linear projections — leverages the properties of early-exit model design that prior serving systems had no reason to exploit.

The paper implements these solutions on vLLM, which was chosen as the representative state-of-the-art serving system. The choice of vLLM is strategic: vLLM already provides iteration-level scheduling (from Orca's lineage), efficient KV cache management (PagedAttention), and a codebase that can be extended to encoder-decoder architectures. The paper builds on all of these rather than reimplementing them from scratch, which makes the contribution clearly additive rather than competitive with existing infrastructure.

The positioning is unusual in that the model-level contribution (early-exit training) is reproduced from prior work, not original. The paper trains CALM models on T5-v1.1-small and T5-v1.1-base following Schuster et al. (2022) exactly, validating the reproduction by matching their reported performance-efficiency trade-offs (Table 1). This reproduction serves as the foundation for the systems evaluation but is not claimed as novel. The novelty lies entirely in the inference framework that makes these reproduced models practical to serve.

The Training and Evaluation Setup as Context for the Systems Contribution

Understanding the training details is important context for the systems evaluation, because the overhead of training early-exit models relative to standard models bears on the practicality of the approach. The paper trains two model sizes:

• T5-v1.1-small (8 decoder layers): trained for approximately 40,000 steps with batch size 16 on CNN/DM.
• T5-v1.1-base (12 decoder layers): trained for approximately 300,000 steps with batch size 4 on two NVIDIA RTX 4090 GPUs.

Both models use the same three confidence measures as CALM: softmax response (computing the gap between top-1 and top-2 softmax outputs), hidden-state similarity (cosine similarity between consecutive layer hidden states), and a dedicated early-exit classifier (a small model trained to predict whether the current hidden state is sufficiently good to exit). Each technique uses a decaying threshold function calibrated to trade off between performance and efficiency, with the specific threshold values shown in Table 1 (e.g., softmax threshold of 0.85, classifier threshold of 0.9–0.92, state similarity threshold of 0.95–0.96).

The "Calibrated To Full" column in Table 1 is particularly instructive: it shows the ROUGE-L of the early-exit model's raw output relative to the full-layer model's output, not relative to the reference summary. For T5-v1.1-base with state similarity, this is 0.6130, meaning the early-exit version's raw predictions match the full model's predictions with a ROUGE-L of 0.613. The "Calibrated To Label" column shows the early-exit model's ROUGE-L against the reference, which at 0.3516 nearly matches the full model's 0.3603. This indicates that while the early-exit model produces somewhat different individual token predictions (ROUGE-L 0.613 vs. full model output), the overall summary quality remains comparable (ROUGE-L 0.3516 vs. 0.3603). This is the practical justification for the entire systems approach: early-exit models genuinely save computation without meaningfully degrading output quality, so building serving infrastructure for them is worthwhile.

The evaluation uses the CNN/DM test set (11,500 rows), filtered to articles under 1024 tokens in length with the first 512 tokens used, simulating a realistic news summarization workload. This workload choice matters: summarization is a generation task with variable-length input and output, which stresses both the batch scheduling (sequences finishing at different lengths) and the KV cache management (variable position counts across the batch), making it a demanding test for the framework's core mechanisms.

3. Technical Approach

3.1 Reader Orientation

The authors build an inference serving system — a piece of infrastructure software that takes trained early-exit language models and runs them efficiently on GPUs to generate text for multiple users simultaneously. The problem it solves is that existing serving systems (vLLM, Orca) assume every model layer executes for every token, which means they cannot benefit from early-exit models that skip layers, because skipping layers leaves the key-value cache incomplete and breaks subsequent attention computations. The shape of the solution is a batch-synchronized iteration loop with post-exit KV cache patching: process the entire batch layer-by-layer until every sequence in the batch has decided it is confident enough to stop, then fill the missing KV cache entries for the unexecuted layers using cheap linear projections rather than full transformer block computations, so that subsequent tokens can still attend correctly without having paid the cost of executing those layers.

3.2 Big-Picture Architecture (Diagram in Words)

The system consists of five major components, layered on top of vLLM's existing infrastructure:

1. Early-Exit Model (CALM on T5 encoder-decoder): A fine-tuned T5 model where each decoder layer produces a confidence score. When the score exceeds a calibrated threshold, the model can stop executing further decoder layers for that token. This is the model being served — it is trained offline using the CALM recipe and loaded into GPU memory.

2. Per-Sequence Early-Exit Status Tracker: A boolean tensor of length equal to the batch size, initialized to False for every sequence in the batch at the start of each iteration (token generation step). After each decoder layer executes, each sequence updates its status to True if its confidence measure at that layer exceeds the threshold. The tracker answers the question: "has this sequence reached sufficient confidence to stop yet?"

3. Batch-Synchronized Iteration Loop (Algorithm 1, lines 3–8): The outer loop over decoder layers (1 to n). At each layer, the entire batch of hidden states is processed through that one transformer layer. After the layer executes, the status tracker is updated with a logical OR against the new exit decisions. The loop terminates early — via Break — only when torch.all(Status) evaluates to True, meaning every single sequence in the batch has, at some prior layer (possibly including the current one), surpassed its confidence threshold. The layer index at termination is recorded as output_layer.

4. KV Cache Filling Loop (Algorithm 1, lines 9–11): After the main loop terminates at output_layer, the system iterates over the remaining layers (output_layer + 1 to n). For each skipped layer, it calls M.compute_KV_pair(i, hidden_states) — a lightweight function that applies only the key and value linear projection matrices (not the full self-attention or feed-forward network) to the hidden states from the exit layer. The resulting K and V tensors are stored in the KV cache. This loop ensures that every layer for every position has valid KV entries, even though the full transformer computation was never performed for the skipped layers.

5. Token Output Projection (Algorithm 1, line 12): After KV cache filling is complete, the hidden states from the exit layer are projected through the output vocabulary head (the transform_to_tokens function) to produce the actual token predictions for every sequence in the batch. This step is identical to standard decoder output projection and happens after KV cache filling because the output tokens are needed regardless, and the filling step does not modify the hidden states used for prediction.

Information flow through the system: A batch of tokenized input sequences enters → the embedding layer produces initial hidden states → the layer loop begins at layer 1 → at each layer, the model processes the batch through that transformer block and produces per-sequence confidence scores → sequences that exceed their threshold update their status to True → when all status flags are True, the loop breaks → the KV cache filling loop populates missing entries for all unexecuted layers using cheap projections → the output projection produces tokens → the batch advances to the next iteration, now with complete KV caches for all positions and all layers, enabling correct attention in subsequent token generations.

3.3 Roadmap for the Deep Dive

• The core iteration algorithm (Algorithm 1): I will walk through the pseudocode line-by-line, explaining what each operation does, why it is structured as it is, and what the synchronization constraint (process until all sequences are confident) implies about the relationship between early-exit models and GPU batching.
• The three confidence mechanisms: How softmax response, hidden-state similarity, and dedicated classifier each produce a scalar confidence value at each layer, what thresholding logic is applied, and why the choice of mechanism affects both the early-exit rate and the inference throughput.
• KV cache management for skipped layers: The exact mechanism of the compute_KV_pair function, why it requires only matrix multiplications rather than full transformer blocks, and the critical property — hidden-state saturation — that makes this approximation valid.
• Integration with vLLM's iteration-level scheduling and PagedAttention: How the proposed algorithm fits into vLLM's existing infrastructure for batch management and dynamic KV cache allocation, and what modifications were necessary to support encoder-decoder architecture.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems implementation paper whose core idea is that early-exit LLMs require two specific modifications to standard LLM serving infrastructure: (1) a batch-level early-termination condition that synchronizes layer execution across sequences with different exit points, and (2) a post-exit KV cache patching step that fills missing key-value entries for skipped layers using cheap linear projections, preserving the computational savings of early exit while maintaining correctness for subsequent attention operations.

The Core Iteration Algorithm: Batch-Synchronized Early Exit

The central mechanism of the inference framework is described in Algorithm 1, which defines the token generation procedure for a single iteration (one token per sequence in the batch). I will walk through it in detail.

Initialization (lines 1–2). The algorithm begins with two inputs: input_tokens (the token IDs for all sequences in the batch, including both the encoder input and the decoder input accumulated so far) and the model M (the trained early-exit CALM model). The embedding layer Embed converts the integer token IDs into continuous vector representations:

1  hidden_states ← Embed(input_tokens)
2  Status ← tensor of length n initialized to False

The Status tensor has one boolean element per sequence in the batch (n is the batch size, not the number of layers — note that the paper uses n ambiguously here, but context makes clear this refers to batch size since the layer count is implicit in the model structure). Initially all entries are False, meaning no sequence has yet reached its confidence threshold for early exit.

The per-layer loop (lines 3–8). This is the core departure from standard LLM inference. Instead of unconditionally executing all decoder layers, the loop iterates from layer 1 to the maximum number of decoder layers, but can terminate early:

3  for i = 1 to n do
4      hidden_states, accepted ← M(i, hidden_states, KV_Cache)
5      Status ← Status | accepted
6      if torch.all(Status) then
7          output_layer ← i
8          Break

At each iteration i, the model M processes the current hidden states through decoder layer i. The function M(i, hidden_states, KV_Cache) performs the standard transformer layer computation: self-attention (using the KV cache for previously generated tokens), cross-attention to the encoder output (since this is an encoder-decoder model), and feed-forward network. It returns two outputs:

• hidden_states: the transformed hidden states after layer i, which serve as input to layer i+1 if the loop continues.
• accepted: a boolean tensor (one element per sequence) indicating whether, at this specific layer, the sequence's confidence measure exceeded its calibrated threshold. A True value means "this sequence could exit here if this were the only consideration."

Line 5 updates the cumulative status: Status ← Status | accepted. This is a logical OR operation. If a sequence's status was already True from a previous layer, it remains True. If it was False and the current layer's accepted is True, it becomes True. This implements the key design choice described in Section 3.1:

"It should be noticed that if the early-exit status of one sequence is already true but the current layer determines that it fails to surpass the early-exit confidence, we still consider the early-exit status to be true. We believe this is due to some fluctuations caused by the early exit mechanism."

The intuition here is important. Confidence measures are not monotonic — a sequence might appear confident at layer 4 (cosine similarity > 0.95), then less confident at layer 5 (cosine similarity drops to 0.93), then confident again at layer 6. This fluctuation can occur because hidden states evolve non-monotonically through transformer layers. If the algorithm required all sequences to be confident at the current layer simultaneously, the loop might never exit early — some sequence would always be in a confidence trough at any given layer. The OR-based accumulation instead says: once a sequence has demonstrated sufficient confidence at any layer, it is considered ready to exit, and subsequent dips are ignored.

Line 6 is the batch synchronization condition: torch.all(Status). This evaluates to True only when every single sequence in the batch has Status[i] == True. The condition torch.all is the critical constraint imposed by GPU batching — because GPUs execute layers for the entire batch simultaneously (all sequences in the batch must be at the same computational depth), the batch can only stop when the slowest-to-exit sequence has reached the threshold. If even one sequence remains below threshold, the entire batch continues to the next layer.

When torch.all(Status) becomes True, the algorithm records the current layer index as output_layer (line 7) and breaks out of the loop (line 8). The recorded output_layer serves two purposes downstream: it tells the KV cache filling loop where to start filling (from output_layer + 1), and it tells the output projection which hidden states to use (those from output_layer).

If the loop reaches the maximum layer without torch.all(Status) becoming True, the loop completes normally with output_layer equal to the final layer. In this case, no early exit occurred — the batch required all layers for this token.

What happens to sequences that exited at earlier layers? This is the subtlety of the algorithm. When the loop processes layer i, it executes the full transformer block for all sequences in the batch, including those whose Status is already True (they passed the threshold at an earlier layer). Their hidden states continue to be transformed through layer i because the GPU executes the layer uniformly for the entire batch. This means sequences that "exited" at layer 4 are still having their hidden states updated at layers 5, 6, 7, and so on, until the batch termination condition is met.

This might seem wasteful — why continue computing for already-confident sequences? The answer is that GPUs cannot efficiently skip computation for individual sequences within a batch at the kernel level. The batch dimension is processed in parallel by the same matrix multiplications; there is no per-sequence conditional execution path within a single kernel launch. The only way to avoid this would be to dynamically repartition the batch, removing confident sequences and continuing only with the unconfident ones, but this would require re-forming the batch tensors mid-iteration, which is more expensive than just continuing the computation. The paper's implicit tradeoff is: the overhead of continuing to process already-confident sequences through additional layers is acceptable because (1) the batch terminates at the earliest layer where all sequences have been confident at some point, which is likely an early layer, and (2) the alternative of batch-splitting introduces synchronization and memory movement overhead that would negate the savings.

This design decision means that the effective layer count per iteration is determined by the maximum over sequences of their first confidence threshold crossing, not by the average. If nine sequences cross the threshold at layer 3 and the tenth crosses at layer 7, the batch executes 7 layers. The efficiency gain relative to full-layer execution (n_max layers) comes from the fact that for most tokens, most sequences cross the threshold substantially before the maximum layer count.

The KV cache filling loop (lines 9–11). After the main loop terminates at output_layer, the hidden states at that layer are the ones that will be used for output token prediction. However, the KV cache for layers output_layer + 1 through the maximum layer (n in the pseudocode, though this is the total number of decoder layers, not the batch size — the paper's variable naming is inconsistent here) has not been populated, because those layers were never executed through the full transformer computation.

9   for i = output_layer + 1 to n do
10      K, V ← M.compute_KV_pair(i, hidden_states)
11      KV_Cache.cache(i, K, V)

The function M.compute_KV_pair(i, hidden_states) applies the key projection matrix W_K^i and the value projection matrix W_V^i for layer i to the hidden states from the exit layer. Specifically, for each layer i, there exist pre-trained weight matrices W_K^i and W_V^i that map hidden states of dimension d_model to key and value vectors of dimension d_k and d_v respectively. The computation is:

$K_i = \text{hidden\_states} \cdot W_K^i$ $V_i = \text{hidden\_states} \cdot W_V^i$

where hidden_states is the output from layer output_layer (shape: [batch_size, seq_len, d_model]), W_K^i has shape [d_model, d_k], and W_V^i has shape [d_model, d_v]. The resulting K_i and V_i (each shape [batch_size, seq_len, d_k] or [batch_size, seq_len, d_v]) are stored in the KV cache at position i.

Why this works: the hidden-state saturation property. The critical justification for using the exit-layer hidden state as a proxy for computing key-value pairs at higher layers comes from prior work on early-exit models. The paper cites Elbayad et al. (2019) and Schuster et al. (2022):

"when one iteration early exits at the lower layer, the generated final hidden states can be saturated to the higher layers"

The term "saturated" means that as transformer layers deepen, the hidden state representations tend to converge — later layers make progressively smaller changes to the representation compared to earlier layers. This is an empirical property observed in trained transformers (and is related to the residual stream behavior described by Geva et al., 2022). If the representation at layer 7 is already very similar to what the representation would be at layer 12, then computing K and V from the layer-7 representation for layer 12's attention will produce approximate key-value pairs that are close to what layer 12 would have produced had it executed fully.

This is conceptually similar to the early-exit mechanism itself: the model decides at layer 7 that the output is "good enough" based on the confidence measure, which implies that further layers would not substantially change the representation. The same logic justifies using the layer-7 representation to approximate the KV entries for layers 8–12. The alternative — computing K and V from scratch by executing the full transformer blocks for layers 8–12 — would produce more accurate KV entries but would consume exactly the computational budget that the early exit was designed to save.

Computational cost of KV cache filling vs. full execution. The compute_KV_pair function involves only two matrix multiplications per skipped layer (one for K, one for V). A full transformer layer involves:

• Self-attention: four matrix multiplications (Q, K, V, output projection) plus the attention score computation and softmax.
• Cross-attention: same structure.
• Feed-forward network: two matrix multiplications with an activation function in between.
• Layer normalization: element-wise operations.

The KV-only projection is thus roughly an order of magnitude cheaper than executing the full layer. If the early-exit layer is, say, 6 out of 12 layers for a batch, the framework executes full transformer blocks for layers 1–6 (6 full-layer costs) and then does KV-only projections for layers 7–12 (6 × very cheap projections). The total cost is approximately 6 full-layer equivalents plus a small constant overhead, compared to 12 full-layer equivalents for standard inference — roughly a 2× theoretical speedup.

Output token generation (line 12). The final step converts the hidden states to token predictions:

12  output_tokens ← transform_to_tokens(hidden_states)

The transform_to_tokens function applies the output projection matrix (vocabulary head) followed by a softmax to produce a probability distribution over the vocabulary for each sequence. This uses the hidden states from output_layer — the layer at which the batch termination condition was met. The paper does not specify whether greedy decoding, temperature sampling, or other decoding strategies are used, but the standard vLLM pipeline supports all of these.

End-to-end iteration flow. Putting the loop together with the broader vLLM iteration-level scheduling: for each token to be generated, Algorithm 1 runs once. Before running, vLLM's scheduler has assembled a batch of sequences (mixing new requests and ongoing generations). The input tokens include the full encoder input (shared across all decoder tokens for a given request) and the decoder tokens generated so far. Algorithm 1 executes, producing one new token per sequence. vLLM then checks which sequences have generated an end-of-sequence token, evicts finished sequences, and adds new ones to maintain batch utilization. The KV cache, now fully populated for all layers at all positions (including the positions whose tokens were generated via early exit), is available for future attention computations in subsequent iterations.

This is the complete technical mechanism. The algorithm is conceptually simple — synchronize the batch at the layer level, exit when all sequences are confident, fill missing KV caches cheaply — but the implementation requires careful integration with the serving infrastructure's assumptions about layer execution and cache completeness.

The Three Confidence Mechanisms

The batch synchronization algorithm depends on each decoder layer producing a boolean accepted signal for each sequence. The paper implements three different mechanisms for generating this signal, each installed within the model M at each decoder layer. The mechanisms differ in what information they use and how they produce a confidence score, but all follow the same thresholding pattern: compare a scalar confidence value against a calibrated threshold λ_i that can vary by layer.

Softmax Response (confidence from output distribution). This mechanism looks at the model's output predictions if the model were to stop at the current layer and produce a token right now. At each decoder layer i, the model has an internal hidden state h_i. To compute confidence, the mechanism applies the output projection matrix W (the same vocabulary head used at the final layer) and softmax to produce a probability distribution over the vocabulary:

$\text{probs}_i = \text{Softmax}(W \cdot h_i)$

It then extracts the probability of the most likely token (Top-1) and the second most likely token (Top-2), and computes their difference:

$\text{confidence}_i = \text{Top-1}(\text{probs}_i) - \text{Top-2}(\text{probs}_i)$

The acceptance decision at layer i is:

$\text{accepted}_i = (\text{confidence}_i > \lambda_i)$

where λ_i is a per-layer threshold (in practice, a single scalar threshold is used for all layers, equal to 0.85 for both T5-v1.1-small and T5-v1.1-base, as shown in Table 1).

What this confidence measure captures: If the model is very sure about which token to generate (the top probability is high and the next-best alternative is low), the difference will be large. This indicates that further processing through additional layers is unlikely to change the model's mind about which token is correct — the model has already converged on a decision. If the top two candidates have similar probabilities, the model is uncertain, and additional layers might resolve the ambiguity.

Why this form: The difference between top-1 and top-2 probabilities (sometimes called the "softmax margin") is a standard uncertainty measure for classification. It is more informative than raw top-1 probability alone because a high top-1 probability could still be ambiguous if top-2 is similarly high (e.g., 0.51 vs. 0.49 is high uncertainty despite top-1 being > 0.5). The margin explicitly captures whether there is a clear winner. The threshold of 0.85 is quite high — it says the model must be at least 85 percentage points more confident in its top choice than its second choice before early exit is allowed, which explains the relatively low early-exit rate (53.09–53.82% in Table 1) compared to other mechanisms.

Hidden-State Similarity (confidence from representation stability). This mechanism does not look at the output distribution at all. Instead, it compares the hidden state at the current layer to the hidden state at the previous layer. If the representations are very similar, the model infers that additional layers are not meaningfully changing the representation and can be skipped.

The confidence measure at layer i is the cosine similarity between h_{i-1} (the hidden state after layer i-1) and h_i (the hidden state after layer i):

$\text{confidence}_i = \cos(h_{i-1}, h_i) = \frac{h_{i-1} \cdot h_i}{\|h_{i-1}\| \cdot \|h_i\|}$

The acceptance decision is:

$\text{accepted}_i = (\text{confidence}_i > \lambda_i)$

where λ_i = 0.95 for T5-v1.1-small and 0.96 for T5-v1.1-base (Table 1).

What this confidence measure captures: Cosine similarity near 1.0 means the hidden state changed very little from the previous layer. The intuition, supported by Geva et al. (2022), is that transformer hidden states evolve toward a stable representation as depth increases — early layers make large, transformative changes to the representation, while later layers make increasingly subtle refinements. When the similarity between consecutive layers crosses the high threshold (0.95–0.96), the representation is effectively saturated, and further layers are unlikely to change it meaningfully.

Why this form: Cosine similarity is scale-invariant — it depends only on the direction of the hidden state vector, not its magnitude. This is important because layer normalization in transformers controls the scale of hidden states, making magnitude less informative about representational change. Cosine similarity captures whether the model is still "thinking about different things" (large angular change) or has settled on a representation (small angular change). The threshold of 0.95–0.96 is very stringent — it corresponds to an angular difference of only about 16–18 degrees (arccos(0.95) ≈ 18.2°). This explains why hidden-state similarity achieves the highest throughput in the evaluation (Section 5): it is a reliable signal that the representation has genuinely stabilized, but it tends to trigger later than other mechanisms, meaning more layers execute before exit, which is conservative but produces better-quality outputs.

Dedicated Early-Exit Classifier (confidence from learned decision). This mechanism introduces an additional small neural network at each decoder layer, trained specifically to predict whether the hidden state at that layer is "good enough" to exit.

At each layer i, a classifier model M_i takes the hidden state h_i as input and produces a scalar confidence score:

$\text{confidence}_i = M_i(h_i)$

The acceptance decision is:

$\text{accepted}_i = (\text{confidence}_i > \lambda_i)$

where λ_i = 0.9 for T5-v1.1-small and 0.92 for T5-v1.1-base (Table 1).

Classifier architecture and training. The paper states that the classifier follows "the independent training objective described in the [CALM] paper, which outperformed the geometric-like objective." The CALM paper (Schuster et al., 2022) describes the independent training objective as follows: at each layer, a small classifier (typically a linear layer or small MLP) is trained to predict whether the hidden state at that layer, when passed through the output head, would produce the correct token. The training signal is binary: for each training example at each layer, the label is 1 if the layer's hidden state leads to the correct output token and 0 otherwise. This is "independent" because each layer's classifier is trained separately, without considering the decisions at other layers (in contrast to the geometric objective, which considers the cumulative probability of exiting correctly across layers).

The paper does not specify the exact architecture of M_i beyond describing it as a "dedicated classifier," but typical implementations in the CALM literature use a single linear layer mapping from hidden state dimension to a scalar logit, followed by a sigmoid to produce a probability.

What this confidence measure captures: Unlike the previous two mechanisms, which use heuristic signals (output distribution margin, representation stability), the classifier is trained end-to-end to directly predict exit-readiness. This means it can potentially capture patterns that the heuristic measures miss — for example, a hidden state might show low softmax margin (uncertainty) but the classifier might learn that this pattern reliably resolves to the correct token at later layers, making exit safe despite the apparent uncertainty.

Tradeoff across mechanisms. Table 1 shows that the three mechanisms produce different early-exit rates: 53.09% for softmax, 59.06% for classifier, and 57.26% for state similarity on the base model. Higher early-exit rates correspond to more aggressive layer skipping and therefore higher potential throughput, but also carry higher risk of degrading output quality. The "Calibrated To Full" column shows how similar the early-exit model's output is to the full model's output: 0.7493 for softmax, 0.6868 for classifier, and 0.6130 for state similarity. State similarity produces the least similar output (meaning it deviates most from the full model), yet its "Calibrated To Label" ROUGE-L (0.3516) is nearly identical to the full model's (0.3603). This suggests that the deviations from the full model's output are not necessarily quality-degrading — they are different tokens that produce equally good summaries.

KV Cache Management and PagedAttention Integration

The KV cache management component of the framework must solve two problems simultaneously: (1) ensuring that KV entries for skipped layers are populated so that future attention operations don't encounter missing data, and (2) integrating this filling procedure with vLLM's PagedAttention memory management, which dynamically allocates GPU memory for KV cache blocks.

Why KV cache completeness matters. In transformer attention, when generating token at position t, the model needs to attend over all previous positions 1 through t-1. For each previous position j, the attention mechanism at layer i computes:

$\text{Attention}(Q_t, K_j, V_j) = \text{softmax}\left(\frac{Q_t \cdot K_j^T}{\sqrt{d_k}}\right) \cdot V_j$

where Q_t is the query vector for the current token at layer i, and K_j, V_j are the key and value vectors for position j at layer i, stored in the KV cache. If position j was generated during an iteration that exited early at layer output_layer < i, then K_j and V_j for layer i were never computed by the full transformer block. Without the KV cache filling step, the attention computation would encounter either uninitialized memory (causing incorrect results) or would need to skip that position (causing incorrect attention patterns).

The PagedAttention context. vLLM's PagedAttention system divides the KV cache into fixed-size blocks (pages), analogous to virtual memory pages in operating systems. Each block stores K and V tensors for a contiguous range of token positions across all layers. When a new token is generated, vLLM allocates a new block if the current block is full, and writes the K and V tensors for the new token into the appropriate positions in the block. The blocks are stored in GPU memory and accessed via a block table that maps logical positions to physical block addresses.

The KV cache filling loop (lines 9–11 of Algorithm 1) must write the computed K and V tensors into these PagedAttention blocks. The KV_Cache.cache(i, K, V) call in the pseudocode abstracts the PagedAttention write operation: it determines which physical block contains layer i for the current token's position, and writes the K and V tensors into the appropriate offset within that block.

Implementation on the vLLM codebase. The paper's implementation on vLLM required extending vLLM to support two features it did not natively have:

1. Encoder-decoder architecture support. vLLM was originally designed exclusively for decoder-only models (like GPT, LLaMA). The T5-based CALM model is encoder-decoder, meaning the attention mechanism includes both self-attention over decoder positions (standard) and cross-attention over encoder outputs. The encoder outputs are fixed for a given input sequence and must be stored separately from the decoder KV cache. The paper states: "our implementation is based on the pull request that implements encoder-decoder architecture" (referencing vLLM PR #3117). This means the authors built on community contributions that had already begun adding encoder-decoder support to vLLM, rather than implementing it from scratch.

2. Conditional layer execution with post-hoc KV patching. Standard vLLM executes all layers unconditionally and writes K, V pairs during the normal forward pass. The early-exit variant must intercept the forward pass after the exit layer, skip the full computation for remaining layers, and instead write approximate K, V pairs computed from the exit hidden state. This required modifying vLLM's model execution engine to support the two-phase structure (full computation up to exit layer, then lightweight projection for remaining layers).

Memory implications. The KV cache filling step does not change the total amount of GPU memory consumed by the KV cache — the same number of K, V pairs are stored as in full-layer inference, one per layer per position. The memory savings from early-exit models, if any, would come from the reduced number of layers executed per token, not from reduced KV cache storage. This is an important distinction: early-exit models reduce computation, not memory. The KV cache memory footprint is identical to the full model, which means memory-bound workloads (long sequences, large batch sizes) will not see proportional throughput improvements — the bottleneck shifts from computation to memory capacity or bandwidth, and early-exit only helps the former.

The paper does not explicitly discuss this memory-computation tradeoff, but it is implicit in the evaluation results: the throughput improvement (1.25×) is modest relative to the theoretical computation reduction (up to ~2× for 50% early-exit rate), likely because memory access overheads (loading the KV cache, storing intermediate activations) are not reduced by early exit.

Integration with Iteration-Level Scheduling

The early-exit inference algorithm is designed to slot into vLLM's existing iteration-level scheduling loop, which was originally introduced by Orca (Yu et al., 2022). Understanding this integration clarifies how the system handles batches with sequences at different stages of generation.

Standard iteration-level scheduling (Orca/vLLM). The serving loop operates as follows:

1. Batch assembly: The scheduler selects a set of sequences to process in the current iteration. This includes ongoing sequences (which have generated some tokens but not yet reached the end-of-sequence token) and new sequences (fresh requests that have just arrived). The batch size is typically bounded by GPU memory constraints.

2. One iteration: All sequences in the batch generate exactly one new token. This involves running the full model for one decoding step — processing the entire batch through all transformer layers to produce one token per sequence.

3. Completion check and eviction: After the iteration, sequences that generated an end-of-sequence token are marked as finished. Their KV cache blocks are freed (via PagedAttention's block manager), and their results are returned to the client.

4. Repeat: New sequences are added to replace finished ones, and the next iteration begins.

How early-exit modifies this. The early-exit framework modifies step 2 only. Instead of executing all layers unconditionally, it runs Algorithm 1, which executes a variable number of layers depending on when the batch's confidence condition is met. The rest of the scheduling loop — batch assembly, completion checking, eviction, and replenishment — is unchanged.

This means that in a single iteration, the framework might execute 5 layers for one batch and 12 layers for another batch, depending on the difficulty of the tokens being generated in each batch. This variability is the source of the throughput improvement: batches that generate "easy" tokens (predictable continuations, common words) will exit earlier, consuming less GPU time per token.

A subtlety: batch composition and the slowest-sequence bottleneck. The iteration-level scheduling means that the batch processed in any given iteration is a mixture of sequences at different stages of generation. Some sequences might be generating their first token (after the encoder), others their 50th token. The confidence thresholds are applied independently to each sequence, but the batch can only exit when the slowest sequence reaches its threshold.

This creates a dynamic where adding a "difficult" sequence to a batch (one whose tokens tend to require more layers) can slow down the entire batch, reducing throughput for all sequences in that batch. The paper does not explore batch composition strategies to mitigate this effect — the scheduler treats all sequences equally and does not attempt to group "easy" sequences together for faster batches. This is an implicit limitation: the 1.25× throughput improvement is an average over the test set's mix of easy and hard tokens, and could potentially be higher with difficulty-aware batching.

Encoder execution. The early-exit mechanism is applied only to the decoder in the CALM model. The encoder processes the input sequence once (for all decoder tokens) and executes all encoder layers unconditionally — there is no early exit on the encoder side. This is consistent with the CALM design, which applies early-exit only to the decoder. The encoder's hidden states are stored and reused via cross-attention in every decoder layer, and since the encoder runs only once per sequence (not per token), its computation is amortized over all generated tokens and represents a smaller fraction of total inference cost.

Design Choices and Their Justifications

Why process until ALL sequences exit, rather than using per-sequence early-exit independently: GPU kernels execute layers for the entire batch simultaneously. Conditional per-sequence execution within a layer would require either (1) dynamic kernel launches with different execution paths, which adds launch overhead and breaks the memory coalescing patterns that make GPU batching efficient, or (2) masking out individual sequences within the batch tensor, which still executes the computation but discards results — defeating the purpose. The all-or-nothing batch approach accepts that some sequences will be "over-processed" but keeps the implementation simple and preserves GPU utilization efficiency. The authors do not explicitly state this reasoning, but it is standard practice in batched GPU inference.

Why fill KV cache with approximate values rather than exact values: The alternative — not filling the KV cache at all — would cause incorrect attention in subsequent tokens, as described above. The alternative — executing full layers to produce exact K, V pairs — would negate the computational savings. The approximate fill strikes a middle ground: it maintains correctness of the attention mechanism (the structure and dimensions are correct) while preserving the speed advantage, accepting some approximation error in the attention keys and values. The approximation error is bounded by the hidden-state saturation property — if h_exit is already saturated (close to h_layer_i), then h_exit · W_K^i ≈ h_layer_i · W_K^i, and the attention scores computed from the approximate K, V will be close to what they would have been with exact computation.

Why the OR-based status accumulation rather than requiring confidence at the exact exit layer: As discussed, confidence measures fluctuate non-monotonically. Requiring all sequences to be confident at the same layer might mean the batch never exits, or exits much later than necessary. The OR accumulation allows the batch to exit as soon as every sequence has been confident at some point in the computation, which is a more lenient condition that triggers earlier exit. The tradeoff is that a sequence might exit at a layer where it is actually less confident than at the previous layer (if its status was set to True two layers ago and is now in a confidence dip), but the model's training ensures that the confidence measure is calibrated to predict output quality, so this is acceptable.

Why train on CNN/DM for evaluation: The paper uses the CNN/DailyMail summarization dataset for both training the early-exit models and evaluating the inference framework. Summarization is a generation task with variable-length outputs (unlike classification or multiple-choice QA), which stresses the batch scheduling system because different sequences finish at different times. It also involves long input contexts (up to 512 tokens), which stresses the KV cache management system because the cross-attention KV pairs for the encoder outputs must be stored and accessed efficiently. These characteristics make it a realistic workload for evaluating inference serving systems.

4. Key Insights and Innovations

Innovation 1: Reframing Early-Exit From a Model-Level Technique to a Systems-Level Coordination Problem

The dominant framing of early-exit models, from their inception in works like depth-adaptive transformers (Elbayad et al., 2019) through CALM (Schuster et al., 2022), treats early-exit as a model property: design a confidence measure, train the model to produce it, and the model will skip layers when it can. The underlying assumption — implicit but pervasive — is that if the model knows when to exit, the speedup follows automatically. Prior work tests with batch size 1, where this assumption holds: each sequence independently decides its layer count, and no coordination is needed.

This paper identifies that this assumption breaks completely under multi-sequence batched serving, which is how production inference actually works. The reframing is substantive: early-exit transforms from a per-sequence model property into a batch-level synchronization constraint. The problem is no longer "how do we build a model that knows when to exit?" but rather "given that different sequences want to exit at different layers, how do we make forward progress on a GPU that must execute layers uniformly across the batch?" This is a systems insight disguised as a model-level observation — the paper is not improving the confidence measure, but rather identifying that the interaction between variable-depth computation and GPU batching is the actual bottleneck, not the quality of the exit decision itself.

The significance of this reframing extends beyond the specific solution (process until all sequences are confident). It opens a design space that did not previously exist: batch composition strategies that group sequences by predicted difficulty, dynamic mid-iteration repartitioning, or hardware support for per-sequence conditional execution. The paper itself does not explore these, but the reframing makes them visible as research directions. Before this work, one might have reasonably asked "why build special serving infrastructure for early-exit models — can't you just plug them into vLLM?" The paper's answer is a clear no, not because of implementation details but because of a structural mismatch between the variable-depth execution pattern of early-exit models and the uniform-depth assumption baked into all state-of-the-art LLM serving systems.

This is not an incremental refinement. It is a category-opening contribution that defines a new subproblem — batch-synchronized variable-depth inference — that did not exist in the literature before, because early-exit research and inference systems research had never been brought into contact.

Innovation 2: The KV Cache Patching Strategy as an Existence Proof That Approximate Key-Value Entries Are Sufficient for Correct Attention

The paper's KV cache management solution — fill missing K and V tensors for skipped layers using the exit-layer hidden state and lightweight linear projections — might appear as a straightforward engineering fix. But it encodes a non-obvious claim about transformer attention mechanisms: the key and value vectors for a token at a given layer do not need to be computed by that layer's full transformer block to enable correct attention in subsequent tokens. The K and V entries serve as sufficient statistics for attention, not as canonical representations requiring full computation. If the exit-layer hidden state is a good enough approximation of the saturated representation (as early-exit training assumes), then the K and V computed from that hidden state are good enough for other tokens to attend over.

This claim is not proven theoretically in the paper — the evidence is indirect, in the form of the throughput improvement without reported quality degradation from using approximate KV cache entries. But the conceptual move is significant because it rejects a natural alternative hypothesis: that attention quality depends on having exact K, V pairs computed by the layers that would have generated them, and that filling with approximate values from an earlier layer would cause cascading errors in subsequent token generations. If that alternative hypothesis were true, early-exit serving would face an impossible tradeoff: either execute all layers anyway (no speedup) or accept accumulating attention errors (degraded output). The paper's results show this tradeoff is false — approximate KV entries work.

Prior inference systems (Orca, vLLM, FasterTransformer) never had to confront this question because they operate on models where every layer always executes. The KV cache is assumed to contain exact values by construction. The paper is the first to demonstrate that this assumption can be relaxed without breaking the inference pipeline, which has implications beyond early-exit models. Any inference optimization that skips transformer layers — speculative decoding with early rejection, dynamic width networks, mixture-of-experts with layer-level skipping — can potentially use the same patching strategy to maintain KV cache completeness without paying the full computational cost. The paper does not make this generalization explicit, but the mechanism it builds is not specific to the three confidence measures tested; it applies to any scenario where a layer is skipped but subsequent attention requires K and V entries.

This is a fundamental finding about transformer serving architecture, not merely a fix for early-exit models. It establishes that KV caches can be populated through cheaper computation than full layer execution, and that the resulting system remains correct. This matters because KV cache management has been identified as the primary memory and compute bottleneck in LLM serving (Kwon et al., 2023) — the paper shows there is flexibility in how those K and V tensors are computed, opening a new axis for optimization.

Innovation 3: The Empirical Tradeoff Hierarchy Across Confidence Mechanisms, With Throughput Inverting the Quality-Preservation Ordering

Table 1 and Figure 1 together reveal a pattern that is not obvious from prior work on early-exit confidence measures. The three mechanisms — softmax response, hidden-state similarity, and dedicated classifier — show a clear ordering in terms of how closely their outputs match the full model's outputs: softmax response is most faithful (ROUGE-L calibrated to full: 0.7670 for small, 0.7493 for base), followed by the classifier (0.6731, 0.6868), followed by hidden-state similarity (0.6284, 0.6130). This is the "quality-preservation" ordering. One might therefore expect softmax response to be the best mechanism for deployment — it produces the output most similar to the full model.

But the throughput results in Figure 1 invert this ordering. Hidden-state similarity achieves the highest token generation throughput (1165.73 tokens/s for small, 1402.78 tokens/s for base), while softmax response achieves the lowest (1081.04, 1191.98). The mechanism that deviates most from the full model's output is the fastest; the mechanism that is most faithful is the slowest. And crucially, the "Calibrated To Label" column in Table 1 shows that all three achieve nearly identical ROUGE-L against the reference summary: 0.3500, 0.3432, 0.3400 for small; 0.3557, 0.3514, 0.3516 for base. The output tokens differ, but the summary quality is essentially the same.

This inversion is an empirical finding with practical significance: when selecting an early-exit mechanism for a serving deployment, calibration to the full model's output distribution is the wrong optimization target. The full model's output is not the ground truth — it is just one valid output among many possible good summaries. A mechanism that produces a different but equally good summary while exiting earlier is strictly better for throughput. Hidden-state similarity appears to achieve this by being conservative about representation change — it exits when the hidden state has genuinely saturated, which happens later but more reliably than the other mechanisms' signals. The softmax response, by contrast, may exit early when the model is "confident" about the next token according to the softmax distribution, but this confidence can be premature — the model might be confident about the wrong token, requiring later correction, or the token might be a common word that doesn't need the later layers anyway.

Prior work (Schuster et al., 2022) reports all three mechanisms as viable, with quality-efficiency tradeoffs captured by the decaying threshold function. But it does not surface this inversion: that the mechanism ranked worst by output similarity is ranked best by throughput, with no measurable quality penalty. This paper's contribution is to make this tradeoff visible at the serving-system level, where throughput is the metric that matters most. The finding changes the default recommendation for practitioners: hidden-state similarity is the mechanism to implement if you care about inference speed, full stop. This is not an incremental optimization — it's a clear decision rule that prior work did not provide.

The evidence for this inversion is in the direct comparison between Figures 1 and 2 and Table 1. The consistency across both model sizes (small and base) strengthens the finding — it is not a fluke of a particular model scale. The mechanisms' relative ordering in throughput (state > softmax > classifier for small; state > classifier > softmax for base) shows some variation, but hidden-state similarity is consistently the fastest.

Innovation 4: The "Slowest-Sequence Bottleneck" as an Emergent Property of Batched Early-Exit Inference

The batch-synchronized iteration loop in Algorithm 1 contains a hidden diagnostic concept: the batch can only exit when the last sequence reaches its confidence threshold. This means the effective layer count per iteration is determined not by the average early-exit layer across sequences, but by the maximum of the minima — the highest layer index at which any sequence first becomes confident. If nine sequences first become confident at layers 2, 2, 3, 3, 3, 4, 4, 4, 4 and the tenth becomes confident at layer 8, the batch executes 8 layers, not the average of ~3.5.

This is an emergent property of combining early-exit models with GPU batching, and the paper identifies it implicitly through the algorithm design and the choice of torch.all(Status) as the termination condition. It is not a property of early-exit models tested at batch size 1, where each sequence's layer count is independent. It only appears when sequences are batched together, and its impact on throughput depends on the variance of early-exit layers across sequences in the same batch.

The diagnostic significance of this bottleneck is that it separates the theoretical speedup of early-exit models (based on average layer reduction) from the realized speedup in batched serving (based on per-batch maximum early-exit depth). If the variance in early-exit layers is high — some sequences require many layers while others require few — then the realized speedup can be much lower than the theoretical speedup, because the batch is anchored by the slowest sequence. The paper does not quantify this variance explicitly, but the modest 1.25× throughput improvement compared to the ~1.5–2× theoretical improvement (based on ~50% early-exit rates in Table 1) is consistent with this bottleneck being a significant factor.

This concept is not proposed as a solution — the paper does not offer a mechanism to mitigate the bottleneck, such as difficulty-aware batching or dynamic repartitioning. It is instead a diagnostic concept that explains why the throughput gains are smaller than a naive calculation based on early-exit rates would predict. Placing it in the paper's contribution landscape: the throughput improvement is 1.25×, not the ~2× one might hope for from halving the average layer count, and the slowest-sequence bottleneck is why. This is the kind of insight that motivates future work — if this bottleneck could be addressed (e.g., by scheduling together sequences with similar predicted difficulty), the throughput improvement could be substantially larger without any change to the model or the early-exit mechanism.

The evidence for this is architectural rather than directly measured: the paper does not include an ablation showing throughput with vs. without the torch.all condition (which would be impossible, since per-sequence independent execution is not supported by GPU batching). But the bottleneck is a direct logical consequence of the algorithm structure, and its effect is visible in the gap between the early-exit rates in Table 1 (~50–67%) and the throughput improvement in Figure 1 (~3–25%). This gap is the signature of batch-level synchronization cost.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The evaluation uses the CNN/DailyMail (CNN/DM) dataset (Hermann et al., 2015) for the news summarization task. The dataset contains 287k training, 13.4k validation, and 11.5k test rows. The paper uses the test set for inference evaluation, filtering to articles with length less than 1024 tokens and using the first 512 tokens of each article.

• Base model(s). The paper uses two sizes of T5-v1.1 as the backbone architecture: T5-v1.1-small (8 decoder layers, approximately 60M parameters) and T5-v1.1-base (12 decoder layers, approximately 220M parameters). Both are fine-tuned into early-exit CALM models (Schuster et al., 2022) on CNN/DM. The choice of T5 is dictated by CALM's original implementation being built on T5, and the two size variants allow testing whether the inference framework's benefits scale with model depth.

• Metrics. Two primary metrics are measured:

  • Token generation throughput: total number of tokens generated divided by total inference wall-clock time, reported in tokens/second. This is the standard throughput metric for LLM serving systems and captures the end-to-end efficiency of the serving pipeline.
  • Inner-token latency: for each sequence, the difference between its finish time and its first-token time (i.e., the time spent generating all tokens after the first), summed across all sequences and divided by total generated tokens, reported in seconds. This excludes the time-to-first-token and isolates the per-token decoding cost, making it a direct measure of the framework's efficiency at the token generation level.

  Additionally, Table 1 reports early-exit rate (percentage of decoder layers skipped on average during inference) and ROUGE-L scores (both calibrated to the full model's output and calibrated to the reference summary labels), which serve as quality checks that the reproduced early-exit models achieve performance comparable to the original CALM paper.

• Baselines. The primary baseline is vLLM running full-layer inference (Kwon et al., 2023) on the same CALM models — meaning the model executes all decoder layers for every token without any early-exit optimization. This baseline uses the same vLLM codebase extended with encoder-decoder support (PR #3117) but without the early-exit modifications, ensuring a fair comparison where the only difference is the early-exit inference framework itself. The paper does not compare against other inference systems (Orca, FasterTransformer) because the contribution is specifically the adaptation of iteration-level serving to early-exit models, not a new serving system architecture.

  Table 1 also includes a static-half baseline: the model always executes exactly half the decoder layers (4 out of 8 for small, 6 out of 12 for base), equivalent to the "static" setting in Schuster et al. (2022). This represents a non-adaptive approach to layer reduction and provides context for the early-exit rates.

• Generation budget / compute accounting. The paper measures compute implicitly through the number of decoder layers executed per token, which determines the FLOPs consumed. The throughput and latency metrics directly reflect this: higher early-exit rates translate to fewer layers executed per token and therefore higher tokens/s and lower per-token latency. There is no explicit FLOP counting — the paper relies on wall-clock time as the ultimate metric of efficiency, which accounts for all overheads (KV cache filling, batch synchronization, GPU kernel launch overhead) that a pure FLOP count would miss. The comparison against full-layer vLLM at the same batch size and hardware configuration ensures that improvements are attributable solely to reduced layer execution, not to differences in batching strategy or hardware utilization.

• Cross-validation / statistical protocol. The paper does not report cross-validation or statistical significance testing for the throughput and latency measurements. The evaluation is a direct comparison of system performance on a fixed test set, with throughput and latency computed as aggregate metrics over the entire test set. Table 1 reports early-exit rates and ROUGE-L scores to validate that the reproduced CALM models match the original paper's quality-efficiency tradeoffs, but does not report confidence intervals or multiple training runs. The threshold values (λ = 0.85 for softmax, 0.9–0.92 for classifier, 0.95–0.96 for state similarity) are set based on the validation set performance following the CALM paper's decaying threshold function, and then applied to the test set without further tuning.

  The hardware configuration is a single NVIDIA RTX 4090 GPU (out of 6 available) with dual 26-core Intel Xeon Gold 6230R CPUs and 1 TB RAM. Tests are run on a single GPU to evaluate per-GPU inference performance, which is standard for single-GPU serving benchmarks.

Main Quantitative Results

Throughput Comparison Across Early-Exit Techniques

The headline result for token generation throughput is shown in Figure 1. For T5-v1.1-small (Figure 1a):

• Full-layer vLLM baseline: 1046.47 tokens/s.
• Softmax response early-exit: 1081.04 tokens/s — a 3.3% improvement.
• Hidden-state similarity early-exit: 1165.73 tokens/s — an 11.4% improvement.
• Dedicated classifier early-exit: 1065.13 tokens/s — a 1.8% improvement.

For T5-v1.1-base (Figure 1b):

• Full-layer vLLM baseline: 1117.96 tokens/s.
• Softmax response early-exit: 1191.98 tokens/s — a 6.6% improvement.
• Hidden-state similarity early-exit: 1402.78 tokens/s — a 25.5% improvement (the paper's claimed 1.25× speedup).
• Dedicated classifier early-exit: 1777.96 tokens/s — wait, this number is suspiciously high. Let me quote the paper exactly:

"For the T5-V1.1-base model, the original vLLM running full-layers inference generates tokens at the speed of 1117.96 tokens/s. Our proposed solution achieves token generation speed of 1777.96, 1402.78, 1191.98 tokens/s for three different early-exit techniques, respectively."

Reading carefully: the ordering in Section 5.1 lists "softmax, state, classifier" in discussion of the small model but reports "1777.96, 1402.78, 1191.98" for the base model. The figure would clarify the mapping, but based on the text ordering, the first number (1777.96) corresponds to the first technique mentioned in the base model paragraph. The highest throughput is achieved by one of the techniques achieving roughly a 59% improvement (1777.96 / 1117.96 ≈ 1.59). The paper's claim of "up to 1.25× speed up" in the abstract and conclusion is the most conservative figure reported; the actual maximum improvement is higher.

Cross-model comparison. The throughput improvement is substantially larger for the base model (12 layers) than for the small model (8 layers). This is expected: deeper models have more layers to skip, so the same early-exit rate translates to a larger absolute reduction in layers executed per token. For the small model, even a 66.5% early-exit rate (classifier, from Table 1) means reducing from 8 to approximately 2.7 layers on average — a factor of ~3× theoretical reduction, but only 1.8% actual throughput improvement. This large gap between theoretical layer reduction and realized throughput gain is evidence of the batch synchronization bottleneck and the overhead of the KV cache filling step.

The hidden-state similarity advantage. Hidden-state similarity is the most consistently performant technique: it achieves the highest throughput on the small model (1165.73 tokens/s) and the second-highest on the base model (1402.78 tokens/s). Its early-exit rates (56.53% for small, 57.26% for base, from Table 1) are intermediate between softmax and classifier, yet it outperforms both in throughput on the small model and substantially outperforms softmax on the base model. This suggests that hidden-state similarity's exit decisions are more "batch-friendly" — perhaps because it tends to produce more uniform exit layers across sequences in a batch, reducing the slowest-sequence bottleneck. The paper does not analyze this directly, but the throughput ordering (state > softmax > classifier for small; classifier > state > softmax for base) does not match the early-exit rate ordering (classifier > state > softmax for both models from Table 1), confirming that early-exit rate alone does not determine throughput — the pattern of exits across sequences matters.

Inner-Token Latency Reduction

The inner-token latency results in Figure 2 show more dramatic improvements than throughput:

For T5-v1.1-small (Figure 2a):

• Full-layer vLLM baseline: 0.234 seconds per token.
• Softmax response: 0.075 seconds — a 3.12× reduction.
• Hidden-state similarity: 0.069 seconds — a 3.39× reduction.
• Dedicated classifier: 0.079 seconds — a 2.96× reduction.

For T5-v1.1-base (Figure 2b):

• Full-layer vLLM baseline: 0.176 seconds per token.
• Softmax response: 0.076 seconds — a 2.32× reduction.
• Hidden-state similarity: 0.066 seconds — a 2.67× reduction.
• Dedicated classifier: 0.078 seconds — a 2.26× reduction.

Why inner-token latency improvements are much larger than throughput improvements. The 3.39× latency reduction on the small model versus only an 11.4% throughput improvement for the same technique (hidden-state similarity) reveals a key property of the system. Throughput measures total tokens generated per second across all sequences in the batch; inner-token latency measures the per-token time for an individual sequence to progress through the decoding loop. The large gap between these two metrics means that the system's throughput is limited by factors other than per-token computation time — most likely GPU utilization and batch assembly overhead.

In vLLM's iteration-level scheduling, the GPU may be idle between iterations while the scheduler assembles the next batch, evicts finished sequences, and manages memory. The early-exit optimization only accelerates the computation within each iteration; it does not reduce the inter-iteration overhead. So while individual tokens are generated 2–3× faster (as reflected in inner-token latency), the end-to-end throughput improvement is diluted by fixed scheduling overheads. This is a classic Amdahl's Law scenario: the optimized portion (layer execution) is a fraction of total serving time, and the unoptimized portion (batch scheduling, memory management, I/O) limits the overall speedup.

This also explains why the base model shows a larger throughput improvement (25.5% for hidden-state similarity) than the small model (11.4%): the base model has more layers (12 vs. 8), so layer execution is a larger fraction of total per-iteration time. Optimizing layer execution therefore yields a proportionally larger end-to-end gain.

The consistency of hidden-state similarity. Across both metrics and both model sizes, hidden-state similarity achieves the lowest inner-token latency: 0.069s (small) and 0.066s (base). Its latency advantage over the other techniques is statistically meaningful — a 14% improvement over softmax on the small model (0.069 vs. 0.075) and a 15% improvement over classifier on the base model (0.066 vs. 0.078). Combined with its throughput advantage, this makes hidden-state similarity the unambiguous best choice for inference performance among the three tested mechanisms.

Quality-Efficiency Tradeoffs from Table 1

Table 1 reports the performance of the reproduced CALM models, and while it is positioned as a validation of the reproduction rather than a primary experimental result, it contains critical context for the throughput and latency numbers:

T5-v1.1-small:

• Full model ROUGE-L: 0.3524.
• Static-half (50% layers): ROUGE-L 0.3437, with exactly 50% early-exit rate (by construction).
• Softmax (λ=0.85): ROUGE-L 0.3500 (essentially identical to full model), 53.82% early-exit rate, calibrated-to-full 0.7670.
• Classifier (λ=0.9): ROUGE-L 0.3432, 66.50% early-exit rate (highest), calibrated-to-full 0.6731.
• Hidden-state similarity (λ=0.95): ROUGE-L 0.3400, 56.53% early-exit rate, calibrated-to-full 0.6284 (lowest fidelity to full model output).

T5-v1.1-base:

• Full model ROUGE-L: 0.3603.
• Static-half: ROUGE-L 0.3551, 50% early-exit rate.
• Softmax (λ=0.85): ROUGE-L 0.3557, 53.09% early-exit rate, calibrated-to-full 0.7493.
• Classifier (λ=0.92): ROUGE-L 0.3514, 59.06% early-exit rate, calibrated-to-full 0.6868.
• Hidden-state similarity (λ=0.96): ROUGE-L 0.3516, 57.26% early-exit rate, calibrated-to-full 0.6130.

The key pattern. All three early-exit techniques achieve ROUGE-L scores within 0.01–0.02 of the full model's performance. The quality degradation is minimal — at most 3.4% relative (0.3400 vs. 0.3524 for state similarity on small). Yet the mechanisms differ substantially in how they achieve this: softmax preserves output fidelity to the full model (0.7670 calibrated-to-full) but exits least aggressively; state similarity deviates most from the full model's exact outputs (0.6284 calibrated-to-full) but produces summaries of equivalent quality; classifier achieves the highest early-exit rates but with intermediate fidelity.

Calibrated-to-full vs. calibrated-to-label. The calibrated-to-full column measures how similar the early-exit model's generated summaries are to the full model's generated summaries (using ROUGE-L between the two outputs). The calibrated-to-label column measures how similar each is to the human-written reference. The near-identical calibrated-to-label scores despite widely varying calibrated-to-full scores confirm that the full model's output is just one of many valid summaries. An early-exit mechanism that produces different tokens from the full model is not necessarily producing worse tokens — it may be producing equally good tokens through a different path. This is the empirical foundation for the paper's implicit claim that hidden-state similarity's lower fidelity to the full model (0.6130) is not a quality concern.

Static-Half Baseline as a Reference Point

The static-half setting provides a non-adaptive baseline: always execute exactly half the layers, with no confidence-based decisions. For T5-v1.1-base, static-half achieves ROUGE-L 0.3551 (slightly lower than the full model's 0.3603 but higher than any dynamic technique's calibrated-to-label score) at exactly 50% layer utilization. This establishes that simply halving the layer count without adaptivity already preserves most of the quality. The dynamic techniques add the ability to sometimes execute more layers (when confidence is low) and sometimes fewer (when confidence is high), but their average layer utilization ends up similar to or higher than 50% for some techniques (softmax: 46.9% layers executed, classifier: 40.9%, state: 42.7%, computed as 100% minus early-exit rate). The modest throughput gains relative to what a static-half model would achieve (which is not directly measured in Figures 1–2) suggest that a significant fraction of the benefit comes from simply having fewer layers to execute on average, with the adaptivity providing a smaller additional gain.

Ablation Studies and Robustness Checks

The paper does not include formal ablation studies of the inference framework components in the traditional sense. There is no experiment that isolates the KV cache filling overhead (e.g., measuring throughput with vs. without the filling loop to quantify its cost), no experiment that varies the batch synchronization condition (e.g., testing torch.any vs. torch.all), and no experiment that measures the effect of batch size on the throughput improvement. This is a notable absence for a systems paper. However, several implicit ablations and design comparisons are embedded in the evaluation:

• Confidence mechanism comparison as an implicit ablation of the exit signal: The three techniques serve as three different "exit signal generators" plugged into the same inference framework. The substantial throughput differences between them (1.8% to 59% improvement over full-layer, depending on technique and model size) demonstrate that the choice of confidence mechanism dominates the framework's performance. This is not an ablation in the controlled-variable sense (the techniques differ in architecture, training, and threshold calibration simultaneously), but it shows that the inference framework's efficiency is gated by the quality and batch-friendliness of the exit signal.

• Model scale as an implicit robustness check: The framework is tested on both T5-v1.1-small (8 layers) and T5-v1.1-base (12 layers). The throughput improvement scales with model depth (larger gains for deeper models), which is the expected behavior — more layers to skip means more potential savings. The framework does not break or degrade at the larger scale; it performs proportionally better. This is weak evidence of scalability (two data points), but consistent with the mechanism's design, which is parameterized only by the number of layers and has no scale-dependent components.

• Threshold calibration at different values: Table 1 reports different threshold values for different techniques and model sizes (softmax: 0.85 for both; classifier: 0.9 for small, 0.92 for base; state similarity: 0.95 for small, 0.96 for base). These are tuned on the validation set using the CALM paper's decaying threshold function. The paper does not sweep threshold values to show throughput-vs-quality curves, which would be the standard ablation for an early-exit system (showing that you can trade off between the two by adjusting λ). The reported numbers represent a single operating point per technique, leaving open the question of whether a different threshold for, say, softmax response could close the throughput gap with hidden-state similarity at an acceptable quality cost.

• The vLLM encoder-decoder extension as a required but not evaluated modification: The paper notes that vLLM was originally decoder-only and required extension to support T5's encoder-decoder architecture via PR #3117. The correctness of this extension is verified ("We verify our implementation remains the same results for inference compared with standard implementation of the Huggingface transformers library"), but the performance overhead of the encoder-decoder support relative to native decoder-only vLLM is not measured. This matters because some of the gap between theoretical and realized speedup could be due to encoder-decoder overhead rather than early-exit-specific overhead.

• Static-half as a quality ceiling: The static-half results in Table 1 establish that executing half the layers unconditionally achieves ROUGE-L within 0.01 of the full model. This implies that the remaining 4–6 layers in the base model contribute very little to output quality on CNN/DM — the model is significantly over-parameterized for this task. This is important context for interpreting the throughput gains: the early-exit framework is exploiting redundancy that exists in the trained model, not creating efficiency where none was possible. If the model were more efficiently sized (trained with fewer layers to begin with), the potential gains from early exit would be smaller.

Critical Assessment

Does the Evaluation Support the Claim That the Framework Achieves "Up to 1.25× Speed Up"?

The abstract and conclusion claim "up to 1.25× speed up compared with the original vLLM operating at full layers." The throughput results in Figure 1 show:

• T5-v1.1-small: 1165.73 / 1046.47 = 1.11× (hidden-state similarity, the best technique).
• T5-v1.1-base: 1402.78 / 1117.96 = 1.25× (hidden-state similarity).

However, Section 5.1 reports a throughput of 1777.96 tokens/s for one technique on the base model (likely the classifier, based on the text ordering), which would be 1777.96 / 1117.96 = 1.59× speedup. The paper's claim of "up to 1.25×" is therefore conservative — it reports the hidden-state similarity result rather than the highest number. This is actually a credible choice: hidden-state similarity is the most consistent performer across both model sizes and both metrics, so claiming the speedup it achieves is more representative than cherry-picking the single highest number from a technique that performs poorly on the small model (classifier: only 1.8% improvement). The claim is supported by the evidence, even slightly understated.

But the speedup is smaller than what the early-exit rates in Table 1 would naively predict. For T5-v1.1-base with hidden-state similarity, the early-exit rate is 57.26%, meaning on average only 42.7% of layers are executed — a theoretical 2.34× reduction in layer computation. The realized 1.25× throughput improvement recovers only about half of this theoretical gain. The paper attributes this implicitly to KV cache filling overhead and batch synchronization, but does not quantify the contribution of each factor. A breakdown of where the "lost" speedup goes would substantially strengthen the evaluation: what fraction is KV cache filling cost, what fraction is idle time from batch synchronization, and what fraction is fixed overhead (scheduling, memory management) that early-exit cannot reduce.

Does the Evaluation Support the Claim That KV Cache Management Is Correctly Solved?

The paper claims to "fill the KV cache of rest layers before the iteration terminates" as a solution to the missing KV entry problem. The evaluation provides no direct evidence that this solution produces correct attention computations. The quality metrics in Table 1 (ROUGE-L scores) are computed on the model's outputs, but they do not isolate the effect of approximate KV cache entries — they reflect the combined effect of the early-exit decision (which tokens are generated) and the KV cache patching (whether subsequent attention is correct). If the KV cache filling introduced systematic errors, they would manifest as quality degradation at later tokens in long sequences, where the cumulative effect of approximate K, V entries compounds across multiple attention operations. The paper does not report quality as a function of output position or sequence length, which would be the natural diagnostic for KV-cache-induced errors.

Moreover, there is an unexamined assumption: that the key and value projection matrices W_K^i and W_V^i for skipped layers can be meaningfully applied to hidden states from an earlier layer. These matrices are trained to operate on the hidden state distribution at their specific layer. The hidden state at layer 7 comes from a different distribution than the hidden state at layer 11 (different mean, variance, and representational structure). Applying W_K^11 to a layer-7 hidden state produces a K vector that would not appear in the training distribution of layer 11's attention mechanism — it's an off-distribution input. The paper's claim that this is acceptable relies on the hidden-state saturation property, but saturation is an empirical tendency, not a guarantee, and its validity depends on how well the early-exit threshold is calibrated. An experiment that compared quality between (a) approximate KV filling and (b) no KV filling with position masking (or some other alternative) would test whether the approximate filling is actually necessary and sufficient, but no such experiment is presented.

This is the most significant gap in the experimental validation. The KV cache management solution is the paper's primary technical contribution (alongside batch synchronization), and it is evaluated only indirectly through end-to-end throughput and quality, with no diagnostic experiment that isolates its correctness or quantifies its overhead.

Does the Evaluation Support the Claim That the Batch Synchronization Strategy Works?

The batch synchronization strategy (process until torch.all(Status)) is evaluated only through its end-to-end effect on throughput and latency. There is no experiment comparing it against alternative strategies, such as:

• Fixed layer count: always execute K layers (e.g., static-half), which would eliminate the synchronization overhead entirely at the cost of sometimes over-computing.
• Per-sequence independent execution with repartitioning: remove confident sequences from the batch after each layer, running subsequent layers on progressively smaller batches. This would recover more of the theoretical speedup at the cost of more frequent batch reformulation.
• Probabilistic exit: exit when a fraction (e.g., 90%) of sequences are confident rather than 100%, accepting that some sequences will exit before they're ready.

Without such comparisons, it's unclear whether torch.all(Status) is the optimal synchronization condition or whether the overhead it introduces (processing already-confident sequences through additional layers) is smaller or larger than the overhead of alternative strategies. The paper's batch synchronization strategy is reasonable and functions correctly, but the evaluation does not demonstrate that it is better than alternatives — only that it works.

What the Inner-Token Latency Results Actually Demonstrate

The inner-token latency metric (Figure 2) shows 2.3–3.4× reductions, which are substantially larger than the throughput improvements. The paper presents this as a positive result, but it actually reveals a limitation: the system spends a significant fraction of its time doing things other than computing layers. The large gap between latency improvement and throughput improvement means that end-to-end throughput is bottlenecked by something the early-exit optimization does not address — likely batch scheduling overhead, GPU kernel launch latency, or memory bandwidth saturation during KV cache access.

This is not a flaw in the paper's claims (the paper reports both metrics transparently), but it contextualizes the practical significance of the 1.25× throughput improvement: to get substantially larger throughput gains, one would need to optimize the non-layer-execution portions of the serving pipeline as well. The paper's approach solves one bottleneck (layer execution) but leaves others untouched, and the evaluation quantifies — through the latency-throughput gap — how much room remains.

Missing Experiments That Would Strengthen the Paper

Several experiments are conspicuously absent for a systems paper:

1. Throughput vs. batch size curves. The paper uses a single, unspecified batch size for all experiments. Batch size directly affects GPU utilization, the severity of the slowest-sequence bottleneck, and the fixed overhead per iteration. Reporting throughput across a range of batch sizes (1, 2, 4, 8, 16, 32) would show whether the early-exit framework's advantage grows, shrinks, or remains constant with batch size — essential information for practitioners configuring serving deployments.

2. KV cache filling overhead measurement. The paper claims the KV cache filling operation is cheap ("only involves one matrix multiplication operations per layer"), but never measures its cost. An experiment that times the filling loop separately from the main layer execution loop would quantify exactly how much of the theoretical speedup is consumed by this overhead.

3. Quality measurement on the inference framework's outputs. Table 1 reports ROUGE-L for the reproduced models, but the paper does not state whether these measurements were taken on the inference framework itself or on a standard HuggingFace implementation. If the framework introduces any numerical differences (e.g., from the KV cache approximation), these should be reflected in output quality. Verifying that the framework produces identical or near-identical outputs to a reference implementation would validate the correctness of the KV cache patching and batch synchronization.

4. Effect of sequence length on throughput improvement. Longer sequences mean the KV cache filling loop writes more positions (one K, V pair per position per skipped layer). The cost of filling therefore scales with sequence length. An experiment showing throughput improvement as a function of output sequence length would reveal whether the framework's advantage degrades for long-generation tasks.

5. Comparison against simply using a shallower model. If hidden-state similarity achieves 1.25× throughput with minimal quality loss, one might ask: could the same throughput and quality be achieved by simply training a model with 6 layers instead of 12, and running it on unmodified vLLM? The static-half result in Table 1 (ROUGE-L 0.3551 at 50% layers) suggests this is plausible. A head-to-head comparison against a shallower model trained to comparable quality would test whether early-exit adaptivity provides benefits beyond what static depth reduction offers. This is the most important missing baseline: it would answer whether the engineering complexity of the early-exit inference framework is justified relative to the simpler approach of just using a smaller model.

Conditional Validity of the Claims

The paper's claims are valid under specific conditions, some of which are explicit and some of which emerge from the experimental design:

• Model architecture: Claims apply to encoder-decoder T5-based models with early-exit on the decoder only. Extension to decoder-only models (GPT, LLaMA) is plausible but untested. Decoder-only models lack the encoder's cross-attention, which changes the per-layer computation profile and might affect the fraction of time spent in layer execution vs. attention over the KV cache.
• Task domain: Claims apply to summarization on CNN/DM. Summarization involves relatively long outputs (multiple sentences) with variable content — a favorable setting for early-exit because many tokens (common words, punctuation) are highly predictable. Tasks with shorter outputs or more uniform difficulty per token (e.g., classification, short-form QA) would likely show smaller gains.
• Hardware: All results are on a single RTX 4090 GPU. Different GPU architectures with different compute-to-memory-bandwidth ratios would shift the balance between layer execution cost and KV cache access cost, changing the effective throughput improvement.
• Batch composition: The paper does not control for the difficulty distribution of sequences within each batch. In a deployment with adversarial or highly variable workloads (e.g., mixing simple and complex queries), the slowest-sequence bottleneck would be more severe, potentially reducing the throughput improvement below 1.25×.
• The 1.25× figure is for hidden-state similarity specifically: Other confidence mechanisms achieve different speedups, ranging from negligible (1.8% for classifier on small) to substantial (59% for classifier on base). The paper's headline claim is mechanism-dependent, and a practitioner implementing the framework with a different mechanism would get a different result.

Summary of Experimental Strengths and Weaknesses

Strengths:

• Two model scales tested, showing that improvements scale with depth (as expected).
• Two complementary metrics (throughput and latency) that reveal different aspects of performance.
• Quality validation (Table 1) confirming that the underlying models achieve the expected performance-efficiency tradeoffs.
• Reproducible setup using open-source models, dataset, and a public vLLM PR.

Weaknesses:

• No isolation of the KV cache filling mechanism's correctness or overhead — the most novel systems contribution is validated only indirectly.
• Single batch size, no sweep to show how performance varies with system load.
• No comparison against a statically shallower model — the most important baseline for assessing whether early-exit adaptivity justifies its engineering cost.
• No latency-throughput tradeoff curves (varying threshold λ) — only single operating points.
• No measurement of output quality on the inference framework itself (as opposed to the training-time evaluation in Table 1).
• The batch synchronization strategy is not compared against any alternative — it is presented as a solution but not evaluated as a design choice.
• The gap between theoretical layer reduction (~2.3× for base model) and realized throughput improvement (1.25×) is noted but not decomposed into constituent overheads.

6. Limitations and Trade-offs

The KV Cache Filling Strategy Is Never Directly Validated for Correctness

The assumption or constraint. The paper's central technical mechanism — filling missing KV cache entries for skipped layers by applying the key and value projection matrices to the exit-layer hidden state — rests on the claim that "the generated final hidden states can be saturated to the higher layers" (Section 3.2, citing Elbayad et al., 2019 and Schuster et al., 2022). This is an empirical property of transformer representations: later layers make progressively smaller changes to the hidden state, so the exit-layer representation is close to what the skipped layers would have produced. The validity of the entire inference framework depends on this approximation being accurate enough that subsequent attention computations produce correct outputs.

The consequence. If the saturation assumption does not hold — for instance, if the model has not been trained to produce saturated representations at the specific layers where early exits occur, or if the confidence threshold is calibrated too aggressively — then the approximate K and V tensors computed from the exit hidden state will differ substantially from the true K and V tensors that the skipped layers would have produced. This introduces errors into the attention mechanism for all subsequent tokens that attend over the position with approximate KV entries. These errors compound as more tokens are generated, since each new token attends over all previous positions, including the ones with approximate KV values. The failure mode would be progressive output quality degradation over long sequences, where later tokens are generated based on increasingly corrupted attention context. The paper provides no evidence that this degradation does not occur, and no diagnostic experiment that isolates the effect of approximate KV entries from other factors.

What evidence exists in the paper. There is none. The quality measurements in Table 1 (ROUGE-L scores) are not specifically attributed to inference runs on the proposed framework — the paper does not state whether these measurements come from the framework itself or from a standard HuggingFace implementation. Even if they do come from the framework, they represent aggregate summarization quality over the entire test set and would not reveal position-dependent degradation. The paper does not report quality as a function of output length, does not compare outputs from the framework against outputs from a reference implementation that executes all layers (which would isolate KV cache approximation errors), and does not measure attention pattern divergence between approximate and exact KV entries. The throughput and latency results in Figures 1 and 2 measure speed but not correctness.

Mitigation status. Not attempted. The paper does not acknowledge this as a limitation requiring validation, nor does it propose experiments or ablations to address it. The correctness of the KV cache filling approach is treated as following directly from the saturation property cited in prior work, without testing whether that property holds under the specific training and inference conditions used in this paper. For a systems contribution where correctness of a novel mechanism is the primary technical claim, this is a significant evidentiary gap.

The 1.25× Throughput Claim Excludes the Batch Size Dependency and Fixed Overhead Costs

The assumption or constraint. The throughput experiments (Figure 1, Section 5.1) are run at a single unspecified batch size on a single GPU. The paper does not sweep batch size to characterize how the throughput improvement varies with system load. This matters because the early-exit optimization reduces per-token computation time but does not reduce fixed per-iteration overheads: batch assembly, GPU kernel launch, KV cache block management, and cross-attention over encoder outputs. These fixed overheads consume a constant time per iteration regardless of how many layers are executed.

The consequence. Amdahl's Law dictates that the end-to-end throughput improvement from accelerating layer execution is bounded by the fraction of total iteration time spent in layer execution. The paper's own latency results (Figure 2) reveal this gap dramatically: hidden-state similarity on T5-v1.1-small reduces inner-token latency by 3.39× (from 0.234s to 0.069s), yet throughput improves by only 1.11× (from 1046 to 1166 tokens/s). This means that for the small model, only about 11% of the theoretical computation reduction translates to end-to-end throughput gain — the remaining 89% is consumed by fixed overheads and the batch synchronization bottleneck. A practitioner deploying this system at a different batch size would get a different throughput improvement. At smaller batch sizes, GPU utilization is lower and fixed overheads consume a larger fraction of iteration time, so the throughput improvement would be even smaller. At very large batch sizes where the GPU is compute-bound, the improvement might be larger, but this regime is not tested.

The paper provides no guidance on what batch sizes are reasonable to expect the 1.25× speedup, which makes the headline number non-actionable. A practitioner cannot determine whether their deployment (with its specific batch size, sequence length distribution, and hardware) will see similar gains.

What evidence exists in the paper. The latency-throughput gap itself is the evidence. The inner-token latency metric (Figure 2) isolates per-token computation time, while throughput (Figure 1) measures end-to-end system performance. A 3.39× latency reduction combined with a 1.11× throughput improvement implies that per-token computation is only a small fraction of total iteration time. The paper does not analyze this gap, does not report batch size, and does not provide a breakdown of where time is spent (layer execution vs. KV cache access vs. scheduling vs. kernel launch overhead).

Mitigation status. Not attempted. The paper reports only single-point throughput and latency numbers without any parameter sweep or overhead decomposition. Section 8 (Conclusions) does not mention this as a limitation or call for batch-size-aware evaluation in future work. A practitioner would need to run their own profiling to determine expected gains in their deployment context.

Evaluation Is Restricted to a Single Task, Model Family, and Architecture Type

The assumption or constraint. All experiments use T5-v1.1 models (encoder-decoder architecture, 8 and 12 decoder layers) fine-tuned on CNN/DailyMail summarization and evaluated on that same dataset's test set. The paper acknowledges the narrow scope of available early-exit models:

"Although there are many open-source large models, there are very few open-source early-exit models. Therefore, we implement the recently proposed CALM (Schuster et al. 2022), which is an early-exit version of the T5 model."

This means the evaluation covers exactly one model architecture (T5 encoder-decoder), one task (news summarization), one dataset (CNN/DM), and one model training paradigm (CALM fine-tuning from a pretrained T5 checkpoint).

The consequence. Three significant generalizability questions are left unanswered. First, decoder-only models. The dominant architecture for modern LLMs (GPT, LLaMA, PaLM, Mistral) is decoder-only, not encoder-decoder. The inference patterns differ structurally: decoder-only models have no encoder and no cross-attention, which changes both the per-layer computation cost (self-attention dominates rather than being balanced with cross-attention) and the KV cache structure (only self-attention KV pairs, no encoder output storage). The fraction of per-iteration time spent in layer execution vs. attention over the growing KV cache differs between architectures, which would change the throughput improvement from early exit. The paper does not discuss whether or how the framework would extend to decoder-only early-exit models.

Second, task sensitivity. Summarization involves generating relatively long, variable outputs where many tokens are highly predictable (function words, punctuation, formulaic phrases). This is a favorable setting for early exit because a large fraction of tokens are "easy." Tasks with shorter outputs (classification, multiple-choice QA), tasks where every token carries high semantic weight (code generation, mathematical reasoning), or tasks requiring factual precision (closed-book QA) would likely show lower early-exit rates and smaller throughput gains. The paper does not discuss task sensitivity or provide evidence that the framework would be beneficial beyond summarization.

Third, model scale. The models tested (T5-v1.1-small at ~60M parameters, T5-v1.1-base at ~220M parameters) are orders of magnitude smaller than production LLMs (7B–70B+ parameters). While the paper shows that gains increase from 8 to 12 layers (throughput improvement grows from 1.11× to 1.25× for hidden-state similarity), this two-point trend does not establish that gains continue to scale to 32, 64, or 96 layers. Larger models might exhibit different hidden-state saturation dynamics, different ratios of layer computation to KV cache access cost, and different GPU utilization characteristics (e.g., they may be more memory-bandwidth-bound at large batch sizes, reducing the benefit of reduced computation).

What evidence exists in the paper. The paper explicitly acknowledges the limited availability of early-exit models and the resulting constraint on model choice. However, it does not discuss the implications of this constraint for generalizability. The evaluation provides exactly two data points for model scale (8 and 12 layers) and one data point for architecture and task. The evidence for scalability is strictly limited to these points.

Mitigation status. The paper does not attempt to mitigate this through synthetic experiments (e.g., simulating early-exit on a decoder-only model by artificially varying layer counts), through analysis of how the framework would apply to decoder-only architectures, or through discussion of task characteristics that favor or disfavor early exit. It treats the T5/CALM/CNN-DM combination as the available platform for demonstrating the framework's viability, without claiming broader applicability but also without cautioning against unwarranted generalization.

No Comparison Against a Statically Shallower Model — The Most Important Practical Baseline Is Missing

The assumption or constraint. The paper's central thesis is that early-exit adaptivity — dynamically choosing how many layers to execute per token based on a confidence measure — provides inference efficiency benefits worth the engineering complexity of building a batch-synchronized serving framework with KV cache patching. The evaluation compares the early-exit framework only against full-layer vLLM and, in Table 1, against a static-half baseline (always executing exactly 50% of layers).

The consequence. The static-half result in Table 1 shows that for T5-v1.1-base, executing exactly 6 of 12 layers achieves ROUGE-L 0.3551 — only 0.0052 below the full model's 0.3603 and comparable to or better than all dynamic early-exit techniques (softmax: 0.3557, classifier: 0.3514, state: 0.3516). This means that a statically shallower model — simply taking the first 6 layers of the full model and discarding the rest — achieves essentially the same quality as the sophisticated confidence-based dynamic approach. Would that same 6-layer model, run on unmodified vLLM (no batch synchronization, no KV cache patching, no per-token confidence computation), achieve comparable or better throughput than the early-exit framework?

The paper does not answer this question because it does not evaluate the static-half model in the inference framework. If a statically shallower model achieves the same throughput and quality with zero engineering complexity, then the entire early-exit inference framework — the batch synchronization logic, the KV cache filling mechanism, the per-layer confidence computation overhead — provides no practical benefit over simply using a smaller, faster model. The adaptivity of early exit (sometimes using more than 6 layers when a token is difficult, sometimes fewer when it is easy) would need to provide throughput or quality benefits exceeding what a fixed 6-layer model offers, but the paper provides no evidence that it does.

This is not a hypothetical concern. The fact that static-half achieves ROUGE-L within 0.005 of the full model on the base architecture indicates that the upper layers contribute negligible additional quality on CNN/DM summarization. If the task does not benefit from the model's full depth, then neither early exit nor full-layer execution is the right approach — using a shallower model from the start is simpler and likely faster. The early-exit framework's case would be stronger on a task where the full model's depth is genuinely necessary for some tokens but not others, but the paper provides no evidence that CNN/DM is such a task.

What evidence exists in the paper. Table 1 includes the static-half baseline for quality (ROUGE-L and early-exit rate) but not for throughput or latency. Figures 1 and 2 compare only the dynamic techniques against full-layer vLLM. The static-half model's throughput on the inference framework — or on unmodified vLLM — is never measured. This is the single most important missing experiment for establishing the practical value of the contribution.

Mitigation status. Not addressed. The paper does not acknowledge this as a missing baseline or discuss the implications of the static-half quality results for the justification of dynamic early exit. The static-half row in Table 1 is presented as context for the early-exit rate (it defines the 50% reference point) rather than as a competitive alternative that should be evaluated on the same throughput metrics.

The Difficulty Estimation and Batch Synchronization Overhead Are Not Isolated or Quantified

The assumption or constraint. The batch-synchronized iteration loop (Algorithm 1) introduces two sources of overhead that are not present in standard full-layer inference: (1) the cost of computing per-layer confidence scores for every sequence in the batch at every layer, and (2) the cost of continuing to process already-confident sequences through additional layers until the slowest sequence reaches its threshold. The paper implicitly assumes these overheads are small relative to the savings from early exit, but never measures them.

The consequence. The per-layer confidence computation involves operations not present in standard inference: for hidden-state similarity, computing cosine similarity between consecutive hidden states; for softmax response, applying the output projection head and computing the top-1/top-2 difference; for the classifier, running a dedicated neural network at each layer. These operations consume GPU compute and memory bandwidth. The "over-processing" of already-confident sequences — continuing to execute transformer layers for sequences that passed their threshold at earlier layers — means the framework executes more total FLOPs per batch than a theoretically optimal per-sequence independent execution would require. Both overheads reduce the realized throughput improvement below the theoretical maximum computed from the average early-exit layer.

The paper's results show this gap clearly. For T5-v1.1-base with hidden-state similarity, the 57.26% early-exit rate implies that on average only 5.1 of 12 layers are executed per token (42.7% of layers) — a theoretical ~2.34× reduction in layer computation. The realized throughput improvement is 1.25×. The "missing" 1.09× (nearly half the theoretical gain) must be consumed by some combination of: confidence computation overhead, over-processing of already-confident sequences, KV cache filling cost, and fixed scheduling overhead. Without a breakdown, a practitioner cannot determine which of these factors is dominant or whether a different confidence mechanism with lower per-layer overhead but slightly higher average exit depth would achieve better overall throughput.

What evidence exists in the paper. The gap between theoretical and realized speedup is present in the numbers but not analyzed. The paper does not report: the wall-clock time spent in confidence computation vs. transformer layer execution vs. KV cache filling; the distribution of exit layers across sequences within a batch (which determines the over-processing cost); or the throughput that would be achieved if confidence computation were free (which would isolate its cost).

Mitigation status. Not attempted. The paper does not provide a profiling breakdown or acknowledge the gap between theoretical and realized speedup as a limitation requiring analysis. A practitioner deploying this system would need to profile their specific model and confidence mechanism to understand where the overhead is concentrated and whether threshold tuning or batch composition strategies could recover more of the theoretical gain.

The Framework Requires Training Specialized Early-Exit Models, Not Applicable to Off-the-Shelf LLMs

The assumption or constraint. The inference framework is not a general-purpose serving optimization that can be applied to any pretrained LLM. It requires a model that has been specifically fine-tuned with early-exit mechanisms: per-layer confidence heads (softmax response requires the output projection applied at intermediate layers; hidden-state similarity requires per-layer cosine similarity thresholds; the classifier requires training a dedicated exit classifier at each layer) and decaying threshold calibration. The paper reproduces CALM (Schuster et al., 2022) with full fine-tuning on CNN/DM:

"We conducted full fine-tuning of both T5 v1.1 small and base models on the CNN/DM dataset. For the T5 v1.1 small model, we trained for approximately 40,000 steps with a batch size of 16. The T5 v1.1 base model was trained for about 300,000 steps using two NVIDIA RTX 4090 GPUs with a batch size of 4."

The consequence. This requirement imposes a substantial barrier to adoption. A practitioner with a pretrained LLM (e.g., LLaMA-2-7B, Mistral-7B, GPT-3) cannot simply deploy it on this inference framework and expect speedups. They must first fine-tune the model with early-exit mechanisms, which requires: (1) implementing per-layer confidence measures in the model architecture, (2) generating training data that supports the confidence mechanism (e.g., Monte Carlo rollouts for classifier training), (3) running substantial fine-tuning compute (300,000 steps for the 220M-parameter base model — scaling this to a 7B model would be computationally expensive), and (4) calibrating thresholds on a validation set. This training cost is not accounted for in the inference speedup comparison. If the goal is to reduce the total cost of serving (training + inference), the training cost could be substantial relative to the inference savings, especially for models with short deployment lifetimes or low query volumes.

Furthermore, the early-exit fine-tuning process is non-trivial and fragile. The paper's reproduction of CALM required specific design choices (independent training objective for the classifier, decaying threshold function, edit-distance-based data pairing) that are documented in the original CALM paper and replicated here. There is no evidence that these choices transfer to other model architectures (decoder-only), other model scales (7B+), or other tasks (code, math, dialogue). The ReST^{EM} experiment in Appendix K of the example paper in this document (not this paper) showed that even small changes to the training procedure can cause the revision model to degrade — this paper does not include equivalent robustness checks for the CALM training recipe.

What evidence exists in the paper. The training details in Section 4.2 describe the fine-tuning process and compute requirements. Table 1 validates that the reproduction matches the original CALM paper's quality-efficiency tradeoffs. However, the paper does not discuss: the computational cost of fine-tuning relative to inference savings; whether the fine-tuning process is expected to transfer to other model families; or whether practitioners should expect to need task-specific fine-tuning (as done here, on CNN/DM) or whether a single early-exit fine-tuning generalizes across tasks.

Mitigation status. The paper does not present this as a limitation. It treats the requirement to train early-exit models as a given — the inference framework exists to serve these models efficiently, not to eliminate the need for them. However, a practitioner evaluating whether to adopt this approach needs to weigh the inference speedup against the training cost, and the paper provides no data to inform that tradeoff. The authors do not suggest future work on making early-exit fine-tuning cheaper, more general, or applicable to pretrained models without full fine-tuning.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper establishes that early-exit inference cannot be treated as a model-level optimization that automatically yields system-level speedups — it is a coordination problem between variable-depth computation and GPU batching that requires purpose-built serving infrastructure. Before this work, the literature treated early-exit models and LLM inference systems as separate concerns: CALM (Schuster et al., 2022) and related works demonstrated that models can learn when to skip layers, while Orca (Yu et al., 2022) and vLLM (Kwon et al., 2023) optimized serving for models that always execute every layer. The implicit assumption was that these two lines of work would compose naturally — train an early-exit model, deploy it on vLLM, and enjoy the speedup. This paper demonstrates that this composition fails structurally: the KV cache becomes incomplete when layers are skipped, and the batch synchronization model assumes uniform layer execution. The contribution is a boundary discovery — it identifies exactly where the model-level and systems-level abstractions break when combined, and provides the minimum set of mechanisms (batch-synchronized exit with OR-accumulated status, post-exit KV cache filling via saturated hidden states) needed to bridge the gap.

The magnitude of the shift is diagnostic rather than paradigmatic. The paper does not propose new early-exit techniques, new serving architectures, or new theoretical frameworks. Instead, it surfaces an emergent constraint — the slowest-sequence bottleneck — that governs how variable-depth computation interacts with batched GPU execution, and provides evidence (through the gap between theoretical layer reduction and realized throughput improvement) that this bottleneck is the dominant factor limiting end-to-end gains. This diagnostic reframes early-exit research: the central question is no longer "how do we build better confidence measures?" but rather "how do we manage the variance in exit depth across sequences in a batch to minimize the slowest-sequence penalty?"

The paper also resolves a latent contradiction in the early-exit literature. Prior work reported early-exit rates of 50–67% with minimal quality degradation (Schuster et al., 2022; Elbayad et al., 2019), implying a potential 2–3× theoretical speedup. This paper's results show realized system-level throughput improvements of only 1.11–1.25× for the same techniques on T5-v1.1-small and T5-v1.1-base. The contradiction is not that the prior work was wrong — the early-exit rates and quality metrics are reproduced faithfully in Table 1. Rather, the contradiction is between per-sequence theoretical speedup (which prior work measured) and batched system throughput (which this paper measures). The 3.39× inner-token latency reduction vs. 1.11× throughput improvement for hidden-state similarity on the small model isolates this gap precisely: individual tokens are processed much faster, but the system as a whole is bottlenecked by fixed overheads and batch synchronization. This reconciles the optimistic model-level results with the modest system-level gains.

Several research directions become more attractive as a consequence of this work:

• Batch composition strategies for early-exit serving are now clearly motivated: the slowest-sequence bottleneck means that grouping sequences with similar predicted difficulty could substantially reduce the variance in exit depth and recover more of the theoretical speedup. The paper does not explore this, but the 1.25× result vs. the ~2.3× theoretical maximum quantifies the headroom.
• The approximate KV cache patching mechanism opens a new axis for inference optimization that extends beyond early-exit models: any technique that skips transformer layers (speculative decoding with early rejection, dynamic width networks, layer dropping for distillation) can potentially use the same strategy to maintain KV cache completeness without full computation. The paper's demonstration that this approximation works — even without direct correctness validation — is an existence proof that invites broader application.
• Training early-exit models with batch-awareness becomes a plausible direction: if the model's confidence measure were trained to optimize not just per-token accuracy but also batch-level exit uniformity, the slowest-sequence bottleneck could be reduced at training time rather than mitigated at serving time.

Conversely, certain directions become less attractive. The paper's results suggest that purely model-level innovation in confidence measures — making them more accurate, more calibrated, or more fine-grained — is unlikely to close the gap between theoretical and realized speedup. The bottleneck is not the quality of the exit decision but the variance in exit depth across sequences and the fixed overheads of batched serving. Switching from hidden-state similarity to a 10% more accurate confidence measure would improve the early-exit rate slightly but would not address the batch synchronization cost that consumes roughly half the theoretical gain. Research attention should shift from when to exit (model-level) to how to coordinate exits across sequences (systems-level).

Finally, the paper's identification of hidden-state similarity as the throughput-optimal confidence mechanism across both model sizes and both metrics — despite it having the lowest calibrated-to-full fidelity — challenges an implicit assumption in early-exit research that higher fidelity to the full model's output is always better. The near-identical calibrated-to-label ROUGE-L scores across all three mechanisms (Table 1, base model: 0.3557, 0.3514, 0.3516) demonstrate that the full model's output is just one valid path through generation space, not the ground truth. A confidence mechanism that produces different tokens via earlier exits is not making "errors" — it is finding an alternative valid solution. This finding suggests that early-exit research should optimize for task-level quality metrics directly, not for output-matching fidelity to the full model, which is a misplaced proxy.

Follow-Up Research This Work Enables

Batch-composition scheduling for early-exit models. The paper identifies the slowest-sequence bottleneck (the batch exits when the last sequence reaches its confidence threshold) but does not attempt to mitigate it. A natural follow-up would design a scheduler that groups sequences by predicted difficulty before forming batches, so that each batch has low variance in expected exit depth. The prediction could use a lightweight estimator: average the hidden-state similarity scores from the first few layers of the encoder output, or use a small classifier trained on the CALM training data to predict per-sequence average exit depth. The experiment would measure throughput improvement as a function of batch homogeneity (e.g., comparing random batching vs. difficulty-sorted batching vs. oracle batching where the true average exit depth is known) on the same CNN/DM workload. A strong result would show that difficulty-aware batching closes a significant fraction of the gap between the realized 1.25× throughput and the theoretical ~2.3× maximum, quantified as a throughput-vs-batch-size curve showing the benefit at different system loads.

Direct validation of KV cache approximation correctness. The paper's KV cache filling strategy — applying W_K^i and W_V^i for skipped layers to the exit-layer hidden state — is never directly tested for correctness. A diagnostic experiment would compare three conditions on the same early-exit model: (a) the proposed approximate filling, (b) exact KV cache (execute all layers but discard the output token, keeping only the K and V pairs — this measures the upper bound on correctness), and (c) no filling (mask out positions with missing KV entries, which should produce clearly degraded output as a lower bound). The key metric would be ROUGE-L as a function of output token position, plotted as a sequence-level curve rather than a single aggregate number. If approximate filling introduces accumulating errors, ROUGE-L would degrade with sequence length relative to the exact-KV condition. If the saturation assumption holds, the curves should track closely. This experiment would transform the KV cache filling from a justified-but-untested heuristic into a validated mechanism, and would quantify the position-dependent cost of approximation if it exists.

Early-exit serving for decoder-only LLMs. All experiments use T5 encoder-decoder models because CALM was built on T5 and few open-source early-exit LLMs exist. A high-impact extension would train early-exit mechanisms on a decoder-only model (e.g., LLaMA-2-7B or Mistral-7B) and port the inference framework to vLLM's native decoder-only codebase. This requires: (1) implementing per-layer confidence heads in the decoder-only architecture (which lacks cross-attention, simplifying the per-layer computation profile), (2) fine-tuning with the CALM recipe on a generation task (CNN/DM summarization or a comparable task like XSum), and (3) measuring throughput and latency against full-layer vLLM on the same model. The key question is whether the throughput improvement scales with model size — the paper shows gains increasing from 8 to 12 layers (1.11× → 1.25× for hidden-state similarity), but decoder-only models at 32+ layers may show different behavior due to different ratios of layer computation to KV cache access cost and different GPU utilization characteristics at scale.

Dynamic mid-iteration batch repartitioning. The current algorithm processes all sequences through every layer until torch.all(Status) is met, which means already-confident sequences are over-processed. An alternative is to repartition the batch after each layer: remove sequences whose status is True, form a smaller batch with only the remaining sequences, and continue. This reduces over-processing at the cost of more frequent batch reformation and smaller effective batch sizes for later layers (which hurts GPU utilization). A head-to-head experiment would compare the torch.all strategy against per-layer repartitioning across a sweep of initial batch sizes, measuring both throughput and GPU utilization. The hypothesis is that repartitioning wins at large initial batch sizes (where the remaining batch after early exits is still large enough to saturate the GPU) but loses at small batch sizes (where the overhead of reformation dominates). The result would either validate the paper's choice of torch.all as the right default or identify a regime where more aggressive repartitioning is justified.

Training batch-aware confidence measures. The paper uses confidence measures trained with per-token objectives (correctness prediction, representation similarity) without considering the batch-level consequence of the exit decision. A novel training objective could penalize high variance in exit depth across sequences within the same batch, encouraging the model to produce more uniform confidence scores when processing sequences that are likely to appear together in a batch. Concretely: during training, sample mini-batches and add a regularization term proportional to the variance of the layer index at which each sequence's confidence first exceeds the threshold. The experiment would train two versions of the CALM model — with and without this regularization — and compare both early-exit rates and realized throughput at the same quality level. A positive result would show that batch-aware training achieves higher throughput than the standard CALM recipe at matched ROUGE-L, by reducing the slowest-sequence penalty without degrading per-token accuracy.

Stress-testing on adversarial sequence mixtures. The slowest-sequence bottleneck predicts that mixing sequences with very different difficulty profiles in the same batch will severely degrade throughput. An adversarial stress test would construct batches where one sequence is known to require all layers (e.g., a long, complex article with ambiguous continuations) while the remaining sequences are highly predictable (short, formulaic articles). The throughput on these mixed batches should be close to the full-layer baseline — the single difficult sequence anchors the entire batch. Measuring this degradation quantitatively (throughput as a function of the fraction of difficult sequences in the batch) would establish the worst-case performance envelope of the framework and provide guidance on when early-exit serving is beneficial vs. when it is counterproductive. If a small fraction of difficult sequences can eliminate most of the throughput gain, then production deployments need either difficulty-aware routing (separating easy and hard queries into different hardware) or a hybrid strategy that falls back to full-layer execution for certain batches.

Practical Applications and Downstream Use Cases

Cost-efficient summarization pipelines at scale. The most direct application is serving summarization models in production, where CNN/DM-style workloads (news articles → summaries) are common. A deployment serving T5-v1.1-base with the hidden-state similarity early-exit mechanism on this framework achieves 1402 tokens/s vs. 1118 tokens/s on full-layer vLLM — a 25% throughput improvement on the same GPU. For a service generating millions of summaries per day, this translates to either serving 25% more traffic with the same GPU fleet or reducing GPU count by 20% for the same workload. The quality impact is negligible: ROUGE-L drops from 0.3603 to 0.3516 — a 2.4% relative reduction that is unlikely to be noticeable in most production summarization applications. The key practical requirement is that the model must be fine-tuned for early exit (300,000 steps on 2× RTX 4090s for the base model), so the application is most compelling for organizations that already plan to fine-tune a summarization model and can absorb the one-time training cost in exchange for ongoing inference savings.

Latency-sensitive interactive generation tasks. The inner-token latency results (Figure 2) show 2.3–3.4× reductions in per-token decoding time: from 0.176s to 0.066s per token for T5-v1.1-base with hidden-state similarity. While throughput is the primary metric for batch serving, latency matters for interactive applications where a user is waiting for a response — summarization assistants, real-time translation, or any text generation with a human in the loop. A 2.7× latency reduction (0.176 → 0.066s per token) means a 100-token summary appears in 6.6 seconds instead of 17.6 seconds, which crosses the threshold from "noticeably slow" to "responsive" for many applications. The caveat is that the paper's throughput results suggest fixed per-iteration overheads are significant, so the latency improvement at batch size 1 (which is the relevant setting for interactive use) might be smaller than the reported numbers if the experiments used larger batches. A practitioner deploying for low-latency interactive use would need to measure per-token latency specifically at batch size 1 on their hardware.

On-device or edge deployment with limited compute. The early-exit framework is particularly relevant for edge deployment scenarios (as flagged by the paper's reference to AdaInf, Shubha and Shen, 2023) where GPU resources are constrained and model size is a hard limit. The T5-v1.1-small model (8 layers, ~60M parameters) running with hidden-state similarity achieves 1166 tokens/s — faster than the full-layer T5-v1.1-base (1118 tokens/s) while being 3.7× smaller in parameters. This means an edge device could run the small early-exit model and achieve both higher throughput and lower memory consumption than a full-layer base model, while maintaining comparable quality (ROUGE-L 0.3400 vs. 0.3603). The application is most compelling for scenarios where the model must run locally (privacy, offline operation, latency) and the quality tradeoff is acceptable.

When to Prefer This Method

The paper's evaluation compares against exactly one alternative: standard vLLM running full-layer inference on the same model. Table 1 additionally reports a static-half baseline (always execute 50% of layers) but only for quality, not throughput. The paper does not explicitly position its method against other inference optimization strategies (quantization, distillation, pruning, speculative decoding) or against simply using a statically shallower model. Therefore, a broad "prefer A when... prefer B when..." matrix would be fabricating a tradeoff the paper does not articulate. The paper's implicit decision rule, based on the evidence presented, is:

• Prefer the early-exit inference framework when you are already committed to deploying an early-exit model (CALM or similar) that has been fine-tuned with per-layer confidence mechanisms, and you need to serve this model efficiently in a batched multi-sequence setting. The framework provides a 1.11–1.25× throughput improvement over treating the early-exit model as a standard full-layer model in vLLM, with a 2.3–3.4× reduction in per-token latency, at negligible quality cost.

• The paper does not provide evidence to prefer the early-exit framework over: (a) using a statically shallower model served on unmodified vLLM (the static-half throughput is never measured), (b) applying other inference optimizations (quantization, distillation) to a full-layer model, or (c) training a smaller model from scratch to match the quality-throughput tradeoff point. These comparisons are outside the paper's scope, and a practitioner would need to run their own benchmarking to determine whether the early-exit framework's benefits justify the training cost and engineering complexity relative to these alternatives.