Inference Optimization2025~123 minarXiv:2503.01840

EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test

score 10Yuhui Li, Fangyun Wei, Chao Zhang +1↑ 6★ 2,205Mar 3, 2025cs.CLhttps://github.com/safeailab/eagle

🎯 Pitch

EAGLE-3 achieves up to ~6.5× speedup over standard autoregressive decoding—a ~1.4× gain over its predecessor—while establishing, for the first time, a scaling law where inference acceleration improves with more draft-model training data. The key was replacing feature prediction with direct token prediction via multi-layer fusion, solving a previously unseen expressiveness bottleneck that prevented prior systems from benefiting from larger datasets.

1. Executive Summary

This paper introduces EAGLE-3, an enhanced speculative sampling framework that accelerates LLM inference by replacing the feature-level autoregression of prior EAGLE methods with direct token prediction and multi-layer feature fusion, trained via a training-time test procedure that simulates multi-step generation during training (the draft model feeds its own predictions back as input for subsequent steps). Evaluated on five tasks—MT-bench, HumanEval, GSM8K, Alpaca, and CNN/Daily Mail—using LLaMA-Instruct and Vicuna chat models plus a DeepSeek-R1 distilled reasoning model, EAGLE-3 achieves speedup ratios up to ~6.5× over vanilla autoregressive decoding, representing a ~1.4× improvement over EAGLE-2, and attains a 1.38× throughput gain at batch size 64 in the SGLang production framework. Critically, the removal of the feature prediction constraint and the adoption of fused multi-layer features enable a scaling law for inference acceleration—increasing the draft model's training data yields proportional speedup gains—establishing that the expressiveness bottleneck in prior feature-predicting draft models limited their capacity to benefit from additional data, a limitation EAGLE-3 resolves.

2. Context and Motivation

The Core Problem: LLM Inference Is Fundamentally Expensive and Slow

The paper addresses a problem that has become increasingly acute as LLMs have scaled: the sequential, autoregressive nature of decoding makes inference slow and expensive. When an LLM generates text, it must produce one token at a time, with each token requiring a full forward pass through all model parameters. As the authors state in Section 1, "each token requires accessing all model parameters, making LLM inference slow and costly." For modern models with hundreds of billions of parameters, this means hundreds of billions of memory accesses and floating-point operations for every single token generated.

This problem is not merely an inconvenience—it has direct consequences for deployment feasibility, user experience, and the economics of LLM-powered applications:

Latency sensitivity: Users expect near-instantaneous responses in interactive settings (chatbots, code assistants, search). Each additional token of latency compounds user frustration, and the sequential bottleneck means that response time scales linearly with output length regardless of how much parallel compute is available.
Cost structure: For API-based LLM services, inference costs typically dominate training costs at scale. Reducing the FLOPs per generated token directly translates to lower operational costs.
The rise of reasoning models: The paper specifically highlights that models like ChatGPT o1 and DeepSeek-R1 "engage in deliberate reasoning before responding, pushing the boundaries of LLM capabilities at the cost of longer inference time." These reasoning models, which generate lengthy chain-of-thought traces before producing a final answer, dramatically increase the proportion of total inference time relative to prefill. As the authors note, "these reasoning models significantly increase the proportion of inference costs in the overall LLM pipeline, driving researchers to explore cheaper and faster inference optimization methods."

This third point—the emergence of reasoning models that deliberately extend generation length to improve quality—creates heightened urgency. When inference was a small fraction of the total compute budget, optimization was nice-to-have. When inference can require thousands of tokens of intermediate reasoning, acceleration becomes essential.

The Hardware Disconnect: Memory-Bound Execution Meets Underutilized Compute

Autoregressive decoding exhibits a fundamental hardware inefficiency that creates an opportunity for optimization. During decoding, the model is memory-bound: the primary bottleneck is loading weights and KV cache entries from memory, not performing computations. Modern GPUs have abundant parallel compute (FLOPS) that sits idle during the decode phase because the arithmetic intensity—the ratio of compute operations to memory accesses—is so low. Each token's forward pass requires reading all model parameters from memory but performs relatively few operations per parameter.

Speculative sampling exploits this compute surplus by doing more total work (generating draft tokens, verifying them in parallel) in exchange for reducing the number of sequential forward passes through the target model. The intuition: because the GPU's compute units are underutilized during memory-bound decoding, we can use that "free" compute to run a cheaper draft model that proposes multiple tokens at once. If those tokens are verified successfully by the target model in a single parallel forward pass, we've amortized the cost of one expensive memory-bound pass across multiple generated tokens.

This is the critical insight that makes speculative sampling work: it converts underutilized compute into reduced sequential steps. The paper builds on this insight by making the draft model more effective (higher acceptance rates) and therefore better at converting compute into latency savings.

The Gap: Prior Speculative Sampling Methods Are Expressiveness-Limited

The paper positions itself against the lineage of speculative sampling methods, tracing the evolution from vanilla speculative sampling through EAGLE and EAGLE-2, while identifying specific limitations that EAGLE-3 overcomes.

Vanilla Speculative Sampling: Independent Draft Models

Standard speculative sampling (Leviathan et al., 2023; Chen et al., 2023) uses a separate, smaller LLM—typically a lower-parameter version from the same model family—as the draft model. The draft model autoregressively generates a sequence of candidate tokens (the "draft"), and the target model verifies all draft tokens in a single parallel forward pass. Accepted tokens skip the sequential bottleneck; rejected tokens are resampled from an adjusted distribution.

The drawback is fundamental: a smaller independent model has only limited capacity to approximate the larger target model's behavior. The draft model operates on the same input tokens as the target model but lacks access to the target model's internal representations. It must predict what the target model would have said based solely on the text, without the rich internal state that the target model computes during its forward pass. This limits acceptance rates and, consequently, speedup ratios.

EAGLE: Feature-Level Autoregression with Target Model Reuse

EAGLE (Li et al., 2024c) addresses the independent draft model limitation by reusing the target model's top-layer features—the hidden states immediately before the LM head—as input to the draft model. Rather than predicting tokens directly, EAGLE performs autoregression at the feature level: it predicts the next feature vector from previous feature vectors, then feeds that predicted feature through the target model's LM head (which is cheap and reused without modification) to obtain token probabilities.

The key advantage is informational richness: the top-layer features encode the target model's full understanding of the context and its predictions for the next token. By conditioning on these features, the EAGLE draft model operates on a much more informative signal than raw tokens alone. As the paper notes, EAGLE "achieves significantly better acceleration compared to vanilla speculative sampling."

However, EAGLE's design introduces a specific constraint that the paper identifies as the root cause of limited scaling behavior:

"EAGLE's loss function consists of two components: the feature prediction loss $l_{fea}$ and the token prediction loss $l_{token}$ . Thanks to the feature prediction loss, the draft model trained only at Step 1 can adapt to Step 2 and acquire multi-step prediction capabilities."

The feature prediction loss serves a specific purpose: it ensures that the draft model's output at step 1 is close to the true top-layer feature $f_{t+1}$ , which means that when that predicted feature is used as input for step 2 (along with the true features from earlier positions), the input distribution remains close to what the model saw during single-step training. This is what gives EAGLE its multi-step generation capability without requiring explicit multi-step training.

But the paper identifies a tradeoff:

"With token prediction as the ultimate goal, feature prediction can be seen as an additional constraint, which limits the expressiveness of the draft model and makes it difficult to benefit from increased data."

This is the central insight that motivates EAGLE-3. The feature prediction loss forces the draft model to not just produce the right token (via the LM head) but also to produce a feature vector that matches the target model's internal representation—an intermediate objective that may be partially orthogonal to the final token prediction goal. It's a regularization that helps with multi-step generalization but constrains the model's capacity to improve with more data.

EAGLE-2: Dynamic Draft Trees (Orthogonal Improvement)

EAGLE-2 (Li et al., 2024b) addresses a different limitation: the static, context-independent draft tree structure. In EAGLE and Medusa, the draft tree—which specifies how many tokens to generate and at which positions—is predefined and fixed for all inputs. EAGLE-2 introduces context-aware dynamic draft trees that estimate acceptance rates based on the draft model's confidence scores and prune unpromising branches, making more efficient use of the drafting budget.

EAGLE-3 adopts this dynamic draft tree technique from EAGLE-2 but targets the more fundamental bottleneck: the draft model's expressiveness and data scaling behavior. The two improvements are orthogonal and complementary.

The Data Scaling Gap: Why EAGLE Hits a Wall

The paper's most striking motivation comes from an empirical observation that directly demonstrates the expressiveness bottleneck:

"Recent LLMs have increasingly relied on larger training datasets to achieve better performance. … Similarly, we aim to improve the acceptance rate and acceleration ratio of EAGLE by increasing its training data. Unfortunately, we observe that the gains from additional training data for EAGLE are limited."

This is shown concretely in Figure 1 (left panel): EAGLE-2's speedup ratio essentially plateaus as training data increases from 1× to 8× (relative to the ShareGPT dataset size), hovering around 3.2–3.4× for LLaMA-Instruct 3.1 8B on MT-bench. The scaling curve is essentially flat—more data provides negligible returns. In contrast, EAGLE-3's speedup ratio rises from approximately 4.1× at 1× data to roughly 4.4× at 8× data, with the trend suggesting continued improvement with further data scaling.

The paper diagnoses why this plateau occurs. When the feature prediction constraint is removed (the middle configuration in Figure 3, labeled "EAGLE + $l_{fea}$ removal"), something interesting happens:

"As shown in Figure 4, the acceptance rate 0-α of the first draft token improves significantly. However, the output of the draft model in Step 1, denoted as $\hat{a}_{t+1}$ , is far away from the ground-truth $f_{t+1}$ , causing the input sequence $f_1, f_2, \cdots, f_t, \hat{a}_{t+1}$ in Step 2 to deviate significantly from the training distribution, resulting in a very low acceptance rate 1-α for the second draft token."

In plain language: removing the feature constraint frees the model to optimize purely for token prediction, which improves first-token accuracy. But this freedom means the draft model's output vector is no longer constrained to be close to the target model's true feature vector. When this (now-unconstrained) output is fed back as input for the next drafting step, the model encounters an out-of-distribution input—it was trained on true target model features, but at test time it receives its own predictions. This distribution shift causes the second token's accuracy (acceptance rate 1-α) to collapse.

Figure 4 quantifies this: the 0-α rate (first token) improves noticeably with "EAGLE without fea pred," but the 1-α rate (second token) drops to dramatically lower levels than with feature prediction. The net effect is that removing the constraint alone doesn't help—the single-step gain is offset by multi-step degradation.

This sets up the paper's key methodological contribution: training-time test, which combines (a) removing the feature prediction constraint to gain expressiveness with (b) simulating multi-step generation during training to close the train-test distribution gap. The bottom configuration in Figure 3 shows this approach: during training, the draft model's own predictions from Step 1 are fed back as input for Step 2, and the loss is computed on the token predictions at all steps. This ensures the model learns to handle its own prediction errors as input, eliminating the distribution shift that plagued the naive removal of feature prediction.

The Top-Layer Feature Limitation: Information Bottleneck for Multi-Token Prediction

EAGLE and related methods (Medusa, Hydra) specifically reuse the top-layer features of the target model—the hidden states from the final transformer layer, immediately before the LM head projection. The authors identify a subtle but important limitation of this design choice:

"For an LM head with a full-rank weight matrix, the top-layer features corresponding to the logits of the next token are unique, ensuring that the information contained in these features aligns directly with the logits of the next token."

This is a correctness argument: the top-layer features are informationally sufficient for predicting the next token, because they are the input to the LM head that produces that token's logits. No information is lost by using only the top layer.

"However, predicting the next-next token based solely on top-layer features—which are inherently limited to the next token—poses a significant challenge."

This is the bottleneck. The top-layer features are optimized to predict one token ahead. When the draft model needs to predict two or more tokens ahead (which is essential for multi-step drafting), the features that are sufficient for the immediate next token may lack the richer semantic and syntactic information needed to plan further ahead. Lower and middle layers of the transformer encode different kinds of information—syntactic structure, entity relationships, discourse-level patterns—that may be more useful for long-range prediction.

The paper argues that the training-time test technique makes it possible to address this limitation:

"Fortunately, the training-time test technique described above enables the use of features from intermediate layers instead of relying solely on the top layer, as the feature prediction loss $l_{fea}$ has been removed during training."

Why does removing $l_{fea}$ enable multi-layer feature fusion? Because the feature prediction loss required the draft model's output to approximate the top-layer feature specifically (since that's what gets passed to the LM head). When the loss was $l_{fea} + l_{token}$ , the feature target was $f_{t+1}$ (the true top-layer feature), and the token target was derived by passing that feature through the LM head. With only $l_{token}$ remaining, there is no longer any requirement that the draft model's intermediate output match any specific layer of the target model. The draft model can construct its input from any combination of layers, learn its own internal representation, and optimize directly for the downstream token prediction loss.

Positioning Relative to HASS: Shared Mechanics, Divergent Motivations

The paper explicitly distinguishes EAGLE-3 from HASS (Zhang et al., 2024), which also modifies attention during training to simulate multi-step drafting. This comparison is important because the methods share surface-level similarities (both simulate test-time behavior during training) but differ fundamentally in motivation and outcome:

"HASS still performs feature prediction, includes a feature prediction loss $l_{fea}$ , and the input to the draft model must be the top-layer features. In contrast, the motivation behind EAGLE-3 is to remove unnecessary constraints to enhance the model's expressive power."

HASS's goal is to mitigate error accumulation from inaccurate feature predictions while retaining the feature prediction framework. EAGLE-3's goal is to eliminate the feature prediction framework entirely because it imposes an expressiveness bottleneck. HASS patches the existing approach; EAGLE-3 replaces it.

The outcomes reflect this difference: "Figure 2 also shows the speedup of EAGLE-3 and HASS, with EAGLE-3 demonstrating significantly better performance." Across Vicuna-13B, LLaMA-Instruct 3.1 8B, and other models, EAGLE-3 achieves speedup ratios substantially higher than HASS (and all other speculative sampling methods), validating the decision to remove rather than merely mitigate the feature prediction constraint.

How EAGLE-3 Positions Itself

The paper frames EAGLE-3 as a successor that resolves fundamental limitations of the EAGLE lineage rather than a competitor to prior speculative sampling methods. The positioning has three components:

Expressiveness argument: The feature prediction constraint in EAGLE/EAGLE-2 served a purpose (enabling multi-step generation from single-step training) but imposed a ceiling on how much the draft model could improve with additional data. EAGLE-3 removes this ceiling through training-time test, allowing the draft model to benefit from data scaling.
Information argument: Top-layer features are sufficient for next-token prediction but suboptimal for multi-token prediction. Multi-layer feature fusion—enabled by the removal of the feature prediction loss—provides richer representations for longer-range drafting.
Compatibility argument: EAGLE-3 is not competing with EAGLE-2's dynamic draft tree mechanism; it adopts it. The two improvements are orthogonal—EAGLE-3 improves the draft model's per-token accuracy and multi-step generalization, while EAGLE-2's dynamic trees improve how the drafting budget is allocated across candidate tokens. The paper demonstrates that combining them yields the best of both.

The paper also draws an interesting connection to a broader trend in the LLM community: scaling training data to improve performance without changing inference cost. The authors note that LLaMA 1, 2, and 3 each scaled training tokens substantially (1T → 2T → 15T) while keeping inference architecture and cost constant, resulting in significant capability improvements. EAGLE-3's key contribution is to enable a parallel scaling law for inference acceleration—more training data for the draft model yields faster inference, not better task performance, but the principle is analogous: invest in training data to improve a downstream metric without changing the deployment architecture.

3. Technical Approach

3.1 Reader Orientation

EAGLE-3 is a draft model for speculative sampling that accelerates LLM inference by rapidly proposing candidate tokens for the target model to verify in parallel. The system replaces the feature-level autoregression of prior EAGLE methods with direct token prediction and multi-layer feature fusion, trained using a training-time test procedure where the draft model learns to handle its own prediction errors as input for subsequent steps—solving the expressiveness bottleneck that prevented prior methods from benefiting from additional training data.

3.2 Big-Picture Architecture (Diagram in Words)

The EAGLE-3 system has five major components that operate in an alternating draft-then-verify cycle:

Target LLM — The large language model being accelerated (e.g., LLaMA-Instruct 3.1 8B, DeepSeek-R1-Distill-LLaMA 8B). Its forward pass is expensive and memory-bound. During inference, it runs two types of forward passes: (a) one during the prefilling/verification stage to check draft tokens and produce new features, and (b) none during the drafting stage when only the draft model runs.
Feature Extraction — During the target model's forward passes, three feature vectors are captured from different transformer layers: low-level ( $l$ , input to the first decoder layer), middle-level ( $m$ , input to a middle decoder layer), and high-level ( $h$ , input to the top decoder layer). Each is a $k$ -dimensional vector where $k$ is the target model's hidden size.
Feature Fusion FC Layer — A fully connected layer that concatenates the three $k$ -dimensional vectors $l$ , $m$ , and $h$ into a $3k$ -dimensional vector, then projects it back down to $k$ dimensions, producing a fused feature $g$ . This $g$ integrates semantic information from multiple levels of the target model's processing hierarchy.
Draft Model (Single Transformer Decoder Layer) — A lightweight autoregressive model composed of one transformer decoder layer plus input/output projections. It takes the fused feature $g$ from the current position and the token embedding $e$ from the previously sampled token, concatenates them, projects down to $k$ dimensions via a second FC layer, then passes the result through self-attention and feed-forward sublayers to produce an output vector $a$ . This output is fed to the target model's LM head (reused without modification) to obtain logits, from which a draft token is sampled.
LM Head (Reused from Target Model) — The target model's final projection layer that maps $k$ -dimensional hidden states to vocabulary-sized logits. EAGLE-3 feeds the draft model's output $a$ through this LM head to produce draft token probabilities. Since the LM head is just a matrix multiplication, this is computationally cheap and introduces no additional parameters.

Information flow during drafting (Figure 5): The prefix "How can" is processed by the target model (prefill or prior verification), producing features $l_{how}, m_{how}, h_{how}$ and $l_{can}, m_{can}, h_{can}$ . These are fused into $g_{how}$ and $g_{can}$ . The draft model takes $[g_{how}; e_{how}]$ and $[g_{can}; e_{can}]$ as input (where $e$ denotes token embeddings of sampled tokens) and produces output $a_{can}$ . The LM head converts $a_{can}$ to logits for the next token; sampling yields "I". In Step 1 of drafting (the first draft token after the verified prefix), the draft model takes $[g_{how}; e_{how}]$ , $[g_{can}; e_{can}]$ , and $[g_{I}; e_{I}]$ as input, producing $a_{I}$ , which goes through the LM head to sample "do". In Step 2, $g_{do}$ is unavailable (the target model hasn't verified "do" yet), so EAGLE-3 substitutes the draft model's own output $a_{I}$ as a proxy for $g_{do}$ . The input for Step 2 combines $[g_{how}; e_{how}]$ , $[g_{can}; e_{can}]$ , $[g_{I}; e_{I}]$ , and $[a_{I}; e_{do}]$ , producing $a_{do}$ and sampling "it". In Step 3, $a_{do}$ substitutes for the unavailable $g_{it}$ , combined with $e_{it}$ to predict the next token. This self-feeding of the draft model's own outputs as substitutes for missing target model features is the core inference-time mechanism, and the training-time test procedure ensures the model is prepared for it.

3.3 Roadmap for the Deep Dive

First, the training-time test mechanism — the paper's central technical innovation. We trace how EAGLE's feature prediction loss constrained expressiveness, why naive removal broke multi-step generalization, and how training-time test resolves the train-test distribution gap by simulating self-feeding during training.
Second, the feature extraction and fusion architecture — how $l$ , $m$ , and $h$ are selected from the target model, how they are concatenated and projected to form $g$ , and why multi-layer fusion is only possible after removing $l_{fea}$ .
Third, the draft model architecture — the single decoder layer, the input projection from concatenated feature+embedding pairs, the output path through the reused LM head, and the attention mask modifications for training-time test.
Fourth, the attention mechanism modifications — the tree-structured causal mask that enables training on multi-step self-generated inputs, and the computational efficiency technique (vector dot products instead of matrix multiplication for diagonal attention patterns).
Fifth, the loss function — what is being optimized, the removal of $l_{fea}$ and retention of $l_{token}$ , and how gradients flow through the self-feeding loop during training-time test.
Sixth, the inference pipeline — how the training architecture maps to inference behavior, the alternating draft-verify cycle, the integration with EAGLE-2's dynamic draft trees, and the computational cost model.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and methods paper whose core idea is that removing the feature prediction constraint and simulating multi-step self-feeding during training enables a more expressive draft model that scales with training data, while multi-layer feature fusion provides richer representations for multi-token prediction.

The Feature Prediction Bottleneck: Why EAGLE Hit a Scaling Ceiling

To understand training-time test, we must first understand what it replaces. In the original EAGLE (Li et al., 2024c), the draft model is trained with two loss functions operating simultaneously on its output:

Loss 1: Feature prediction loss ( $l_{fea}$ ). The draft model's output vector $\hat{f}_{t+1}$ is trained to directly match the target model's true top-layer feature $f_{t+1}$ at the next position. This is typically a regression loss (e.g., smooth L1 or MSE) between the draft model's predicted feature vector and the ground-truth hidden state from the target model:

$l_{fea} = \text{SmoothL1}(\hat{f}_{t+1}, f_{t+1})$

where $\hat{f}_{t+1} \in \mathbb{R}^k$ is the draft model's output and $f_{t+1} \in \mathbb{R}^k$ is the true top-layer feature from the target model at the position being predicted.

What it computes: a distance metric between the draft model's predicted feature representation and the target model's actual internal representation. The smooth L1 loss is less sensitive to outliers than MSE and provides stable gradients when predictions are far from targets.

Why this form: the feature prediction loss serves a distribution-matching purpose. By forcing $\hat{f}_{t+1} \approx f_{t+1}$ , it ensures that when the draft model's output is substituted for a true feature as input for the next drafting step, the input distribution at step 2 remains close to what the model was trained on (true features). This is what gives EAGLE multi-step capability from single-step training: the model never actually sees its own predictions during training, but because those predictions are constrained to be near the truth, the distribution shift at test time is small.

Loss 2: Token prediction loss ( $l_{token}$ ). The predicted feature $\hat{f}_{t+1}$ is passed through the target model's frozen LM head to produce token logits, and a cross-entropy loss is computed against the true next token:

$l_{token} = \text{CrossEntropy}(\text{LMHead}(\hat{f}_{t+1}), t_{t+1})$

where $\text{LMHead}(\hat{f}_{t+1})$ produces a probability distribution over the vocabulary and $t_{t+1}$ is the ground-truth next token.

The total EAGLE loss is a weighted combination:

$l_{\text{EAGLE}} = l_{token} + \lambda \cdot l_{fea}$

where $\lambda$ controls the trade-off between token accuracy and feature matching.

The expressiveness problem. The authors identify that $l_{fea}$ acts as a constraint on the draft model's representational capacity. The draft model must simultaneously optimize for two potentially conflicting objectives: (a) produce a feature vector that matches the target model's internal representation at that position, and (b) produce a feature vector that, when passed through the LM head, gives high probability to the correct token. These objectives are not perfectly aligned—there may exist feature vectors that produce better token predictions but don't match the target model's internal features, especially when the draft model has more flexibility to learn its own representations.

The constraint becomes more severe with scale: as training data increases, a constrained model hits diminishing returns because it cannot exploit additional data to learn representations that deviate from the target model's feature space. This is the mechanism behind the flat scaling curve in Figure 1 (the EAGLE-2 line): more data can't help if the model is forced to predict an intermediate representation rather than the final objective.

Naive Removal of Feature Prediction and Its Failure Mode

The obvious fix is to remove $l_{fea}$ and train the draft model with only $l_{token}$ . The draft model's output $\hat{a}_{t+1}$ is no longer constrained to match any target model feature—it is optimized purely to produce correct token probabilities when fed through the LM head.

This is shown in the middle configuration of Figure 3 ("EAGLE + $l_{fea}$ removal"). During training, the draft model receives true features $f_1, f_2, \ldots, f_t$ as input and predicts $\hat{a}_{t+1}$ . The loss is:

$l_{\text{token-only}} = \text{CrossEntropy}(\text{LMHead}(\hat{a}_{t+1}), t_{t+1})$

where $\hat{a}_{t+1} \in \mathbb{R}^k$ is now an unconstrained vector (the authors denote this with $a$ rather than $\hat{f}$ to emphasize it is not approximating a target model feature).

What happens to first-token prediction. Figure 4 (left panel, 0-α) shows that the first draft token's acceptance rate improves significantly with this change. For example, at 1× data scale, the 0-α acceptance rate rises from approximately 0.74 (EAGLE) to roughly 0.78 (EAGLE without feature prediction). This is the expressiveness gain: the model, freed from the feature-matching constraint, learns representations that better optimize token prediction.

What happens to second-token prediction. Figure 4 (right panel, 1-α) reveals the catastrophic failure: the second draft token's acceptance rate plummets. At 1× data scale, 1-α drops from approximately 0.65 (EAGLE) to roughly 0.35 (EAGLE without feature prediction)—nearly halved. The reason, as the paper explains:

"the output of the draft model in Step 1, denoted as $\hat{a}_{t+1}$ , is far away from the ground-truth $f_{t+1}$ , causing the input sequence $f_1, f_2, \cdots, f_t, \hat{a}_{t+1}$ in Step 2 to deviate significantly from the training distribution."

During training, the draft model always receives true features $f$ as input—it never sees its own unconstrained output $\hat{a}$ as part of an input sequence. At inference time, Step 2's input includes $\hat{a}_{t+1}$ (Step 1's output) as a substitute for the unavailable $f_{t+1}$ . Because $\hat{a}_{t+1}$ is optimized purely for token prediction through the LM head, it can drift far from the distribution of true features that the model was trained to process. The model encounters out-of-distribution inputs and produces poor predictions.

This is the classic train-test distribution mismatch: the model is trained to predict given true features but tested given its own predictions. The feature prediction loss in EAGLE was an implicit solution to this problem—it constrained the output to stay near the training distribution. Removing it without addressing the mismatch breaks multi-step generation.

Training-Time Test: Bridging the Train-Test Distribution Gap

Training-time test is the technique that enables EAGLE-3 to simultaneously remove the feature prediction constraint (gaining expressiveness) and maintain multi-step generation capability (by eliminating the distribution shift). The core idea is simple: during training, simulate exactly what happens during inference—feed the draft model's own predictions back as input for subsequent steps, and compute losses on the predictions at all steps.

The procedure, shown in the bottom configuration of Figure 3, operates as follows:

Step 1 (training mode, first draft token). Given the prefix token sequence $T_{1:t}$ , the target model's forward pass produces true features $f_1, f_2, \ldots, f_t$ (in EAGLE-3, these become fused features $g_1, g_2, \ldots, g_t$ , but we use $f$ here for consistency with Figure 3's notation). The draft model takes $g_{1:t}$ and token embeddings $e_{1:t}$ as input, produces output $\hat{a}_{t+1}$ , which goes through the LM head to predict token $t_{t+1}$ . Loss $l^{(1)}_{token}$ is computed between this prediction and the ground truth.

Step 2 (training-time test mode, second draft token). Instead of using the true $g_{t+1}$ (which would hide the test-time distribution shift), the training procedure uses the draft model's own output $\hat{a}_{t+1}$ from Step 1 as a substitute for $g_{t+1}$ . The input sequence for Step 2 is $g_1, g_2, \ldots, g_t, \hat{a}_{t+1}$ combined with token embeddings $e_1, e_2, \ldots, e_t, e_{t+1}$ (where $e_{t+1}$ is the embedding of the ground-truth token at position $t+1$ —this uses teacher forcing for the token but self-generated features). The draft model processes this input and produces $\hat{a}_{t+2}$ , which goes through the LM head to predict $t_{t+2}$ . Loss $l^{(2)}_{token}$ is computed.

Step 3 and beyond. The same pattern continues: each step's input includes the outputs from all previous steps ( $\hat{a}_{t+1}, \hat{a}_{t+2}$ , etc.) as substitutes for the unavailable true features, combined with ground-truth token embeddings for the teacher-forced token sequence.

The total loss is the sum of token prediction losses across all steps of the training-time test procedure:

$l_{\text{EAGLE-3}} = \sum_{s=1}^{S} l^{(s)}_{token}$

where $S$ is the number of training-time test steps (the paper mentions simulating "two simulated training steps (the second and third steps)" in Figure 6, and uses chain lengths of up to 2 or 3 in the SGLang/vLLM experiments; the exact $S$ used during training is not explicitly stated but can be inferred to be at least 3 from Figure 6).

Why this works. During training, the draft model experiences exactly the input distribution it will encounter during inference: a mix of true features from the target model (for positions that have been verified) and self-generated output vectors (for positions being drafted ahead). The model learns to compensate for its own prediction errors—if $\hat{a}_{t+1}$ is systematically different from true features in some subspace, the model learns to interpret those differences correctly at Step 2. The distribution shift is eliminated because the training distribution now matches the test distribution.

Why this enables removal of $l_{fea}$ . In EAGLE, $l_{fea}$ was necessary because the model was never exposed to its own predictions during training—the loss provided a proxy guarantee (if predictions are close to truth, the distribution shift is small). In EAGLE-3, the model learns directly from its own predictions, making the proxy unnecessary. The constraint can be removed entirely, freeing the model to optimize purely for token prediction accuracy at all steps.

Computational cost during training. Training-time test requires multiple forward passes through the draft model per training example (one per simulated step), similar to truncated backpropagation through time. However, the draft model is a single transformer decoder layer, making this cost modest relative to the target model. The target model is only run once per training example to produce the features $g$ for all prefix positions—its forward pass is not repeated for each simulated drafting step. This is a key efficiency consideration: the expensive target model serves as a feature extractor for the cheap draft model training.

Feature Extraction: What Changes from EAGLE to EAGLE-3

In EAGLE and EAGLE-2, the draft model receives the top-layer features of the target model—the hidden states from the final transformer layer, denoted as $h$ in EAGLE-3's notation but historically called $f$ . These are the representations immediately before the LM head, so they directly encode the information needed to predict the next token.

EAGLE-3 introduces two changes to feature extraction:

Change 1: Multi-layer instead of top-layer only. Instead of using only the top-layer hidden state, EAGLE-3 extracts features from three levels of the target model's transformer stack:

Low-level features ( $l$ ): the input to the first decoder layer (or, equivalently, the output of the embedding layer after positional encoding). These capture token-level and local syntactic information.
Middle-level features ( $m$ ): the input to a middle decoder layer (the exact layer index is not specified in the paper, but it is described as "the input to a middle decoder layer" in Figure 5). These capture intermediate semantic representations.
High-level features ( $h$ ): the input to the top decoder layer (the final layer before the LM head). These capture the information most directly relevant to next-token prediction, as in EAGLE.

All three are $k$ -dimensional vectors where $k$ is the target model's hidden size.

Why multi-layer fusion was not possible in EAGLE. The feature prediction loss $l_{fea}$ in EAGLE required the draft model's output to approximate the top-layer feature specifically, because the LM head operates on the top-layer representation. If EAGLE used features from other layers as input, the feature prediction target would still need to be the top-layer feature (for the LM head to work), creating a mismatch between input and output representation spaces that would complicate training. With $l_{fea}$ removed, there is no requirement that the draft model's output resemble any specific layer of the target model—it can construct its own internal representation optimized for token prediction through the LM head.

Change 2: The fusion FC layer. The three feature vectors $l$ , $m$ , and $h$ (each $\in \mathbb{R}^k$ ) are concatenated into a single $3k$ -dimensional vector, then passed through a fully connected layer that projects it back to $k$ dimensions:

$g = \text{FC}([l; m; h])$

where $[l; m; h] \in \mathbb{R}^{3k}$ is the concatenated feature vector, $\text{FC}$ is a learned linear projection (with optional bias, not explicitly mentioned), and $g \in \mathbb{R}^k$ is the fused feature that serves as input to the draft model.

What the FC layer does. It learns a compressed representation that combines information from different abstraction levels into a single $k$ -dimensional vector. This is a bottleneck operation—the $3k$ dimensions of raw features are reduced to $k$ —forcing the layer to extract and combine the most relevant information from each level.

Why not concatenate and keep $3k$ dimensions? Keeping the full $3k$ -dimensional vector would increase the draft model's input size and parameter count, defeating the purpose of a lightweight draft model. The fusion to $k$ dimensions preserves computational efficiency while still allowing the model to benefit from multi-level information.

When features are captured. During the target model's forward pass—either in the prefill phase (processing the input prompt) or during verification (checking draft tokens)—the target model produces per-token hidden states at all layers. EAGLE-3 records $l$ , $m$ , and $h$ for each position. In the verification stage, these are computed as a byproduct of checking the draft tokens and incur no additional forward passes through the target model—only additional memory to store the intermediate activations.

The Draft Model Architecture in Detail

The draft model in EAGLE-3 is a single transformer decoder layer surrounded by input and output projections. Despite its simplicity relative to the target model (which may have dozens of layers), it is carefully designed to process the specific input format of fused features plus token embeddings.

Input representation. At each position $i$ , the draft model receives two pieces of information:

The fused feature $g_i$ (either from the target model, if position $i$ has been verified, or from the draft model's own output $a_i$ from a previous drafting step, if position $i$ is being drafted ahead).
The token embedding $e_i$ of the token at position $i$ (either the verified token from the target model's sampling, or the draft model's own sampled token from a previous step).

These two $k$ -dimensional vectors ( $g_i$ is $k$ -dimensional after the fusion FC layer, and $e_i$ is $k$ -dimensional from the target model's embedding table) are concatenated to form a $2k$ -dimensional vector:

$\text{input}_i = [g_i; e_i] \in \mathbb{R}^{2k}$

Input projection FC layer. The concatenated $2k$ -dimensional vector is passed through a second fully connected layer that projects it back to $k$ dimensions:

$\text{projected}_i = \text{FC}_{\text{input}}([g_i; e_i]) \in \mathbb{R}^k$

This serves two purposes: (1) it reduces the dimensionality to match the decoder layer's expected input size, and (2) it learns to combine the feature and embedding information into a single representation.

The transformer decoder layer. The $k$ -dimensional projected vector enters a standard transformer decoder layer consisting of:

Self-attention sublayer: multi-head self-attention over the sequence of projected inputs at all positions (with causal masking to prevent attending to future positions). This allows each position to gather information from previous positions—a critical capability for autoregressive prediction.
Feed-forward sublayer: a position-wise feed-forward network (typically two linear transformations with an activation function between them) that applies non-linear transformations to the attention output.
Residual connections and layer normalization: standard transformer architectural elements that stabilize training.

The output of the decoder layer at position $i$ is the vector $a_i \in \mathbb{R}^k$ .

Why a single decoder layer? The draft model must be lightweight because its forward pass is on the critical path of inference—every drafting step runs the draft model sequentially. Multiple decoder layers would increase drafting latency, eating into the speedup gains from parallel verification. A single layer with multi-head attention provides sufficient capacity when paired with the rich input features, while keeping the drafting cost low. The paper validates this design choice implicitly through its speedup results: the draft model's forward pass is fast enough that the gains from more accepted tokens outweigh the drafting overhead.

Output path to LM head. The draft model's output $a_i$ is passed through the target model's frozen LM head to produce logits over the vocabulary:

$\text{logits}_i = W_{\text{LM}} \cdot a_i + b_{\text{LM}}$

where $W_{\text{LM}} \in \mathbb{R}^{|\mathcal{V}| \times k}$ is the LM head weight matrix, $b_{\text{LM}} \in \mathbb{R}^{|\mathcal{V}|}$ is the bias (if present), and $|\mathcal{V}|$ is the vocabulary size. A token $\hat{t}_{i+1}$ is then sampled from the distribution $\text{softmax}(\text{logits}_i / T)$ where $T$ is the sampling temperature.

Why reuse the LM head? The LM head is a $k \times |\mathcal{V}|$ matrix multiplication—computationally cheap compared to the target model's decoder layers. Reusing it ensures that the mapping from hidden states to token probabilities is identical between drafting and verification, which is important for the speculative sampling acceptance criterion (which compares draft probabilities to target probabilities). Training a separate LM head for the draft model would introduce approximation error in the probability estimates, reducing acceptance rates.

The e (embedding) and g (fused feature) interface. The paper distinguishes between two types of inputs to the draft model, corresponding to two sources of information:

$g$ (fused feature): represents the target model's internal state. For verified positions, this comes directly from the target model's forward pass (the concatenation and projection of $l$ , $m$ , $h$ ). For drafted positions where the target model hasn't run yet, this comes from the draft model's own output $a$ from the previous step.
$e$ (token embedding): represents the discrete token that was sampled. For verified positions, this is the embedding of the token verified by the target model. For drafted positions, this is the embedding of the token the draft model sampled at the previous step.

This dual-input design is inherited from EAGLE and addresses a specific problem: the target model's features contain rich information about the context but don't directly encode the random sampling outcome. By also providing the token embedding, the draft model knows which specific token was chosen, which matters because the same feature vector $g$ can lead to different sampled tokens due to randomness in the sampling process.

The Attention Mechanism Modifications for Training-Time Test

The training-time test procedure requires the draft model to process sequences where the input at some positions comes from the target model (true $g$ ) and at other positions comes from the draft model's own previous outputs (self-generated $a$ ). This creates a non-standard attention pattern that must be explicitly handled during training.

The problem. In a standard autoregressive transformer, each position attends to all previous positions (including itself) in a lower-triangular causal pattern. During training-time test, the draft model generates outputs at multiple steps: Step 1 produces $a_{t+1}$ , Step 2 produces $a_{t+2}$ , Step 3 produces $a_{t+3}$ , etc. These outputs are then fed back as inputs for subsequent steps, but they are not in a simple sequential relationship. As shown in Figure 6:

Step 1 (native training step): The input is the prefix "How can I" with true features $g_{how}, g_{can}, g_I$ . The model predicts outputs corresponding to "are", "we", "do" (where "are" is the next token after "I" in the training data, "we" is the token after "are", and "do" is after "we"). The attention mask is a standard lower-triangular matrix over positions "How", "can", "I", "are", "we", "do".
Step 2 (first training-time test step): The input now includes $a_{how}, a_{can}, a_I$ (the draft model's outputs from Step 1 at the corresponding positions) as substitutes for the true features of the training tokens "are", "we", "do". The training tokens themselves serve as queries, and they attend to both the original prefix (as keys) and the self-generated $a$ vectors (as keys). The attention pattern becomes tree-structured: "you" attends to "How", "can", "I", and the Step 1 output at its position; "help" attends to the prefix and its corresponding Step 1 output; "it" does the same.
Step 3 (second training-time test step): The input now includes $a$ vectors from both Step 1 and Step 2, creating a deeper tree of dependencies.

The attention mask visualization (Figure 6). The paper represents this as a sequence of attention matrices where:

Gray tokens ("How", "can", "I") represent the original training prefix—they attend to each other causally.
Blue tokens ("are", "we", "do") represent Step 1 predictions—they attend to the gray tokens causally and attend to themselves causally.
Yellow tokens ("you", "help", "it") represent Step 2 predictions—they attend to gray and blue tokens causally.

The resulting attention mask is not a simple triangular matrix but a block-structured pattern where different "branches" of the prediction tree can attend to their ancestors but not to siblings in other branches.

The computational optimization: vector dot products for diagonal cases. The paper notes an efficiency consideration:

"All attention masks are diagonal, except when the original training data is used as the key. Using matrix multiplication in this case would result in significant computational waste, so we can use vector dot products to calculate the attention score only for the corresponding positions."

When the query and key both come from the same step's predictions (e.g., both are blue tokens), the attention pattern is diagonal—each position only needs to attend to itself and previous positions within the same step, not to positions in parallel branches. In these cases, computing the full query-key matrix multiplication would compute many attention scores that are masked out anyway (corresponding to cross-branch attention that isn't allowed). The paper implements a sparse attention computation that only computes the dot products for valid (query, key) pairs, reducing FLOPs during training.

The purpose of the attention modifications. These modifications ensure that during training-time test, the self-attention mechanism operates on exactly the same pattern of dependencies it will encounter during inference. At inference, the draft model generates tokens sequentially in a tree structure (using EAGLE-2's dynamic draft trees), where each drafted token depends on:

All verified prefix tokens (the common ancestors)
Previously drafted tokens in the same branch (sequential dependencies within a chain)
But NOT tokens in other branches (parallel chains are independent during drafting)

Training with the tree-structured attention mask teaches the model's self-attention to respect these dependency patterns, preventing information leakage across branches that would be unavailable at inference time.

The Loss Function: Pure Token Prediction with Multi-Step Optimization

EAGLE-3's training objective is simpler than EAGLE's precisely because it removes the feature prediction constraint. The only loss is cross-entropy on token predictions, summed over all training-time test steps.

For a single training example with prefix $T_{1:t}$ and continuation $T_{t+1:t+S}$ (where $S$ is the number of training-time test steps), the loss at step $s$ (where $s \in \{1, 2, \ldots, S\}$ ) is:

$l^{(s)}_{token} = -\log p(\hat{t}_{t+s} = t^*_{t+s})$

where:

$t^*_{t+s}$ is the ground-truth token at position $t+s$ (from the training data)
$\hat{t}_{t+s}$ is the draft model's predicted distribution over tokens at position $t+s$ , obtained by passing $a_{t+s}$ through the LM head and applying softmax
$p(\hat{t}_{t+s} = t^*_{t+s})$ is the predicted probability assigned to the correct token

The total loss is:

$L_{\text{EAGLE-3}} = \frac{1}{S} \sum_{s=1}^{S} l^{(s)}_{token}$

What this computes: the average negative log-likelihood of the correct tokens at each drafting step, given the prefix and the draft model's own predictions from previous steps. It measures how well the draft model predicts tokens at various horizons (1 step ahead, 2 steps ahead, ..., $S$ steps ahead) when conditioned on its own potentially imperfect earlier predictions.

Why equal weighting across steps? The paper does not explicitly justify equal weighting, but it is the natural choice when all draft tokens are equally important for the final speedup ratio. An alternative would be to weight earlier steps more heavily (since earlier tokens are more likely to be accepted), but the paper's results suggest equal weighting works well.

What is NOT in the loss. Crucially, there is no $l_{fea}$ term—no regression loss forcing $a_i$ to approximate any target model feature. The draft model's internal representation is entirely unconstrained; it is optimized end-to-end for token prediction accuracy at all horizons.

Gradient flow through self-feeding. During training-time test, the loss at Step 2 depends on the input at Step 2, which includes $a_{t+1}$ (the output from Step 1). The gradient from $l^{(2)}_{token}$ therefore flows backward through:

The LM head (frozen, no gradient)
The draft model's Step 2 forward pass
The input projection at Step 2
The $a_{t+1}$ vector (which was the output of Step 1)
The draft model's Step 1 forward pass
The input projection at Step 1

This means the model receives a training signal about how its Step 1 output affects Step 2 predictions. If a particular type of error in $a_{t+1}$ (e.g., systematic overconfidence in a certain feature subspace) causes poor Step 2 predictions, the gradients will push Step 1 to produce $a_{t+1}$ vectors that are more useful for downstream prediction, even if they don't match any target model feature.

This is the key mechanism that replaces $l_{fea}$ . In EAGLE, the constraint that $a_i \approx f_i$ (via $l_{fea}$ ) was a hard-coded solution to the distribution shift problem. In EAGLE-3, the gradients flowing through the self-feeding loop learn a more flexible solution: produce $a$ vectors that are optimized for the downstream task (helping future steps predict correctly) rather than for matching a specific intermediate representation. The feature prediction loss enforced one particular kind of consistency; training-time test lets the model discover whatever consistency properties are actually needed.

Teacher forcing of token embeddings. Note an important asymmetry in the training-time test procedure: while the features $g$ at drafted positions are replaced with self-generated $a$ vectors, the token embeddings $e$ at those positions use the ground-truth tokens from the training data (teacher forcing). This is visible in Figure 6: the blue tokens "are", "we", "do" at Step 1 are the actual training data tokens, not the draft model's own samples. The model learns to predict the next token given the true previous token but its own internal features—this is the standard teacher forcing approach adapted to the dual-input setting. At inference time, the draft model uses its own sampled tokens as embeddings, which introduces a small train-test gap in the token dimension. However, this is standard in autoregressive training (teacher forcing is the norm for LLM pretraining) and is mitigated by the fact that the draft model's token sampling should approximate the target model's token distribution.

Training data construction. The paper uses instruction-following datasets (ShareGPT, ~68K entries; UltraChat-200K, ~464K entries) as the source of prefixes and continuations. For each data entry, the target model generates a response (rather than using a fixed pre-existing response), ensuring that the training data reflects the target model's actual generation behavior. The target model is run once per training example to produce features at all positions. The draft model training then uses training-time test to simulate multi-step generation from those features. For the reasoning model (DeepSeek-R1-Distill-LLaMA 8B), the OpenThoughts-114k-math dataset is additionally used. The data scaling experiments in Figure 1 use 1×, 2×, 4×, and 8× the ShareGPT dataset size, with UltraChat-200K presumably providing the scaled-up data.

The Inference Pipeline: Drafting and Verification

At inference time, EAGLE-3 alternates between two stages, identical in structure to other speculative sampling methods but with a different drafting mechanism.

Stage 1: Drafting. The draft model runs autoregressively but cheaply to generate a draft token tree (using EAGLE-2's dynamic tree construction). The procedure for one drafting-verification cycle:

Start state. The target model has just verified (or prefilled) a prefix of length $t$ . The features $g_1, g_2, \ldots, g_t$ are available from the target model's last forward pass.
Step 1 (first drafted token). The draft model processes the sequence $[g_1; e_1], [g_2; e_2], \ldots, [g_t; e_t]$ through its input projection, decoder layer, and the target model's LM head. The output is a probability distribution over the vocabulary. Multiple tokens may be sampled at this position (branching factor determined by EAGLE-2's dynamic tree logic). For each sampled token $\hat{t}_{t+1}$ , an output vector $a_{t+1}$ is produced by the draft model.
Step 2 (second drafted token). For each branch from Step 1, the draft model processes $[g_1; e_1], \ldots, [g_t; e_t], [a_{t+1}; e_{\hat{t}_{t+1}}]$ , using the output $a_{t+1}$ as a substitute for $g_{t+1}$ and the embedding $e_{\hat{t}_{t+1}}$ of the sampled token. It produces $a_{t+2}$ and samples tokens $\hat{t}_{t+2}$ for each branch.
Subsequent steps. The process continues for a tree depth of up to 8 (increased from EAGLE-2's depth of 6 because EAGLE-3's higher acceptance rate justifies deeper trees). At each step, the input for a position includes its ancestors' $a$ vectors and the embeddings of the corresponding sampled tokens.
Tree pruning (EAGLE-2 mechanism). After generating the draft tree, EAGLE-2's confidence-based pruning removes low-probability branches to keep the total number of draft tokens within budget (e.g., 60 total draft tokens for 7B/8B models, 50 for 13B, 48 for 70B, with a tree depth of 6 for EAGLE-2—EAGLE-3 increases depth to 8 while keeping total nodes constant).

Stage 2: Verification. The target model processes all draft tokens in the tree in a single parallel forward pass using tree attention. For each draft token, the target model computes its probability under the target distribution $p$ . The speculative sampling acceptance criterion is applied sequentially along each branch: a draft token $\hat{t}_{j}$ with draft probability $\hat{p}_j$ is accepted with probability $\min(1, p_j(\hat{t}_j) / \hat{p}_j(\hat{t}_j))$ . If accepted, verification proceeds to the next token; if rejected, a token is resampled from $\text{norm}(\max(0, p_j - \hat{p}_j))$ and all subsequent tokens in that branch are discarded.

The verification forward pass also produces new target model features (the $l$ , $m$ , $h$ at all verified positions), which are fused into $g$ vectors for the next drafting cycle.

The critical role of $a$ as a substitute for $g$ . The entire drafting pipeline hinges on the fact that $a$ (the draft model's output) can stand in for $g$ (the fused target model feature) at positions where the target model hasn't run yet. During training-time test, the model learned to produce $a$ vectors that, when used as input for subsequent steps, enable accurate prediction. At inference, this trained behavior generalizes: even when $a$ vectors are imperfect (the model makes prediction errors), the subsequent steps have learned to handle this imperfection.

What happens when a drafted token is wrong. If the draft model samples an incorrect token at Step 1, the input to Step 2 includes $a_{t+1}$ (based on the wrong token's context) and $e_{\hat{t}_{t+1}}$ (the wrong token's embedding). The draft model at Step 2 may produce a poor prediction, or it may partially recover if the attention mechanism can discount the erroneous token. Either way, the verification stage will detect the error through the acceptance criterion: the target model's probability $p_{t+1}(\hat{t}_{t+1})$ will be low, causing rejection and discarding the erroneous branch. This is the safety net that makes speculative sampling lossless—no amount of draft model error compromises the final output quality, only the speedup ratio.

The dynamic draft tree integration. EAGLE-3's higher acceptance rates (shown in Figure 7) change the optimal draft tree shape compared to EAGLE-2. Because more tokens are accepted per branch on average, EAGLE-3 can profitably explore deeper trees (depth 8 vs. depth 6) without increasing the total node budget. This is a direct consequence of the improved draft model quality: when branches are more likely to be accepted, it's worth investing in deeper exploration. The paper maintains the same total number of draft tokens as EAGLE-2 ("60, 50, and 48" for 7B/8B, 13B, and 70B models respectively) but redistributes them across deeper trees.

Batch size considerations. The paper includes experiments in production frameworks (SGLang, vLLM) that illuminate an important practical aspect: speculative sampling is often thought to only help at small batch sizes where the GPU is most memory-bound. At large batch sizes, compute utilization increases and the "free" compute that speculative sampling exploits diminishes. The paper shows that EAGLE-3 significantly extends the batch size range where speculative sampling is beneficial: in SGLang (Table 3), EAGLE-3 achieves 1.38× throughput at batch size 64, whereas EAGLE drops below 1.0× (becomes detrimental) at batch size 24. This is attributed to EAGLE-3's higher acceptance rates, which mean more tokens are generated per target model forward pass, better amortizing the drafting overhead even when compute utilization is higher.

Training Configuration and Hyperparameters

The paper provides specific training details that are essential for reproducibility:

Optimizer. AdamW with $\beta_1 = 0.9$ , $\beta_2 = 0.95$ , and gradient clipping of 0.5. The learning rate is $5 \times 10^{-5}$ (5e-5). These are standard choices for transformer fine-tuning.

Training data. ShareGPT (~68K conversations) and UltraChat-200K (~464K conversations) serve as the instruction-following training corpora. The target model generates responses to these prompts (rather than using pre-existing responses), ensuring the training distribution matches the target model's generation behavior. For DeepSeek-R1-Distill-LLaMA 8B, the OpenThoughts-114k-math dataset provides math-specific training data, which explains why this model sees its highest speedup on GSM8K (5.01× at temperature=0) rather than on code generation tasks like the chat models.

Data scaling. The scaling experiments in Figure 1 use 1×, 2×, 4×, and 8× the base data scale, with the x-axis representing "data scale relative to ShareGPT." The exact composition of the scaled dataset (whether UltraChat is scaled, additional datasets are added, or ShareGPT is repeated) is not specified, but the trend is clear: EAGLE-3's speedup improves with data scale while EAGLE-2's plateaus.

Draft tree configuration. For EAGLE-3: total draft tokens are set to 60 (7B/8B models), 50 (13B), and 48 (70B), with a draft tree depth of 8 (increased from 6 in EAGLE-2). During tree expansion, 10 nodes are selected per step (matching EAGLE-2). For the SGLang and vLLM experiments, no tree structure was used—a simple chain of length 2 or 3 was employed, likely for simplicity of integration. The tree structure provides additional speedup in the single-batch setting (Table 1) but is not essential for demonstrating the core improvement.

No task-specific fine-tuning. The paper emphasizes that "we evaluate on five common tasks, using the same weights for all tasks without fine-tuning on the respective tasks." The draft model is trained once on general instruction-following data and then applied to conversation, code generation, math reasoning, instruction following, and summarization tasks without any task-specific adaptation. This demonstrates that the improvements from training-time test and multi-layer fusion are not task-specific but reflect a general improvement in the draft model's ability to approximate the target model's behavior.

Evaluation setup. The metrics (speedup ratio, average acceptance length, acceptance rate n-α) are measured on actual hardware (GPUs) rather than simulated. The speedup ratio is relative to vanilla autoregressive decoding using the same target model and hardware. For the acceptance rate measurements, a chain-like draft (not tree-like) is used to isolate the per-step acceptance behavior, following EAGLE's evaluation methodology.

4. Key Insights and Innovations

Innovation 1: The Feature Prediction Constraint as a Diagnosed Expressiveness Bottleneck, Not an Innocent Regularizer

The paper's most intellectually distinctive contribution is not the removal of EAGLE's feature prediction loss per se—anyone could ablate a loss term—but rather the diagnosis that this loss was the root cause of a previously unobserved phenomenon: a flat scaling law for inference acceleration. Prior work treated the feature prediction loss as a benign enabler of multi-step generalization (EAGLE's key trick: train on single steps, generalize to multiple steps because outputs are constrained to resemble inputs). The dominant assumption in the field was that draft model quality was primarily limited by capacity (the draft model is small) or data quantity, not by an architectural constraint that actively prevented the model from exploiting additional data.

EAGLE-3 reframes that assumption entirely. By showing that EAGLE-2's speedup plateaus at ~3.2–3.4× regardless of whether training data is scaled 1×, 2×, 4×, or 8× (Figure 1), the paper demonstrates that the feature prediction loss is not merely a training aid—it is an expressiveness ceiling. The mechanism behind this ceiling is subtle: the loss forces the draft model's output to live in the target model's feature space, which is optimized for the target model's architecture and scale, not for the draft model's more constrained representational capacity. The draft model cannot learn representations that maximally benefit token prediction because they must also satisfy a distribution-matching constraint to the target model's internal features. This is a form of representational coupling between draft and target models that was previously invisible because no one had systematically scaled draft model training data while measuring speedup.

The significance of this diagnosis extends beyond EAGLE. It establishes a design principle for any method that reuses target model internals (Medusa, Hydra, Falcon, HASS): intermediate supervision signals that enable desirable training behaviors (like multi-step generalization) may simultaneously impose hard limits on how much the draft model can improve with scale. The paper does not merely show that removing the constraint helps; it identifies why it helps and why the constraint was there in the first place (to close the train-test distribution gap without multi-step training). This is a conceptual contribution to the speculative sampling literature: it explains the tradeoff between trainability (easy multi-step via feature matching) and scalability (expressiveness freed from distribution matching), and it provides a diagnostic framework—scale training data and watch the speedup curve—that future methods can use to detect similar bottlenecks.

The comparison to HASS (Zhang et al., 2024) sharpens this contribution. HASS observed that feature prediction errors accumulate across drafting steps, and addressed this by simulating multi-step training while retaining feature prediction. HASS treats the feature prediction framework as fundamentally sound and patches its failure mode (error accumulation). EAGLE-3 identifies the framework itself as the problem: the constraint, not its imperfect satisfaction, is what limits scaling. This is a more fundamental critique, and Figure 2 validates it empirically—EAGLE-3 substantially outperforms HASS across models and tasks, suggesting that removing the constraint is more effective than mitigating its consequences.

Anchoring evidence: Figure 1 (left panel) shows EAGLE-2 speedup plateauing while EAGLE-3 rises; Figure 4 shows that naive removal of feature prediction improves first-token acceptance (0-α) but catastrophically degrades second-token acceptance (1-α), confirming that the constraint was solving a real problem that training-time test must address.

Innovation 2: Training-Time Test as a Generalizable Principle for Closing Train-Test Distribution Gaps in Autoregressive Draft Models

The training-time test technique is not merely an implementation detail—it represents a principled solution to a general class of distribution-shift problems that arise whenever a learned component feeds its own outputs back as inputs for subsequent steps, but was trained only on ground-truth inputs. This pattern appears throughout machine learning (exposure bias in sequence models, distribution shift in imitation learning, compounding errors in model-based RL), and the standard solutions—scheduled sampling (Bengio et al., 2015), DAgger (Ross et al., 2011), data aggregation—all involve interleaving self-generated data during training.

What makes EAGLE-3's instantiation of this principle distinctive is the specific diagnostic decomposition of the distribution shift problem in speculative drafting. The paper identifies that the shift is not uniform across drafting steps: removing the feature constraint improves Step 1 prediction (because the model is more expressive) but devastates Step 2 prediction (because Step 2 encounters out-of-distribution inputs). This is visible in Figure 4's comparison of 0-α and 1-α acceptance rates. A naive approach that simply removed the constraint would see net-negative results because the Step 1 gain is more than offset by the Step 2 loss. Training-time test specifically targets the multi-step degradation by exposing the model to its own prediction errors during training, allowing it to learn robustness.

The conceptual contribution is the decoupling of two previously intertwined objectives: (1) producing accurate single-step predictions, and (2) producing outputs that serve as good inputs for subsequent steps. In EAGLE, the feature prediction loss conflated these: matching the target model's features served both to improve token prediction (by staying close to the target's representation) and to ensure inputs for the next step were in-distribution. Training-time test separates them: the token prediction loss handles objective (1) at all horizons, and the self-feeding training loop handles objective (2) by directly teaching the model what inputs to expect at each step. This decoupling is what enables the model to simultaneously achieve higher 0-α (better single-step via expressiveness) and higher 1-α (better multi-step via trained robustness to self-generated inputs), as shown in Figure 7 where EAGLE-3's acceptance rate remains nearly flat across successive self-feeding steps while EAGLE's drops significantly.

The paper explicitly contrasts this with HASS's motivation: HASS uses multi-step training to "mitigate the error accumulation caused by inaccurate feature predictions," i.e., to make the feature prediction framework work better. EAGLE-3 uses multi-step training to eliminate the feature prediction framework entirely. The technique is the same (simulate test-time behavior during training), but the purpose is fundamentally different: patching an existing constraint versus enabling its removal. This distinction matters because it suggests training-time test is not tied to any particular draft model architecture—it is a general technique for any setting where a model's outputs will be fed back as inputs and the training distribution (ground-truth inputs) differs from the test distribution (self-generated inputs).

Anchoring evidence: Figure 4 demonstrates the failure mode that training-time test addresses; Figure 7 shows that EAGLE-3's acceptance rate is nearly invariant to the number of self-generated inputs, whereas EAGLE's degrades sharply, confirming that the technique closes the distribution gap.

Innovation 3: Multi-Layer Feature Fusion as an Information-Theoretic Improvement Over Top-Layer-Only Drafting

The shift from top-layer-only features to fused multi-layer features (low, middle, high) is not just an architectural tweak—it reflects an information-theoretic argument about what information is needed for multi-token prediction. The paper articulates a specific limitation of top-layer features that was not previously recognized in the speculative sampling literature: these features are optimized to predict exactly one token ahead (they feed directly into the LM head), and while they are sufficient for that purpose, they may lack the richer semantic and structural information needed to predict two, three, or more tokens ahead.

The key conceptual move is recognizing that information sufficient for next-token prediction is not necessarily sufficient for next-next-token prediction, even though both tasks ultimately operate through the same LM head. Top-layer features represent the model's state after all transformer layers have processed the context with the objective of predicting the immediate next token. Information that is useful for longer-range prediction—syntactic patterns that span multiple tokens, discourse structure, entity relationships that persist across several words—may be distributed across intermediate layers, where representations are less specialized for the immediate next-token task and retain more general linguistic structure.

By fusing features from multiple layers, EAGLE-3 provides the draft model with complementary information from different levels of abstraction: low-level features capture token identity and local syntax, middle-level features capture phrase-level semantics and syntactic relations, and high-level features capture the information most directly relevant to the immediate next token. The fusion FC layer learns to extract and combine these into a single representation optimized for the draft model's prediction task.

Why was this not done before? The paper identifies a specific coupling: the feature prediction loss $l_{fea}$ required the draft model's output to approximate the top-layer feature specifically (because that's what the LM head expects). If intermediate-layer features were used as input but the output target remained the top-layer feature, there would be a representational mismatch. Removing $l_{fea}$ breaks this coupling: the draft model's output is no longer required to approximate any target model layer, so the input can be drawn from any combination of layers. Training-time test enables the removal of $l_{fea}$ , which in turn enables multi-layer fusion. These innovations are therefore not independent—they form a chain of enablement.

The ablation study (Table 2) quantifies the contribution of multi-layer fusion: on MT-bench with LLaMA-Instruct 3.1 8B, removing the feature constraint alone (EAGLE-2 + "remove fea con") yields a speedup of 3.82× (up from EAGLE-2's 3.16×), while adding fused features ("+ fused features (ours)") further improves to 4.40×. On GSM8K, the corresponding progression is 3.39× → 3.77× → 4.48×. The gain from feature fusion is substantial (~15% relative speedup on MT-bench, ~19% on GSM8K) even after the feature constraint is removed, confirming that top-layer features were indeed a bottleneck.

Anchoring evidence: Table 2 isolates the contribution of multi-layer fusion; the conceptual argument about information sufficiency for multi-token prediction appears in Section 3's discussion of why top-layer features are limited.

Innovation 4: The Discovery of a Scaling Law for Inference Acceleration as an Empirical Phenomenon

The paper's finding that EAGLE-3's speedup ratio increases with training data scale (Figure 1) is not just a performance result—it establishes a new empirical scaling relationship that was previously unknown and, under prior architectures, unobservable. This is significant at the level of scientific discovery rather than engineering improvement: prior to this work, there was no evidence that inference acceleration could be improved by scaling draft model training data, because the architectures being tested contained a bottleneck (feature prediction) that capped the benefits.

The scaling behavior is shown in Figure 1: on MT-bench with LLaMA-Instruct 3.1 8B, EAGLE-3's speedup rises from approximately 4.1× at 1× data scale to roughly 4.4× at 8× data scale, while EAGLE-2's speedup is essentially flat at ~3.2–3.4×. The accept length (right panel) shows a corresponding trend, rising from ~5.6 tokens to ~6.1 tokens for EAGLE-3 while EAGLE-2 stays at ~4.0. While the absolute improvement is modest in this particular setup (a few percent speedup from 8× data), the existence of a positive scaling relationship is the finding of interest. It implies that EAGLE-3 has not saturated—further data scaling may yield further speedup gains, and the ceiling is not yet visible.

This finding parallels the broader trend in LLM research where scaling training data improves model capabilities (the LLaMA series: 1T → 2T → 15T tokens yielding better benchmarks at fixed inference cost). The paper explicitly draws this parallel, noting that "similarly, we aim to improve the acceptance rate and acceleration ratio of EAGLE by increasing its training data." The analogy is apt but with an important difference: the metric being scaled is not task performance (accuracy, benchmark scores) but inference efficiency (speedup ratio). This is a novel axis for scaling laws, distinct from the pretraining scaling laws (Hoffmann et al., 2022) that relate model size, data, and loss.

The intellectual significance is that it opens a new research direction: data-scaling laws for inference acceleration components. If draft models can be systematically improved by scaling their training data—and if this scaling behavior depends on architectural choices (no bottleneck) rather than being universal—then designing draft model architectures becomes not just about achieving good speedup at a fixed data scale but about enabling favorable scaling properties. The paper demonstrates that EAGLE had unfavorable scaling properties and EAGLE-3 has favorable ones, establishing that scaling behavior is an architectural property that can be optimized for.

The paper is appropriately cautious about the magnitude: "we expect larger data size would lead to further improved speedup ratio" is stated as an expectation, not a proven law. The scaling curve in Figure 1 shows a positive trend but with only four data points (1×, 2×, 4×, 8×) and a relatively shallow slope at the high end, suggesting that the functional form of the scaling relationship (logarithmic? power-law? saturating?) is not yet characterized. This is a limitation but does not diminish the conceptual contribution: establishing that the relationship exists at all, and that it was previously masked by an architectural constraint, is the key finding.

Anchoring evidence: Figure 1 (both panels) and the accompanying discussion in Section 1; the flat EAGLE-2 curve serves as the counterfactual demonstrating that the scaling law is a property of EAGLE-3's architecture, not an inherent property of draft models.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper evaluates on five tasks: MT-bench (multi-turn conversation, 80 questions from Zheng et al., 2023), HumanEval (code generation, 164 problems from Chen et al., 2021), GSM8K (mathematical reasoning, ~1,319 test questions from Cobbe et al., 2021), Alpaca (instruction following, from Taori et al., 2023), and CNN/Daily Mail (summarization, from Nallapati et al., 2016). These span diverse generation types—conversation, code, math, instruction following, and summarization—providing broad coverage of LLM use cases. For the reasoning model (DeepSeek-R1-Distill-LLaMA 8B), the authors additionally use the OpenThoughts-114k-math dataset for training the draft model, which explains the model's particularly strong GSM8K results (5.01× speedup at temperature=0 vs. 4.59× on HumanEval, reversing the pattern seen in chat models where code tasks yield the highest speedups).
Base model(s). The experiments use four target models spanning different scales and types: Vicuna 13B (Chiang et al., 2023), LLaMA-Instruct 3.1 8B (Dubey et al., 2024), LLaMA-Instruct 3.3 70B (Dubey et al., 2024), and DeepSeek-R1-Distill-LLaMA 8B (DeepSeek-AI et al., 2025). This selection covers small (8B), medium (13B), and large (70B) parameter scales for chat models, plus a distilled reasoning model that represents the emerging class of o1-style deliberate-reasoning LLMs. The authors note they were unable to test 405B or 671B models due to GPU constraints. The rationale for this model selection is that it demonstrates the method's generality across model scales and architectures (LLaMA vs. Vicuna vs. DeepSeek-R1), and the reasoning model is particularly important because these models "significantly increase the proportion of inference costs in the overall LLM pipeline" (Section 1), making acceleration especially valuable.
Metrics. Three metrics assess acceleration performance without evaluating generation quality (since speculative sampling with strict acceptance conditions is lossless):
- Speedup Ratio: The wall-clock time of vanilla autoregressive decoding divided by the wall-clock time of the accelerated method, measured on actual hardware. A ratio of 5.0× means the accelerated method generates tokens 5 times faster.
- Average Acceptance Length τ: The average number of draft tokens accepted per drafting-verification cycle. This directly determines the speedup potential—each accepted token avoids one sequential forward pass of the target model.
- Acceptance Rate n-α: The proportion of draft tokens accepted under the condition that the input to the draft model contains n self-generated (estimated) vectors, given that all previous estimated tokens have been accepted. For EAGLE, this refers to input sequences containing n estimated features: f₁, f₂, …, fᵢ, \hat{f}ᵢ₊₁, …, \hat{f}ᵢ₊ₙ. For EAGLE-3, it refers to input sequences with n self-predicted a vectors: g₁, g₂, …, gᵢ, aᵢ₊₁, …, aᵢ₊ₙ. This metric isolates how well the draft model handles its own prediction errors as inputs. For acceptance rate measurements specifically, a chain-like draft (not tree-like) is used to cleanly measure per-step behavior.
Baselines. The paper compares against a comprehensive set of speculative sampling and related methods, all evaluated under the same hardware conditions:
- Vanilla autoregressive decoding: the baseline for all speedup ratios (1.00×).
- Standard speculative sampling (SpS) (Leviathan et al., 2023; Chen et al., 2023; Gante, 2023): uses a separate smaller model (Vicuna-68M for Vicuna-13B experiments) as the draft model.
- PLD (Prompt Lookup Decoding; Saxena, 2023): generates draft tokens by searching for matching n-grams in the prompt.
- Medusa (Cai et al., 2024): uses multiple decoding heads on top of the target model to predict several future tokens in parallel. The paper notes that Medusa relaxes acceptance conditions under non-greedy settings, meaning it does not guarantee lossless acceleration at temperature=1.
- Lookahead (Fu et al., 2024): generates draft tokens by looking ahead in the generation process.
- Hydra (Ankner et al., 2024): extends Medusa with sequentially-dependent draft heads.
- HASS (Zhang et al., 2024): simulates multi-step drafting during training while retaining feature prediction, designed to mitigate error accumulation in EAGLE.
- EAGLE (Li et al., 2024c): the original feature-level autoregressive draft model.
- EAGLE-2 (Li et al., 2024b): extends EAGLE with context-aware dynamic draft trees. This is the most important baseline since EAGLE-3 adopts EAGLE-2's tree mechanism and represents the direct predecessor.
For all non-EAGLE methods, the authors use officially released weights and default settings where available. For EAGLE on LLaMA3-Instruct, they train using the ShareGPT dataset to match Medusa and Hydra's training protocol. The comparison set covers the major speculative sampling paradigms: independent draft models (SpS), n-gram methods (PLD), parallel head methods (Medusa, Hydra), feature-level methods (EAGLE, EAGLE-2, HASS), and lookahead methods.
Generation budget / compute accounting. The primary unit of comparison is the wall-clock speedup ratio measured on actual GPUs, which accounts for all costs: the draft model's forward passes, the target model's verification forward passes, the feature extraction and fusion operations, and the dynamic tree construction. This is a realistic cost accounting because it captures hardware-specific factors (memory bandwidth, compute utilization, kernel launch overhead) that FLOPs-based accounting would miss. The generation budget is implicitly controlled by the draft tree configuration: total draft tokens per cycle are set to 60 (7B/8B models), 50 (13B), and 48 (70B), with tree depth of 8 for EAGLE-3 (increased from 6 for EAGLE-2 due to higher acceptance rates enabling deeper trees). The paper does not use a FLOPs budget for fairness comparison because different methods use different computational primitives (feature extraction vs. independent model forward passes vs. multiple LM head invocations) that don't translate cleanly to a common FLOPs metric. The speedup ratio naturally captures the net effect of all computational costs.
Cross-validation / statistical protocol. The paper does not report formal cross-validation or statistical significance testing. The evaluation uses standard benchmark test sets (MT-bench, HumanEval, GSM8K, Alpaca, CNN/Daily Mail) with fixed splits, following the evaluation protocol established in EAGLE (Li et al., 2024c) and Spec-Bench (Xia et al., 2024). Results are reported as point estimates without confidence intervals. The five-task evaluation provides some robustness against task-specific variance, and the consistent ranking of EAGLE-3 above all baselines across tasks and models (Table 1) functions as an informal replication across conditions. However, the lack of statistical measures means that small differences in speedup ratios (e.g., 4.40× vs. 4.48× between MT-bench and GSM8K for LLaMA-Instruct 3.1 8B) cannot be assessed for significance.

Main Quantitative Results

Aggregate Speedup Performance Across All Methods, Models, and Tasks

Table 1 presents the paper's central performance results, reporting speedup ratios and average acceptance lengths for all methods on all tasks with both greedy (temperature=0) and non-greedy (temperature=1) sampling. The headline numbers are unambiguous: EAGLE-3 achieves the highest speedup ratio across every model-task-temperature combination tested, with speedups ranging from 3.07× (LLaMA-Instruct 3.1 8B at temperature=1 on MT-bench) to 6.47× (LLaMA-Instruct 3.1 8B at temperature=0 on HumanEval).

The aggregate picture across all 40 reported comparisons (4 models × 5 tasks × 2 temperatures) shows:

Vicuna 13B (temperature=0): EAGLE-3 achieves a mean speedup of 5.51× across all five tasks, compared to EAGLE-2's 4.22× and EAGLE's 3.05×. The improvement over EAGLE-2 is 1.31× (i.e., EAGLE-3 is 31% faster than EAGLE-2), and over standard speculative sampling (1.92× mean) it is 2.87×. The highest individual task speedup is 6.47× on HumanEval, and the lowest is 5.01× on CNN/Daily Mail.
LLaMA-Instruct 3.1 8B (temperature=0): Mean speedup 4.44× for EAGLE-3 vs. 3.23× for EAGLE-2—a 37% improvement. The task range spans 3.65× (CNN/Daily Mail) to 4.85× (HumanEval). This model is the primary testbed for the ablation study (Table 2) and scaling law analysis (Figure 1).
LLaMA-Instruct 3.3 70B (temperature=0): Mean speedup 4.12× for EAGLE-3 vs. 2.85× for EAGLE-2—a 45% improvement. The larger model shows a larger relative gain from EAGLE-3, suggesting that the expressiveness bottleneck was more severe for larger target models (where the draft model must approximate a more complex function) or that multi-layer features provide richer information when the target model has more layers.
DeepSeek-R1-Distill-LLaMA 8B (temperature=0): Mean speedup 4.16× for EAGLE-3 vs. 3.26× for EAGLE-2—a 28% improvement. Unlike other models where HumanEval (code) yields the highest speedup, DeepSeek-R1 hits 5.01× on GSM8K (math), attributed to the OpenThoughts-114k-math training data. This is a notable result: it demonstrates that domain-specific draft model training can push acceleration higher on in-domain tasks, an important practical consideration for specialized deployments.

At temperature=1 (non-greedy), the absolute speedups are uniformly lower (e.g., Vicuna 13B EAGLE-3 mean 4.65× vs. 5.51× at temperature=0), which is expected because non-greedy sampling reduces predictability and therefore acceptance rates. However, the relative advantage of EAGLE-3 over EAGLE-2 is similar or larger: 24% improvement for Vicuna 13B (4.65× vs. 3.76×), 23% for LLaMA-Instruct 3.1 8B (3.45× vs. 2.80×), 49% for LLaMA-Instruct 3.3 70B (3.95× vs. 2.65×), and 27% for DeepSeek-R1 (3.52× vs. 2.77×).

The paper does not report speedup ratios for Medusa, Lookahead, PLD, or Hydra at temperature=1 because these methods "relax acceptance conditions under non-greedy settings, which do not guarantee lossless acceleration" (Table 1 note). This is a methodological choice that strengthens the comparison: EAGLE-3 is evaluated against the strictest baseline (vanilla speculative sampling with exact acceptance criteria) rather than against methods that trade output fidelity for speed.

Average acceptance length τ tells a consistent story. For Vicuna 13B at temperature=0, EAGLE-3 achieves τ = 6.62 tokens per cycle on average across tasks, compared to 4.83 for EAGLE-2 and 3.96 for EAGLE. For LLaMA-Instruct 3.1 8B, the corresponding values are 6.23 vs. 4.11 vs. (EAGLE not reported for this model). The τ improvement is the direct mechanism behind the speedup: each additional accepted token saves one sequential forward pass through the target model. EAGLE-3 generates roughly 37% more accepted tokens per cycle than EAGLE-2 on average across models and tasks.

Comparing EAGLE-3 to non-EAGLE methods on Vicuna 13B (temperature=0): the best non-EAGLE method is Hydra at 2.80× mean speedup with τ = 3.50, followed by Medusa at 2.12× (τ = 2.51). EAGLE-3's 5.51× mean places it at roughly 2× the speedup of the best non-EAGLE method and over 2.6× the acceptance length. This substantial gap validates the paper's core architectural choices (feature reuse, training-time test, multi-layer fusion) over alternative paradigms like independent draft heads (Medusa/Hydra) or independent draft models (standard speculative sampling).

For a direct benchmark with standard speculative sampling using a draft model from the same model series: Vicuna 13B with Vicuna-68M as draft achieves 1.93× speedup on MT-bench (τ = 2.27) at temperature=0. EAGLE-3 on the same task achieves 5.58× (τ = 6.65)—a 2.9× improvement. This stark difference underscores how much richer target model features are as input signals compared to raw tokens processed by an independent model.

Acceptance Rate Analysis: Characterizing Multi-Step Stability

Figure 7 provides the diagnostic insight into why EAGLE-3 achieves higher speedups: the acceptance rate as a function of how many self-generated inputs are in the draft model's context. On MT-bench with LLaMA-Instruct 3.1 8B as the target model:

EAGLE (blue bars): The acceptance rate degrades markedly as the number of self-generated inputs increases. The 0-α rate starts around 0.77, but by 7-α the rate has fallen to approximately 0.52—a drop of roughly 25 percentage points. This is the error accumulation phenomenon that HASS was designed to mitigate and that EAGLE's feature prediction loss partially addressed.
EAGLE-3 (red bars): The acceptance rate remains nearly flat across all self-generation depths. The 0-α rate starts around 0.79 (slightly higher than EAGLE), and even at 7-α the rate is approximately 0.77—a drop of only about 2 percentage points. The near-horizontal line is direct evidence that training-time test has solved the distribution-shift problem: the draft model is robust to receiving its own (potentially imperfect) outputs as inputs because it was trained on exactly this condition.

The 0-α improvement (~0.79 vs. ~0.77) reflects the expressiveness gain from removing the feature prediction constraint (the model learns better representations for single-step prediction). The dramatic difference at higher n-α values (e.g., ~0.77 vs. ~0.60 at 3-α, ~0.77 vs. ~0.52 at 7-α) reflects the solution to the train-test distribution mismatch. Both effects are necessary for the overall speedup gain: without the 0-α improvement, first-token acceptance would be lower; without the flat n-α curve, deep draft trees (which EAGLE-3 extends from depth 6 to 8) would be counterproductive because later tokens would have very low acceptance rates.

The paper does not provide acceptance rate curves for other models or tasks, which is a minor limitation—the flatness of the n-α curve might vary with model scale or task type, and this would be informative for understanding where EAGLE-3's advantages are largest.

Data Scaling Analysis: The Emergent Scaling Law

Figure 1 (and the underlying Figure 4) presents the paper's most conceptually significant result: EAGLE-3 exhibits a positive scaling relationship between training data quantity and speedup ratio, while EAGLE-2 does not. On MT-bench with LLaMA-Instruct 3.1 8B:

EAGLE-2: Speedup stays essentially constant at 3.2–3.4× across 1×, 2×, 4×, and 8× the ShareGPT data scale. At 1×: ~3.2×; at 2×: ~3.3×; at 4×: ~3.3×; at 8×: ~3.4×. The average acceptance length (right panel) similarly plateaus at approximately 4.0 tokens. This is the flat scaling curve that motivated the paper.
EAGLE-3: Speedup rises from ~4.1× at 1× data to ~4.2× at 2×, ~4.3× at 4×, and ~4.4× at 8×. The acceptance length rises from ~5.6 to ~6.1 across the same range. The trend is positive at all measured data points, with no visible plateau, suggesting continued improvement is possible with further data scaling.

The absolute improvement from 1× to 8× data is modest (~0.3× speedup, or about 7% relative improvement), which might seem underwhelming. However, the scientific contribution is not the magnitude but the existence and direction of the trend. EAGLE-2's complete flatness demonstrates that the feature prediction constraint qualitatively prevents scaling—not just that it reduces the slope, but that it eliminates the relationship entirely. EAGLE-3's positive slope demonstrates that the new architecture has removed this barrier and enabled a scaling behavior that was previously impossible to observe.

The paper appropriately states this as an observation rather than a characterized law: "we expect larger data size would lead to further improved speedup ratio" (Section 1). The functional form of the scaling relationship (logarithmic? power-law? linear?) cannot be determined from four data points, and the paper does not attempt to fit a curve. This is honest about the preliminary nature of the finding.

Figure 4 decomposes the scaling behavior into first-token (0-α) and second-token (1-α) acceptance rates, comparing EAGLE, EAGLE without feature prediction, and EAGLE-3 across 1×, 2×, 4×, and 8× data scales:

0-α (left panel): Both EAGLE-3 and "EAGLE without fea pred" show clear improvement with data scale, starting around 0.76 at 1× and rising to ~0.79 at 8×. EAGLE with feature prediction is lower (starting ~0.73) and improves more slowly. This confirms that the feature prediction constraint specifically limits the first-token scaling benefit—models without it learn better token predictions from additional data.
1-α (right panel): "EAGLE without fea pred" shows catastrophic degradation—acceptance rates in the 0.25–0.35 range across all data scales, far below EAGLE's 0.55–0.65. EAGLE-3 shows acceptance rates of 0.72–0.77 across data scales, substantially higher than EAGLE. This panel is the key evidence that training-time test successfully addresses the multi-step degradation: removing feature prediction alone destroys 1-α, but adding training-time test not only recovers it but improves it beyond the constrained EAGLE baseline.

The interaction between the two panels explains the net speedup results: "EAGLE without fea pred" would have worse overall performance than EAGLE despite better 0-α because the 1-α (and presumably higher n-α) collapse dominates. Only EAGLE-3, which combines removal of feature prediction with training-time test, achieves improvements in both 0-α and n-α.

Throughput in Production Frameworks: SGLang and vLLM

Tables 3 and 4 report throughput measurements in the SGLang framework (Zheng et al., 2024), while Table 5 reports vLLM (Kwon et al., 2023) results. These experiments are conducted by the SGLang team and the authors respectively, and they address a critical practical concern: speculative sampling is often thought to be primarily beneficial at small batch sizes where the GPU is memory-bound, with diminishing and eventually negative returns as batch size increases and compute utilization rises.

SGLang results (Tables 3, 4):

Batch size 1 throughput (Table 4): SGLang without speculative sampling achieves 158.34 tokens/s on H100 with LLaMA-Instruct 3.1 8B. SGLang + EAGLE-2 achieves 244.10 tokens/s (1.54×). SGLang + EAGLE-3 achieves 373.25 tokens/s (2.36×). The throughput improvement from EAGLE-2 to EAGLE-3 is 1.53× at batch size 1.
Batch size scaling (Table 3): EAGLE's throughput improvement declines steadily from 1.40× at batch size 2 to 0.99× at batch sizes 56 and 64—essentially no benefit, and at intermediate batch sizes (24, 32, 48) it actually reduces throughput (0.93×, 0.94×, 0.88×). This is because the drafting overhead (running the draft model sequentially) exceeds the savings from reduced target model forward passes when the GPU is already well-utilized.

In contrast, EAGLE-3 shows substantial throughput improvements across a much wider range of batch sizes: 1.81× at batch size 2, declining gradually to 1.38× at batch size 64. EAGLE-3 never drops below 1.0× in this experiment (up to batch size 64), and the improvement at batch size 64 (1.38×) is comparable to EAGLE's improvement at batch size 2 (1.40×). This is a practically significant result: it means EAGLE-3 can be deployed in throughput-oriented serving scenarios with larger batch sizes, not just latency-sensitive single-request scenarios.

The batch size sweep was conducted without tree structure (chain length set to 3), meaning these numbers represent a lower bound on achievable throughput—the tree structure used in the single-batch experiments (Tables 1, 4) provides additional speedup. The paper's explanation for EAGLE-3's better batch-size scaling is implicit: higher acceptance rates mean more tokens are verified per target model forward pass, better amortizing the drafting overhead even when compute utilization is higher.

vLLM results (Table 5):

On A100 with LLaMA-Instruct 3.1 8B, the pattern is similar but with overall lower absolute improvements compared to SGLang (likely due to framework differences, GPU differences, or both). EAGLE-3 achieves 1.75× at batch size 2, declining to 1.01× at batch size 56 (still slightly above 1.0×). EAGLE drops below 1.0× starting at batch size 32 (0.93×) and falls to 0.71× at batch size 56—a 29% throughput reduction.

The maximum throughput improvement for EAGLE occurs at batch size 24 (1.03×), while for EAGLE-3 it's at batch size 56 (1.01×)—confirming that EAGLE-3 extends the beneficial batch size range. The chain length was set to 2 for these experiments (one step shorter than SGLang), and no tree structure was used, making these conservative estimates of EAGLE-3's throughput benefits.

The production framework results are important for two reasons beyond the raw numbers. First, they demonstrate that EAGLE-3's improvements are not artifacts of the experimental evaluation setup—they transfer to real serving systems with practical batching, memory management, and kernel optimizations. Second, they challenge the conventional wisdom that speculative sampling only helps at batch size 1. EAGLE-3's 1.38× throughput at batch size 64 is a result that would change deployment decisions for high-traffic LLM services.

Comparison with HASS and Other Feature-Reusing Methods

Figure 2 (bar chart) provides a visual comparison of speedup ratios across methods for Vicuna 13B, LLaMA-Instruct 3.1 8B, LLaMA-Instruct 3.3 70B, DeepSeek-R1-Distill-LLaMA 8B, and an additional LLaMA 8B (base) model. The chart shows:

Vanilla speculative sampling (gray): consistently the lowest speedup, ranging from ~1.0× (by definition, the baseline for the bar chart) to ~1.9× depending on the model.
Medusa (yellow): ~2.1× for Vicuna 13B and LLaMA-Instruct 3.1 8B, ~3.1× for LLaMA-Instruct 3.3 70B.
HASS (green): ~2.8× for Vicuna 13B, ~3.2× for LLaMA-Instruct 3.1 8B, ~4.1× for LLaMA-Instruct 3.3 70B.
EAGLE (orange): ~3.1× for Vicuna 13B, ~3.6× for LLaMA-Instruct 3.1 8B.
EAGLE-2 (red): ~4.1× for Vicuna 13B, ~3.2× for LLaMA-Instruct 3.1 8B, ~4.4× for LLaMA-Instruct 3.3 70B.
EAGLE-3 (dark red): ~5.6× for Vicuna 13B, ~4.4× for LLaMA-Instruct 3.1 8B, ~4.1× for LLaMA-Instruct 3.3 70B, ~5.0× for DeepSeek-R1-Distill-LLaMA 8B, and ~3.4× for LLaMA 8B (base).

The comparison with HASS is particularly informative because HASS represents the alternative approach of patching EAGLE's error accumulation while retaining feature prediction. Across all five models, EAGLE-3 substantially outperforms HASS. The gap is largest on Vicuna 13B (~5.6× vs. ~2.8×) and smallest on LLaMA-Instruct 3.3 70B (~4.1× vs. ~4.1×—note that HASS matches EAGLE-3 on this model in the bar chart, though Table 1 doesn't report HASS results for the 70B model, making this a chart-only comparison). The consistent advantage validates the paper's claim that removing the feature prediction constraint is more effective than mitigating its consequences.

The chart also shows an interesting pattern for LLaMA 8B (the base, non-instruction-tuned model): EAGLE-3 achieves ~3.4× speedup, compared to EAGLE-2's unreported value and EAGLE's ~3.4× (they appear equal in the chart). The instruction-tuned variants (LLaMA-Instruct 3.1 8B) see larger gains from EAGLE-3, suggesting that the information fusion and expressiveness benefits are more impactful when the target model's generation distribution is shaped by instruction fine-tuning rather than raw pretraining.

Ablation Studies and Robustness Checks

The two architectural improvements independently contribute to speedup. Table 2 reports the ablation study on MT-bench and GSM8K with LLaMA-Instruct 3.1 8B as the target model:

EAGLE-2 (baseline): 3.16× speedup (τ = 4.05) on MT-bench; 3.39× (τ = 4.24) on GSM8K.
EAGLE-2 + remove feature constraint (+ "remove fea con"): 3.82× (τ = 5.37) on MT-bench; 3.77× (τ = 5.22) on GSM8K. This is a 21% improvement on MT-bench and 11% on GSM8K, confirming that the feature prediction loss was indeed a significant constraint on expressiveness. The τ improvement (from ~4.1 to ~5.3) shows that the gain comes from longer accepted sequences, not just faster per-token drafting.
EAGLE-2 + remove feature constraint + fused features (EAGLE-3): 4.40× (τ = 6.13) on MT-bench; 4.48× (τ = 6.23) on GSM8K. This adds 15% on MT-bench and 19% on GSM8K beyond the feature constraint removal alone.

The two improvements are not perfectly additive (3.16 → 3.82 → 4.40 on MT-bench; the total improvement of 1.24× is less than the product of the individual improvements of 1.21× and 1.15× = 1.39×), suggesting some overlap in their benefits—both contribute to better representations that enable longer acceptance chains—but each provides substantial independent value.

The feature constraint removal alone would fail without training-time test. While not presented as a formal ablation comparing "remove feature constraint with training-time test" vs. "remove feature constraint without training-time test," this is demonstrated implicitly by Figure 4. The "EAGLE without fea pred" condition in Figure 4 removes the feature constraint without training-time test (since EAGLE's original training procedure doesn't include self-feeding). The 1-α acceptance rate collapses to ~0.25–0.35. The "EAGLE-3" condition in Figure 4 removes the feature constraint with training-time test, and 1-α is high (~0.72–0.77). The training-time test is therefore not optional—it is required to prevent the multi-step degradation that would otherwise make the feature constraint removal net-negative. The ablation in Table 2 ("remove fea con") must include training-time test because otherwise the 3.82× speedup would not be achievable (the 1-α collapse would reduce it below EAGLE-2's 3.16×). This is confirmed by the fact that the "remove fea con" configuration achieves τ = 5.37, which requires multi-step acceptance that the Figure 4 "without fea pred" configuration lacks.

Model scale interacts with the benefit of EAGLE-3. While not a formal ablation, comparing results across model scales reveals an interaction: the relative improvement from EAGLE-2 to EAGLE-3 at temperature=0 is 31% for Vicuna 13B, 37% for LLaMA-Instruct 3.1 8B, 45% for LLaMA-Instruct 3.3 70B, and 28% for DeepSeek-R1-Distill-LLaMA 8B. The pattern suggests larger models benefit proportionally more from EAGLE-3's improvements, possibly because larger target models have richer multi-layer features (more layers to fuse from) and the expressiveness bottleneck is more severe (the draft model must approximate a more complex function). The DeepSeek-R1 model is an outlier in this pattern, showing a smaller relative improvement (28%), which may reflect its different architecture, training procedure, or the domain-specific draft model training (OpenThoughts-114k-math) creating a different baseline.

Temperature affects absolute speedup but not the relative advantage of EAGLE-3. Comparing temperature=0 and temperature=1 results in Table 1:

For Vicuna 13B, EAGLE-3's mean speedup drops from 5.51× to 4.65× (16% reduction) when moving to non-greedy sampling, while EAGLE-2's drops from 4.22× to 3.76× (11% reduction). The relative advantage of EAGLE-3 over EAGLE-2 is 31% at temperature=0 and 24% at temperature=1—slightly smaller but still substantial.
For LLaMA-Instruct 3.1 8B, the corresponding drops are 4.44× → 3.45× (22% reduction for EAGLE-3) and 3.23× → 2.80× (13% for EAGLE-2), with the relative advantage going from 37% to 23%.
The pattern that EAGLE-3's advantage narrows at temperature=1 is consistent across models and suggests that non-greedy sampling introduces randomness that partially masks the expressiveness gains (since the target model itself is less predictable). Nevertheless, EAGLE-3 continues to substantially outperform all baselines at temperature=1.

Task difficulty affects absolute speedup but EAGLE-3 consistently leads. Across all models and both temperatures, the ranking of tasks by speedup follows a consistent pattern: HumanEval (code generation) > GSM8K (math) ≈ MT-bench (conversation) > Alpaca (instruction following) > CNN/Daily Mail (summarization). For example, LLaMA-Instruct 3.1 8B EAGLE-3 at temperature=0: 4.85× (HumanEval) > 4.48× (GSM8K) > 4.40× (MT-bench) > 4.82× (Alpaca—note this breaks the pattern slightly; Alpaca at 4.82× is higher than MT-bench at 4.40×) > 3.65× (CNN/DM). The paper attributes the HumanEval advantage to "many fixed templates in code generation tasks, generating drafts is the easiest." The CNN/Daily Mail disadvantage likely reflects the open-ended nature of summarization, where there are many valid continuations and draft token prediction is inherently more difficult. EAGLE-3 maintains its advantage over EAGLE-2 across all tasks, confirming that the improvements are not task-specific.

The SGLang throughput results are robust to reduced draft complexity. The SGLang experiments (Tables 3, 4) use a simpler draft configuration than the main experiments: no tree structure, chain length of 3. Despite this simplification, EAGLE-3 achieves 1.38× throughput at batch size 64 and 373.25 tokens/s at batch size 1. These results are important because they demonstrate that the core architectural improvements (training-time test, multi-layer fusion) provide benefits even without the dynamic draft tree optimization from EAGLE-2. The draft tree provides additional gains (as evidenced by the speedup differences between Table 1's batch-1 results and Table 4's batch-1 results: 4.40× speedup vs. 2.36× throughput improvement, which are not directly comparable metrics but suggest the tree structure adds substantial value), but the fundamental improvements stand on their own.

Critical Assessment

The Core Claim: EAGLE-3 achieves ~1.4× speedup over EAGLE-2 and up to 6.5× over vanilla decoding

The experiments in Table 1 directly support this claim with comprehensive evidence across 4 models, 5 tasks, and 2 temperature settings. The mean speedup improvement over EAGLE-2 ranges from 24% (Vicuna 13B at temperature=1) to 49% (LLaMA-Instruct 3.3 70B at temperature=1), with the 31–45% range at temperature=0 corresponding well to the "about 1.4× improvement" claim in the abstract. The maximum speedup of 6.47× (LLaMA-Instruct 3.1 8B on HumanEval at temperature=0) slightly exceeds the abstract's claimed "up to 6.5×."

However, two aspects of this claim warrant qualification. First, the 1.4× figure is a round number that varies by model and task—a practitioner deploying EAGLE-3 should expect something in the 1.2–1.5× range over EAGLE-2 depending on their specific setup. Second, the speedup is measured against vanilla autoregressive decoding, not against batched vanilla decoding. The 6.5× figure is for single-request latency, not throughput. For throughput-oriented deployments, the relevant numbers are in Tables 3 and 5 (1.38–1.75× over no speculative sampling at batch sizes 2–64).

The Claim: Removing the feature prediction constraint and using training-time test enables a scaling law for inference acceleration

Figure 1 and Figure 4 provide clear evidence of a positive scaling relationship for EAGLE-3 and a flat relationship for EAGLE-2. The claim that a scaling law has been "discovered" is qualitatively supported: EAGLE-3's speedup improves with data while EAGLE-2's does not.

However, the characterization as a "scaling law" is ambitious for several reasons. First, four data points (1×, 2×, 4×, 8×) establish a direction but not a functional form. The paper does not fit a curve, report scaling exponents, or predict performance at higher data scales. "Law" typically implies a characterized mathematical relationship (e.g., the power-law relationships in Hoffmann et al., 2022), which the paper does not provide. Second, the absolute improvement from 8× data is modest: ~0.3× speedup (7% relative). This is a positive slope, but whether it represents a practically meaningful scaling relationship (as opposed to diminishing returns that asymptote quickly) cannot be determined from the available data. Third, the experiment is only shown for one model (LLaMA-Instruct 3.1 8B) on one task (MT-bench). It is unknown whether the scaling behavior generalizes to other model scales, architectures, or tasks.

The more conservative—and fully supported—claim would be: "EAGLE-3's architectural changes remove a bottleneck that prevented the draft model from benefiting from additional training data, as demonstrated by the positive correlation between data scale and speedup in Figure 1, in contrast with EAGLE-2's flat scaling curve." The paper's actual language ("scaling law," "scaling up training data provides limited improvements for EAGLE") slightly overstates what the experiments establish.

The Claim: Multi-layer feature fusion improves over top-layer-only features

Table 2 provides clean evidence: adding fused features to the feature-constraint-removed baseline improves speedup from 3.82× to 4.40× on MT-bench (15% relative) and from 3.77× to 4.48× on GSM8K (19% relative). This is a substantial and well-isolated contribution.

Two missing experiments would strengthen this claim. First, an ablation comparing multi-layer fusion with top-layer-only under the training-time test regime would isolate the fusion benefit more cleanly than the current comparison (which adds fusion on top of feature constraint removal). The current ablation shows that "remove constraint + fusion" outperforms "remove constraint alone," but it doesn't show whether fusion without constraint removal would help (which would test whether the benefit of fusion is independent of the training procedure). Second, the paper does not ablate which layers to fuse from—would low+high be sufficient? Is the middle layer necessary? Is there a benefit to fusing from more than three layers? These questions about the specific instantiation of multi-layer fusion are unanswered.

The Claim: EAGLE-3 maintains throughput benefits at larger batch sizes where prior methods fail

Tables 3 and 5 provide strong evidence for this claim, which is one of the paper's most practically important findings. EAGLE drops below 1.0× throughput at batch size 24 (SGLang) or 32 (vLLM), while EAGLE-3 maintains >1.0× throughput up to at least batch size 64 (SGLang) and 56 (vLLM). The 1.38× throughput at SGLang batch size 64 is a result that genuinely challenges conventional wisdom about speculative sampling's batch-size limitations.

The experiments were conducted by the SGLang team, which adds credibility (independent validation) but also raises questions about reproducibility in other frameworks. The vLLM results (Table 5) replicate the pattern—EAGLE-3 is beneficial at larger batch sizes where EAGLE is not—though with lower absolute throughput improvements, likely due to framework or GPU differences (A100 vs. H100). The paper does not explain these differences.

A limitation is that the maximum batch size tested is 64, and EAGLE-3's throughput improvement is declining with batch size (1.81× at 2 → 1.38× at 64 in SGLang). It is unclear at what batch size EAGLE-3 would drop below 1.0×, and whether this threshold is high enough for the largest-scale deployments (batch sizes of 128, 256, or higher). The trend suggests eventual crossover, but the paper doesn't explore this boundary.

The Claim: Training-time test solves the train-test distribution mismatch

Figure 7 provides compelling evidence: EAGLE-3's acceptance rate is nearly flat across self-generation depths (0-α through 7-α), while EAGLE's drops sharply. This is exactly what training-time test is designed to achieve, and the experimental result is clean.

The missing evidence is an explicit ablation comparing training-time test to EAGLE's original training procedure (single-step training with feature prediction loss) under otherwise identical conditions. Table 2 compares EAGLE-2 (which has feature prediction) with "remove fea con" (which presumably uses training-time test, since otherwise the speedup would collapse), but there is no comparison of "EAGLE-3 with training-time test" vs. "EAGLE-3 with single-step training only." This ablation would directly quantify the benefit of the multi-step training procedure independent of the feature constraint removal and multi-layer fusion. The Figure 4 comparison of "EAGLE without fea pred" vs. "EAGLE-3" provides this evidence implicitly (since "EAGLE without fea pred" lacks training-time test and performs poorly on 1-α), but a clean ablation within the EAGLE-3 architecture would be more informative.

Overarching Strengths

The experimental design has several genuine strengths. The evaluation across four diverse models (including a reasoning model), five tasks, and two temperature settings provides unusually broad coverage for a systems paper. The production framework experiments (SGLang, vLLM) address the gap between academic benchmarking and real-world deployment, and the independent validation by the SGLang team adds credibility. The ablation study cleanly isolates the two architectural improvements. The acceptance rate analysis (Figure 7) provides mechanistic insight, not just aggregate metrics, explaining why EAGLE-3 is faster.

Overarching Weaknesses

No statistical measures. All results are reported as point estimates without confidence intervals, standard deviations, or significance tests. Speedup ratios are measured on actual hardware and can vary across runs due to GPU boost clocks, thermal throttling, memory allocation patterns, and operating system noise. For a paper whose central claims involve numerical comparisons (1.4× improvement, scaling law), the lack of any error bars or variance reporting is a notable omission. The five-task evaluation provides informal robustness, but it cannot substitute for statistical rigor when comparing close numbers (e.g., 4.40× vs. 4.48×).

Single draft model training recipe. The draft model is trained once on a fixed combination of ShareGPT and UltraChat-200K (plus OpenThoughts-114k-math for the reasoning model) and then applied to all tasks without task-specific fine-tuning. While the paper frames this as a strength ("using the same weights for all tasks"), it leaves open the question of how much additional speedup could be achieved with task-specific or domain-specific draft model training. The DeepSeek-R1 result (5.01× on GSM8K, higher than HumanEval, after math-specific training) hints that domain adaptation matters. A systematic study of how draft model training data composition affects speedup on different downstream tasks would be valuable but is absent.

Limited exploration of the draft tree depth increase. EAGLE-3 increases tree depth from 6 to 8 while keeping total nodes constant. The paper attributes this to higher acceptance rates enabling deeper trees, but there is no ablation comparing depth 6 vs. depth 8 for EAGLE-3. It is possible that some of EAGLE-3's speedup gain comes from the deeper tree rather than the improved per-token acceptance, and this contribution is not isolated. A depth sweep for both EAGLE-2 and EAGLE-3 would disentangle these effects.

No latency breakdown. The paper reports end-to-end speedup but does not decompose it into drafting time, verification time, and feature extraction/fusion overhead. Such a breakdown would help practitioners understand where the compute goes and identify bottlenecks for further optimization. For example, the multi-layer feature fusion requires extracting and concatenating features from three layers—what is the overhead of this operation relative to EAGLE's single-layer feature extraction?

Absence of the largest models. The authors acknowledge GPU constraints prevented testing on 405B and 671B models. The trend across 8B → 13B → 70B suggests EAGLE-3's relative advantage over EAGLE-2 increases with model scale (31% → 37% → 45% at temperature=0), making the omission of larger models unfortunate—they would likely show the strongest results and provide the most practically valuable data for large-scale deployment decisions. The scaling behavior of EAGLE-3's benefits with target model size is an important open question.

The scaling law experiment is preliminary. As discussed above, four data points on one model-task combination establish a direction but not a law. A more comprehensive scaling study—multiple models, multiple tasks, larger data range (e.g., 16×, 32×), fitted curves—would substantially strengthen what is presented as a major contribution. The current evidence is suggestive rather than definitive.

Despite these limitations, the experimental evidence consistently supports the paper's central claims about EAGLE-3's superiority over prior speculative sampling methods, and the mechanistic analysis (acceptance rate curves, ablation study) provides a satisfying explanation for why the improvements occur. The production framework results and batch-size scaling behavior are particularly valuable contributions that address real-world deployment concerns beyond academic benchmarking.

6. Limitations and Trade-offs

Limitation 1: The Scaling Law Is Preliminary and Uncharacterized, Not an Established Predictive Relationship

The assumption or constraint. The paper claims discovery of "a scaling law for inference acceleration" where "increasing the amount of training data for the draft model leads to a proportional increase in the speedup ratio" (Section 1). However, the evidence for this claim consists of exactly four data points (1×, 2×, 4×, 8× the ShareGPT data scale) on a single model (LLaMA-Instruct 3.1 8B) evaluated on a single task (MT-bench), as shown in Figure 1. No functional form is fitted to the data, no scaling exponent is reported, no predictions are made or validated at held-out data scales, and no confidence intervals are provided. The absolute improvement across this range is modest—speedup rises from ~4.1× to ~4.4×, a roughly 7% relative improvement—and the curve's shape at the upper end (4× to 8×) appears to be shallowing, consistent with saturating rather than proportional returns. The paper itself hedges this claim, stating "we expect larger data size would lead to further improved speedup ratio" (Section 4.3), an expectation rather than a demonstrated fact.

The consequence. The term "scaling law" implies a characterized mathematical relationship that can be used to predict performance at unobserved data scales—this is what distinguishes scaling laws from simple observations of positive correlation. The paper provides no such characterization. A practitioner cannot use Figure 1 to estimate what speedup EAGLE-3 would achieve at 16×, 32×, or 100× data scale, nor can they determine whether the relationship follows a power law (which would imply continued meaningful returns), is logarithmic (diminishing returns), or will saturate entirely (a plateau similar to EAGLE-2's, just at a higher level). The flat EAGLE-2 curve demonstrates that some architectures exhibit zero scaling, but it does not establish that EAGLE-3's scaling is well-behaved or practically exploitable at larger data scales. The modest absolute gain from 8× data (0.3× speedup for 8× the data) already raises questions about cost-effectiveness: collecting and training on 8× more data for a ~7% speedup improvement may not be economically rational in many deployment scenarios, especially when the draft model training requires running the target model to generate responses for all that data.

What evidence exists in the paper. Figure 1 (both panels) and Figure 4 provide the four-point scaling data. The paper does not report scaling experiments on any model other than LLaMA-Instruct 3.1 8B, on any task other than MT-bench, or at data scales beyond 8×. There is no curve fitting, no scaling law parameter estimation, and no held-out validation of predicted speedups at unobserved data scales. The acceptance rate decomposition in Figure 4 (0-α and 1-α as functions of data scale) uses the same four data points and shows positive but similarly mild trends. The paper does not report the composition of the scaled dataset (whether additional data comes from UltraChat scaling, added datasets, or repetition), which matters because data diversity and quality may affect scaling behavior independently of quantity.

Mitigation status. The paper partially acknowledges the limitation implicitly by framing the finding as an observation rather than a fully characterized law: "we expect larger data size would lead to further improved speedup ratio" is forward-looking language. The abstract describes the scaling behavior as an "observation" rather than an established law. However, the Introduction uses stronger language ("discovery of a scaling law," "scaling law for inference acceleration") that overstates what the experiments establish. The paper does not suggest specific future work to characterize the scaling relationship more rigorously (functional form fitting, extrapolation validation, multi-model scaling studies), which would be necessary to elevate the observation to a genuine scaling law. The limitation is partially self-aware but the strength of the claim is not fully aligned with the evidence presented.

Limitation 2: The Difficulty Estimation Cost of Prior EAGLE Methods Is Eliminated, But No Analysis of Draft Model Training Cost or Inference Overhead Is Provided

The assumption or constraint. EAGLE-3 makes architectural changes that affect both training and inference computational costs: (1) multi-layer feature extraction requires capturing and storing features from three target model layers instead of one, (2) the fusion FC layer adds a 3k × k matrix multiplication per token per drafting cycle, (3) training-time test requires multiple forward passes through the draft model per training example (one per simulated step), and (4) the draft model's input representation is expanded to include concatenated feature+embedding pairs processed through an additional FC layer. The paper reports no measurements of any of these costs—neither the training time for draft models, nor the per-token inference overhead of the new components, nor a latency breakdown showing where time is spent during drafting. The only cost metric is the end-to-end speedup ratio, which folds all overheads into the numerator and denominator but does not decompose them.

The consequence. A practitioner deciding whether to adopt EAGLE-3 cannot answer several critical questions. What is the wall-clock time to train an EAGLE-3 draft model compared to an EAGLE-2 draft model? The training-time test procedure runs the draft model S times per training example (where S is the number of simulated steps, at least 3 based on Figure 6), which multiplies the draft model's training FLOPs by roughly S—but the draft model is a single decoder layer, so the absolute cost may still be modest. Without measurements, this remains unknown. At inference time, what fraction of the drafting latency is spent on feature extraction from three layers, concatenation, and the fusion FC layer, versus the draft model's own forward pass? If the overhead is significant, it eats into the speedup gains and may explain why the production framework throughput improvements (1.38× at batch size 64 in SGLang) are lower than the single-request speedup ratios (4.40× for the same model on MT-bench). The paper does not even report the number of parameters in the draft model, the FC layers, or the fusion component, making it impossible for a reader to estimate memory requirements or FLOPs.

What evidence exists in the paper. The paper reports zero measurements of training cost, inference overhead decomposition, or parameter counts. The SGLang and vLLM experiments (Tables 3, 4, 5) provide end-to-end throughput numbers that implicitly capture all overheads, but they do not decompose where the time goes. The paper notes that the SGLang experiments used a simplified configuration (no tree structure, chain length of 3) but does not explain why this simplification was necessary—was the full EAGLE-3 configuration too slow or too complex to integrate into SGLang? The absence of this information makes it difficult to assess whether EAGLE-3's overheads are well-justified or whether further optimization is needed.

Mitigation status. The paper does not acknowledge this as a limitation, nor does it discuss training cost, inference overhead decomposition, or parameter counts. The speedup ratio metric is standard in the speculative sampling literature, and most prior work (including EAGLE and EAGLE-2) similarly reports only end-to-end speedup without detailed overhead breakdowns. However, EAGLE-3 introduces more components (multi-layer extraction, fusion FC, training-time test) than its predecessors, making the omission more consequential. The paper would be strengthened by a simple table reporting: draft model parameter count, draft model training time relative to EAGLE-2, and a latency breakdown (feature extraction, fusion, draft model forward pass, LM head, tree construction) for at least one model-task combination. None of this is provided.

Limitation 3: No Experiments on the Largest Models (405B, 671B), Leaving Open the Question of Scalability to Frontier-Scale Deployments

The assumption or constraint. The paper evaluates EAGLE-3 on models up to 70B parameters (LLaMA-Instruct 3.3 70B), explicitly stating: "Due to the GPU constraint, we are unable to test EAGLE-3 on the 405B and 671B models" (Section 4). This means the method's behavior at the scale of the largest deployed LLMs—where inference cost is highest and acceleration is most valuable—is entirely uncharacterized. The largest models tested (70B) are substantial but are not in the same class as LLaMA-3.1 405B or DeepSeek-V3 671B, which represent the frontier of open-weight LLM deployment.

The consequence. Two features of EAGLE-3's design raise questions about scaling to very large models that cannot be answered from the available data. First, the trend across 8B → 13B → 70B shows that EAGLE-3's relative improvement over EAGLE-2 increases with model scale: roughly 37% for 8B, 31% for 13B, and 45% for 70B at temperature=0 (computed from Table 1 mean speedups). This is a suggestive but inconsistent trend (13B shows a smaller improvement than 8B), and extrapolating to 405B is unreliable. Second, larger models have more transformer layers, meaning the choice of which three layers to extract features from (l, m, h) becomes a higher-dimensional design decision. The paper's selection strategy (input to first, middle, and top decoder layers) is a natural default, but its optimality—and whether it generalizes to models with 100+ layers where "middle" is ambiguous—is unexamined.

There is also a practical concern: at very large model scales, the memory overhead of storing intermediate features from three layers (for all tokens in the KV cache) may become significant. The paper does not report the memory footprint of feature storage, but for a 405B model with large context lengths, storing per-token hidden states from three layers in addition to the KV cache could meaningfully increase GPU memory pressure, potentially reducing the maximum batch size or context length that can be served.

What evidence exists in the paper. The scaling across model sizes is reported in Figure 2 and Table 1. The trend is generally positive (larger models benefit more from EAGLE-3), but the evidence is based on only four models in the 8B–70B range. There are no experiments, analyses, or even projections about 405B+ scale behavior. The feature selection strategy for large models is described once for all models ("low, middle, and high-level features") without discussion of how it should be adapted for models with different layer counts.

Mitigation status. The authors honestly disclose the GPU constraint preventing larger-model experiments but do not discuss the implications of this omission, suggest heuristics for applying EAGLE-3 to larger models, or propose how the method might need to be adapted. Given that the strongest practical case for inference acceleration exists at the largest model scales (where inference costs dominate total expenditure), this is a significant gap. Future work with access to larger GPU clusters could address it, but the paper provides no guidance for practitioners deploying EAGLE-3 at the 405B scale.

Limitation 4: The Method Is Evaluated on a Narrow Task Distribution (Predominantly English Instruction-Following), with No Evidence for Broader Generalization

The assumption or constraint. EAGLE-3 is evaluated on five tasks: MT-bench (multi-turn conversation), HumanEval (code generation), GSM8K (math reasoning), Alpaca (instruction following), and CNN/Daily Mail (summarization). All five are English-language tasks. All five involve relatively structured generation where there are "correct" or at least predictable continuations. The method is not evaluated on: open-ended creative writing, long-form document generation, multilingual tasks, tasks requiring extensive factual recall (where the target model's internal features may encode knowledge differently), or tasks with very long contexts (where the draft model's single decoder layer may struggle to capture long-range dependencies via self-attention). The draft model is trained on ShareGPT and UltraChat-200K—both English instruction-following datasets—with OpenThoughts-114k-math added for the reasoning model.

The consequence. The paper's claim that EAGLE-3 "achieves a speedup ratio up to 6.5×" (abstract) implicitly suggests this is representative of general LLM inference acceleration. But the task distribution tested skews toward settings where the target model's behavior is relatively predictable: code generation has fixed templates, math reasoning follows structured chains of deduction, and summarization operates on constrained input-output pairs. In more open-ended generation scenarios—dialogue that drifts across topics, creative writing with high entropy, factual generation where token-level features may be less predictive of multi-token continuations—the acceptance rate behavior characterized in Figure 7 may not hold. The near-flat n-α curve (acceptance rate remains stable across self-generation depths) is demonstrated only on MT-bench for one model. Whether this robustness extends to high-entropy or out-of-distribution generation tasks is unknown.

Additionally, the training data composition is English-only and instruction-following-focused. A draft model trained on this data and then applied to, say, code completion in a non-English programming language, or translation between non-English languages, may exhibit different acceptance characteristics because the target model's internal features in those settings would encode linguistic structures not represented in the training data. The paper does not assess this.

What evidence exists in the paper. The five-task evaluation is a reasonable breadth for a systems paper, and the tasks do cover distinct generation types (conversation, code, math, instruction, summarization). However, they are all English and all relatively structured. The highest speedup on HumanEval (6.47×) is attributed to "many fixed templates in code generation tasks, generating drafts is the easiest," which indirectly acknowledges that task structure affects speedup. The paper does not discuss task coverage limitations, nor does it suggest that the reported speedups should be interpreted as task-dependent rather than universal.

Mitigation status. The limitation is not acknowledged in the paper. The evaluation breadth exceeds prior work in the speculative sampling literature—most papers evaluate on 1-3 tasks—and the inclusion of code, math, conversation, instruction, and summarization is a genuine strength. However, the lack of multilingual evaluation, long-context evaluation, or high-entropy generation evaluation means the paper provides no evidence about where EAGLE-3 might underperform. Future work on broader task coverage, particularly including tasks where the target model's token-level predictability is lower, would establish the robustness and boundaries of the method.

Limitation 5: The Dynamic Draft Tree Integration and Depth Increase Are Not Ablated, Making It Unclear How Much of the Speedup Comes from the Core Architectural Changes Versus Better Tree Configuration

The assumption or constraint. EAGLE-3 adopts EAGLE-2's context-aware dynamic draft tree mechanism but increases the tree depth from 6 to 8 while keeping the total number of draft tokens constant (60 for 7B/8B models, 50 for 13B, 48 for 70B). The paper justifies this change: "EAGLE-3's draft model achieves a significantly higher acceptance rate, allowing us to increase the draft tree depth from 6 to 8 while keeping the number of nodes the same as in EAGLE-2" (Appendix A). However, there is no ablation experiment comparing EAGLE-3 with depth 6 versus depth 8, or EAGLE-2 with depth 8 (to test whether some of the gain comes from the deeper tree alone, independent of the architectural changes). The depth increase and the architectural changes are confounded in the main comparisons (Table 1).

The consequence. A portion of EAGLE-3's reported speedup improvement over EAGLE-2 may come from the deeper draft tree rather than from training-time test and multi-layer fusion. If EAGLE-2 could also benefit from depth 8 trees (perhaps with a smaller gain, since its acceptance rates are lower), then the 1.4× improvement attributed to EAGLE-3's architectural innovations would be overstated. Conversely, if the depth increase is only beneficial because of the higher acceptance rates (deeper trees with low acceptance rates would waste compute on tokens that get rejected), then the depth change is not a confound but a direct consequence of the architectural improvements—however, this remains an untested assumption. A practitioner reading the paper cannot determine whether they should adopt EAGLE-3's architecture, increase their tree depth, or both when deploying their own speculative sampling system.

The paper also does not ablate the tree construction parameters: the number of nodes selected during expansion (10), the total draft token budget, or the depth budget. These hyperparameters were tuned for EAGLE-2 and adopted unchanged for EAGLE-3 (except depth), but EAGLE-3's different acceptance rate profile may warrant different optimal settings. The SGLang and vLLM experiments use a simplified chain with no tree structure at all (depth limited to 2 or 3, no branching), which provides a partial control—EAGLE-3 still outperforms EAGLE in those experiments (Tables 3, 5)—but the chain-based results are not directly comparable to the tree-based main results, and the chain configuration is not systematically varied.

What evidence exists in the paper. No ablation of tree depth, tree structure, or tree construction parameters is reported. The Appendix A note about increasing depth from 6 to 8 is the only discussion of this design choice. The SGLang/vLLM experiments (no tree, chain length 2–3) partially isolate the architectural improvements from the tree mechanism, but they use different metrics (throughput vs. speedup), different hardware, and different configurations, preventing direct comparison. The paper would benefit from a simple experiment: EAGLE-3 at depth 6 vs. EAGLE-3 at depth 8, and EAGLE-2 at depth 8 vs. EAGLE-2 at depth 6, on at least one model-task combination, to isolate the contribution of the tree depth change.

Mitigation status. The limitation is not acknowledged. The paper presents the depth increase as a natural consequence of higher acceptance rates rather than as a design choice that requires validation. While the reasoning is plausible (higher acceptance rates make deeper trees more worthwhile), the absence of an ablation leaves open the question of how much of the ~1.4× improvement over EAGLE-2 is attributable to the core innovations (training-time test, multi-layer fusion) versus the tree configuration change. The SGLang/vLLM results provide suggestive evidence that the architectural improvements matter independently, but the confound remains in the headline results.

Limitation 6: The Method Is Only Demonstrated on LLaMA-Family and Vicuna Architectures, with No Evidence for Generalization to Architecturally Distinct Models

The assumption or constraint. All target models evaluated in the paper belong to the LLaMA architectural family: Vicuna 13B (based on LLaMA 1), LLaMA-Instruct 3.1 8B, LLaMA-Instruct 3.3 70B, and DeepSeek-R1-Distill-LLaMA 8B (a distilled version of DeepSeek-R1 fine-tuned from LLaMA 8B). These models share the same basic transformer architecture (pre-normalization, SwiGLU activations, rotary position embeddings, grouped query attention in some variants). The paper does not evaluate on models with different architectural paradigms: mixture-of-experts (e.g., Mixtral, DeepSeek-V2/V3 base), non-LLaMA architectures (e.g., Gemma, Phi, Qwen, Falcon), encoder-decoder models (e.g., T5, BART), or models with different normalization schemes, attention mechanisms, or positional encodings.

The consequence. EAGLE-3's design makes specific assumptions about the target model's architecture that may not hold generally. The multi-layer feature extraction assumes a standard transformer with distinct decoder layers from which low, middle, and high features can be extracted. In a mixture-of-experts model, the "features" at a given layer may be routed through different expert sub-networks depending on the token, potentially creating discontinuities or distribution shifts in the feature space that the fusion FC layer and draft model were not designed to handle. The feature extraction also assumes that the hidden state dimensionality is constant across layers (all features are k-dimensional), which holds for standard transformers but may not for architectures with varying internal dimensions. The reliance on the target model's LM head assumes a standard linear projection from hidden states to vocabulary logits, which may not apply to models with factored embeddings, tied weights, or alternative output parameterizations.

The paper also implicitly assumes that the target model's features are informative and well-behaved when used as input to the draft model. If a non-LLaMA architecture learns internal representations with different statistical properties (different variance scaling, different sparsity patterns, different information content across layers), the draft model trained on LLaMA features may not transfer. The training-time test procedure is architecture-agnostic in principle, but its effectiveness depends on whether the draft model can learn to interpret and predict from the target model's specific feature representation.

What evidence exists in the paper. The evaluation is restricted to LLaMA-family models. The paper does not discuss architectural assumptions or claim generalization beyond the tested architectures. The model selection is diverse within the LLaMA family—different sizes, different fine-tuning stages (base, instruct, reasoning-distilled), and different generations (LLaMA 1-based Vicuna, LLaMA 3.1, LLaMA 3.3)—but all share the same fundamental architecture. There is no theoretical analysis of which architectural properties EAGLE-3 depends on, and no empirical test on non-LLaMA architectures.

Mitigation status. The paper does not acknowledge this as a limitation. The LLaMA family dominates the open-weight LLM landscape, making the evaluation highly relevant for practitioners—most real-world deployments using open models will use LLaMA-derived architectures. However, the method is presented as a general speculative sampling technique ("EAGLE-3, an enhanced version of EAGLE"), not as a LLaMA-specific optimization. The lack of non-LLaMA evaluation means a practitioner using, for example, Gemma or Phi cannot be confident that EAGLE-3 will work as described, and the paper provides no guidance for adapting the method to different architectures. Future work testing on Mixture-of-Experts models, models with alternative attention mechanisms, or encoder-decoder architectures would establish the scope of applicability.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper initiates a reframing of speculative sampling research around expressiveness bottlenecks rather than architectural complexity. Prior work in the EAGLE lineage (and speculative sampling more broadly) operated under an implicit assumption that draft model quality was primarily limited by capacity—the draft model is intentionally small, so of course it cannot perfectly approximate the target model. The standard approach to improving speedup was therefore to add mechanisms: dynamic draft trees (EAGLE-2), error accumulation mitigation (HASS), multiple decoding heads (Medusa, Hydra), or richer feature reuse (Falcon). Each of these added complexity on top of the feature prediction framework that EAGLE established.

EAGLE-3 demonstrates that this complexity was, in an important sense, patching a fundamentally constrained architecture. The feature prediction loss that enabled EAGLE's elegant trick—single-step training generalizing to multi-step inference—was simultaneously a hard ceiling on expressiveness that prevented the draft model from benefiting from additional training data. The paper's key reframing is: the question is not "how can we make the draft model better at predicting features?" but rather "what architectural constraints are preventing the draft model from fully exploiting available training data?" This shifts the optimization target from better feature prediction to greater expressiveness, and the training-time test technique is the mechanism that makes this shift possible without sacrificing multi-step capability.

The magnitude of this contribution is not a paradigm shift—speculative sampling remains the same fundamental approach, and EAGLE-3 is demonstrably an evolution of EAGLE/EAGLE-2 rather than a clean-slate redesign. But it is more than an incremental refinement. It is a diagnostic contribution: the paper identifies a specific mechanism (representation coupling via feature prediction loss) as the root cause of a previously unobserved phenomenon (flat scaling law for inference acceleration), and then demonstrates that removing this mechanism and replacing it with training-time test enables a qualitatively different scaling behavior. The diagnostic framework—scale training data, watch the speedup curve, and if it's flat, look for expressiveness bottlenecks—is transferable to other speculative sampling methods and potentially to other domains where intermediate supervision losses are used.

The work also reconciles a tension in the speculative sampling literature between two approaches to multi-step drafting. HASS (Zhang et al., 2024) argued that error accumulation from inaccurate feature predictions was the key problem and addressed it by simulating multi-step training while retaining feature prediction. EAGLE had already implicitly recognized the distribution-shift issue and addressed it through the feature prediction loss itself. These approaches patched the feature prediction framework from different angles. EAGLE-3's insight is that the framework itself is the problem—removing it entirely and replacing it with training-time test yields better results than any patch. Figure 2 validates this empirically: EAGLE-3 substantially outperforms HASS across all tested models and tasks. The resolution is not that HASS or EAGLE were wrong about the existence of error accumulation or distribution shift; they correctly identified real problems. But they misdiagnosed the solution. The correct solution was not to mitigate the consequences of feature prediction constraints but to eliminate the constraints and handle the distribution shift through a different mechanism (self-feeding during training).

This finding redirects research attention in several ways:

Away from increasingly complex draft model architectures and toward training procedures that maximize expressiveness. If a single transformer decoder layer with training-time test can outperform more complex architectures (HASS's harmonized representations, Medusa's multiple heads), then the bottleneck was never architectural complexity—it was the training objective. Future work should investigate what other training objectives or constraints in speculative sampling might be imposing hidden expressiveness ceilings, and whether training-time test can replace them.
Away from treating draft model training data scale as a fixed resource and toward viewing it as a scaling axis. Prior speculative sampling papers did not systematically vary training data scale, likely because there was no reason to expect it would matter—EAGLE-2's flat scaling curve (Figure 1) would have been the expected result if anyone had tested it. EAGLE-3 shows that data scale matters, but only when the architecture is unconstrained enough to exploit it. This opens the door to draft model training at scales comparable to target model pretraining, with the attendant questions about data quality, diversity, and composition.
Toward understanding the relationship between target model architecture and draft model design. EAGLE-3's multi-layer feature fusion implicitly assumes that information useful for multi-token prediction is distributed across layers—that top-layer features are not sufficient. This is an empirically validated assumption for LLaMA-family models, but whether it holds for other architectures (mixture-of-experts, alternative attention patterns, different depth-width ratios) is unknown. The paper provides a methodology for testing this: compare speedup with top-layer-only features versus fused multi-layer features under the training-time test regime.
Toward production deployment of speculative sampling at larger batch sizes. The SGLang and vLLM results (Tables 3, 5) challenge the conventional wisdom that speculative sampling is only useful at batch size 1. EAGLE-3's 1.38× throughput improvement at batch size 64 is practically significant and, if generalizable to other models and frameworks, would change deployment decisions for high-traffic LLM services. This finding shifts speculative sampling from a latency optimization (for interactive applications) to a throughput optimization (for batch processing), expanding its applicability.

Follow-Up Research This Work Enables

Characterizing the functional form of the EAGLE-3 data scaling relationship. The paper establishes a positive correlation between training data scale and speedup (Figure 1), but four data points on one model-task combination is a preliminary observation, not a characterized law. A strong follow-up would train EAGLE-3 draft models at data scales ranging from 1× to at least 32× (possibly higher, using large-scale instruction datasets like OpenHermes, Dolphin, or synthetic data generation from the target model itself), measure speedup on multiple tasks (MT-bench, HumanEval, GSM8K, plus an out-of-distribution task not represented in training), and fit power-law and logarithmic functions to the speedup-vs-data curves. The key question: does the scaling relationship follow a power law (speedup ∝ data^α) with a predictable exponent, or does it saturate? If a power law holds, the exponent would be a fundamental quantity characterizing draft model data efficiency; if it saturates, the saturation point defines the practical limit of data scaling. The paper's own data hints at shallowing between 4× and 8×, consistent with saturation, but four points cannot distinguish a saturating curve from a low-exponent power law. The experiment should also measure scaling on a model where EAGLE-3's baseline speedup is lower (e.g., CNN/Daily Mail task, where speedups are consistently the lowest), since scaling behavior may differ when the starting point is lower. If the scaling curve saturates quickly on harder tasks, it would suggest that data scaling helps most where the draft model is already strong—a practically important boundary condition.

Training-time test as a drop-in replacement for feature prediction in other speculative sampling architectures. The paper demonstrates training-time test in the specific context of EAGLE-3 (single decoder layer, fused multi-layer features, reused LM head), but the principle—simulate self-feeding during training to close the train-test distribution gap—should apply to any architecture where a learned component's outputs are fed back as inputs. A direct follow-up would implement training-time test in Medusa (replacing Medusa's parallel head training with sequential self-feeding), in Hydra (where the sequential dependency between draft heads creates an analogous distribution shift problem), and in Falcon (which also predicts features autoregressively). The experiment would measure whether training-time test improves speedup in these architectures without any other architectural changes—i.e., is the principle portable, or does it depend on EAGLE-3's specific design choices (removal of feature prediction, single decoder layer, dual feature+embedding input)? A negative result (training-time test doesn't help Medusa) would be informative because it would suggest that eliminating feature prediction is a necessary precondition for training-time test to be effective, refining our understanding of when and why the technique works.

Difficulty-adaptive or entropy-adaptive draft tree configuration. The paper notes that EAGLE-3 increases tree depth from 6 to 8 because higher acceptance rates justify deeper trees, and that HumanEval achieves the highest speedups because "fixed templates" make drafting easier. This suggests that optimal tree configuration (depth, branching factor, total nodes) is context-dependent, but EAGLE-3 uses a single static configuration per model (depth 8, 60/50/48 nodes). A follow-up would implement dynamic tree configuration that adapts to the estimated difficulty or entropy of the current generation context. For example: use the draft model's confidence (softmax probability of the top-1 token) at each drafting step to decide whether to expand deeper (high confidence → continue the chain) or branch wider (low confidence → explore alternatives). This is a natural extension of EAGLE-2's confidence-based pruning, but operating at the tree construction level rather than post-hoc pruning. The experiment would compare static depth-8 trees against adaptive trees on a task with mixed difficulty (e.g., GSM8K, where some problems require long reasoning chains and others are single-step), measuring both speedup and the correlation between local confidence and optimal depth. If adaptive trees outperform static trees substantially, it would suggest that the next frontier in speculative sampling is not better draft models but smarter allocation of the drafting budget—paralleling the compute-optimal test-time scaling insight from the LLM inference literature.

Cross-architecture generalization: testing EAGLE-3 on mixture-of-experts and non-LLaMA models. The paper's evaluation is restricted to dense LLaMA-family models, leaving open the question of whether EAGLE-3's design assumptions hold for other architectures. A targeted follow-up would implement EAGLE-3 on (a) a mixture-of-experts model like Mixtral 8×7B, where features at each layer are routed through different expert sub-networks depending on the token, potentially creating discontinuities in the feature space that the fusion FC layer and draft model must handle; (b) a non-LLaMA dense model like Gemma 2 or Phi-3, which use different normalization schemes, activation functions, or positional encodings; and (c) an encoder-decoder model like FLAN-T5, where the decoder's cross-attention to encoder outputs creates a fundamentally different feature structure. For each architecture, the experiment would report: whether EAGLE-3 trains successfully without architecture-specific modifications, the speedup ratio relative to vanilla decoding and to the best available speculative sampling baseline for that architecture, and whether multi-layer fusion provides the same benefit as on LLaMA (or whether a different layer selection strategy is needed). A failure on mixture-of-experts models—if the draft model cannot learn to interpret expert-routed features—would indicate a fundamental limitation that requires architectural innovation. A success on Gemma or Phi would demonstrate that EAGLE-3's principles generalize across dense transformer variants, substantially expanding its applicability.

Scaling the draft model itself: multiple decoder layers, larger hidden dimensions, or separate LM head training. EAGLE-3 uses a single transformer decoder layer as the draft model, following EAGLE's design. The training-time test technique removes the expressiveness bottleneck, which raises a natural question: if the draft model had more capacity (more layers, wider hidden dimensions), could it exploit additional training data even more effectively? A follow-up would train EAGLE-3 draft models with 2, 3, and 4 decoder layers at multiple data scales (1×, 4×, 16×), measuring whether deeper draft models show steeper scaling curves (larger speedup improvement per data doubling) or whether the single-layer model already captures most of the achievable gain. The experiment would also test whether a separately trained LM head (rather than reusing the target model's LM head) improves acceptance rates—since the draft model's internal representation a is no longer constrained to approximate the target model's feature space, the frozen LM head may be suboptimal for the draft model's learned representations, and a fine-tuned or jointly trained LM head could close this gap. The cost tradeoff would be critical: a deeper draft model with a separate LM head would increase drafting latency per token, and the experiment must measure whether the higher acceptance rate compensates. This is the natural endpoint of the expressiveness argument: if removing the feature prediction constraint enables scaling, what is the optimal scale for the draft model itself, and how does this interact with data scale?

Characterizing the failure modes of EAGLE-3: when does the n-α acceptance rate curve stop being flat? Figure 7 shows that EAGLE-3's acceptance rate is nearly invariant to the number of self-generated inputs (n-α values 0 through 7 are similar), but this is shown only for MT-bench with LLaMA-Instruct 3.1 8B. A systematic failure-mode analysis would measure n-α curves (out to n = 10 or 15) across multiple tasks, models, and temperatures, identifying conditions where the curve does drop. Specific hypotheses to test: (a) on high-entropy tasks (creative writing, open-ended dialogue), the acceptance rate curve may degrade at higher n because the draft model's predictions become increasingly unmoored from any ground truth; (b) at higher temperatures, the randomness in token sampling introduces larger variance in the a vectors, potentially causing distribution shift even with training-time test; (c) on out-of-distribution prompts (e.g., testing a draft model trained on English instruction data on Chinese or code-switched inputs), the feature representations from the target model may be sufficiently different that the draft model's self-feeding loop breaks down. Identifying where the flat n-α curve stops being flat would define the operating envelope of EAGLE-3 and guide practitioners on when to trust its robustness.

Practical Applications and Downstream Use Cases

Batch inference for LLM API services at moderate batch sizes. EAGLE-3's most practically significant result is the sustained throughput improvement at batch sizes where prior speculative sampling methods fail. In SGLang on H100 with LLaMA-Instruct 3.1 8B (Table 3), EAGLE-3 achieves 1.38× throughput at batch size 64, while EAGLE drops below 1.0× at batch size 24. For an LLM API provider processing millions of requests daily, a 38% throughput improvement at batch size 64 directly translates to serving 38% more requests on the same hardware, or equivalently, reducing GPU costs by approximately 28% for the same request volume. The batch size range (16–64) is representative of production serving where requests are grouped to amortize GPU idle time but not so large that latency becomes unacceptable. The vLLM results on A100 (Table 5) show a similar pattern with somewhat lower absolute gains, suggesting the benefit is framework-dependent but directionally robust. The deployment scenario is: an API service running LLaMA-Instruct 3.1 8B (or similar-scale models) with dynamic batching, where EAGLE-3 is integrated into the serving framework (SGLang already has native support) and provides automatic throughput improvement without any per-request configuration. The draft model is trained once offline on general instruction-following data and deployed alongside the target model.

Latency reduction for interactive reasoning model applications. The paper demonstrates 5.01× speedup on GSM8K with DeepSeek-R1-Distill-LLaMA 8B (Table 1), a distilled reasoning model that generates lengthy chain-of-thought traces. As the paper notes, reasoning models "significantly increase the proportion of inference costs in the overall LLM pipeline" because they generate hundreds or thousands of reasoning tokens before producing a final answer. For an interactive application using a reasoning model—an AI tutor that walks through math problems step by step, a code assistant that reasons about architecture before writing code, a scientific QA system that explains its reasoning—the 5× latency reduction transforms user experience from "noticeable wait" to "near-instantaneous." The specific deployment scenario is: a single-user interactive session with a reasoning model, where the dominant cost is the sequential generation of reasoning tokens. EAGLE-3's draft model is trained on domain-specific data (OpenThoughts-114k-math for math reasoning) to maximize acceptance rates on the target domain. The GPU is running at effectively batch size 1, so the full speedup ratio (4–5×, not the reduced throughput gains at larger batches) is realized. The draft model's training cost is amortized across many user sessions.

On-device or edge deployment of mid-size models with cloud-level latency. While the paper evaluates models in the 8B–70B range on datacenter GPUs, the speedup ratios suggest a deployment model where a mid-size model (e.g., LLaMA-Instruct 3.1 8B) running on a consumer GPU or even a high-end mobile processor could achieve latency comparable to a much larger model without speculative sampling. If EAGLE-3 provides 4.4× speedup on 8B (Table 1 mean), the effective tokens-per-second could approach what an unaccelerated 35B model would deliver—not because the 8B model is as capable, but because the user experiences similar responsiveness. This is particularly relevant for privacy-sensitive applications (on-device medical QA, local code assistants, offline document processing) where sending data to a cloud API is unacceptable. The deployment scenario is: a laptop or workstation with a single consumer GPU running a local LLM with EAGLE-3 for interactive use. The draft model is pre-trained and distributed alongside the target model weights. The key constraint is GPU memory: EAGLE-3 requires storing features from three layers in addition to the KV cache, and the memory overhead (not quantified in the paper) must fit within the available VRAM.

Cost-efficient data generation for LLM training and evaluation. When using LLMs to generate synthetic training data (for distillation, instruction tuning, or preference optimization) or to evaluate on large test sets, the dominant cost is inference. EAGLE-3's speedup directly reduces this cost by the speedup factor: generating 1 million tokens with a 4.4× speedup requires approximately 23% of the GPU-hours compared to vanilla decoding. For a research lab or company generating billions of tokens for training data, this translates to substantial compute savings. The deployment scenario is: offline batch generation where throughput per GPU is the key metric. EAGLE-3's batch-size scaling behavior (Tables 3, 5) determines the optimal batch size for throughput, which may differ from the batch size that maximizes vanilla throughput. The draft model can be trained on the same data distribution being generated, potentially achieving even higher acceptance rates than the general-purpose draft models evaluated in the paper. The quality of generated data is unaffected because speculative sampling with strict acceptance criteria is lossless.

When to Prefer This Method

The paper positions EAGLE-3 as a direct successor to EAGLE and EAGLE-2 within the speculative sampling paradigm, and the empirical comparisons in Figure 2 and Table 1 consistently show EAGLE-3 outperforming all prior speculative sampling methods across all tested conditions. The paper does not articulate a tradeoff where a practitioner might prefer EAGLE-2, HASS, Medusa, or standard speculative sampling over EAGLE-3 for any specific deployment scenario. The method is presented as strictly dominant on the metrics measured (speedup ratio, acceptance length, throughput) while maintaining the same lossless guarantee as prior speculative sampling approaches.

The paper also does not position EAGLE-3 against non-speculative acceleration methods (quantization, distillation, pruning, flash attention, kernel fusion) or discuss conditions where speculative sampling in general is preferable to these alternatives. The comparison framework is entirely within speculative sampling.

Given this, a conditional decision matrix based on the paper's content would be:

Prefer EAGLE-3 over EAGLE/EAGLE-2 when deploying speculative sampling for LLaMA-family models (8B–70B). The method provides 1.2–1.5× speedup improvement over EAGLE-2 with no identified downside—same lossless guarantee, same inference framework compatibility, and comparable implementation complexity (the paper provides code, and SGLang has integrated support). The only practical consideration is the need to train a new draft model (the paper does not provide pre-trained EAGLE-3 draft model weights for all configurations, though the code repository may include them).
Prefer EAGLE-3 over HASS, Medusa, Hydra, or standard speculative sampling across all tested model scales and tasks based on the comprehensive superiority in Table 1 and Figure 2. No condition is identified where these methods outperform EAGLE-3.
The open question is models beyond 70B and non-LLaMA architectures. The paper provides no evidence for or against EAGLE-3 on 405B, 671B, mixture-of-experts, or architecturally distinct models. A practitioner deploying speculative sampling on these models cannot rely on the paper's results and must conduct their own evaluation. The paper's methodological framework (training-time test, multi-layer fusion) provides a recipe but not a guarantee.

Dependency tree

Papers this one builds on, walking backwards through the same domain and shared topics.

EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty2024· inference-optimization← LLM-judged similar
- Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads2024· inference-optimization← in-domain predecessor
- TAPS: Task Aware Proposal Distributions for Speculative Sampling2026· inference-optimization← similar topic
LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attention2025· inference-optimization← in-domain predecessor
- InfiniteHiP: Extending Language Model Context Up to 3 Million Tokens on a Single GPU2025· inference-optimization← in-domain predecessor
- Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps2026· inference-optimization← similar topic
Reward-Guided Speculative Decoding for Efficient LLM Reasoning2025· reasoning← topic thread
- SpecInfer: Accelerating Large Language Model Serving with Tree-based Speculative Inference and Verification2023· serving← LLM-judged similar
- Trading Inference-Time Compute for Adversarial Robustness2025· reasoning← in-domain predecessor
- Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Models2025· multimodal← topic thread

What it influenced

Later papers in the same chain. Direction reversed.

Quantization for OpenAI's Whisper Models: A Comparative Analysis2025· inference-optimization→ in-domain predecessor
- Hogwild! Inference: Parallel LLM Generation via Concurrent Attention2025· inference-optimization→ in-domain predecessor
- EfficientLLM: Efficiency in Large Language Models2025· evaluation→ topic thread
Eagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language Models2025· multimodal→ topic thread
- Kimi-Audio Technical Report2025· multimodal→ in-domain predecessor
- QwenLong-L1: Towards Long-Context Large Reasoning Models with Reinforcement Learning2025· rl-training→ topic thread
- Beat the long tail: Distribution-Aware Speculative Decoding for RL Training2025· training-methods→ topic thread
- Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Models2025· multimodal→ similar topic
- PaddleOCR-VL: Boosting Multilingual Document Parsing via a 0.9B Ultra-Compact Vision-Language Model2025· multimodal→ similar topic
- Training Long-Context Vision-Language Models Effectively with Generalization Beyond 128K Context2026· multimodal→ similar topic
Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads2024· inference-optimization→ similar topic
- Speculative Streaming: Fast LLM Inference without Auxiliary Models2024· inference-optimization→ LLM-judged similar
- Kangaroo: Lossless Self-Speculative Decoding via Double Early Exiting2024· inference-optimization→ LLM-judged similar
- Clover: Regressive Lightweight Speculative Decoding with Sequential Knowledge2024· inference-optimization→ LLM-judged similar
- Set Block Decoding is a Language Model Inference Accelerator2025· inference-optimization→ LLM-judged similar
- EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty2024· inference-optimization→ in-domain predecessor
- Fast Inference from Transformers via Speculative Decoding2022· inference-optimization→ similar topic
- SkipDecode: Autoregressive Skip Decoding with Batching and Caching for Efficient LLM Inference2023· inference-optimization→ similar topic
- Accelerating LLM Inference with Staged Speculative Decoding2023· inference-optimization→ similar topic
Distributed Speculative Inference (DSI): Speculation Parallelism for Provably Faster Lossless Language Model Inference2024· inference-optimization→ similar topic
- A Simple and Effective $L_2$ Norm-Based Strategy for KV Cache Compression2024· inference-optimization→ in-domain predecessor
- Cascade Speculative Drafting for Even Faster LLM Inference2023· inference-optimization→ similar topic
SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Acceleration2024· inference-optimization→ similar topic
- PrefixQuant: Eliminating Outliers by Prefixed Tokens for Large Language Models Quantization2024· inference-optimization→ in-domain predecessor
- Prompt Cache: Modular Attention Reuse for Low-Latency Inference2023· inference-optimization→ similar topic
- SparQ Attention: Bandwidth-Efficient LLM Inference2023· inference-optimization→ similar topic
- SageAttention3: Microscaling FP4 Attention for Inference and An Exploration of 8-Bit Training2025· inference-optimization→ similar topic