SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding

ArXiv: 2604.09557

🎯 Pitch

Synthetic random-token inputs overestimate real-world throughput by 23% in speculative decoding, and vocabulary pruning in state-of-the-art drafters silently kills performance on long-tail domains. SPEED-Bench introduces the first benchmark that exposes these failures by evaluating across semantically diverse data and production engines under varying batch sizes.

---

1. Executive Summary

This paper introduces SPEED-Bench, a unified benchmark suite for evaluating Speculative Decoding (SD) algorithms across diverse semantic domains and realistic serving regimes. The benchmark comprises a Qualitative Split—curated via a greedy selection algorithm with local swap refinement to maximize semantic diversity across 11 categories (e.g., Math, Coding, Multilingual)—and a Throughput Split that aggregates samples into fixed Input Sequence Length buckets (1k–32k) across three entropy levels, enabling throughput-latency Pareto curve construction under production-grade engines like vLLM and TensorRT-LLM. SPEED-Bench reveals that synthetic random-token inputs overestimate real-world throughput by an average of 23% and that vocabulary pruning in state-of-the-art drafters like EAGLE3 disproportionately degrades acceptance lengths in long-tail domains—dropping by 9–10% in Summarization and RAG—establishing that SD performance is deeply data-dependent and sensitive to both semantic diversity and concurrency regime, with optimal draft lengths shifting from longer chains at low batch sizes to shorter chains as the system becomes compute-bound.

2. Context and Motivation

The Core Problem: We Don't Know How to Evaluate Speculative Decoding Properly

Speculative Decoding (SD) has rapidly become one of the most important techniques for accelerating LLM inference. The core idea is elegant: rather than generating tokens one at a time—a process that leaves GPU compute units severely underutilized because memory reads from High-Bandwidth Memory (HBM) dominate latency—SD uses a lightweight "draft model" to predict multiple future tokens, then verifies them all in a single forward pass through the large "target model." If the draft model is accurate, the system generates multiple tokens for roughly the memory-access cost of one, yielding substantial speedups with no loss in output quality (since rejection sampling ensures exact distribution matching with the target model).

However, the paper identifies a critical gap: the evaluation of SD algorithms is fragmented, inconsistent, and often unrepresentative of real-world deployment conditions. This is not merely a methodological nuisance—it means that researchers and practitioners cannot reliably compare methods, cannot predict how algorithms will behave in production, and may be making design decisions based on misleading benchmarks.

Why Evaluation Quality Matters for Speculative Decoding Specifically

The reason evaluation is uniquely challenging for SD—and why getting it wrong has outsized consequences—stems from a fundamental property the paper emphasizes throughout: SD performance is inherently data-dependent. Unlike deterministic system optimizations (e.g., kernel fusion, quantization, KV-cache management) that improve inference uniformly regardless of the input text, SD's effectiveness depends critically on the predictability of the text being generated.

This data-dependence operates at multiple levels:

Domain sensitivity. Draft acceptance rates—the probability that the draft model's predicted token matches what the target model would have generated—vary dramatically across semantic domains. A drafter that achieves high acceptance rates on structured code (where syntax and conventions are highly predictable) may perform poorly on creative writing (where the next token is inherently uncertain) or on multilingual text (where token distributions differ from the predominantly English training data). If a benchmark over-represents easy, predictable domains, it will overstate the SD method's real-world effectiveness.

Entropy-dependence and batch-size interaction. The paper explains that SD's speedup mechanism is rooted in the memory-bound vs. compute-bound distinction. At low batch sizes (e.g., single-user inference), the dominant cost is reading model weights from HBM to on-chip caches—the actual computation is cheap by comparison. In this regime, verifying multiple drafted tokens costs only marginally more than verifying one, so even moderate acceptance rates yield large speedups. However, as batch size increases (as in multi-user production serving), the system shifts toward being compute-bound: the arithmetic cost of processing many tokens across many requests begins to dominate. In this regime, the cost of verifying speculative tokens—especially if many are rejected—can outweigh the gains, potentially causing slowdowns relative to standard autoregressive decoding.

This means that a benchmark reporting speedups at batch size 1 (a common practice the paper critiques) tells you almost nothing about performance in a production serving environment with hundreds of concurrent users. The optimal draft length—how many tokens the draft model should predict ahead—is not a fixed constant but varies with the batch size and the domain entropy.

Input sequence length effects. The paper notes an industry trend toward long-context applications (coding assistants analyzing entire repositories, document Q&A, long-form dialogue), yet existing benchmarks predominantly feature short Input Sequence Lengths (ISLs). The behavior of drafters at long contexts is poorly understood: draft models trained on short sequences may degrade in accuracy when the context extends far beyond their training distribution, and the computational dynamics of verification change as the context length grows.

Where Existing Benchmarks Fall Short

The paper systematically catalogs the deficiencies of current SD evaluation practices, using SpecBench (Xia et al., 2024)—the most significant prior attempt at standardized SD evaluation—as a primary point of comparison.

Insufficient sample volume and intra-category diversity. The most commonly used evaluation datasets for SD are alarmingly small. The paper uses EAGLE3, "arguably the most widely adopted speculation method," as a case study: it is validated on MT-Bench (10 samples per category, minimal intra-category variance), HumanEval for coding (simple Python-only tasks), GSM8K for math (grade-school level only), and a handful of other domain-specific subsets. This creates a situation where:

• Statistical noise dominates category-level results. With only 10 samples in a category like Coding or Reasoning, a single lucky or unlucky prediction can swing the measured acceptance rate by several percentage points. The paper shows (Figure 5) that on SpecBench's 10-sample categories, a lightweight EAGLE3 drafter appears comparable to the more robust Vanilla SD method, but when evaluated on SPEED-Bench's larger, more diverse splits, the expected advantage of the external drafter at long draft lengths becomes clear.

• The benchmark cannot capture the variance in drafter performance within a domain. A coding benchmark with 10 Python functions tells you nothing about performance on Java, C++, Go, or Rust—languages that real code assistants encounter daily.

Problematic data source quality and structural monoculture. SpecBench inherits most of its categories directly from MT-Bench, which the paper argues lacks the complexity of modern LLM benchmarks. The authors found MT-Bench's Coding and Humanities categories to be "quite simple and not representative of the complexity found in modern LLM benchmarks." More critically, the paper identifies a structural monoculture problem: SpecBench's multilingual subset is sourced entirely from WMT14 DE-EN, consisting exclusively of German-to-English translation prompts ("Translate German to English:"). This constitutes approximately 15% of SpecBench's total dataset yet captures only a single task type in a single language pair. Real-world multilingual usage spans dozens of languages and task types—question answering, summarization, code generation in non-English contexts, creative writing—all of which exert different pressures on draft model accuracy.

The paper quantifies this diversity gap in Figure 2, showing that SPEED-Bench achieves a 40% reduction in average pairwise semantic similarity between samples compared to SpecBench, with the largest improvement (83%) in the Multilingual category. This is not just a nice-to-have; the paper demonstrates in Section 8.3 that on SpecBench, the performance gap between EAGLE3 and Vanilla SD on multilingual text is moderate, but on SPEED-Bench's genuinely diverse multilingual split, the gap is substantial. A benchmark that artificially narrows the measured performance gap between methods is actively misleading.

Short input sequence lengths. Table 3 (Appendix A) reveals that the median ISL in SpecBench is approximately 57 tokens per sample, compared to 141 tokens in SPEED-Bench's Qualitative Split. This matters because long-context applications are increasingly central to LLM deployment, and the paper shows (Section 8.5, Figure 8) that drafter accuracy degrades rapidly when inference ISL exceeds the training ISL—a phenomenon invisible on short-context benchmarks.

No throughput-oriented evaluation. The paper emphasizes that prior work "predominantly focus on BS = 1 in non-optimized environments." This is a critical omission because, as explained above, the cost-benefit calculus of SD fundamentally changes with batch size. A method that reports 2× speedup at batch size 1 may provide negligible benefit or even cause slowdown at batch size 128. Without a benchmark that systematically evaluates across concurrencies, practitioners cannot make informed decisions about whether and how to deploy SD in production.

Reliance on high-level libraries that mask production realities. Many papers evaluate SD using HuggingFace's native PyTorch implementation, which lacks the extensive optimizations found in production engines (CUDA Graphs, continuous batching, kernel fusion, efficient KV-cache management). The paper notes that different engines handle the draft-verification loop differently: TensorRT-LLM supports a unified CUDA Graph that captures the entire cycle in a single launch, while vLLM's multi-engine design incurs host communication overhead but offers greater flexibility for dynamic drafting strategies. Results from a HuggingFace implementation do not translate directly to either production engine, making cross-method comparisons based on such implementations unreliable.

The synthetic data trap. A common practice in inference benchmarking—one the paper forcefully argues against—is using random token inputs to simulate prompt load. The logic seems reasonable: if you need high-volume throughput measurements, generating random tokens is convenient and avoids data dependencies. But for SD, this practice is "fundamentally flawed" (Section 6). The paper identifies two failure modes:

1. Trivial Response: The model identifies the input as noise and defaults to predictable, generic acknowledgments (e.g., "It looks like you've pasted a very long block of mixed-language text... I'm happy to help, but I need a bit more guidance"). This pattern is trivially predictable by the draft model, artificially inflating acceptance rates. The paper provides a concrete example: GPT-OSS 120B with EAGLE3 achieves an average acceptance length of 3.44 on such inputs—far higher than on any real domain.

2. Topic Latching: Random sampling occasionally produces tokens that the model interprets as coherent signals, causing it to hallucinate elaborate but arbitrary responses on random topics. These responses have idiosyncratic token distributions that produce abnormally low acceptance rates—also unrepresentative of real usage.

Section 8.4 demonstrates empirically that synthetic benchmarking with SD enabled overestimates real-world throughput by an average of 23% compared to SPEED-Bench's real-data measurements (Figure 6). Even more insidiously, the paper reveals that random tokens cause problems even for baseline autoregressive decoding on Mixture-of-Experts (MoE) architectures: the expert routing network, trained on natural text distributions, exhibits "router collapse" on out-of-distribution random inputs, favoring a subset of experts and leaving 20–30% of experts never activated in some layers (Appendix F, Figures 11–12). This violates the load-balancing assumptions of the inference engine, producing inaccurate step latency measurements regardless of SD.

Missing metadata for fine-grained analysis. The paper notes that SpecBench lacks subcategory classifications, difficulty labels, and multi-turn indicators (beyond a binary two-turn flag). This makes it impossible to conduct the kind of targeted analysis the paper demonstrates—for instance, identifying that vocabulary pruning disproportionately affects the Multilingual category (Section 8.2) or that drafters trained on short sequences degrade at long contexts (Section 8.5).

How SPEED-Bench Positions Itself

The paper frames SPEED-Bench not as a replacement for research-oriented toolkits like SpecBench—which it explicitly describes as excelling at evaluating methods using native PyTorch/HuggingFace—but as a complementary system focused on production viability. The positioning is stated clearly: "SPEED-Bench focuses on the viability of these methods in deployment."

This distinction drives the benchmark's design:

• The Qualitative Split uses a principled selection algorithm (Greedy Selection with Local Swap Refinement, Algorithm 1) operating on embedding-space distances to construct a compact (880 samples) but maximally diverse dataset across 11 categories. The goal is not exhaustive coverage but efficient, high-signal measurement of drafter accuracy across fine-grained domains. The split is designed for computing acceptance rates and acceptance lengths—metrics that depend on semantic domain but do not require massive data volume to measure reliably.

• The Throughput Split takes the opposite approach: it aggregates samples into broad entropy categories (Low, Mixed, High) across fixed ISL buckets (1k, 2k, 8k, 16k, 32k) with 512 samples per category per bucket (1,536 total per bucket, 7,680 total). This volume is necessary to construct stable throughput-latency Pareto curves—measuring system speedups requires controlling for ISL and batch size effects, and small datasets produce noisy latency measurements.

• The Measurement Framework operates as a thin client, handling all tokenization and prompt formatting externally before transmitting pre-tokenized inputs to the inference engine. This bypasses the internal preprocessing logic of different engines (which may inconsistently append BOS tokens, apply chat templates, or handle special tokens differently), ensuring that the draft and target models process identical token sequences regardless of the backend. The framework captures fine-grained metrics—acceptance rates per chunk, Time To First Token, step latency, total request latency—from streaming response objects, enabling the derivation of aggregate throughput metrics.

The paper also introduces a practical methodology for estimating domain-specific speedups without constructing exhaustive throughput benchmarks for every category. By noting that per-step latency ($t_{ar}$ for baseline decoding, $t_{sd}$ for SD) is primarily governed by system constraints (memory bandwidth, batch size, ISL) rather than the specific domain, while acceptance length (AL) is domain-dependent, practitioners can measure latency on the Throughput Split's realistic workloads, measure AL on their target domain's prompts, and analytically compute speedup as $S = (t_{ar} \cdot \text{AL}) / t_{sd}$ (Appendix G). This decouples system benchmarking from domain-specific accuracy measurement, making the benchmark more practical for real-world deployment decisions.

The Broader Significance

Beyond the immediate technical contributions, SPEED-Bench addresses a structural problem in the SD literature: the inability to reconcile conflicting results or make reliable cross-method comparisons. When EAGLE3 reports results on MT-Bench + HumanEval + GSM8K and another method reports on a different ad-hoc mix of datasets, there is no way to determine whether performance differences stem from algorithmic superiority or from differences in evaluation data. This fragmentation slows progress by making it difficult to identify which ideas genuinely advance the state of the art.

The paper's empirical demonstrations—that synthetic inputs overestimate throughput by 23%, that vocabulary pruning has dramatically domain-dependent effects (dropping AL by ~10% in Summarization and RAG but negligibly in Math and Coding), that optimal draft length shifts with batch size, and that SpecBench's limited diversity masks differences between methods—collectively make the case that the field needs a more rigorous evaluation standard. SPEED-Bench is proposed as that standard.

3. Technical Approach

3.1 Reader Orientation

SPEED-Bench is a benchmark suite and measurement framework—not an SD algorithm—designed to evaluate Speculative Decoding methods under conditions that reflect real-world production deployment. The system solves the evaluation problem: given an SD algorithm and a target model, how do you measure its speculation quality across diverse text domains and its throughput performance under realistic serving loads (large batches, long contexts, production engines) in a standardized, reproducible way? The "shape" of the solution is a two-part dataset (Qualitative Split for accuracy, Throughput Split for system speedups) plus a unified measurement client that sits between the data and the inference engine, ensuring that all backends process identical token sequences and that the reported metrics are comparable across methods.

3.2 Big-Picture Architecture (Diagram in Words)

Figure 1 provides the visual overview. In prose, SPEED-Bench has four major components:

1. Data Sourcing and Curation Pipeline — Aggregates raw prompts from 24 publicly available datasets across 18 sources. For the Qualitative Split, applies a greedy selection algorithm with local swap refinement to choose a compact (880 samples, 80 per category) subset that maximizes semantic diversity within each of 11 categories. For the Throughput Split, aggregates and processes samples into three entropy categories (Low, Mixed, High) and pads/truncates them into fixed Input Sequence Length buckets (1k, 2k, 8k, 16k, 32k tokens), with 512 samples per category per bucket.

2. Pre-tokenized Dataset Storage — The curated prompts are tokenized externally (using the o200k base tokenizer for ISL calculations) and stored as tokenized sequences with rich metadata (category, subcategory, difficulty, multi-turn indicator, ISL bucket). This pre-tokenization is the critical design choice that decouples data from engine implementation.

3. Measurement Framework — A thin client written with Python's asyncio that reads the pre-tokenized prompts, dispatches them to the target inference engine (vLLM, TensorRT-LLM, SGLang, or SpecBench/HuggingFace), and captures streaming response objects. It records per-chunk token counts (to compute acceptance rates), timestamps (for Time To First Token, step latency, total request latency), and derives aggregate metrics (throughput in output tokens per second, user TPS as a proxy for per-request latency).

4. Inference Engine Backend — The production-grade engine (vLLM, TensorRT-LLM, SGLang) or research library (SpecBench) that performs actual inference. The measurement framework transmits pre-tokenized inputs, so the engine's internal chat templating and preprocessing are bypassed—ensuring the draft and target models process identical token sequences regardless of engine.

Information flows: raw data sources → curation/selection algorithm → pre-tokenized dataset → measurement framework dispatches requests → inference engine generates tokens → streaming response objects → metrics extraction (acceptance rates, latencies, throughput).

3.3 Roadmap for the Deep Dive

• First, the Qualitative Split's data composition and the greedy selection algorithm (Algorithm 1), because semantic diversity is the core challenge this split addresses, and the algorithm's formulation (minimizing pairwise embedding similarity) is the intellectual contribution.
• Second, the Throughput Split's construction, covering ISL bucketing, entropy categorization, and why random synthetic inputs are fundamentally broken for SD benchmarking.
• Third, the Measurement Framework—how it achieves engine-agnostic evaluation, what metrics it captures, and the technical design for bypassing engine preprocessing.
• Fourth, the speedup estimation methodology (Appendix G equation) that decouples system latency from domain-specific acceptance length, since this is how practitioners use the Throughput Split in practice.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a benchmarking infrastructure paper whose core idea is that SD evaluation requires two distinct data splits serving different purposes—qualitative analysis of drafter accuracy across diverse domains, and throughput measurement under controlled system conditions—plus a framework that ensures fair comparison across production engines by externalizing text processing.

---

Qualitative Split: Maximizing Semantic Diversity with Limited Samples

The Qualitative Split is designed to answer a specific question: how accurate is this drafter across different types of text? Accuracy here means acceptance rates (ARs) and acceptance lengths (ALs)—the probability that draft tokens match the target model's distribution, and the expected number of tokens generated per verification step. These metrics depend critically on the domain and entropy of the prompt. Generating code completions in Python is highly predictable (structured syntax, limited vocabulary at each position); generating creative fiction is not.

The key design tension is that exhaustive evaluation across dozens of data sources would be computationally prohibitive—each configuration (model, drafter, draft length) requires running the full inference pipeline. The Qualitative Split resolves this by constructing a small but maximally diverse subset: 80 samples per category across 11 categories (880 total), selected via a principled algorithm that explicitly minimizes semantic redundancy.

Data composition and sources. The 11 categories are: Coding, Humanities, Math, Multilingual, QA, RAG, Reasoning, Roleplay, STEM, Summarization, and Writing. These are inspired by SpecBench's categories but with important refinements: the Math and Math Reasoning categories are consolidated into Math, Extraction and RAG into a single RAG category, and the Translation category is replaced with Multilingual to capture a broader set of languages (23 distinct languages) and task types (not just translation but QA, summarization, and other multilingual queries).

The data is sourced from 18 publicly available datasets, compared to 5 in SpecBench (Table 2). The guiding principle is "semantic heterogeneity"—within each category, the prompts should span multiple task types, difficulty levels, and structural formats. For example:

• Coding draws from LiveCodeBench Lite (competitive programming tasks across Python, Java, C++, Go, JavaScript, Rust), Code Contests (algorithmic problem-solving), and HumanEvalPack (code completion across multiple languages). This yields prompts spanning 7 programming languages and 3 task formats.

• Multilingual draws from MMATH (multilingual math reasoning), OPUS-100 (machine translation across many language pairs), and MCIF (multilingual QA, translation, and summarization from scientific talks). This captures 23 languages (DE, ZH, IT, MG, FR, JA, PT, AR, MK, DA, NL, KO, ES, NN, TH, VI, BN, GU, CS, GD, EU, RU, EN) across translation, QA, and summarization tasks.

• Roleplay draws from RoleBench (multi-turn character roleplay) and CoSER (book character simulation, filtered to public-domain works). The multi-turn samples span 1–5 turns, with system prompts randomly sampled from 8 templates instructing the model to embody the character.

Each sample includes metadata beyond what SpecBench provides: a subcategory classification (e.g., within STEM: Physics, CS/AI, Biology/Medicine, Chemistry, Engineering), a multi-turn binary indicator (approximately 20% of samples feature 2–5 turns), and a difficulty field for Coding, Humanities, Math, and STEM categories (focused on hard problems, approximately 80%, while retaining easier tasks for coverage). The prompts are verified to generate a mean of approximately 650 output tokens when processed by GPT-4, ensuring there is sufficient response length for SD metrics to be meaningful.

The selection algorithm: Greedy Selection with Local Swap Refinement (Algorithm 1). With 18 data sources aggregated into 11 categories, there are far more candidate prompts than the target of 80 per category. Simply taking 80 random samples would produce a subset that might over-represent certain sub-domains or task types—for instance, if 60% of the candidate coding prompts are Python, a random sample would likely over-represent Python. The selection algorithm's goal is to choose a subset $S$ of size $k = 80$ from $N$ candidates that maximizes semantic coverage.

The algorithm operates on dense vector embeddings. Each candidate prompt $t_i$ is mapped to a vector $x_i \in \mathbb{R}^d$ using OpenAI's text-embedding-3-large model. These vectors are row-normalized such that $\lVert x_i \rVert = 1$, making cosine similarity computable as a simple dot product $x_i^\top x_j$. The objective is to minimize the total pairwise similarity within the selected subset:

$\mathcal{L}(S) = \sum_{i \in S} \sum_{j \in S, j \neq i} x_i^\top x_j$

where $S \subset \{1, \ldots, N\}$ is the subset of indices to select, $|S| = k$ is the target size (80 per category), $x_i \in \mathbb{R}^d$ is the normalized embedding of the $i$-th candidate prompt, and $x_i^\top x_j$ is the cosine similarity between prompts $i$ and $j$.

What it computes: The sum of all pairwise cosine similarities among selected prompts. A low value indicates that the selected prompts are far apart in embedding space—semantically diverse. A high value indicates clustering—semantically redundant prompts. Minimizing this objective directly penalizes selecting prompts that are near each other, forcing the selection to span the embedding space.

Why this form: Embedding-space distance (cosine similarity) is a well-established proxy for semantic similarity in modern language models—prompts that are semantically similar (e.g., two translation tasks, two Python function completions) map to nearby vectors, while prompts from different domains map to vectors with lower similarity. Using cosine similarity rather than Euclidean distance is important because normalized embeddings live on a unit hypersphere, and the dot product captures angular separation (which correlates with semantic distance better than absolute L2 distance for high-dimensional text embeddings).

Algorithm mechanics. Finding the exact subset that minimizes $\mathcal{L}(S)$ is NP-hard (it is a combinatorial optimization over all $\binom{N}{k}$ possible subsets). The paper uses a two-phase heuristic:

Phase 1: Greedy construction. Initialize $S$ with a single random index $i_{\text{rand}}$. Maintain a running sum vector $m \in \mathbb{R}^N$ where $m_j = \sum_{i \in S} x_i^\top x_j$—the total similarity of candidate $j$ to all currently selected prompts. At each step, append the candidate with the minimum $m_j$: $i^* = \arg\min_{j \notin S} \; m_j; \quad S \leftarrow S \cup \{i^*\}; \quad m \leftarrow m + Xx_{i^*}$ where $X \in \mathbb{R}^{N \times d}$ is the matrix of all candidate embeddings. This greedily minimizes the sum of similarities to the current set, which is a standard approximation for maximum-diversity subset selection. Repeat until $|S| = k$.

Phase 2: Local swap refinement. The greedy construction can get stuck in local minima—early selections might be suboptimal in hindsight. The swap phase iteratively considers swapping one element $i_{\text{out}} \in S$ with one element $i_{\text{in}} \notin S$. For each candidate swap, compute the change in the objective $\Delta$: $\Delta = \sum_{j \in S \setminus \{i_{\text{out}}\}} \left(x_{i_{\text{in}}}^\top x_j - x_{i_{\text{out}}}^\top x_j\right)$

If $\Delta < 0$ (the swap strictly decreases the total similarity), accept the swap: $S \leftarrow (S \setminus \{i_{\text{out}}\}) \cup \{i_{\text{in}}\}$. Repeat until no swap reduces the objective, or until a maximum iteration count.

Why this two-phase approach? The greedy phase provides a reasonable initialization quickly (time complexity roughly $O(kN)$ dot products). The swap phase escapes local minima by considering pairwise replacements, which is more computationally intensive (examining $k(N-k)$ possible swaps per full pass) but needed only for refinement, not for the bulk of the selection. The authors explored a quadratic programming (QP) convex relaxation as an alternative (Appendix C), which minimizes $w^\top G w$ subject to $\sum w_i = k$, $w \in [0,1]^N$, where $G = XX^\top$ is the Gram matrix. The QP approach yields similar diversity scores but is less scalable—the greedy + swap method is faster and simpler to implement.

Empirical validation. Figure 2 compares the average pairwise similarity of SPEED-Bench's selected subsets against SpecBench and against random selection from the same data sources. SPEED-Bench achieves a 40% reduction in average pairwise similarity compared to SpecBench overall, with the largest improvement in Multilingual (83% reduction). The fact that random selection from SPEED-Bench's data sources also outperforms SpecBench on most categories demonstrates that data source quality matters independently of the selection algorithm—SpecBench's reliance on MT-Bench produces intrinsically less diverse candidate pools.

Appendix D provides qualitative confirmation via pairwise similarity heatmaps (Figures 9–10). SpecBench's Multilingual/Translation heatmap shows dense blocks of near-identical prompts (all German-to-English translations with identical structure), while SPEED-Bench's heatmap shows a diffuse pattern with few dark regions. This visual difference directly corresponds to the quantitative gap in Figure 2.

---

Throughput Split: Controlled System-Level Evaluation Across Concurrencies and Context Lengths

The Throughput Split addresses a different evaluation question: how fast is the system when serving many users simultaneously with long prompts? Unlike acceptance rates (which depend primarily on the semantic domain of the prompt), system throughput depends primarily on hardware constraints (memory bandwidth, compute throughput), serving parameters (batch size, ISL), and engine optimizations (CUDA Graphs, continuous batching). The split is designed to construct stable throughput-latency Pareto curves—plots of total output tokens per second versus per-user tokens per second, where each point corresponds to a different batch size.

Why fixed ISL buckets are necessary. In real serving environments, requests arrive with varying prompt lengths. However, for controlled benchmarking, the paper argues that isolating the effect of ISL is essential. If you benchark with a mixed distribution of ISLs, you cannot determine whether throughput differences stem from the ISL or from the SD configuration. The Throughput Split uses exact ISL buckets at powers-of-two-ish boundaries: 1k, 2k, 8k, 16k, and 32k tokens. ISLs are calculated using the o200k base tokenizer (OpenAI's tokenizer) to ensure consistency.

To achieve fixed ISLs, the paper does not simply take chunks of text. The approach respects semantic integrity: prompts are truncated to the target length where possible (if the raw prompt is longer than the target), or padded with a neutral suffix ("please answer now") if the raw prompt is shorter. This preserves the prompt's semantic content while ensuring deterministic prefill load. Unlike random truncation mid-sentence, padding with a neutral suffix avoids creating semantically broken inputs that might trigger unusual model behavior.

Entropy categorization. Samples within each ISL bucket are further classified into three broad categories based on the expected entropy of the response:

• Low Entropy — Tasks with highly structured, predictable outputs. Examples: code completion from Long Code Arena and RepoBench (where syntax and API usage are constrained), text sorting from AdaLEval (where the output format is deterministic). These tasks should yield high acceptance rates.

• Mixed Entropy — Tasks with moderate unpredictability. Examples: STEM problems from Humanity's Last Exam (where reasoning is structured but the exact wording varies), StackOverflow answer evaluation from AdaLEval (where the model provides structured analysis but with natural language variation).

• High Entropy — Tasks with inherently unpredictable outputs. Examples: dialogue continuation from BAMBOO, book continuation from Project Gutenberg, creative writing from WritingBench. These tasks should yield lower acceptance rates.

This categorization follows the taxonomy proposed by Li et al. (2025a). It matters because the entropy category interacts with batch size: at large batch sizes where the system is compute-bound, verifying speculative tokens for high-entropy domains (where many are rejected) may be counterproductive, while low-entropy domains may still benefit.

Data volume. Each of the 5 ISL buckets contains 512 samples per entropy category, totaling 1,536 samples per bucket and 7,680 across the entire Throughput Split. This volume is necessary for stable throughput measurements: throughput is measured by running many concurrent requests and measuring aggregate token generation rate. With too few samples, individual outlier requests (an unusually long or short generation) distort the measurement. The 512-sample volume per condition provides sufficient statistical stability for constructing smooth Pareto curves. The prompts are verified to generate a mean of approximately 2.4k output tokens on GPT-4 with 16k ISLs.

Why not fine-grained categories in the Throughput Split? Unlike the Qualitative Split (where fine-grained categories are essential for measuring domain-dependent acceptance rates), the Throughput Split uses only three broad entropy categories. The paper explains this is because "replicating such granularity at this scale with high quality is both impractical and perhaps redundant for speedup estimations." System latency in the verification step depends primarily on memory bandwidth and batch size, not on whether the prompt is about physics vs. chemistry. The entropy categorization is sufficient to capture the broad interaction between domain predictability and system throughput.

The pitfalls of synthetic token benchmarking. A commonly used shortcut in LLM throughput benchmarking—one the paper forcefully critiques—is generating random token sequences to simulate prompt load. The logic is that if you need 512 prompts of exactly 8k tokens, you can generate random token IDs, decode them into (gibberish) text, and use that as input. For standard autoregressive decoding on dense models, this is arguably fine because the model's per-step latency is independent of the semantic content of the prompt.

For SD, this practice is "fundamentally flawed" (Section 6) because draft acceptance rates depend on the predictability of the output, which depends on the semantic content of the input. The paper identifies two specific failure modes:

1. Trivial Response — When the input is random noise, the model often identifies it as such and produces a generic acknowledgment ("It looks like you've pasted a very long block of mixed-language text that doesn't form a clear question or request..."). This response pattern is highly predictable—the draft model can anticipate the boilerplate sequence with near-certainty. The paper provides a concrete example (Appendix E): GPT-OSS 120B with EAGLE3 achieves an average AL of 3.44 on random tokens, far higher than on any real domain. This artificially inflates apparent SD speedups.

2. Topic Latching — Occasionally, the random token sampling produces a subsequence that the model interprets as a coherent signal (a technical term, a common name). The model then "latches" onto this signal and generates an elaborate, arbitrary response on that topic. Because the topic is effectively random, the token distribution is idiosyncratic, producing abnormally low acceptance rates. The paper provides an example where random tokens triggered a multi-paragraph response about building a Unity 2D platformer game, with an AL of only 1.877—lower than real-domain performance.

Both failure modes produce unreliable measurements because they do not represent any real usage pattern. The paper demonstrates this quantitatively in Section 8.4 (Figure 6): when measuring GPT-OSS 120B with EAGLE3 drafting on the Throughput Split's 8k ISL data, synthetic benchmarking overestimates throughput by an average of 23% with SD enabled compared to real SPEED-Bench data.

Expert imbalance in MoE architectures. Appendix F reveals an even more subtle problem with random token inputs that affects baseline autoregressive decoding (without SD) for Mixture-of-Experts models. MoE architectures use a gating network (router) to select a sparse subset of experts for each token. Because this router is trained on natural text distributions, random token inputs—which are statistically out-of-distribution—cause "router collapse": the router disproportionately favors a subset of experts. Figure 11 shows that on GPT-OSS 120B (which has 128 experts), processing 8k random tokens at batch size 32 causes significant activation imbalance, with certain experts receiving up to 4–5× more tokens than others. Figure 12 shows that 20–30% of experts in some layers are never activated at all when processing random tokens, despite the high token volume (32 requests × 8000 tokens = 256,000 tokens). This violates the load-balancing assumptions of the inference engine's expert-parallel scheduling, producing inaccurate step latency measurements regardless of SD. The Throughput Split's real semantic data avoids this problem by providing in-distribution inputs.

---

Measurement Framework: Engine-Agnostic Evaluation with Externalized Text Processing

The measurement framework is a Python client that sits between the pre-tokenized dataset and the inference engine, handling all text processing externally so that different backends process identical sequences. Its design addresses a practical problem: different inference engines handle chat templating, special token insertion, and prompt formatting differently.

Why external tokenization matters. Consider a chat prompt with system and user messages. vLLM might prepend a BOS token and format the conversation with <|im_start|> tags. TensorRT-LLM might handle the same conversation with a different internal representation. If you send raw text to each engine and let them apply their own preprocessing, the actual token sequences seen by the draft and target models differ—sometimes subtly (a different BOS token), sometimes substantially (different chat template structures). This makes it impossible to attribute performance differences to the SD algorithm vs. the preprocessing.

SPEED-Bench solves this by pre-tokenizing all prompts and transmitting token IDs directly to the inference engine. The engine's internal tokenizer, chat template, and special token handling are completely bypassed. The draft and target models across all backends process identical token sequences, isolating the performance impact of the speculation algorithm and the engine's system optimizations.

Concurrency model. The framework uses Python's asyncio event loop to dispatch requests concurrently, mimicking high-throughput serving scenarios. Requests are sent to the inference engine's API endpoint (OpenAI-compatible or engine-specific), and the framework captures streaming response objects as they arrive. For each response chunk, it records:

• Token count — How many newly generated tokens appear in this chunk. In SD, a single verification step may produce multiple accepted tokens; the framework infers acceptance by counting tokens per chunk. A chunk containing multiple tokens indicates a successful speculation step.

• Timestamps — Recorded upon receipt of every streaming object, enabling computation of Time To First Token (TTFT), step latency (time between consecutive chunks), and total request latency (end-to-end wall-clock time).

From these per-request measurements, the framework derives aggregate metrics:

Output TPS (Throughput): Total tokens generated across all concurrent requests per second. This is the primary system-level efficiency metric.

User TPS: Tokens generated per second for a single request, computed as total output tokens divided by total request latency. This serves as a proxy for per-user latency—higher User TPS means faster individual responses.

Acceptance Rate (AR): Conditional acceptance rate: the probability that draft token $x_i$ is accepted given that the draft prefix $x_{<i}$ was accepted. Computed by inspecting the number of accepted tokens per verification step across all requests.

Acceptance Length (AL): The expected number of generated tokens per verification step $L_t$, including the "free" verification token (the target model always generates at least one token—the first token after the draft prefix). For a draft length $\gamma$ and conditional acceptance rates $\text{AR}_i$:

$\text{AL} = \mathbb{E}[L_t] = 1 + \sum_{i=1}^{\gamma} \prod_{j=1}^{i} \text{AR}_j$

where $\gamma$ is the draft length (how many tokens the draft model speculates ahead), $\text{AR}_j$ is the conditional acceptance rate at position $j$ (the probability that draft token $x_j$ matches the target distribution, given that all previous draft tokens $x_1, \ldots, x_{j-1}$ were accepted), and the sum runs over all draft positions.

What it computes: The expected number of tokens produced per verification forward pass. The leading "1" accounts for the free token—the target model always produces at least one token (the verification token that follows the accepted draft prefix). The sum iterates over the draft: for each position $i$, the product $\prod_{j=1}^{i} \text{AR}_j$ is the probability that all $i$ draft tokens are accepted. Summing these products gives the expected number of accepted draft tokens. The total AL is 1 (the guaranteed verification token) plus the expected number of accepted draft tokens.

Why this form: This formulation captures the sequential nature of SD verification. Tokens are accepted left-to-right: if the first draft token is rejected, no subsequent draft tokens are even considered. The conditional acceptance rates capture this dependency—$\text{AR}_2$ only matters if $\text{AR}_1$ causes acceptance. A naive alternative that sums independent acceptance probabilities would overestimate AL because it would count tokens that could never be reached (if an earlier token were rejected). The product-of-conditionals correctly accounts for the forward-masking property of rejection sampling.

Integration with production engines. The framework provides native integration with vLLM, TensorRT-LLM, and SGLang. Each backend receives pre-tokenized prompts through its API and returns streaming responses. The framework also integrates with SpecBench for research-oriented evaluation, providing a bridge between the production-focused SPEED-Bench and the PyTorch/HuggingFace-focused SpecBench. The supplementary material includes an example for SpecBench's Medusa implementation, with instructions for extending to other models.

Limitation at extreme batch sizes. The paper acknowledges a current limitation: at extremely high throughputs (batch size > 256), Python's Global Interpreter Lock (GIL) can introduce client-side overhead in the measurement framework itself. The asyncio event loop is single-threaded, and the GIL limits the rate at which response objects can be processed. The authors state they are "actively extending the framework to leverage more advanced parallelism" for these regimes. This is relevant for edge cases—most practical batch sizes (1–256) are within the framework's reliable operating range.

---

Domain-Specific Speedup Estimation (Appendix G)

A practical capability of the Throughput Split is that it enables practitioners to estimate SD speedups for fine-grained domains without constructing exhaustive throughput benchmarks for every specific category. This works by decoupling the two factors that determine speedup: system-dependent per-step latency and domain-dependent acceptance length.

The speedup $S$ is defined as the ratio of effective token generation speed with SD to the baseline autoregressive speed:

$S = \frac{\text{Tokens}_{\text{sd}} / \text{Time}_{\text{sd}}}{\text{Tokens}_{\text{ar}} / \text{Time}_{\text{ar}}} = \frac{\text{AL} / t_{\text{sd}}}{1 / t_{\text{ar}}} = \frac{t_{\text{ar}} \cdot \text{AL}}{t_{\text{sd}}}$

where $t_{\text{ar}}$ is the average time per decoding step in the baseline autoregressive system (generating exactly 1 token per step), $t_{\text{sd}}$ is the average time per decoding step in SD (generating on average AL tokens per step), and AL is the acceptance length for the target domain at the chosen draft length.

What it computes: The multiplicative speedup: how many times more tokens per second the SD system produces compared to baseline. The numerator $t_{\text{ar}} \cdot \text{AL}$ is the effective token generation rate under SD normalized to the baseline step time. Dividing by $t_{\text{sd}}$ accounts for the fact that each SD verification step takes a different amount of time than a baseline step (usually longer, due to the cost of processing draft tokens).

Why this form: The decoupling is the key insight. $t_{\text{ar}}$ and $t_{\text{sd}}$ depend primarily on system factors (memory bandwidth, batch size, ISL, engine optimizations, draft model overhead) and can be measured once on the Throughput Split's realistic workloads at the target serving configuration (e.g., BS=32, ISL=8k). AL depends on the semantic domain and can be measured on a small set of representative prompts from the target domain using the Qualitative Split's evaluation pipeline. The product formula cleanly separates these concerns.

Why this proxy is accurate. The paper argues that per-step latency is "primarily governed by system constraints and serving parameters, not by the specific domain." This is because a verification step involves a fixed computational graph—loading model weights, computing attention over the full context, and producing output logits—whose cost scales with the batch size and context length, not with what the prompt is about. There may be small second-order effects (cache behavior, padding patterns) but the first-order approximation is domain-independent. Given reliable measurements of $t_{\text{ar}}$ and $t_{\text{sd}}$ on realistic data (not random tokens, which the paper showed are unreliable), and an AL measurement on the target domain, the formula provides an accurate speedup estimate without requiring a full throughput benchmark for that domain.

The practical protocol (Appendix G.1) is: (1) use the Throughput Split at the target ISL and batch size to measure $t_{\text{ar}}$ and $t_{\text{sd}}$; (2) use the Qualitative Split (or custom prompts) to measure AL on the target domain at the target draft length; (3) plug into the formula. This makes SPEED-Bench practical for organizations evaluating SD across many deployment scenarios without exhaustive re-benchmarking.

4. Key Insights and Innovations

Innovation 1: The Problem of SD Evaluation Is a Diversity Problem, Not a Scale Problem

The paper's most fundamental conceptual move is reframing what makes SD evaluation difficult. The dominant assumption in the field—visible in SpecBench and EAGLE3's validation methodology—was that evaluation quality is primarily a matter of task coverage breadth (having many categories) and that small samples per category are acceptable if the categories are well-chosen. The paper diagnoses this as wrong: the problem is not that we lack enough categories, but that the samples within categories are insufficiently diverse, creating statistical noise and masking systematic failures that appear in production.

This is a diagnostic insight, not an algorithm. It matters because it explains a pattern of confusing results in the SD literature—methods that appear comparable on narrow benchmarks diverge sharply on broader, more diverse data (Figure 5)—and because it provides a principled criterion for what constitutes a good SD benchmark: not the number of categories, but the semantic spread within each category. Prior work (SpecBench, the ad-hoc dataset mixes used in EAGLE3 and Vanilla SD papers) implicitly treated categories as homogeneous blocks. SPEED-Bench's core argument is that intra-category diversity—different languages in Multilingual, different programming languages in Coding, different task formats within Math—is the dominant factor determining whether a benchmark can distinguish methods.

The evidence is in Figure 2 and Figure 5. Figure 2 shows that SPEED-Bench's selection algorithm achieves a 40% reduction in average pairwise similarity compared to SpecBench, with the largest gains in categories where SpecBench's narrowness was most acute (83% in Multilingual, where SpecBench had only German-to-English translation). Figure 5 shows the consequence: on SpecBench's narrow categories, EAGLE3 and Vanilla SD appear comparable; on SPEED-Bench's diverse splits, Vanilla SD's advantage at long draft lengths emerges clearly. The cost of narrow evaluation is not just imprecision but qualitatively wrong conclusions about relative method performance.

This is a fundamental reframing, not an incremental improvement. It changes the design objective from "add more categories" to "maximize intra-category semantic spread," and it provides both the evidence that the former fails and a concrete algorithmic mechanism (the greedy selection + swap refinement of Algorithm 1) for achieving the latter. A benchmark designer reading this paper should come away understanding that 10 carefully diversified samples per category may provide more signal than 80 homogeneous ones—a non-obvious result with implications for how all inference benchmarks are constructed.

Innovation 2: Synthesizing the Privacy of Synthetic Inputs as a Systematic Measurement Bias, Not Just a Convenience

The critique of random-token benchmarking in Section 6 and Section 8.4 goes deeper than a methodological best practice—it identifies a structural failure mode that corrupts both SD and baseline autoregressive measurements in different ways, and in doing so exposes a hidden coupling between model architecture and evaluation data that the field had overlooked.

Prior work implicitly assumed that synthetic random inputs are a harmless convenience for throughput benchmarking—since per-step latency for dense models is independent of input semantics, random tokens seemed like a valid way to generate high-volume, fixed-length prompts without data dependencies. The paper systematically dismantles this assumption along two previously unrecognized dimensions:

For SD specifically, the coupling is direct: draft acceptance rates depend on output predictability, which depends on input semantics. Random inputs trigger two degenerate response patterns—Trivial Response (generic acknowledgment, artificially high AR) and Topic Latching (hallucinated topical response, artificially low AR)—neither of which represents any real deployment scenario. The 23% throughput overestimation in Figure 6 is not an accident of one configuration; it reflects a fundamental mismatch between the noise distribution and the text distribution the draft model was trained to predict.

For MoE architectures generally, the coupling is more subtle and more broadly applicable: the expert routing network, trained on natural text, exhibits "router collapse" on out-of-distribution inputs, leaving 20–30% of experts unactivated in some layers (Appendix F, Figures 11–12). This means that even baseline autoregressive decoding throughput measurements on random tokens are inaccurate for MoE models—a finding with implications far beyond SD evaluation. Any benchmark that uses synthetic inputs to measure MoE inference performance inherits this bias, regardless of whether SD is involved.

This is a diagnostic innovation whose significance lies in what it rules out as valid methodology. It establishes that synthetic throughput benchmarking is not merely suboptimal but actively misleading, and that realistic semantic data is a necessary precondition for any throughput measurement on modern LLM architectures (dense or MoE, with or without SD). This finding, if adopted by the community, would change how all inference benchmarks are constructed—not just those for SD.

Innovation 3: The Throughput-Qualitative Split as an Evaluation Architecture for Data-Dependent System Optimizations

The paper's most significant architectural contribution is the recognition that evaluating data-dependent system optimizations requires two fundamentally different types of benchmarks serving different purposes, and that conflating them leads to benchmarks that fail at both. This insight, while emergent from the SD domain, generalizes to any inference optimization whose effectiveness varies with input properties.

The Qualitative Split is optimized for statistical signal: with only 80 samples per category, it cannot provide stable throughput measurements (latency varies too much with individual sample characteristics), but it can provide high-fidelity acceptance rate measurements because those metrics average cleanly over moderate sample counts. The selection algorithm maximizes the information-per-sample ratio by minimizing redundancy, making the split compact enough for rapid evaluation while diverse enough to expose domain-dependent behaviors.

The Throughput Split is optimized for systematic control: fixed ISL buckets and controlled entropy categories enable the construction of stable throughput-latency Pareto curves that isolate the effects of context length, batch size, and domain entropy on system performance. The large sample counts (512 per condition) are necessary because throughput is a system-level measurement that aggregates over many concurrent requests, and statistical noise from small samples produces unstable curves.

The insight—which prior benchmarks like SpecBench did not operationalize—is that you cannot serve both purposes with a single dataset. A dataset large enough for stable throughput measurement would be too expensive for rapid qualitative evaluation of many draft-model configurations. A dataset diverse enough for qualitative analysis would be too small and uncontrollable for throughput Pareto curves. The two-split architecture is the solution: let each split be optimized for its specific measurement task, and provide the analytical framework (the domain-specific speedup estimation formula of Appendix G) to combine their results.

This is an incremental but practically significant innovation. It provides a template for evaluating any data-dependent inference optimization—not just SD, but also adaptive computation methods, dynamic pruning, or any acceleration technique whose cost-benefit ratio varies with input properties. The paper doesn't claim this generalization explicitly, but the architecture implies it: anytime you have an optimization whose effectiveness $\text{Eff}(x)$ depends on the input $x$, you need (1) a diverse qualitative split to measure $\text{Eff}(x)$ across the input space, and (2) a controlled throughput split to measure the system-level costs under realistic load.

Innovation 4: Establishing Over-Optimization Regimes in Two Under-Studied Dimensions of SD Deployment

The paper's empirical analysis surfaces two deployment phenomena that, prior to this work, were either undocumented or underestimated in the SD literature: vocabulary pruning's domain-dependent degradation and long-context drafter accuracy collapse. These are not methodological contributions per se, but they are novel empirical findings that change how practitioners should think about SD deployment tradeoffs—and they were only discoverable because SPEED-Bench's diversity and ISL coverage exposed them.

On vocabulary pruning (Section 8.2): EAGLE3 and similar drafters commonly apply vocabulary pruning (filtering to the top K most frequent tokens, typically 32k) to reduce the computational cost of the final projection layer. The paper quantifies what was previously an anecdotal concern: this optimization degrades acceptance lengths by domain-dependent margins, from negligible in Math and Coding (structured, predictable, English-heavy vocabularies) to ~10% in Summarization and RAG. By providing the first systematic measurement of this effect across domains, the paper transforms vocabulary pruning from a simple "on/off" optimization to a deployment decision with tradeoffs that depend on the expected query distribution. A service that handles primarily code generation might prune aggressively; a summarization service might need the full vocabulary. This is actionable practical knowledge that did not exist before SPEED-Bench's cross-domain analysis capability.

On long-context drafter degradation (Section 8.5): The paper documents that publicly available EAGLE3 checkpoints suffer substantial accuracy loss at inference ISLs beyond their training ISL, and that this degradation interacts with RoPE scaling configuration in subtle ways. Figure 8 quantifies the effect: acceptance lengths drop sharply when inference ISL exceeds the training ISL, but applying YaRN scaling at inference time recovers significant accuracy even for models trained on relatively short sequences (2k–4k). This has direct implications: it means practitioners should not assume drafters trained on short-context data will generalize to long-context serving without explicit RoPE configuration, and it provides a concrete mitigation (YaRN scaling) validated across ISL buckets. This finding also implies that current short-context SD benchmarks (most of the literature) are systematically overstating the real-world performance of drafters destined for long-context deployment—a transferability gap that SPEED-Bench's Throughput Split was designed to expose.

These are incremental but practically important findings. They don't change the fundamental SD algorithm, but they identify failure modes and mitigation strategies that directly affect deployment decisions. Their significance lies in making explicit what was previously implicit—that SD optimizations have sharp domain-dependent edge cases—and providing the benchmarking infrastructure to detect them.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The primary evaluation uses SPEED-Bench's own two splits. The Qualitative Split consists of 880 samples (80 per category across 11 categories), curated via greedy selection with local swap refinement from 18 publicly available data sources. The Throughput Split comprises 7,680 samples (512 per entropy category per ISL bucket, across 5 ISL buckets of 1k, 2k, 8k, 16k, and 32k tokens and 3 entropy levels: Low, Mixed, High), sourced from 8 publicly available datasets. For the SpecBench comparison in Section 8.3, the authors use SpecBench's own 480-sample dataset.

• Base model(s). The experiments target five large, modern open-source models: Llama 3.3 70B, GPT-OSS 120B, Qwen3 235B, Qwen3-Next (80B-A3B), and DeepSeek R1. These span a range of architectures including dense (Llama 3.3, GPT-OSS), Mixture-of-Experts (Qwen3 235B, Qwen3-Next, DeepSeek R1), and models with native Multi-Token Prediction heads (DeepSeek R1, Qwen3-Next). The authors justify this selection implicitly by covering the major model families practitioners are likely to encounter in production deployment, though no explicit rationale is stated beyond targeting "large, modern open models."

• Metrics. Three primary metrics are reported. Acceptance Rate (AR) is the conditional probability that draft token x_i is accepted given that the draft prefix x_{<i} was accepted, computed by inspecting the number of newly generated tokens per streaming response chunk. Acceptance Length (AL) is the expected number of generated tokens per verification step, defined as AL = 1 + Σ(i=1 to γ) Π(j=1 to i) AR_j, where γ is the draft length and the leading 1 accounts for the guaranteed verification token. Throughput is reported as Output TPS (total tokens generated across all concurrent requests per second), with User TPS (tokens generated per second for a single request) serving as a proxy for per-user latency. Timestamps are recorded upon receipt of each streaming object to compute Time To First Token, step latency, and total request latency.

• Baselines. The paper compares multiple SD methods against one another rather than using a single fixed baseline. Methods evaluated include: N-Gram speculation (a training-free drafting heuristic using n-gram pattern matching), Vanilla SD (external lightweight draft models: Llama 3.2 1B for Llama 3.3 70B, and Qwen3 0.6B for Qwen3 235B), EAGLE3 (post-trained drafting heads, using publicly available checkpoints except where custom training is mentioned), and Native MTP (Multi-Token Prediction heads integrated into the model architecture for Qwen3-Next and DeepSeek R1). The standard autoregressive decoding without speculation serves as the implicit baseline for speedup calculations.

• Generation budget / compute accounting. The paper uses the draft-verification paradigm's natural unit: one inference step either generates 1 token (autoregressive baseline) or verifies γ draft tokens (SD). The "generation budget" is not explicitly quantified as a compute budget (no FLOPs analysis); instead, speedup is measured as the ratio of effective token throughput with SD versus without. Throughput measurements control batch size, ISL, and draft length to enable fair comparisons. All experiments use a single NVIDIA B200 GPU, except DeepSeek and Qwen models (8 GPUs with Tensor Parallelism and Expert Parallelism as specified in Table 7) and GPT-OSS EAGLE3 training (8 GPUs). GPU configurations are specified in Table 7.

• Cross-validation / statistical protocol. No formal cross-validation or statistical significance testing is reported. The Qualitative Split uses 80 samples per category and reports mean ALs across those samples. The Throughput Split uses 512 samples per condition to construct throughput-latency Pareto curves, where each point on the curve represents a different batch size (BS = 2, 4, 8, 16, 32, 64, 128, 256, 512 depending on the experiment). The paper does not report confidence intervals, standard deviations, or error bars on any metric, which must be considered when interpreting the precision of reported differences (particularly the 23% throughput overestimation figure and the per-category AL differences that often differ by <0.5 tokens).

Main Quantitative Results

Speculator Accuracy Across Qualitative Split Categories (Table 1, Figure 3)

Table 1 presents the headline accuracy results at BS = 32 and DL (draft length) = 3 across all 11 categories for five model families and multiple SD methods. The key numbers at Temperature=0:

• Llama 3.3 70B with EAGLE3 achieves the highest mean speedup of 1.90×, with mean AL of 2.44. N-Gram speculation actually produces a slowdown (0.88× speedup) due to acceptance rates that fail to justify verification costs at this batch size.

• Qwen3-Next's native MTP head achieves the highest mean AL (2.81) and a 1.20× speedup, despite being architecture-integrated rather than post-trained. The paper emphasizes this finding as evidence that pretraining offers significant accuracy gains over post-trained drafting heads like EAGLE3.

• Domain entropy correlates strongly with AL. Low-entropy domains consistently outperform high-entropy ones. For Qwen3-Next MTP: Coding achieves AL = 3.34, Multilingual 3.19, Math 3.13, while Roleplay achieves only 2.09 (Table 1). For Llama 3.3 70B EAGLE3: Coding achieves 3.00 AL vs. Roleplay at 2.04. Writing falls in between at 2.63. This entropy-AL correlation holds across all model families and SD methods.

• Vanilla SD sustains accuracy better than EAGLE3 at longer draft lengths. Figure 3 shows AL scaling as draft length increases from 3 to 7. For Qwen3-Next, MTP maintains higher ALs than EAGLE3 across all draft lengths—evidence of pretraining's advantage. For Llama 3.3 70B, Vanilla SD (using Llama 3.2 1B as drafter) sustains higher ALs at long draft lengths than EAGLE3, consistent at DL = 7. The paper interprets this as indicating that "despite higher draft overhead, external drafting sustains accuracy better than EAGLE3 at longer speculation horizons."

• Temperature=1 reduces ALs and speedups uniformly. The bottom rows of Table 1 report mean ALs at Temperature=1. Llama 3.3 70B EAGLE3 drops from mean AL 2.44 (T=0) to 2.37 (T=1) and speedup from 1.90× to 1.75×. GPT-OSS 120B EAGLE3 drops from 2.25 AL to 2.07 AL and 1.34× to 1.06× speedup. The higher entropy of the target distribution reduces both acceptance rates and—in the speedup calculation—the effective throughput improvement.

Throughput and Latency Analysis Across Batch Sizes (Figures 6, 7)

Figure 6 compares throughput (Output TPS) as a function of User TPS when using random token inputs versus SPEED-Bench's Throughput Split (8k ISL) for GPT-OSS 120B with EAGLE3 drafting at DL = 3. The headline finding: synthetic benchmarking overestimates throughput with SD enabled by an average of 23% compared to SPEED-Bench's real-data measurements. Both with SD (solid lines) and without SD (dashed lines), the SPEED-Bench curves fall below the random-token curves. Points on the curves represent batch sizes from 1 to 128.

The paper also observes a performance gap in the baseline autoregressive setting (without SD), attributed to expert imbalance in GPT-OSS 120B's MoE architecture—random inputs fail to trigger realistic expert routing, producing inaccurate step latency measurements (Appendix F, Figures 11–12).

Figure 7 examines optimal draft length selection across batch sizes for GPT-OSS 120B with EAGLE3 on the Throughput Split (2k ISL). Points represent batch sizes from 2 to 512. The key finding: the optimal DL shifts depending on the concurrency regime. At lower batch sizes where the system is memory-bound, DL = 3 (the longer draft) achieves higher throughput than DL = 1. However, as batch size increases and the system approaches the compute-bound regime, the crossover reverses: the cost of verifying additional tokens outweighs the speculative gains, and DL = 1 becomes preferable. The paper positions this as a direct demonstration of SPEED-Bench's utility for identifying these crossover points—information that practitioners need for deployment configuration but cannot obtain from BS = 1 benchmarks.

Vocabulary Pruning Analysis (Figure 4)

Figure 4 compares average AL for GPT-OSS 120B with EAGLE3 drafters using full vocabulary versus pruned vocabulary (32k tokens) across seven categories at DL = 3. The headline finding: vocabulary pruning has dramatically domain-dependent effects. In Math, the pruning penalty is negligible (-0.65% relative reduction in AL). In Coding, it is small (-2.28%). In Writing, it remains comparable (-2.94%). However, in Summarization, the penalty jumps to -9.51%, in RAG to -10.05%, and in Multilingual to -10.22%. The average penalty across the seven shown categories is -5.53%.

The paper connects this to a theoretical token analysis (Appendix J, Table 10): with a 32k pruned vocabulary, only 76.9% of Multilingual target tokens are present in the vocabulary (versus 94.7% overall), compared to 98.9% in the full vocabulary. This disproportionate impact on the "long tail" of token usage—concentrated in non-English text and specialized domains—is the mechanism driving the domain-dependent degradation.

Comparison with SpecBench (Figure 5)

Figure 5 presents a side-by-side comparison of EAGLE3 versus Vanilla SD (Llama 3.3 70B, DL = 7) on selected SpecBench and SPEED-Bench categories. The headline finding: SpecBench's limited diversity masks differences between methods that SPEED-Bench reveals. On SpecBench's Coding and Reasoning categories (10 samples each), EAGLE3 and Vanilla SD appear comparable—the narrow, homogeneous samples create statistical noise where the two methods' ALs overlap. On SPEED-Bench's larger, more diverse Coding and Reasoning splits, Vanilla SD shows a clear advantage over EAGLE3, reaching higher ALs.

The effect is most pronounced in Multilingual: SpecBench's German-to-English-only translation prompts show a moderate gap between the two methods, while SPEED-Bench's 23-language, multi-task multilingual split reveals a substantially larger advantage for Vanilla SD. The paper connects this directly to Figure 2: Multilingual is where the diversity gap between the benchmarks is largest (83% reduction in pairwise similarity), and correspondingly largest measured performance discrepancy.

The full per-category comparison is in Appendix K (Figure 14), which shows this pattern extending across nearly all categories. The paper argues this demonstrates that "broad-coverage evaluation is essential to expose differences between methods."

Training Data ISL Effects and RoPE Scaling (Figure 8)

Figure 8 evaluates how training ISL affects EAGLE3 drafter accuracy at inference time for GPT-OSS 120B, with drafters trained at maximum ISLs of 1k, 2k, and 4k tokens and evaluated across SPEED-Bench's 1k–32k ISL buckets. The headline finding: accuracy degrades rapidly when inference ISL exceeds training ISL, but YaRN scaling recovers significant performance. Solid lines show unscaled models: the 1k-trained model drops from AL ≈ 2.0 at 1k ISL to AL < 1.5 at 8k ISL. The 2k-trained model holds better but still degrades at 16k and 32k. The 4k-trained model shows less degradation but also drops at 32k. Dotted lines show the same models with YaRN scaling applied at inference: the 2k-trained model with YaRN achieves AL > 1.8 at 32k ISL (compared to AL ≈ 1.5 without YaRN), and the 4k-trained model with YaRN maintains AL ≈ 1.8–1.9 across the full range.

The paper notes in Appendix M that two publicly available EAGLE3 checkpoints for GPT-OSS 120B both suffer from this degradation, with one (lmsys/EAGLE3) having no RoPE scaling configuration at all, and the other (nvidia/gpt-oss-120b-Eagle3-long-context) showing decay even at 8k ISL despite having the original max position embeddings set to 8192—suggesting the default wasn't changed during training and doesn't reflect actual training context length.

AL Stability Across ISLs and Entropy Categories (Figure 13, Appendix I)

Figure 13 presents AL as a function of ISL for three SD setups across the Throughput Split's entropy categories. For Llama 3.3 70B with Vanilla SD and Qwen3-Next with MTP, the expected entropy ordering holds: Low Entropy prompts consistently yield the highest ALs, High Entropy the lowest, and Mixed Entropy falls between. These methods also demonstrate ISL stability—ALs remain relatively constant as ISL grows.

For GPT-OSS 120B with EAGLE3, an anomaly appears: while the ordering holds at 1k ISL, the Low Entropy AL degrades as ISL increases, crossing below the Mixed Entropy curve at longer contexts. The paper attributes this to the training data distribution of the specific EAGLE3 checkpoint, which heavily favors general knowledge prompts over structured coding tasks (trained on UltraChat and Magpie-Llama-3.1-Pro-300K, which contain under 8% coding samples), possibly "exaggerated by incorrect RoPE scaling configuration."

Engine Comparison: TensorRT-LLM vs. vLLM (Figure 15, Appendix L)

Figure 15 compares throughput between TensorRT-LLM and vLLM for GPT-OSS 120B with EAGLE3 on the Throughput Split (2k ISL, DL = 2, batch sizes 2–256). TensorRT-LLM achieves higher peak throughput, which the paper attributes to its support for a one-model runtime paradigm where the speculative head is appended directly to the target model, enabling a single CUDA Graph to capture the entire verification and drafting loop. In contrast, vLLM's two-model approach (draft model as separate engine) incurs host communication overhead between engines, though the paper notes that vLLM's piecewise graph construction may offer greater flexibility for dynamic drafting strategies by reducing static shape requirements.

Ablation Studies and Robustness Checks

Selection algorithm quality: Greedy + Swap vs. QP Approximation vs. Random (Appendix C, Table 6). Both the greedy selection with local swap refinement and the quadratic programming approximation substantially outperform random selection in minimizing pairwise similarity. For Writing, random selection achieves 0.29 average similarity, while both algorithmic methods achieve 0.18. For Humanities, random achieves 0.14 vs. 0.11–0.12 for the algorithmic methods. For RAG, random achieves 0.17 vs. 0.13 for both methods. The Greedy + Swap and QP methods produce nearly identical results, with the QP being less scalable—justifying the choice of the greedy approach as the benchmark's selection mechanism. Appendix D (Figures 9–10) provides qualitative heatmap confirmation of the diversity improvement.

Random selection from SPEED-Bench sources vs. SpecBench (Figure 2). Random selection from SPEED-Bench's higher-quality data sources already outperforms SpecBench on most categories—for example, in Multilingual, random selection from SPEED-Bench's sources achieves lower pairwise similarity than SpecBench's curated subset. This confirms that data source quality matters independently of the selection algorithm: SPEED-Bench's use of 24 diverse data sources provides a better candidate pool than SpecBench's reliance on 5 sources, even before algorithmic curation is applied.

Vocabulary pruning: token coverage analysis (Appendix J, Table 10). For GPT-OSS 120B at Low reasoning effort, the overall token coverage drops from 100% (full vocabulary) to 94.7% at 32k pruning, but Multilingual coverage drops to 76.9%—a 22.3 percentage point gap between overall and worst-affected domain. At Medium reasoning effort, the pattern is virtually identical (94.5% overall, 78.1% Multilingual at 32k). With more aggressive 16k pruning, Multilingual coverage drops to 72.1–73.9%, while overall remains at 89.1–89.7%. This analysis provides the mechanistic explanation for Figure 4's domain-dependent AL degradation: the tokens that Multilingual prompts require are disproportionately excluded from the pruned vocabulary.

Expert routing on real vs. synthetic data (Appendix F, Figures 11–12). At BS = 32 with GPT-OSS 120B processing 8k ISL inputs, SPEED-Bench data produces a relatively uniform expert activation profile during prefill, while random tokens cause significant imbalance—certain experts receive substantially higher activation frequency, and 20–30% of experts in some layers are never activated. Figure 12 tracks the number of unique activated experts across layers: random tokens activate fewer unique experts in most layers. Even though the total token volume is high (32 prompts × 8000 tokens = 256,000), synthetic noise fails to trigger the routing logic that real semantic workloads produce. This is presented as the mechanism behind the overestimation in Figure 6's baseline autoregressive measurements.

Multi-turn sample inclusion (Qualitative Split metadata). Approximately 20% of Qualitative Split samples feature multi-turn interactions spanning 2–5 turns, compared to SpecBench's limitation to 2 turns. The paper does not separately ablate multi-turn vs. single-turn performance—acceptance rates are not broken out by turn count—so the effect of multi-turn structure on SD performance is implied but not quantified.

Temperature sampling robustness (Table 1). The paper reports both Temperature=0 and Temperature=1 results for all model configurations in Table 1. The degradation from T=0 to T=1 is consistent across methods and models but not uniform: for Llama 3.3 70B EAGLE3, mean AL drops from 2.44 to 2.37 (a 2.9% relative reduction) and speedup drops from 1.90× to 1.75× (a 7.9% relative reduction). For GPT-OSS 120B EAGLE3, mean AL drops from 2.25 to 2.07 (8.0% reduction) and speedup drops from 1.34× to 1.06× (20.9% reduction)—a notably sharper degradation than for Llama. This differential temperature sensitivity across model families is noted but not deeply analyzed; the paper presents the T=1 numbers primarily as evidence that the benchmark supports both greedy and sampling-based evaluation.

Critical Assessment

Claim 1: SPEED-Bench reveals that synthetic inputs overestimate throughput by 23%

This claim is supported by Figure 6, but with an important scope limitation: the 23% figure is established for one specific configuration (GPT-OSS 120B with EAGLE3, DL = 3, 8k ISL, TensorRT-LLM, batch sizes 1–128). The paper does not replicate this measurement across other model families, other SD methods, other ISL buckets, or other inference engines. The mechanism behind the overestimation—random inputs producing unrealistic acceptance rates—is well-argued theoretically and supported by the qualitative examples in Appendix E, but the quantitative 23% figure should be understood as a demonstration of the phenomenon's existence, not as a universal constant. The additional finding that random tokens cause expert routing imbalance even for baseline autoregressive decoding (Appendix F) is demonstrated only for GPT-OSS 120B, a MoE architecture—the paper does not test how random inputs affect dense models like Llama 3.3 70B or Qwen3-Next in the baseline setting. This is a genuine limitation: the MoE-specific routing problem is a distinct mechanism from the SD-specific acceptance rate distortion, and the interaction between the two (which contributes how much of the 23%?) is not disentangled.

Claim 2: Vocabulary pruning degrades long-tail domains by ~10%

Strongly supported by Figure 4 and Appendix J, Table 10. The relative AL drops of 9.51% (Summarization), 10.05% (RAG), and 10.22% (Multilingual) at DL = 3 are consistent with the token coverage analysis showing that these domains lose 22–24% of their target tokens under 32k pruning. The near-zero degradation in Math and Coding is consistent with their high token coverage. However, the analysis is limited to a single draft model configuration (GPT-OSS 120B with EAGLE3). The paper does not test whether other draft models or other target models exhibit the same pruning sensitivity, nor does it test intermediate pruning levels (e.g., 48k, 64k) to establish whether the domain-dependence is smooth or exhibits sharp thresholds. The fact that this is demonstrated on models the authors trained themselves (Appendix H) rather than on publicly available checkpoints with standardized pruning configurations makes the finding somewhat specific, though the theoretical token analysis suggests the effect should generalize.

Claim 3: Optimal draft length shifts with batch size

Supported by Figure 7, which shows the crossover between DL = 1 and DL = 3 for GPT-OSS 120B with EAGLE3 on vLLM at 2k ISL. The finding is intuitive given the memory-bound-to-compute-bound transition, and the data supports it. However, the paper tests only two draft lengths (1 and 3) and reports the full Pareto curve for only one configuration. The paper does not sweep multiple draft lengths per batch size to identify the true optimum at each point, does not test whether the crossover batch size varies with ISL (which it almost certainly does, since longer contexts increase the compute burden), and does not test different target models to establish whether the crossover point is architecture-dependent. The paper also does not test whether tree-based verification—which can verify multiple draft branches simultaneously—changes the crossover behavior compared to the draft-chain approach used throughout.

Claim 4: SpecBench masks differences between SD methods

Supported by Figure 5 (and Appendix K, Figure 14), which shows EAGLE3 and Vanilla SD overlapping on SpecBench's narrow categories but diverging on SPEED-Bench's diverse ones. This is a compelling demonstration of the benchmark's value. However, the comparison is made at a single draft length (DL = 7) and a single model pair (Llama 3.3 70B): it demonstrates that diversity changes the measured gap, but does not establish whether the difference changes conclusions about which method is superior. The paper asserts that SPEED-Bench reveals the "expected advantage of the external drafter at long DLs," but this expectation is based on prior understanding—the experiment confirms it rather than discovering it. A stronger demonstration would show that SpecBench leads to a qualitatively different recommendation (e.g., ranking EAGLE3 above Vanilla SD on a specific criterion where SPEED-Bench reverses the ordering).

Claim 5: Long-context drafter degradation can be mitigated by YaRN scaling

Supported by Figure 8 for GPT-OSS 120B with custom-trained EAGLE3 drafters. The finding is clear: YaRN scaling recovers substantial accuracy at inference ISLs beyond training ISL. However, this is demonstrated on models the authors trained themselves at known training ISLs (1k, 2k, 4k). For publicly available checkpoints where the training ISL is uncertain (as discussed in Appendix M for the two existing EAGLE3 models), the diagnosis-and-mitigation workflow is less clean—the paper can only hypothesize about the cause of their degradation. The recommendation to "train with max position embeddings equal to the training context length and apply RoPE scaling at inference" is practical but validated only on the authors' own training runs. The finding is also specific to EAGLE3-style drafters; the paper does not test whether Vanilla SD or MTP drafters exhibit similar long-context degradation patterns, nor whether YaRN scaling helps them.

What was not tested

Several experiments would have strengthened the paper's claims:

• Cross-engine reproducibility of the 23% overestimation. The synthetic-vs-real comparison (Figure 6) uses only TensorRT-LLM. Would vLLM or SGLang show the same gap? The expert routing problem (Appendix F) is engine-independent (it occurs at the model level), but the SD-specific acceptance rate distortion might interact with engine optimizations differently.

• Fine-grained difficulty analysis within categories. The Qualitative Split includes difficulty metadata for Coding, Humanities, Math, and STEM categories, but the paper does not break out acceptance rates by difficulty level. If easy math problems yield AL = 3.5 and hard math problems yield AL = 2.0, reporting only the mean masks important variance—and difficulty-dependent performance might interact with vocabulary pruning or long-context effects in non-obvious ways.

• Statistical reporting. None of the plots include error bars, confidence intervals, or standard deviations. With 80 samples per category in the Qualitative Split and 512 per condition in the Throughput Split, the authors had sufficient data to compute variability estimates. The absence makes it impossible to assess whether per-category AL differences of 0.1–0.2 tokens are statistically meaningful or within sampling noise—particularly relevant for the SpecBench comparison (Figure 5), where SpecBench's 10-sample categories almost certainly have wider confidence intervals than SPEED-Bench's 80-sample categories.

• Interaction effects between vocabulary pruning and long contexts. The pruning analysis (Section 8.2) uses only the Qualitative Split (short-to-moderate ISLs). The long-context analysis (Section 8.5) uses the full vocabulary. What happens when a pruned-vocabulary drafter is deployed at 32k ISL? These two documented failure modes might compound—an important practical question that the independently-run experiments cannot answer.

• Ablation of the measurement framework itself. The paper argues that external tokenization is necessary for fair cross-engine comparison but does not quantify how much variability it eliminates. Comparing AL measurements with and without external tokenization across engines would quantify the benefit of this design choice.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Not Factored Into Any Efficiency Measurement

The assumption or constraint. The Qualitative Split's selection algorithm—Greedy Selection with Local Swap Refinement (Algorithm 1)—requires embedding all candidate prompts using OpenAI's text-embedding-3-large model and computing pairwise similarity matrices for the subset optimization. For the Throughput Split, constructing fixed ISL buckets requires tokenizing, truncating, and/or padding thousands of prompts with the o200k tokenizer. The paper does not account for these preprocessing costs anywhere in its evaluation metrics. This is an unstated assumption: that the benchmark's curation cost is a one-time expense amortized across all subsequent evaluations, and therefore does not affect the reported metrics (acceptance rates, throughput, speedups).

The paper is transparent about the selection algorithm's mechanics but never quantifies its computational cost. Constructing the Qualitative Split required embedding thousands of candidate prompts from 18 data sources, computing the Gram matrix for the swap refinement phase, and iterating the greedy construction for each of the 11 categories. This is a non-trivial compute investment, though likely on the order of hours on a single GPU, not days.

The consequence. For the benchmark's users (researchers evaluating SD methods), this limitation is immaterial—the pre-constructed dataset is downloaded and used directly. For the benchmark's extenders (researchers wanting to add new categories, update data sources, or construct domain-specific variants), the preprocessing cost creates a barrier to adoption. The algorithm's quality depends on having a large candidate pool to select from, and constructing that pool for a new domain requires gathering, embedding, and running the selection pipeline—work that is not trivial and not packaged as a reusable tool in the paper's release.

More importantly, the paper's framing of the Qualitative Split as "compact" and "efficient" (880 samples vs. the thousands of candidate prompts it was culled from) elides the fact that this efficiency is purchased with substantial upstream curation cost. A researcher who naively assumes the benchmark's construction is cheap will underestimate the effort required to extend it.

What evidence exists in the paper. The paper describes the selection algorithm in detail (Algorithm 1, Section 5) and the data sources (Tables 4–5, Appendices A–B), but never quantifies the runtime, FLOP cost, or human effort involved. Appendix C mentions that the QP approximation was "faster and more scalable" than the exact approach, but gives no concrete timing numbers. The paper does not report how many candidate prompts were embedded for each category, how long the embedding step took, or how many swap iterations were required for convergence.

Mitigation status. Not addressed. The paper provides the curated dataset on HuggingFace, so downstream users never pay this cost. But the methodology for extending the benchmark—the selection algorithm, the embedding model choice, the swap refinement parameters—is described but not operationalized as a reusable tool or library. A future version could package the curation pipeline alongside the dataset, reducing the barrier for community extensions.

The 23% Throughput Overestimation Claim Is Demonstrated on Only One Model–Engine Configuration

The assumption or constraint. The headline finding that synthetic random-token inputs overestimate real-world throughput by 23% (Section 8.4, Figure 6) is established for a single configuration: GPT-OSS 120B (a Mixture-of-Experts model) with EAGLE3 drafting at DL = 3, measured on TensorRT-LLM with the Throughput Split's 8k ISL bucket at batch sizes 1–128. The paper argues theoretically that this overestimation stems from two mechanisms: (1) random inputs producing unrealistic acceptance rates (Appendix E) and (2) random inputs causing expert routing imbalance even in baseline autoregressive decoding (Appendix F). Mechanism (1) should generalize to any SD method on any model. Mechanism (2) is specific to MoE architectures—dense models have no router to collapse.

The paper implicitly assumes that the quantitative 23% figure is representative, but does not test whether it varies across model families, SD methods, ISL buckets, or inference engines. It also does not disentangle how much of the 23% comes from mechanism (1) versus mechanism (2)—which would matter for practitioners using dense models, where mechanism (2) does not apply.

The consequence. A practitioner reading the paper might conclude that synthetic throughput benchmarks universally overestimate performance by approximately 23% and adjust their mental models accordingly. But for a dense model like Llama 3.3 70B (which has no expert routing to collapse), the overestimation might be substantially smaller—coming only from the SD-specific acceptance rate distortion, not from the baseline latency error. Conversely, for a different MoE model with a different router architecture, the overestimation might be larger or smaller. The 23% figure risks being cited as a general constant when it is, in fact, configuration-specific.

Similarly, the mechanism (2) finding—that random tokens cause expert routing imbalance—is demonstrated only for GPT-OSS 120B (Appendix F, Figures 11–12). The paper does not test whether Qwen3 235B (also MoE), DeepSeek R1 (also MoE), or any dense model exhibits similar behavior. The finding's generality is therefore unestablished: are all MoE routers similarly vulnerable, or is this specific to GPT-OSS 120B's routing mechanism?

What evidence exists in the paper. Figure 6 (one configuration), Figure 11 (expert activation for GPT-OSS 120B, Layer 17 at 8k ISL, BS=32), Figure 12 (unique activated experts across layers for GPT-OSS 120B). The qualitative examples in Appendix E are for GPT-OSS 120B. No equivalent measurements are reported for Llama 3.3 70B, Qwen3 235B, Qwen3-Next, or DeepSeek R1. The paper does not report synthetic-vs-real throughput comparisons for any configuration other than Figure 6's.

Mitigation status. The paper makes the theoretical argument for why synthetic inputs are problematic in general terms (Section 6: "fundamentally flawed," two failure modes), and the mechanism (2) finding is argued to be architecture-general ("random noise fails to trigger realistic expert routing in MoE architectures"). But the quantitative evidence is from a single point. The paper does not claim generality for the 23% figure explicitly, but the abstract and introduction present it without qualification ("we highlight this by quantifying how synthetic inputs overestimate real-world throughput"), which implies broader applicability. Future work would need to replicate the synthetic-vs-real comparison across multiple model families to establish whether the 23% is typical or anomalous.

No Statistical Reporting of Uncertainty

The assumption or constraint. The paper reports mean acceptance lengths, mean acceptance rates, and aggregate throughput metrics without error bars, confidence intervals, standard deviations, or any other measure of statistical uncertainty. This applies to all figures and tables: Figure 2 (the pairwise similarity comparison), Figure 3 (AL scaling across draft lengths), Figure 4 (vocabulary pruning effects), Figure 5 (SpecBench comparison), Figures 6–7 (throughput curves), Figure 8 (training ISL effects), Table 1 (per-category ALs and speedups), and all appendix plots. The sample sizes—80 per category in the Qualitative Split, 512 per condition in the Throughput Split—are large enough to compute meaningful variability estimates, but none are provided.

The implicit assumption is that the sample sizes are sufficient for the differences between methods or configurations to be statistically significant, and that the reader can trust the reported means as precise. For throughput measurements (Figures 6–7), the curves are constructed by running multiple concurrent requests and measuring aggregate token rates, which provides implicit averaging. But for per-category acceptance lengths (Table 1, Figure 5), the 80-sample averages may have substantial variance, particularly for high-entropy domains where individual prompt difficulty varies widely.

The consequence. Without uncertainty quantification, the reader cannot assess whether reported differences are meaningful. Consider Table 1: for GPT-OSS 120B with EAGLE3, Coding achieves AL = 2.46 while Humanities achieves AL = 2.28—a difference of 0.18 tokens. Is this a real domain effect or within sampling noise? For Llama 3.3 70B EAGLE3, Writing achieves AL = 2.63, Summarization achieves 2.59, RAG achieves 2.76—these differences of 0.04–0.17 tokens are the basis for domain-dependent claims throughout the paper, but without confidence intervals, a practitioner cannot tell whether Coding and Math are genuinely "easier" for the drafter than Writing, or whether the ordering is partly noise.

This is particularly acute for the SpecBench comparison (Figure 5), where SpecBench's 10-sample categories almost certainly have wider confidence intervals than SPEED-Bench's 80-sample categories. The paper claims that SpecBench "masks differences between methods," but part of this masking may simply be that 10-sample averages are too noisy to distinguish methods that genuinely differ. The paper's argument would be stronger if it could demonstrate that the SpecBench-vs-SPEED-Bench discrepancy in Figure 5 exceeds what sampling noise alone would produce.

What evidence exists in the paper. None. No error bars appear in any plot. No standard deviations or confidence intervals are reported in any table. The paper does not discuss statistical methodology, sample size justification, or power analysis.

Mitigation status. Not addressed. The paper's measurement framework captures per-request data (Section 7: "fine-grained performance data by analyzing the streaming response objects"), and the sample sizes are large enough to compute variability estimates from this data. Computing standard deviations or bootstrap confidence intervals from the existing measurements would add substantial rigor at zero additional data collection cost. The paper's release could include variability estimates in the dataset metadata (e.g., per-category AL distributions, not just means), enabling downstream users to perform their own statistical tests.

The Qualitative Split Cannot Measure System-Level Speedups, and the Throughput Split Cannot Measure Fine-Grained Domain Effects

The assumption or constraint. SPEED-Bench's two-split architecture is a design choice motivated by the observation that a single dataset cannot serve both qualitative accuracy analysis and system-level throughput measurement (Section 3.4, implicitly). The Qualitative Split is optimized for semantic diversity and compactness (80 samples per category)—it can measure acceptance rates and acceptance lengths across fine-grained domains, but its small sample counts and uncontrolled ISL distribution make it unsuitable for constructing stable throughput-latency Pareto curves. The Throughput Split is optimized for system-level measurement—fixed ISL buckets, large sample counts (512 per condition), controlled entropy categories—but its three broad entropy bins cannot resolve per-category differences like "how does SD perform specifically on RAG vs. Summarization?"

The paper provides the domain-specific speedup estimation formula (Appendix G) as a bridge: measure system latencies ($t_{ar}$, $t_{sd}$) on the Throughput Split, measure domain-specific AL on the Qualitative Split, and combine analytically via $S = (t_{ar} \cdot \text{AL}) / t_{sd}$. This decoupling is elegant, but it assumes that per-step latency is independent of the specific domain—an assumption the paper argues is justified because "per-step latency is primarily governed by system constraints and serving parameters" (Appendix G).

The consequence. The analytical speedup estimation assumes that $t_{ar}$ and $t_{sd}$ measured on the Throughput Split's data (a mix of coding, STEM, and creative writing prompts) accurately represent the per-step latencies a user would experience on their specific target domain (e.g., only legal document summarization, or only Python code generation). If this assumption holds, the formula works well. If per-step latency has non-negligible domain dependence—for instance, if different prompt types cause different KV-cache memory access patterns, different expert routing distributions in MoE models, or different compute utilization—then the estimated speedup will be biased.

More subtly, the formula captures average speedup but cannot capture variance. A domain where AL is highly bimodal (many prompts are trivially easy, many are impossibly hard) might have the same mean AL as a domain with uniform moderate difficulty, but the tail latency behavior—critical for user experience—would differ substantially.

What evidence exists in the paper. The paper does not directly test whether per-step latency varies across domains within the same ISL and batch size regime. Figure 13 (Appendix I) shows that AL varies substantially across entropy categories for a given ISL, but only measures AL, not $t_{ar}$ or $t_{sd}$. The expert routing analysis (Appendix F) demonstrates that random tokens cause different routing than real data—which supports the claim that domain matters for routing—but does not test whether different real domains (e.g., coding vs. creative writing) cause different routing patterns. The paper implicitly assumes that all real data produces similar routing, which may be true but is unverified.

Mitigation status. Partially addressed. The paper acknowledges the limitation implicitly by providing the analytical speedup formula and noting its assumptions ("for this proxy to be accurate, the latency measurements must be realistic"). The formula is framed as a practical tool, not a theoretical guarantee. The paper does not validate the formula by comparing analytically estimated speedups against directly measured speedups for a specific domain—a simple experiment that would quantify the approximation error. This validation remains future work.

The Benchmark's Coverage of Production Deployment Diversity Is Modeled on Post-2024 Assumptions

The assumption or constraint. SPEED-Bench's design—the categories chosen, the ISL buckets, the batch size ranges, the draft length sweeps, the inference engines supported—reflects the SD deployment landscape as understood in late 2025/early 2026. The paper evaluates on five target models (Llama 3.3 70B, GPT-OSS 120B, Qwen3 235B, Qwen3-Next, DeepSeek R1) and four SD methods (N-Gram, Vanilla SD, EAGLE3, Native MTP). All experiments use NVIDIA B200 GPUs (single GPU or 8-GPU configurations).

This is a snapshot, not a forward-compatible framework. The benchmark's categories were chosen based on analysis of what SpecBench and EAGLE3 evaluation practices missed in 2024–2025 (multilingual diversity, long-context coverage, throughput-oriented measurement). If the SD field shifts—for instance, if tree-based verification becomes the production standard (the paper explicitly restricts itself to draft chains, Section 8), if new model architectures introduce different memory-compute tradeoffs, or if new speculation methods (like retrieval-based drafting or parallel drafter adaptation) become dominant—SPEED-Bench's evaluation categories and metrics may no longer capture the most important dimensions of performance.

The paper does not commit to maintaining or updating the benchmark. The dataset is released on HuggingFace, but there is no stated plan for versioning, community contributions, or periodic data refresh.

The consequence. The benchmark's value decays with time. As new draft models are trained on newer data distributions, the Qualitative Split's semantic diversity may become less representative—if the field converges on a few dominant training datasets, the diversity that SPEED-Bench tests for may become less relevant. More importantly, if tree-based verification (which can verify multiple draft branches simultaneously) becomes standard at BS > 1 (currently, the paper states it is "not standard for BS > 1 speedups in production engines," Section 8), then SPEED-Bench's draft-chain-focused evaluation will miss a major dimension of SD performance. The measurement framework technically supports tree-based evaluation (Section 8), but no tree-based results are reported, and the Throughput Split is not validated for tree-based throughput measurement.

What evidence exists in the paper. The paper reports results for draft chains only (Section 8: "We exclusively utilize draft chains rather than tree-based verification"). The integration with SpecBench models is mentioned as supporting "easier evaluation for the research community" (Section 1), but only Medusa integration is described, and no tree-based results are shown. The paper does not discuss versioning, maintenance, or community contribution mechanisms.

Mitigation status. Not addressed as a limitation. The paper frames SPEED-Bench as a contribution to be used, not maintained. As a static artifact (a dataset on HuggingFace, a measurement framework on what is presumably GitHub), it can be extended by the community, but the curation pipeline—the selection algorithm, the embedding model, the data source choices—is not automated or versioned. The paper does not propose a mechanism for community contributions of new categories, updated data sources, or new inference engine integrations. This is a structural limitation of benchmark papers generally, not specific to SPEED-Bench, but it is worth noting because the paper's value proposition depends on the benchmark remaining representative of real-world deployment conditions.

Vocabulary Pruning Analysis Is Limited to a Single Drafter Type and a Single Pruning Level

The constraint. The vocabulary pruning analysis (Section 8.2, Figure 4) demonstrates that EAGLE3's vocabulary pruning (to 32k tokens) degrades acceptance lengths in a domain-dependent manner, with ~10% drops in Summarization, RAG, and Multilingual, and negligible drops in Math and Coding. This is measured on one configuration: GPT-OSS 120B with a custom-trained EAGLE3 drafter (trained by the authors, Appendix H) at DL = 3, Temperature = 0, using the Qualitative Split. The paper provides a theoretical token coverage analysis (Appendix J, Table 10) that explains why this happens—pruned vocabularies disproportionately exclude tokens needed for multilingual and specialized-domain text.

The consequence. The finding establishes that vocabulary pruning has domain-dependent effects, but does not provide enough information for a practitioner to make a deployment decision. Key questions remain unanswered: (1) Does the degradation scale linearly with pruning aggressiveness (32k vs. 48k vs. 64k vs. full vocabulary), or is there a threshold below which the effect is negligible? (2) Is this specific to EAGLE3's training procedure and data distribution, or does it generalize to other drafters (Vanilla SD, MTP heads, Medusa) that may apply vocabulary pruning differently? (3) Does the degradation compound with other deployment factors—for instance, is a pruned-vocabulary drafter particularly bad at long contexts where the output vocabulary distribution shifts? The paper's long-context analysis (Section 8.5) uses full-vocabulary drafters, and the pruning analysis uses only the Qualitative Split's relatively short prompts, so these two documented failure modes have not been tested in combination.

What evidence exists in the paper. Figure 4 (seven categories at DL = 3) and Appendix J, Table 10 (token coverage at 16k, 32k, 64k, and full vocabulary for two reasoning effort levels). The paper does not sweep pruning levels in the empirical AL measurement, does not test other draft models with pruning, and does not test pruning at long ISLs or high batch sizes.

Mitigation status. The paper's theoretical token coverage analysis (Appendix J, Table 10) provides a vocabulary-level explanation that is drafter-independent—any drafter that prunes its output vocabulary will lose the same tokens. This suggests the finding should generalize, but the magnitude of the AL degradation depends on how the drafter handles missing tokens (does it assign them zero probability? approximate with a related token? fall back to the target model?), which is implementation-specific and not analyzed. The paper does not propose a mitigation beyond the implicit recommendation to use full vocabulary for multilingual and summarization workloads—reasonable but not actionable for practitioners who need vocabulary pruning for latency reasons and want to know the minimal acceptable pruning level for their domain.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper causes a methodological shift in how the field approaches Speculative Decoding evaluation, but the shift is diagnostic rather than algorithmic—it changes what we measure and how we interpret measurements, not how SD itself works. The magnitude is closer to a reframing with practical enforcement than to a paradigm shift: the paper does not introduce a new speculation algorithm or a new theoretical framework, but it provides compelling evidence that existing evaluation practices produce systematically misleading results, and it offers a concrete, usable alternative.

The reframe operates on two levels. First, at the benchmark design level, the paper establishes that SD evaluation requires two fundamentally different types of data—a semantically diverse split for measuring drafter accuracy and a controlled-throughput split for measuring system efficiency—and that conflating them (as prior work implicitly did) produces benchmarks that fail at both. This is not a deep theoretical insight but a practical architectural one that changes how future SD benchmarks should be constructed. The specific evidence is the demonstration that SpecBench's narrow per-category samples mask performance differences between SD methods (Figure 5), and that random-token throughput benchmarks overestimate real-world performance by 23% (Figure 6). A benchmark designer reading this paper should come away understanding that a single dataset cannot serve both purposes, and that the two-split architecture is the correct template.

Second, at the practitioner level, the paper makes three deployment hazards quantitatively visible for the first time: (1) vocabulary pruning's domain-dependent degradation (9–10% AL drops in Summarization and RAG, negligible in Math and Coding; Figure 4), (2) long-context drafter accuracy collapse when inference ISL exceeds training ISL (Figure 8), and (3) the interaction between batch size and optimal draft length (longer drafts help at low concurrency, hurt at high concurrency; Figure 7). Prior to this work, these were anecdotal concerns or undocumented edge cases. SPEED-Bench transforms them into measurable, reproducible phenomena that can inform deployment decisions. This shifts the conversation around SD from "does it work on average?" to "where does it work, and what breaks it?"—a shift toward conditional deployment thinking that is more practically useful than aggregate speedup numbers.

Reconciling prior contradictions. The paper does not explicitly claim to resolve contradictions in the SD literature—there is no deep conflict analogous to the "LLMs can/cannot self-correct" debate that the inference-time scaling paper resolved. Instead, it resolves a subtle but important tension between research evaluation and production reality. The tension is: SD methods that appear to work well in papers (EAGLE3 reporting strong results on MT-Bench + HumanEval + GSM8K) may underperform in deployment on genuinely diverse workloads. The paper demonstrates that this gap is not a failure of the SD methods themselves but an artifact of benchmarks that lack intra-category diversity and throughput-oriented measurement. A practitioner who previously dismissed SD because "it didn't work on my data" and a researcher who was puzzled by inconsistent reproduction of published results both find an explanation here: the benchmarks were too narrow to capture domain-dependent performance variance.

Research directions that become more attractive. The paper makes verifier/drafter robustness research substantially more attractive by providing the tools to measure it. Specifically: (1) research on vocabulary-pruning-aware drafters that dynamically adjust their vocabulary based on detected domain or language becomes testable—SPEED-Bench's Multilingual, RAG, and Summarization categories provide ready-made stress tests; (2) research on long-context drafter robustness becomes measurable—the Throughput Split's ISL buckets (1k–32k) and the training-ISL-vs-inference-ISL protocol demonstrated in Figure 8 provide a template for evaluating context-length generalization; (3) research on adaptive drafting strategies that vary draft length based on estimated batch size or domain entropy becomes evaluable—the throughput-latency Pareto curve methodology of Figure 7 provides the measurement infrastructure.

Research directions that become less attractive. The paper implicitly argues against two practices: (1) synthetic throughput benchmarking with random tokens. The demonstration in Figure 6 and Appendix F that this overestimates throughput by 23% and causes MoE expert routing collapse means that papers relying on this methodology will face credibility challenges. The finding is concrete enough that reviewers can point to it as a reason to require real-data throughput measurements. (2) Single-category or narrow-domain SD evaluation. The demonstration that SpecBench's narrow categories produce qualitatively different conclusions than SPEED-Bench's diverse ones (Figure 5) raises the bar for what constitutes adequate evaluation. A paper that validates an SD method only on MT-Bench's 10-sample categories is now demonstrably insufficient.

What this does not change. The paper does not alter the fundamental SD algorithm or its theoretical guarantees (losslessness via rejection sampling). It does not propose a new SD method, a new drafter architecture, or a new verification strategy. It is purely an evaluation contribution. The ceiling on its impact is therefore bounded: it can improve how the community evaluates SD, which may accelerate progress by enabling better method comparisons and surfacing deployment hazards earlier, but it cannot directly improve SD performance. The paper also does not resolve the open question of whether tree-based verification (vs. draft chains) changes the throughput-latency tradeoffs it documents—it explicitly restricts evaluation to draft chains (Section 8) and leaves tree-based evaluation as future integration.

Follow-Up Research This Work Enables

Validating and bounding the 23% throughput overestimation across model families. The paper demonstrates that synthetic random-token inputs overestimate throughput by 23% for one configuration (GPT-OSS 120B with EAGLE3, 8k ISL, TensorRT-LLM). A natural follow-up is to replicate this comparison across: (a) dense models (Llama 3.3 70B) where the expert routing mechanism does not contribute to the overestimation, (b) other MoE architectures (Qwen3 235B, DeepSeek R1) to establish whether the routing collapse is universal or GPT-OSS-specific, (c) other SD methods (Vanilla SD, MTP, Medusa) to test whether the acceptance-rate distortion from random inputs varies by drafter type, and (d) other inference engines (vLLM, SGLang) to test whether engine-level optimizations interact with the distortion. A strong result would be a table showing that the overestimation is consistently in the 15–25% range across configurations, establishing it as a reliable correction factor; alternatively, if it varies dramatically (e.g., 5% for dense models vs. 30% for some MoE architectures), that would refine the recommendation from "never use random tokens" to "use random tokens only for dense models without SD, and with documented caution."

Disentangling the SD-specific acceptance-rate distortion from the MoE expert routing problem in synthetic benchmarks. Figure 6 shows that random tokens cause throughput overestimation both with SD (attributed to unrealistic acceptance rates) and without SD (attributed to expert routing imbalance). These are two distinct mechanisms operating simultaneously, and the 23% figure conflates them. A careful ablation would measure: (a) baseline autoregressive throughput on random tokens vs. real data for GPT-OSS 120B (isolating the routing effect), (b) SD throughput on random tokens vs. real data for Llama 3.3 70B (isolating the acceptance-rate effect, since dense models have no routing), and (c) the interaction term. The decomposition would tell practitioners whether the routing problem alone is large enough to invalidate synthetic benchmarks for all MoE inference—even without SD—or whether it is a minor effect that only becomes problematic when combined with SD's acceptance-rate distortion. The measurement infrastructure exists (the Throughput Split provides real-data baselines; the framework supports both SD and autoregressive modes), so this is primarily an experiment-design question, not an infrastructure one.

Characterizing the vocabulary pruning sensitivity function across drafters and pruning levels. The paper establishes that vocabulary pruning degrades acceptance lengths in domain-dependent ways at a single pruning level (32k) for a single drafter (EAGLE3 on GPT-OSS 120B). The natural follow-up is to sweep pruning levels—16k, 32k, 48k, 64k, full vocabulary—and measure per-category AL on SPEED-Bench's Qualitative Split for multiple drafter types. This would reveal: (a) whether the degradation is smooth or exhibits thresholds (does AL drop linearly with vocabulary size, or catastrophically below some critical coverage level?), (b) whether the rank-ordering of domain sensitivity is consistent across drafters (is Multilingual always worst-affected, or is this specific to EAGLE3's training data distribution?), and (c) whether finer-grained subcategory analysis (e.g., which languages within Multilingual are most affected? which summarization task types?) reveals pockets of severe degradation that the category average masks. The token coverage analysis of Appendix J (Table 10) provides the theoretical grounding—this experiment would validate whether the token coverage predicts the actual AL degradation in practice. A strong result would be a "safe pruning level" recommendation per domain (e.g., "for code generation, 32k is safe; for multilingual QA, 64k is the minimum"), directly actionable for practitioners.

Stress-testing the domain-specific speedup estimation formula for known-violation domains. The paper proposes that speedup can be estimated as $S = (t_{ar} \cdot \text{AL}) / t_{sd}$, where $t_{ar}$ and $t_{sd}$ are measured on the Throughput Split and AL is measured on the Qualitative Split (Appendix G). The key assumption is that per-step latency is domain-independent. A validation experiment would: (a) select 2–3 fine-grained domains (e.g., Python code completion, legal document summarization, multilingual German QA), (b) construct Throughput-Split-style fixed-ISL workloads for each domain (controlling ISL and batch size), (c) directly measure $t_{ar}$ and $t_{sd}$ for each domain-specific workload, (d) compare the directly-measured speedup against the analytically-estimated speedup using the Throughput Split's mixed-entropy latency measurements and the Qualitative Split's domain-specific AL. If the estimation error is small (<5%), the formula is validated as a practical tool. If the error is large for certain domains (e.g., code generation has different KV-cache memory access patterns than creative writing), that reveals a limitation of the decoupling assumption and suggests that domain-specific throughput benchmarks are necessary for those domains. Even a negative result (the formula doesn't hold for domain X) is valuable because it identifies when the Throughput Split's measurements can and cannot be used as a proxy.

Combining the vocabulary pruning and long-context failure modes to test for compounding effects. The paper documents two distinct SD degradation phenomena: vocabulary pruning hurts long-tail domains (Section 8.2), and long inference contexts hurt drafters trained on short sequences (Section 8.5). These were studied independently—the pruning experiments used the Qualitative Split (relatively short ISLs), and the long-context experiments used full-vocabulary drafters. A natural stress-test is to evaluate a pruned-vocabulary EAGLE3 drafter at 32k ISL using the Throughput Split's 32k bucket. The hypothesis would be that the two degradation mechanisms compound: long contexts may shift the output token distribution toward rarer tokens (as the model needs to reference more context-specific terminology), simultaneously hitting the pruned vocabulary more often and stressing the drafter's ability to generalize beyond its training ISL. If the compounding is superlinear (the combined degradation exceeds the sum of individual degradations), this would be a strong caution for practitioners deploying pruned drafters in long-context applications. If the effects are mostly independent (the AL drop at 32k ISL with pruned vocabulary is approximately the sum of the 32k ISL drop with full vocabulary plus the pruning drop at short ISL), that would be reassuring—practitioners could estimate combined effects by adding independently-measured penalties. The experiment is straightforward given SPEED-Bench's infrastructure: train (or configure) drafters with and without pruning, at short and long training ISLs, and evaluate across the Throughput Split's ISL buckets.

Extending the benchmark's coverage to tree-based verification as production engines adopt it. The paper explicitly restricts evaluation to draft chains, noting that tree-based verification "remains the standard for BS > 1 speedups in production engines" as of its writing (Section 8). However, if tree-based verification becomes widely supported in production engines—as systems like SpecInfer and Sequoia have demonstrated in research settings—SPEED-Bench will need extension to evaluate it. The challenge is that tree-based verification changes both the throughput dynamics (multiple branches can be verified simultaneously, changing the compute-vs-memory tradeoff) and the acceptance-length measurement (the formula AL = 1 + Σ Π AR_j assumes linear chains; tree-based verification requires a different accounting of expected accepted tokens per step). Extending the measurement framework to capture tree-structured speculation and extending the Throughput Split's Pareto curve methodology to compare chain-based and tree-based drafting at equivalent compute budgets would be a significant contribution that builds directly on SPEED-Bench's architecture. The Qualitative Split's diversity would remain valuable for measuring tree-based drafter accuracy; the Throughput Split's controlled ISL buckets would remain valuable for measuring tree-based system throughput. The extension is primarily to the measurement framework's metric computation and the drafting-strategy parameter space.

Practical Applications and Downstream Use Cases

SD deployment configuration for multi-tenant serving platforms. A cloud LLM provider serving thousands of concurrent users with varying query types can use SPEED-Bench to determine per-workload SD configurations rather than applying a uniform draft length and drafter type. Specifically: (1) The provider measures acceptance lengths per category on the Qualitative Split for their target model and available drafters. (2) They construct throughput-latency Pareto curves on the Throughput Split at their expected ISL distribution and batch size range using their production inference engine. (3) Using the domain-specific speedup estimation formula (Appendix G), they compute the optimal draft length per domain category and batch size regime. (4) At serving time, a lightweight classifier estimates the query's domain (or the provider uses the requested endpoint's known domain), and the system selects the corresponding SD configuration. The paper's finding that optimal draft length shifts from DL = 3 to DL = 1 as batch size increases (Figure 7) directly informs this: the system can dynamically switch based on current load. The finding that vocabulary pruning degrades Summarization and RAG by ~10% AL (Figure 4) informs a different configuration for those endpoints vs. coding endpoints. The net benefit is higher aggregate throughput without per-request manual tuning—a direct operational efficiency gain.

Cost-efficient SD drafter selection and training for specialized enterprise deployments. An enterprise deploying LLMs for a specific domain (e.g., internal legal document processing, customer support in 5 languages, repository-level code completion) currently has little guidance on which SD method to adopt or how to train the drafter. SPEED-Bench provides a systematic selection methodology: (1) Evaluate candidate drafters (N-Gram, Vanilla SD with various draft models, EAGLE3 with various checkpoints, MTP if the target model supports it) on the Qualitative Split categories most similar to the enterprise's domain. If the enterprise's workload is multilingual customer support, the Multilingual, QA, and Summarization categories provide relevant signal. (2) For the top candidate drafters, measure throughput on the Throughput Split at the enterprise's expected ISLs (e.g., 8k for typical support conversations, 32k for document processing) and concurrency. (3) If training a custom drafter (e.g., fine-tuning EAGLE3 on internal data), use the training-ISL protocol demonstrated in Figure 8—train at the expected inference ISL and apply YaRN scaling if inference ISL may exceed training ISL. The paper's finding that publicly available EAGLE3 checkpoints degraded at long contexts due to incorrect RoPE configuration (Appendix M) provides a concrete checklist item: verify that the checkpoint's RoPE scaling matches the actual training context length. The finding that vocabulary pruning hurts multilingual domains (Figure 4) directly informs the vocabulary size decision for the multilingual support use case.

Long-context application deployment with validated RoPE scaling. An organization deploying code assistants that analyze entire repositories (ISLs of 16k–32k tokens) or document Q&A systems can use SPEED-Bench's Throughput Split to validate that their SD configuration doesn't silently degrade at long contexts before deploying to users. The paper documents that publicly available EAGLE3 checkpoints can show substantial accuracy loss at ISLs beyond their training distribution (Appendix M, Figure 16), and that YaRN scaling recovers performance when correctly configured (Figure 8). The deployment workflow is: (1) Evaluate the candidate drafter on the Throughput Split's ISL buckets matching the target deployment range (e.g., 16k and 32k). (2) If AL drops substantially at the target ISL relative to shorter ISLs, diagnose: is the training ISL insufficient? Is RoPE scaling misconfigured? (3) Apply YaRN scaling or retrain with extended context length based on the diagnosis. (4) Re-evaluate to confirm the AL recovers. Without this validation step—which SPEED-Bench makes systematic—an organization deploying SD in a long-context application risks shipping a configuration that is substantially slower than expected (due to low acceptance rates) without realizing the cause. The paper's finding that even a "long-context" labeled EAGLE3 checkpoint showed degradation at 8k ISL (Appendix M) demonstrates that metadata alone is insufficient: empirical validation on controlled-ISL data is necessary.