ArXiv: 2603.27027

🎯 Pitch

Speculative decoding speedups collapse when draft models are trained on the wrong data. Merging the lightweight draft trees from two domain specialists at inference time—rather than averaging their weights—yields a ~24% leap in acceptance length over any single drafter, effortlessly reaping the benefits of specialization without the fragility of parameter-space fusion.

1. Executive Summary

This paper studies how the training distribution of lightweight draft models affects speculative decoding quality, evaluating HASS and EAGLE-2 drafters trained on MathInstruct, ShareGPT, and mixed-data variants against a Llama-3-8B-Instruct verifier on MT-Bench, GSM8K, MATH-500, and SVAMP. The work introduces task-aware proposal distributions — drafters whose training data matches the downstream workload — and analyzes three strategies for combining specialized drafters: checkpoint averaging (parameter-space merging), confidence-based routing (selecting the draft tree with higher mean node confidence before verification), and merged-tree verification (packing both specialists' trees under a shared root for joint verification in one parallel pass). Merged-tree verification achieves the highest acceptance length overall, reaching 5.11 for HASS and 5.03 for EAGLE-2 at temperature 0 (outperforming the strongest single-domain checkpoint by up to ~24%), while naive weight-space averaging collapses to 2.34–2.62 — establishing that inference-time composition preserves domain specialization only when drafters are kept separate and combined at test time, not merged in parameter space.

2. Context and Motivation

The Core Problem: Draft Training Distribution Is an Under-Studied Variable

Speculative decoding has emerged as one of the most promising techniques for accelerating LLM inference without sacrificing output quality. The idea is conceptually elegant: use a small, fast draft model to propose several future tokens, then let the large target model verify them all in parallel. When the draft model's proposals match what the target model would have generated, multiple tokens get accepted in a single target-model forward pass — yielding substantial throughput improvements. The formal guarantee is equally appealing: the rejection-sampling acceptance rule (Equation 1 in the paper) ensures the final output distribution is identical to what the target model would produce autoregressively, making speculative decoding a lossless acceleration technique.

However, the paper identifies a gap in how the field has approached speculative decoding: the dominant focus has been on improving draft architectures and verification algorithms, while the draft model's training distribution has been treated as a fixed, uninteresting constant. Most prior work trains draft models on broad generic corpora — typically ShareGPT, a dataset of conversational exchanges — regardless of the downstream workload the system will serve. The implicit assumption is that a well-architected drafter with enough capacity will learn a good enough approximation of the target model's behavior, and that the specific training data matters only insofar as it provides sufficient volume and diversity.

This paper challenges that assumption directly. The central question is whether speculative decoding quality depends not only on how the draft model is designed (its architecture) but also on what it was trained on (its data distribution). More specifically: if a drafter is trained on data from the same domain as the downstream task, does it achieve longer acceptance lengths than a drafter trained on a mismatched domain? And if so, how should practitioners handle systems that need to serve multiple distinct task families?

Why This Matters: Practical Deployment and the Open-Weight Ecosystem

The practical stakes are high for several reasons that extend beyond academic curiosity about draft model behavior.

First, acceptance length directly controls throughput. In speculative decoding, the number of tokens accepted per verifier call determines how much computation the target model saves. Higher acceptance length means fewer target-model forward passes per generated token, translating directly to lower latency and higher throughput. If a mismatched training distribution reduces acceptance length by even 20–30%, the deployment cost penalty is proportional — you are paying for more verifier calls than necessary without any gain in output quality. The paper's results (Table 1) show that domain mismatch can indeed produce differences of this magnitude. For example, under HASS at temperature 0, the MathInstruct drafter achieves 5.35 acceptance length on MATH-500 but only 2.90 on MT-Bench — a 46% drop when the drafter is deployed on a mismatched domain. Conversely, the ShareGPT drafter achieves 3.98 on MT-Bench but 4.09 on GSM8K. These are not small effects; they represent the difference between a deployment that achieves the promised speedup and one that falls substantially short.

Second, the open-weight ecosystem increasingly provides multiple specialized checkpoints. The paper references this trend explicitly (Sun et al., 2025), noting that the landscape of available models has shifted from a few general-purpose LLMs to many specialized variants fine-tuned for particular domains — coding, mathematics, medical reasoning, legal analysis, multilingual translation, and so on. When a practitioner builds a speculative decoding system that must serve diverse workloads (a chatbot that also handles math problems, a coding assistant that answers natural language questions), they face a concrete engineering decision: should they train one draft model on a mixture of all relevant domains, or should they maintain separate specialists and somehow combine them at inference time? This is not a hypothetical question — it is the design problem that any production speculative decoding system serving heterogeneous traffic must solve.

Third, the choice of combination strategy has non-obvious trade-offs. Merging models in weight space is attractive because it produces a single artifact to deploy and maintain. But does naive averaging preserve the specialization that made each drafter useful in the first place? Routing between specialists avoids weight-space interference but introduces a decision problem: how does the system know which drafter to use for each incoming prompt? And can the system do better than choosing one specialist or the other — perhaps by verifying proposals from both simultaneously? The paper provides the first systematic comparison of these alternatives in the context of speculative decoding, and the results are not obvious a priori. Weight averaging, despite its simplicity and popularity in other contexts (Ilharco et al., 2022), turns out to be remarkably ineffective — the averaged checkpoints are consistently the weakest variants in the paper's evaluation, with average acceptance lengths between 2.34 and 2.62 across all methods and temperatures (Table 1). This is worse than either individual specialist and far worse than mixed-data training or inference-time composition. Understanding why this fails and what works instead is directly actionable for practitioners.

Prior Approaches and Where They Fall Short

The paper situates itself within a rich lineage of speculative decoding research, identifying specific limitations that motivate the current study.

Architecture-focused improvements to draft models. The field has invested heavily in designing better drafters. Early speculative decoding methods used a separate lightweight language model as the drafter — essentially a smaller transformer trained to mimic the target model's token distribution (Leviathan et al., 2023; Chen et al., 2023). Subsequent work improved drafting through feature-level prediction rather than token-level autoregression. EAGLE (Li et al., 2024a) introduced the idea of predicting the target model's hidden states rather than discrete tokens, using the target model's LM head to convert predicted features into token probabilities. This reduces the mismatch between training (where the drafter has access to clean target features) and inference (where it must operate on its own potentially drifted predictions). EAGLE-2 (Li et al., 2024b) improved this further with dynamic draft trees — instead of a fixed tree structure, the system expands frontier nodes based on draft confidence, adapting the search to the specific context. EAGLE-3 (Li et al., 2025) continued this trajectory with training-time test strategies. HASS (Zhang et al., 2025) addressed two additional problems: objective mismatch (the drafter's training loss does not focus on the tokens the verifier is most likely to accept) and context mismatch (during training, the drafter sees clean target features at all positions, but during inference, it must condition on its own imperfect feature predictions at earlier steps). HASS's harmonized objective uses Top-K distillation to focus learning on the verifier's most likely next tokens, and its harmonized context alignment trains the drafter to handle its own drifted features.

All of these contributions are about how the draft model processes information — its internal architecture, its training objective, its tree construction logic. None of them systematically vary what data the drafter is trained on. The implicit baseline is always "train on ShareGPT or an equivalent general corpus," and the question of whether different training distributions produce different acceptance behavior is left unexamined. This is the gap the paper fills: it keeps the draft architecture (EAGLE-2 or HASS) and the verifier (Llama-3-8B-Instruct) fixed, and varies only the domain of the training data, to isolate the effect of the draft training distribution.

Verification and tree-search advances. Other lines of work improve speculative decoding through better verification procedures: tree verification (Miao et al., 2024), self-speculative decoding where the target model serves as its own drafter through early exiting (Zhang et al., 2024; Elhoushi et al., 2024), hierarchical drafting with multiple draft stages (Sun et al., 2024), cascaded drafters (Chen et al., 2024), and retrieval-assisted proposals (He et al., 2024). Again, none of these studies the interaction between draft training data and downstream performance. The paper uses fixed verification procedures throughout and focuses exclusively on how the choice of training distribution and composition strategy affects the proposal quality that the verifier sees.

The missing dimension: draft training distribution as a design variable. The paper's key observation is that proposal quality in speculative decoding — as measured by acceptance length — is a function of both draft architecture and draft training data. Prior work has treated the second factor as a constant, implicitly assuming that any reasonably broad training corpus produces a draft model that approximates the target model's behavior well enough. The paper demonstrates that this assumption is false: training data matters, and the effect is large enough to be practically significant. This reframes the draft model from a purely architectural choice ("should I use HASS or EAGLE-2?") to a joint systems choice ("given my workload, which data should I train my drafter on, and how should I combine specialists if my workload is heterogeneous?").

How This Paper Positions Itself

The paper structures its investigation around five research questions that collectively map the relationship between draft training distribution and speculative decoding performance:

  • RQ1 asks whether domain-matching produces measurable improvements — a basic existence question that, if answered negatively, would make the rest of the study moot.
  • RQ2 asks whether mixed-data training can produce a single robust checkpoint that performs well across all domains, which is the natural follow-up if the answer to RQ1 is positive.
  • RQ3 asks how to combine multiple specialists, which is the practical deployment question that arises when RQ1 shows clear specialization and RQ2 shows that mixed training has limitations.
  • RQ4 asks what signals (confidence, entropy) are useful for routing and diagnosing acceptance behavior, which is necessary to build effective routing policies for RQ3.
  • RQ5 asks how speculative depth interacts with task-aware drafting, which helps interpret why the composition strategies work and when each is most valuable.

The paper's methodological stance is deliberately controlled: it fixes the verifier (Llama-3-8B-Instruct), the draft architecture (HASS and EAGLE-2, evaluated separately), the draft model size (one transformer layer, ~0.8B parameters), the training recipe (20 epochs, learning rate 3×10−53 \times 10^{-5}, batch size 8), and the evaluation metric (acceptance length under the lossless speculative decoding constraint). The only variables are the training data domain (MathInstruct vs. ShareGPT), the mixture ratio, and the composition strategy. This isolates the effect of training distribution from all other confounding factors, allowing clean causal claims about how data domain affects draft quality.

The paper is not proposing a new speculative decoding algorithm or a new draft architecture. Its contribution is empirical and analytical: it shows that a factor previously treated as background noise — the draft training distribution — is actually a first-class design variable that significantly affects deployment performance. The composition strategies (confidence routing, merged-tree verification) are presented as natural inference-time approaches that the empirical findings motivate, not as novel algorithmic contributions per se. The paper's value lies in changing how practitioners think about draft model deployment: from "pick the best architecture and train on general data" to "audit your workload composition, train or select domain-matched drafters, and combine them at inference time rather than in weight space."

The positioning relative to prior work is complementary rather than competitive. The paper does not claim that HASS or EAGLE-2 are insufficient; it shows that both backbones exhibit the same domain specialization pattern, and that the composition strategies work across both. The implication is that architectural improvements and data-aware deployment strategies are orthogonal dimensions that can compound: even as better draft architectures emerge, the principle of matching training data to downstream workloads will remain relevant, and the finding that weight-space averaging destroys specialization will apply regardless of the specific architecture being merged.

3. Technical Approach

3.1 Reader Orientation

This paper is an empirical study that systematically varies what data lightweight speculative decoding draft models are trained on, and then measures how domain-specific training and inference-time composition affect acceptance length. The core idea is that speculative decoding quality is a function not only of draft architecture but also of the match between the draft's training distribution and the downstream workload — and that when multiple specialized drafters are available, keeping them separate and combining them at inference time (via routing or merged-tree verification) preserves domain specialization much better than merging them in weight space.

The system solves the practical problem of deploying speculative decoding for heterogeneous workloads: if you need to serve both math reasoning and conversational chat, should you train one mixed-data drafter, average two specialists, or compose them at inference time? The "shape" of the solution is a controlled experimental pipeline where only the training data and composition strategy change, with all other factors (verifier, draft architecture, hyperparameters) held fixed, allowing clean causal attribution of acceptance length differences to the training distribution.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components connected in a pipeline:

  1. Verifier (Target Model): A fixed Meta-Llama-3-8B-Instruct model that serves as the authoritative distribution for text generation. It never changes across experiments — it is the standard against which draft quality is measured. Its role is to verify draft token proposals via the lossless speculative acceptance rule and generate the final output distribution.

  2. Draft Model (Proposal Distribution): A lightweight LLaMA-style decoder with one transformer layer, hidden size 4096, and roughly 0.8B parameters. It shares the verifier's tokenizer and vocabulary. This component is trained to approximate the verifier's next-token behavior, but what varies across experiments is its training data (MathInstruct, ShareGPT, or mixtures). Two backbones are studied: EAGLE-2 (feature-level drafting with dynamic trees) and HASS (harmonized objective with Top-K distillation). The draft model generates candidate future tokens that the verifier will later check.

  3. Composition Module (Inference-Time Only): When multiple specialized draft checkpoints are available (MathInstruct-trained and ShareGPT-trained), this module decides how to combine them. Three strategies are studied: (a) weight-space averaging — merge parameters before inference, producing a single checkpoint; (b) confidence routing — generate draft trees from both specialists and select the one with higher mean node confidence; (c) merged-tree verification — pack both trees under a shared root with ancestor-preserving attention masks, allowing the verifier to check both specialists' proposals in one pass.

  4. Evaluation Harness: Benchmarks (MT-Bench for conversational, GSM8K, MATH-500, SVAMP for math reasoning) at temperatures 0 and 1, with acceptance length as the primary metric. The acceptance length is defined as the average number of consecutively accepted draft tokens per verifier call under the lossless speculative decoding constraint — meaning the final output distribution must match the verifier's autoregressive distribution exactly.

Information flows sequentially: a prompt enters → the draft model (or composition module, if using multiple specialists) generates candidate tokens → the verifier scores them in parallel and applies the speculative acceptance rule → output tokens are committed. The only components that change across experiments are the draft training data and the composition strategy; the verifier, acceptance rule, and evaluation setup remain constant.

3.3 Roadmap for the Deep Dive

  • First, the speculative decoding foundation — the acceptance rule (Equation 1), the residual distribution (Equation 2), and why these are invariant across all experiments — because every subsequent component depends on understanding that the verifier and acceptance procedure are fixed, allowing us to attribute acceptance length differences purely to draft quality.
  • Second, the EAGLE-2 backbone — its feature-level drafting (Equation 3), training objectives (Equation 4), and dynamic tree construction using confidence-based scoring (Equation 6) — because this establishes the internal mechanics of one of the two draft architectures studied.
  • Third, the HASS backbone — its Top-K distillation loss (Equation 7), harmonized context alignment procedure (Equation 8), and training objective (Equation 9) — because this introduces the second backbone and explains why both architectures exhibit the same domain specialization patterns despite different internals.
  • Fourth, the training data configurations (single-domain, mixed-data) and the controlled experimental setup — because the entire paper's argument rests on isolating the effect of training distribution from architecture, model size, and hyperparameters.
  • Fifth, the three composition strategies (checkpoint averaging, confidence routing, merged-tree verification) with their formal definitions — because these are the practical inference-time mechanisms that the paper evaluates as alternatives to single-drafter deployment.
  • Sixth, the correctness guarantees for routing and merged-tree verification (Proposition A.1, Proposition A.2) — because the paper must establish that these composition strategies remain lossless, preserving the target model's output distribution exactly, before any acceptance length comparison is meaningful.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical analysis paper whose core idea is that the draft model's training distribution is a first-class design variable in speculative decoding systems, and that when multiple domain specialists are available, inference-time composition preserves specialization far better than weight-space merging.

Speculative Decoding Foundation: The Invariant Verifier and Acceptance Rule

Throughout the paper, the speculative decoding framework itself never changes. Understanding its mechanics is essential because any measured acceptance length differences must be attributed to draft quality (driven by training data and composition strategy), not to differences in how the verifier operates.

The acceptance rule. Given a prefix x1:nx_{1:n} (the sequence of tokens generated so far), the draft model pp proposes KK future tokens x~n+1:n+K\tilde{x}_{n+1:n+K} autoregressively. The target model qq then computes its own probability distribution over these tokens in parallel. Each drafted token x~n+t\tilde{x}_{n+t} is accepted with probability:

αn+t=min⁡(1,q(x~n+t∣x1:n+t−1)p(x~n+t∣x1:n+t−1))\alpha_{n+t} = \min\left(1, \frac{q(\tilde{x}_{n+t} \mid x_{1:n+t-1})}{p(\tilde{x}_{n+t} \mid x_{1:n+t-1})}\right)

where q(x~n+t∣x1:n+t−1)q(\tilde{x}_{n+t} \mid x_{1:n+t-1}) is the target model's probability of the proposed token given the accepted prefix up to position n+t−1n+t-1, and p(x~n+t∣x1:n+t−1)p(\tilde{x}_{n+t} \mid x_{1:n+t-1}) is the draft model's probability of that same token under the same context.

What it computes: For each draft token in sequence, the rule compares the target model's assessment of how likely that token is against the draft model's assessment. If the target model assigns equal or higher probability than the draft model (ratio ≥1\geq 1), the token is always accepted. If the target model assigns lower probability, the token is accepted with probability equal to the ratio — meaning overconfident draft proposals (where pp assigns high probability but qq assigns low probability) are more likely to be rejected. The min operation caps the acceptance probability at 1, which handles the case where the draft model is underconfident.

Why this form: This is a rejection sampling correction. The ratio q/pq/p is the importance weight that corrects for the difference between the draft distribution (which we sample from) and the target distribution (which we want to sample from). Accepting tokens with probability proportional to this ratio ensures that the accepted token sequence is distributed exactly as if we had sampled from the target model autoregressively, even though we used the draft model to generate proposals. Crucially, this guarantee holds for any draft distribution pp — even a poor one — as long as the acceptance rule is applied correctly. This is why acceptance length (how many tokens get accepted before rejection) is a pure measure of draft quality: it captures how well pp approximates qq, with higher acceptance meaning the draft and target distributions are more aligned.

The residual distribution on rejection. When rejection occurs at position n+tn+t, the system does not simply fall back to the target model's distribution — it must sample from a corrected residual to maintain the lossless guarantee:

r(x)∝max⁡(0,q(x∣x1:n+t−1)−p(x∣x1:n+t−1))r(x) \propto \max(0, q(x \mid x_{1:n+t-1}) - p(x \mid x_{1:n+t-1}))

where xx ranges over the vocabulary, and the normalization is over all tokens.

What it computes: The residual distribution represents the probability mass that the draft model "missed" — tokens that the target model considers more likely than the draft model does. The max(0, ·) operation clips negative values, which occur when the draft model overestimates a token's probability under the target. These overestimated tokens are excluded from the residual because they were already "covered" by the proposal distribution — any token you might accept from the draft has already been handled by the acceptance rule. The residual therefore specializes in tokens the draft model undersampled.

Why this form: This preserves the target distribution exactly. After rejection, sampling from r(x)r(x) produces tokens with the correct marginal probability under qq, given that the acceptance rule has already consumed part of the probability space. Without the residual correction, falling back to q(x)q(x) after rejection would overrepresent tokens that the draft model already proposed with sufficient probability — the distribution would be biased toward using the target model's distribution when the draft model was actually adequate. The combined process (acceptance rule + residual sampling) is provably equivalent to direct autoregressive sampling from qq, a property the paper relies on throughout.

Acceptance length as the primary metric. For an input prompt xx, a verifier MTM_T, and a drafter MDM_D, the paper defines acceptance length as:

Ex∼D[A(x;MD,MT)]\mathbb{E}_{x \sim \mathcal{D}}[A(x; M_D, M_T)]

where D\mathcal{D} is the evaluation benchmark distribution, and the expectation is taken under the lossless speculative decoding constraint. This measures the average number of consecutive draft tokens accepted before rejection occurs across all prompts in the benchmark.

Why this metric: Acceptance length is a pure measure of how well the draft distribution aligns with the target distribution. It is invariant to the specific hardware, batching strategy, or systems implementation — two drafters with the same acceptance length on the same workload will, in principle, achieve the same speedup if all other factors (tree structure, verifier latency) are held constant. This makes it the right metric for isolating the effect of training data on draft quality, separate from systems-level concerns like memory bandwidth or kernel fusion that affect actual wall-clock speedup but confound comparisons of draft model quality.

EAGLE-2 Draft Model Backbone

EAGLE-2 is one of the two drafting architectures studied. The paper uses it in a fixed configuration throughout all experiments — only the training data varies. Understanding its internal mechanism is necessary to see why domain-specific training data matters: the drafter learns to predict the target model's internal representations, and those representations differ systematically between conversational and mathematical domains.

Feature-level drafting instead of token-level drafting. Standard speculative decoding (Leviathan et al., 2023; Chen et al., 2023) uses a draft model that generates tokens directly — it's essentially a smaller language model trained to mimic the target model's token distribution. EAGLE-2 takes a different approach: instead of predicting discrete tokens, it predicts the target model's hidden-state features at the next position, then maps those features to token probabilities through the target model's own LM head (the linear layer that converts hidden states to vocabulary logits). Let hth_t denote the target model's second-to-last-layer hidden state at step tt (the features just before the LM head). The draft model gϕg_{\phi} (parameterized by ϕ\phi) predicts:

h^t+1=gϕ(h^≤t,x≤t+1)\hat{h}_{t+1} = g_{\phi}(\hat{h}_{\leq t}, x_{\leq t+1})

where h^t+1\hat{h}_{t+1} is the predicted feature vector for position t+1t+1, h^≤t\hat{h}_{\leq t} denotes the (possibly predicted) features at all positions up to tt, and x≤t+1x_{\leq t+1} denotes the token sequence up to position t+1t+1.

What it computes: The draft model takes as input the target model's features from previous positions (or its own predicted features from those positions, depending on the stage of autoregressive drafting) plus the token sequence, and outputs a predicted feature vector for the next position. This predicted vector is then fed through the target model's frozen LM head WW to obtain token logits, which are converted to probabilities via softmax. In normal autoregressive decoding, the target model would compute ht+1h_{t+1} itself through multiple transformer layers; the draft model approximates this computation with a much cheaper function gϕg_{\phi} (since the draft model has only one transformer layer vs. the target model's 32 layers).

Why this form: Predicting features rather than tokens has several advantages. First, features live in a continuous vector space where small prediction errors degrade gracefully — if h^t+1\hat{h}_{t+1} is slightly off, the LM head's softmax may still assign high probability to the correct token. Token-level prediction, by contrast, is discrete and high-dimensional (vocabulary size ~128K), making it harder for a small model to calibrate correctly. Second, by using the target model's LM head, the draft model automatically inherits the target model's token embeddings, vocabulary mapping, and output structure — there is no need to learn a separate output layer or handle vocabulary mismatches. Third, the feature prediction task is a regression problem (predicting a real-valued vector) rather than a classification problem (predicting one of 128K tokens), which is computationally lighter and can be trained with simple squared-error loss.

Training objective. The EAGLE-2 draft model is trained with a combination of feature regression loss and cross-entropy loss:

LEAGLE=∑t∥h^t+1−ht+1∥22+λ∑tCE(softmax(Wh^t+1),xt+1)\mathcal{L}_{\text{EAGLE}} = \sum_{t} \|\hat{h}_{t+1} - h_{t+1}\|_2^2 + \lambda \sum_{t} \text{CE}(\text{softmax}(W \hat{h}_{t+1}), x_{t+1})

where ∥h^t+1−ht+1∥22\|\hat{h}_{t+1} - h_{t+1}\|_2^2 is the squared Euclidean distance between predicted and ground-truth target features (this is the feature regression term), CE(⋅,⋅)\text{CE}(\cdot, \cdot) is the cross-entropy loss between the token probabilities derived from h^t+1\hat{h}_{t+1} via the LM head and the ground-truth next token xt+1x_{t+1} (this is the token prediction term), and λ\lambda is a scalar weighting the two terms.

What it computes: The first term penalizes the draft model when its predicted feature vector deviates from the target model's actual feature vector — this is a direct regression on the representation the LM head expects. The second term penalizes the draft model when the token distribution derived from its predicted features assigns low probability to the correct next token — this is the standard language modeling loss. The sum runs over all positions in the training sequence. The combined objective encourages the draft model to produce features that are both close to the target's features (so the LM head receives familiar inputs) and predictive of the correct next token (so the output distribution is accurate).

Why this form: The two-term loss addresses a subtle alignment problem. Feature regression alone might produce features that are close to the target's but map to slightly wrong token distributions — the LM head is a linear map, and small feature errors in certain directions could produce large changes in logits for certain tokens. Cross-entropy loss alone might produce features that give the right token distribution but are far from the target's features — the draft model could learn to output feature vectors in a completely different subspace that the LM head maps to similar probabilities but that don't capture the target model's internal structure. By combining both, the draft model is pushed to produce features that are both accurate (close to the target's internal state) and predictive (yielding correct token probabilities). The paper's experiments use this objective unchanged; what varies is whether the training sequences come from MathInstruct or ShareGPT.

Dynamic tree construction. During inference, EAGLE-2 does not generate a single candidate sequence but a tree of candidate tokens, expanding from promising prefixes. The tree is built by scoring each frontier node (a token at the end of a partially built branch) by its draft confidence and expanding the highest-scoring nodes. Let cjc_j denote the draft model's confidence (probability) in token vjv_j at position jj along a path from the root. The score for a node viv_i is:

Vi≈∏vj∈Path(root,vi)cjV_i \approx \prod_{v_j \in \text{Path}(\text{root}, v_i)} c_j

where Path(root,vi)\text{Path}(\text{root}, v_i) is the sequence of nodes from the tree root to viv_i, and cjc_j is the draft model's predicted probability for the token at node vjv_j given the prefix up to that point.

What it computes: The score approximates the joint probability of the entire token sequence along the path — it is the product of conditional probabilities assigned by the draft model at each step. This represents how confident the draft model is that this particular continuation is correct. Nodes with higher scores represent paths the draft model considers more likely.

Why this form: The product form captures compounding uncertainty. A path where the draft model is 90% confident at 5 consecutive steps has score 0.95≈0.590.9^5 \approx 0.59, while a path where it is 60% confident has score 0.65≈0.0780.6^5 \approx 0.078. The product naturally penalizes paths that include even one low-confidence step, since a single uncertain token multiplies the score by a small value. This makes the scoring function prefer paths where the draft model is consistently confident — and since the verifier's acceptance rule rewards proposals that align with the target distribution, these high-confidence draft paths are precisely where acceptance is most likely. The expansion algorithm greedily grows the tree toward high-confidence regions of the proposal space, which is efficient because it allocates the tree's limited node budget to the most promising continuations.

Verification with EAGLE-2. When the verifier checks the draft tree, each drafted token x^j+i\hat{x}_{j+i} is accepted with probability:

αj+i=min⁡(1,pj+i(x^j+i)p^j+i(x^j+i))\alpha_{j+i} = \min\left(1, \frac{p_{j+i}(\hat{x}_{j+i})}{\hat{p}_{j+i}(\hat{x}_{j+i})}\right)

where pj+i(⋅)=q(⋅∣x1:j+i−1)p_{j+i}(\cdot) = q(\cdot \mid x_{1:j+i-1}) is the target model's distribution given the accepted prefix, and p^j+i(⋅)\hat{p}_{j+i}(\cdot) is the draft model's distribution derived from predicted features. This is the same acceptance rule as Equation 1, just expressed in the paper's notation for the specific case where the draft distribution comes from feature prediction.

HASS Draft Model Backbone

HASS is the second drafting architecture studied. Like EAGLE-2, it operates within the same lossless speculative decoding framework and is used in a fixed configuration with only training data varying. HASS introduces two innovations that address specific failure modes of feature-level drafting: objective mismatch (the training loss doesn't focus on the tokens the verifier is most likely to accept) and context mismatch (during training, the drafter sees clean target features everywhere, but during inference it must condition on its own imperfectly predicted features from earlier steps).

Top-K distillation loss. The first innovation addresses objective mismatch. Standard feature-level drafters are trained with losses (like EAGLE-2's) that treat all vocabulary tokens equally — either through regression on features or cross-entropy over the full vocabulary. However, in speculative decoding, what matters most is whether the drafter assigns high probability to the specific tokens the target model is most likely to generate. HASS's Top-K distillation loss focuses learning exclusively on those tokens:

LTop-K=−∑x∈Ω^q(x)log⁡p(x)\mathcal{L}_{\text{Top-K}} = -\sum_{x \in \hat{\Omega}} q(x) \log p(x)

where Ω^⊂Ω\hat{\Omega} \subset \Omega is the set of top-KK tokens under the target model's distribution q(⋅)q(\cdot), q(x)q(x) is the target model's probability for token xx (treated as a soft target), p(x)p(x) is the draft model's predicted probability for the same token, and KK is a hyperparameter (set to K=10K=10 in all HASS experiments).

What it computes: For each training position, the loss considers only the KK tokens the target model considers most likely (the top-KK set Ω^\hat{\Omega}). For each of those tokens, it computes the cross-entropy between the target model's probability q(x)q(x) (acting as a soft label) and the draft model's probability p(x)p(x). Tokens outside the top-KK set contribute zero to the loss, regardless of how poorly the draft model predicts them. The sum quantifies how much the draft model's distribution on these important tokens disagrees with the target's distribution.

Why this form: The top-KK truncation is both practical and principled. In a vocabulary of 128K tokens, the vast majority have near-zero probability under the target model. Training the draft model to accurately estimate the probability of these long-tail tokens wastes capacity and gradient signal — getting the probability of a token that goes from 0.0001 to 0.0003 correct barely affects acceptance length, because such tokens are neither proposed by the draft nor accepted by the verifier in practice. By focusing loss exclusively on the tokens the verifier might actually produce, HASS ensures the draft model's learning budget is spent on the part of the distribution that matters for acceptance. The use of q(x)q(x) as a soft target (rather than a hard one-hot label) preserves information about the target model's uncertainty among the top candidates — if the target model assigns 60% to one token and 40% to another, the draft model is encouraged to match this distribution rather than simply picking the argmax.

Harmonized context alignment. The second innovation addresses context mismatch. During standard training, the draft model at step t+1t+1 receives the target model's ground-truth features hth_t from step tt as input. During inference, however, the draft model must use its own predicted feature h^t\hat{h}_t from the previous step, because the target model hasn't processed position tt yet (the whole point of speculative decoding is to avoid running the target model at every step). This creates a train-test distribution shift: the draft model is trained to expect clean, accurate features as context, but at inference time it receives drifted, imperfect features that compound error across multiple drafting steps.

HASS addresses this by training the draft model to handle its own predicted features through a multi-step alignment procedure. At alignment step jj (where jj indexes the position within the draft sequence), the draft model predicts:

P(s)(xt+1∣x≤t)=Head(ft+1(sj))=Head(M(s)(ft(sj−1),f1(l)⊕⋯⊕ft−j+1(l)⊕ft−j+2(s1)⊕⋯⊕ft(sj−1)))P^{(s)}(x_{t+1} \mid x_{\leq t}) = \text{Head}(f^{(s_j)}_{t+1}) = \text{Head}\left(M^{(s)}\left(f^{(s_{j-1})}_t, f^{(l)}_1 \oplus \dots \oplus f^{(l)}_{t-j+1} \oplus f^{(s_1)}_{t-j+2} \oplus \dots \oplus f^{(s_{j-1})}_t\right)\right)

where P(s)(xt+1∣x≤t)P^{(s)}(x_{t+1} \mid x_{\leq t}) is the draft model's predicted token distribution at position t+1t+1, Head\text{Head} is the LM head, ft+1(sj)f^{(s_j)}_{t+1} is the draft model's internal feature vector at alignment step jj, M(s)M^{(s)} is the draft model's transformer function, f(l)f^{(l)} denotes features from the target model (the "clean" large-model features), f(s)f^{(s)} denotes features from the draft model itself (the "draft" features), and ⊕\oplus denotes concatenation.

What it computes: The draft model constructs its internal state by mixing clean target features from the more distant past with its own predicted features from recent steps. The input to the draft model at step jj consists of: (1) target features f(l)f^{(l)} for positions far enough in the past that the target model has already computed them (up to position t−j+1t-j+1), and (2) draft features f(s)f^{(s)} for the more recent positions t−j+2t-j+2 through tt, which represent the draft model's own predicted features from earlier alignment steps. At alignment step j=1j=1, the draft model receives only clean target features (since f(s)f^{(s)} hasn't been generated yet). At step j=2j=2, it receives clean features for most positions but draft features for the immediately preceding position. At step j=3j=3, it receives draft features for two preceding positions, and so on.

Why this form: This is a curriculum that gradually exposes the draft model to its own imperfect features. The draft model learns to make predictions not from an idealized clean history but from a mixed history where recent context comes from its own (potentially drifted) outputs. This mirrors the actual inference-time condition: when drafting KK tokens ahead, the first token can condition on clean target features, but the second must condition on the draft-predicted feature for the first token, the third on draft-predicted features for the first two, and so on. By training the draft model to handle this progressive context drift, HASS reduces the performance gap between training and inference — the draft model has already seen (and learned to cope with) the kind of corrupted context it will encounter during deployment.

Full training objective. The complete HASS training loss at alignment step jj is:

LHASS(j)=∑t=1T−1[CE(P(l)(xt+1∣x≤t),P(s)(xt+1∣x≤t))+Laux]\mathcal{L}^{(j)}_{\text{HASS}} = \sum_{t=1}^{T-1} \left[\text{CE}\left(P^{(l)}(x_{t+1} \mid x_{\leq t}), P^{(s)}(x_{t+1} \mid x_{\leq t})\right) + \mathcal{L}_{\text{aux}}\right]

where P(l)(xt+1∣x≤t)P^{(l)}(x_{t+1} \mid x_{\leq t}) is the target model's token distribution, P(s)(xt+1∣x≤t)P^{(s)}(x_{t+1} \mid x_{\leq t}) is the draft model's predicted distribution from the harmonized context alignment, CE(⋅,⋅)\text{CE}(\cdot, \cdot) is the cross-entropy between these two distributions, and Laux\mathcal{L}_{\text{aux}} is an auxiliary loss that includes the Top-K distillation term and a feature regression term (similar to EAGLE-2's combined loss, but with the top-KK focused distillation).

What it computes: For each alignment step jj, the loss encourages the draft model to match the target model's output distribution when conditioned on a progressively more draft-dependent context. The cross-entropy term measures distributional mismatch between target and draft. The auxiliary term Laux\mathcal{L}_{\text{aux}} adds the Top-K distillation and feature regression components, ensuring the draft model not only matches the output distribution but also learns features that are structurally aligned with the target model's internal representations.

Why this form: The multi-step training procedure directly addresses the train-test distribution gap that plagues standard feature-level drafters. A single-step training objective would train the draft model only on clean target features, meaning the model never learns to handle the drifted context that causes catastrophic error accumulation during long draft sequences. By training across multiple alignment steps (the paper uses three forward-alignment steps), HASS learns to produce features that remain useful even when conditioned on earlier imperfect draft predictions. This is a form of scheduled sampling for feature prediction — the model sees increasing fractions of its own predictions during training, preparing it for the entirely self-conditioned regime of inference.

Configuration in this paper's experiments. All HASS runs use the same auxiliary settings throughout: top-K distillation with K=10K = 10, loss weight 1.01.0, and three forward-alignment steps. The draft model is a lightweight LLaMA-style decoder with one transformer layer, hidden size 4096, and roughly 0.8B parameters — matching the EAGLE-2 draft model size so that differences in acceptance length reflect training distribution effects rather than model capacity differences.

Training Data Configurations and Controlled Experimental Setup

The paper's core experimental design isolates the effect of draft training distribution by holding everything else constant. This section details what is varied and what is fixed, establishing the causal identification strategy.

Verifier (fixed across all experiments). The target model is Meta-Llama-3-8B-Instruct, an 8-billion-parameter instruction-tuned language model. It processes all verification steps and defines the ground-truth output distribution through the speculative acceptance rule. The verifier never changes — not its weights, not its architecture, not its decoding parameters — so any measured differences in acceptance length across draft variants are attributable purely to differences in draft quality, not to differences in how the verifier evaluates proposals.

Draft model architecture and scale (fixed across all experiments). Both the EAGLE-2 and HASS drafters use the identical base architecture: a lightweight LLaMA-style decoder with one transformer layer, hidden size 4096, and approximately 0.8 billion parameters. The draft model shares the verifier's tokenizer and vocabulary, eliminating tokenization mismatch as a potential confound — every token the draft proposes is one the verifier can process. Draft model size is deliberately small (roughly 10% of the verifier's parameters, assuming Llama-3-8B has ~30-32 layers) to represent the practical deployment regime where the drafter must be substantially cheaper to run than the verifier for speculative decoding to provide net throughput benefits.

Training hyperparameters (fixed across all experiments). All draft checkpoints are trained for 20 epochs with learning rate 3×10−53 \times 10^{-5}, batch size 8, and gradient accumulation 1. The paper uses these same settings for every training run — single-domain MathInstruct, single-domain ShareGPT, Mixed 35k+35k, and Mixed 70k+70k — so differences in final draft quality cannot be attributed to differences in optimization budget or hyperparameter tuning. The consistent 20-epoch training also means that any differences in convergence rate across domains are embedded in the final checkpoint quality; the paper does not claim that 20 epochs is optimal for each domain, only that it is held constant.

Single-domain checkpoints (RQ1). Two draft models are trained, each on a single homogeneous data source:

  • MathInstruct: 70,000 examples of mathematical reasoning problems with step-by-step solutions, covering arithmetic, algebra, geometry, and other mathematical domains.
  • ShareGPT: 70,000 examples of conversational exchanges, representing the generic chat domain that most prior speculative decoding work uses as the default draft training corpus.

Both are trained for 20 epochs on their respective 70K examples. These checkpoints answer RQ1: does a domain-matched draft model outperform a domain-mismatched draft model on its target domain?

Mixed-data checkpoints (RQ2). Two additional draft models are trained on mixtures of MathInstruct and ShareGPT data, to test whether a single checkpoint can achieve cross-domain robustness:

  • Mixed 35k+35k: 35,000 examples from MathInstruct plus 35,000 examples from ShareGPT, for a total of 70,000 training examples — the same total volume as the single-domain checkpoints but balanced across domains.
  • Mixed 70k+70k: 70,000 examples from each domain, for a total of 140,000 training examples — twice the total volume, testing whether additional data from both domains improves generalization.

These checkpoints answer RQ2: can mixed-data training produce a single robust draft model that performs well on both conversational and mathematical reasoning workloads, or does mixing dilute domain specialization?

Evaluation benchmarks and temperatures. The paper evaluates on four benchmarks chosen to span the conversational-to-mathematical spectrum:

  • MT-Bench: A multi-turn conversational benchmark with 80 questions covering diverse topics. Represents the ShareGPT-style chat domain.
  • GSM8K: Grade-school math word problems requiring multi-step arithmetic reasoning. 1,319 test examples.
  • MATH-500: A 500-question subset of the MATH benchmark covering competition-level mathematics.
  • SVAMP: Math word problems with varying structures, designed to test robustness to surface-level variations. 300 test examples.

All benchmarks are evaluated at two temperatures: temperature=0\text{temperature} = 0 (greedy-like decoding where the draft model's sampling is deterministic) and temperature=1\text{temperature} = 1 (stochastic sampling from the full distribution). Temperature affects both the draft model's proposal distribution and the acceptance dynamics — at temperature 1, the draft model explores more diverse tokens, which may increase or decrease acceptance length depending on domain.

Experimental hardware. All training and evaluation experiments run on a single node with four NVIDIA A100 GPUs. This is a practical constraint — the setup is reproducible on standard academic compute resources — and underscores that the paper's findings are accessible without datacenter-scale infrastructure.

What does not vary. Critically, the paper does not experiment with: different verifier models (only Llama-3-8B-Instruct), different draft model sizes (only ~0.8B parameters, one layer), different numbers of forward-alignment steps in HASS (only three steps), different values of KK for Top-K distillation (only K=10K=10), or different training epochs. This narrow scope is a deliberate choice: it ensures that when the paper attributes acceptance length differences to training data domain, there are no alternative explanations from architectural or hyperparameter variation. The trade-off is that the findings may not generalize to substantially different verifier models, draft model scales, or hyperparameter regimes — a limitation the paper acknowledges implicitly by its scope.

Composition Strategy 1: Checkpoint Weight Averaging

When multiple specialized draft models are available (MathInstruct-trained and ShareGPT-trained), the simplest way to combine them is to average their parameters in weight space, producing a single merged checkpoint. The paper tests this as a baseline because weight averaging has been shown effective in other contexts (model merging for multi-task learning, Ilharco et al., 2022) and is attractive for deployment — it requires no additional inference-time logic, no routing decisions, and no increased memory footprint beyond a single model.

Formal definition. Let θmath\theta_{\text{math}} and θchat\theta_{\text{chat}} denote the parameter vectors (all weights and biases) of the MathInstruct-trained and ShareGPT-trained draft models, respectively. The merged checkpoint is defined by point-wise linear interpolation:

θmerge=λθmath+(1−λ)θchat\theta_{\text{merge}} = \lambda \theta_{\text{math}} + (1 - \lambda) \theta_{\text{chat}}

where λ∈[0,1]\lambda \in [0, 1] controls the contribution of each checkpoint. Each scalar parameter in θmerge\theta_{\text{merge}} is the weighted average of the corresponding parameters in θmath\theta_{\text{math}} and θchat\theta_{\text{chat}}.

What it computes: For every weight matrix, bias vector, and normalization parameter in the draft model, the merged value is a convex combination of the two specialized checkpoints' values at that position. When λ=0.5\lambda = 0.5, the merged parameter is the arithmetic mean of the two specialized values. When λ=0\lambda = 0, the merged model is exactly the ShareGPT checkpoint; when λ=1\lambda = 1, it is exactly the MathInstruct checkpoint.

Why this form: Linear interpolation in weight space is the most direct way to combine two models that share the same architecture and initialization. If the two specialized models have learned complementary features — with different neurons or attention heads specializing in different domains — averaging their parameters might preserve a blend of both specializations, similar to how averaging independently trained models can improve generalization. The convex combination ensures the merged parameters remain in a region of weight space that is plausibly close to a valid model configuration (since both endpoints are valid).

Configuration: The paper uses λ=0.5\lambda = 0.5 for the main results in Table 1 and sweeps λ\lambda across its full range in Figure 6 to assess sensitivity. The averaging is performed element-wise across all trainable parameters; there is no per-layer weighting, no task-vector arithmetic, and no learned merging coefficients.

Key result preview (why this matters for the technical approach): The averaged checkpoints are consistently the weakest variants in Table 1, with average acceptance lengths between 2.34 and 2.62 across methods and temperatures. Figure 6 shows that no interpolation weight λ\lambda recovers performance close to either specialist or to inference-time composition methods. This failure is informative: it suggests that the two specialists have learned fundamentally different internal representations that do not compose linearly. The features that make MathInstruct good at reasoning and ShareGPT good at chat are not additive — averaging them produces a model that is mediocre at both rather than competent at both. This motivates the inference-time composition strategies, which avoid weight-space interference entirely by keeping the specialists separate.

Composition Strategy 2: Confidence-Based Routing

Instead of merging specialists in weight space, confidence-based routing keeps both checkpoints intact and selects which one to use for each incoming prompt based on a simple criterion: which drafter is more confident in its proposed tokens?

Procedure. Given an input prefix (the prompt to be completed), the system decodes one draft tree from the MathInstruct checkpoint and one from the ShareGPT checkpoint. Each tree contains candidate tokens at various depths, with each node annotated by the draft model's confidence (predicted probability) for that token given its path prefix. The system scores each tree by its mean node confidence. Let Tmath\mathcal{T}_{\text{math}} and Tchat\mathcal{T}_{\text{chat}} denote the two draft trees, and let c(v)c(v) denote the confidence (probability) assigned to node v∈Tv \in \mathcal{T} by that tree's draft model. The tree-level score is:

Score(T)=1∣T∣∑v∈Tc(v)\text{Score}(\mathcal{T}) = \frac{1}{|\mathcal{T}|} \sum_{v \in \mathcal{T}} c(v)

where ∣T∣|\mathcal{T}| is the number of nodes in the tree. The selected tree is:

T∗=arg max⁡T∈{Tmath,Tchat}Score(T)\mathcal{T}^* = \argmax_{\mathcal{T} \in \{\mathcal{T}_{\text{math}}, \mathcal{T}_{\text{chat}}\}} \text{Score}(\mathcal{T})

Only T∗\mathcal{T}^* is passed to the verifier; the other tree is discarded.

What it computes: The score is the arithmetic mean of the draft model's predicted token probabilities across all nodes in its generated tree. A high mean confidence means the draft model is consistently assigning high probability to its proposed tokens — it "believes" its proposals are good. A low mean confidence means the draft model is uncertain, assigning moderate or low probabilities even to the tokens it chose to propose. The argmax simply picks whichever specialist is more confident about its tree on this particular prompt.

Why this form (and why confidence over entropy): The intuition is that a draft model's confidence should correlate with its domain expertise. If the prompt is a math word problem, the MathInstruct drafter — which has seen thousands of similar problems — should generate tokens with high confidence because the patterns are familiar. The ShareGPT drafter — which has mainly seen conversations — should generate tokens with lower confidence because it's in unfamiliar territory. The mean confidence aggregates this signal across the entire tree, smoothing out noise from individual token-level fluctuations. The paper's results in Table 2 validate this intuition: under confidence routing, the MathInstruct drafter is selected for 90.8% of GSM8K, 97.0% of MATH-500, and 93.0% of SVAMP prompts, while ShareGPT is selected for 81.2% of MT-Bench prompts. The separation is clear and aligns with domain boundaries.

The paper compares confidence routing against entropy-based routing (also in Table 2) and finds entropy is far less discriminative — it produces near-balanced splits across all benchmarks (e.g., 54.6%/45.4% Math/Share split on GSM8K). This is because entropy captures uncertainty (how spread-out the probability distribution is) rather than peak confidence (how likely the model thinks its top choice is). A model can be uncertain (high entropy) while still picking the mathematically correct token — uncertainty is a property of the distribution, not of correctness. Confidence directly measures the model's estimate of correctness for its chosen token, making it a better proxy for domain match.

Computational cost. Confidence routing requires generating two draft trees (one from each specialist) for every prompt, but only one is verified. This doubles the draft-model computation relative to using a single specialist, but the draft model is lightweight (~0.8B parameters, one layer), so this cost is small compared to the verifier's forward pass. The paper reports that confidence routing reduces average speedup by 0.32×–0.47× relative to the strongest single checkpoint (depending on backbone and temperature), but notes that in a deployment serving two distinct task families, this overhead may be partly offset when the best single checkpoint is weak on one of the tasks.

Correctness guarantee. Proposition A.1 in the appendix establishes that confidence routing preserves the target model's output distribution exactly. The routing decision depends only on draft-side quantities (confidences) computed before verification. The selected tree T∗\mathcal{T}^* is a random valid tree (a function of the two draft trees, which are themselves generated from the prefix). Since the speculative acceptance rule applied to any valid tree preserves the target distribution, and T∗\mathcal{T}^* is such a tree, the combined routing + verification procedure is lossless.

Composition Strategy 3: Merged-Tree Verification

Instead of selecting one specialist's tree and discarding the other (as in routing), merged-tree verification processes both trees simultaneously, giving the verifier a broader set of candidate tokens to choose from. The key insight is that the verifier can evaluate proposals from multiple drafters in a single parallel forward pass if the trees are packed appropriately.

Merging procedure. Given the two draft trees Tmath\mathcal{T}_{\text{math}} and Tchat\mathcal{T}_{\text{chat}} generated from the same root token (the last accepted token of the prefix), the system:

  1. Shares the root: Both trees start from the same root node, which represents the current accepted prefix.

  2. Concatenates subtrees: The non-root nodes of both trees are concatenated into a single flat sequence of tokens for the verifier to process. Nodes from Tmath\mathcal{T}_{\text{math}} occupy the first block of indices (after the root), and nodes from Tchat\mathcal{T}_{\text{chat}} occupy the second block.

  3. Builds ancestor-preserving attention masks: The critical constraint is that nodes within each subtree must attend only to their own ancestors (plus the shared root), not to nodes from the other subtree. The attention mask is a binary matrix where entry (i,j)(i, j) is 1 if node ii is allowed to attend to node jj, and 0 otherwise. The mask is constructed so that: (a) the root attends to itself only (it has no ancestors), (b) every node in Tmath\mathcal{T}_{\text{math}} attends to the root and its own ancestors within Tmath\mathcal{T}_{\text{math}} (as defined by the tree structure), but not to any node in Tchat\mathcal{T}_{\text{chat}}, and (c) symmetrically, every node in Tchat\mathcal{T}_{\text{chat}} attends to the root and its own ancestors within Tchat\mathcal{T}_{\text{chat}}, but not to any node in Tmath\mathcal{T}_{\text{math}}. Cross-subtree attention is completely masked.

  4. Assigns depth-based position IDs: Position embeddings encode each token's depth in its respective tree rather than its sequence position. The root has position 0. A direct child of the root has position 1, its child has position 2, and so on. This means that two nodes at the same depth in different subtrees receive the same position ID, which is correct because they represent tokens at the same speculative step.

  5. Verifies in one pass: The merged tree (root + concatenated subtree nodes) with the constructed attention mask and position IDs is fed through the verifier in a single forward pass. The verifier computes logits for every node in the merged tree, conditional on the appropriate ancestors (as governed by the mask).

  6. Extracts candidates and applies acceptance: After verification, candidate paths are extracted from the merged tree in the standard speculative decoding manner. The acceptance rule is applied sequentially along each path, starting from the root and proceeding depth by depth.

What it computes: The merged tree is functionally equivalent to running the verifier separately on Tmath\mathcal{T}_{\text{math}} and Tchat\mathcal{T}_{\text{chat}} and then combining the results — but it does so in one verifier forward pass. Each node in Tmath\mathcal{T}_{\text{math}} receives exactly the same verifier logits as it would under standalone verification, because its attention is restricted to the same ancestors. The same holds for Tchat\mathcal{T}_{\text{chat}}. The verifier's forward pass is parallel across all nodes, so processing a merged tree of size ∣Tmath∣+∣Tchat∣+1|\mathcal{T}_{\text{math}}| + |\mathcal{T}_{\text{chat}}| + 1 (root) costs roughly the same as processing a single tree of that size — the computational cost is in the attention mechanism, which scales quadratically in the total number of nodes, not linearly with the number of trees.

Why this form: The merged tree increases proposal diversity at each speculative step. Instead of the verifier having to choose among candidates from a single specialist, it sees candidates from both — if the MathInstruct drafter is strong at mathematical reasoning tokens and the ShareGPT drafter is strong at natural language tokens, the merged tree contains both types of proposals. The verifier's parallel evaluation means this diversity comes at minimal additional latency cost (modulo the increased tree size). This is fundamentally different from generating a larger tree from a single drafter, because a single drafter's proposals are correlated — the tokens it considers promising are drawn from its own (potentially narrow) distribution. The merged tree introduces qualitatively different alternatives that a single drafter would never consider because they fall outside its domain of expertise.

The paper's tree-merging implementation uses a specific utility function _merge_trees (shown in Appendix A.3) that handles index remapping, attention mask construction, and position ID assignment. The function takes the token sequences, tree attention masks, position IDs, and retrieval indices (used for extracting candidate paths during verification) from both trees, computes the appropriate offsets and concatenations, and outputs a merged representation ready for verifier processing.

Computational cost considerations. The merged tree is larger than either individual tree, which increases the verifier's attention computation (quadratic in tree size). The paper reports that merged-tree verification incurs a speedup reduction of 0.59×–0.78× relative to the strongest single checkpoint, which is larger than the routing overhead but comes with higher acceptance length. The paper explicitly does not claim an end-to-end latency improvement for merged-tree verification without a separate systems analysis — the acceptance length benefit is established, but whether this translates to wall-clock speedup depends on the trade-off between higher acceptance (fewer verifier calls) and larger per-call trees (more expensive verifier calls). This is flagged as an area for future work.

Correctness guarantee. Proposition A.2 in the appendix establishes that merged-tree verification preserves the target model's output distribution. The proof relies on Lemma A.2 (verifier invariance under masked concatenation), which states that if two packed inputs agree on a subset of indices (tokens, positions, masks) and tokens in that subset do not attend outside it, then the verifier's logits on those indices are identical. The merged tree satisfies these conditions for each subtree separately: nodes from Tmath\mathcal{T}_{\text{math}} have the same tokens, depth-based positions, and within-subtree attention as in standalone verification, and they do not attend to Tchat\mathcal{T}_{\text{chat}} nodes. Therefore, the verifier logits on Tmath\mathcal{T}_{\text{math}} nodes are identical to standalone verification. The same holds for Tchat\mathcal{T}_{\text{chat}}. Since the merged tree is a valid tree in the sense of Assumption A.1 (every node receives the correct target-side conditional distribution), applying the standard speculative acceptance rule to it preserves the target distribution.

Distinction from routing. The key difference between merged-tree verification and confidence routing is what information the verifier receives. In routing, the verifier sees proposals from only one specialist — the one that was more confident. In merged-tree verification, the verifier sees proposals from both. This means merged-tree verification allows the verifier to accept tokens from either specialist at any depth, potentially mixing proposals: for example, the verifier might accept the first two tokens from the MathInstruct subtree and then switch to the ShareGPT subtree's candidate for the third token if the third MathInstruct token gets rejected but the third ShareGPT token is accepted. Routing forecloses this possibility — once a specialist is selected, only its proposals are available for the entire tree depth.

This explains why merged-tree verification achieves higher acceptance length than routing (Table 1: 5.11 vs. 4.80 for HASS at temperature 0; 5.03 vs. 4.63 for EAGLE-2). The merged tree gives the verifier more options at every speculative step, increasing the probability that at least one candidate aligns with the target distribution. The trade-off is increased per-call computation (larger tree), which the paper quantifies but does not fully resolve in terms of end-to-end latency.

Composition Strategy Correctness: Why Routing and Merged-Tree Verification Are Provably Lossless

The paper includes formal correctness arguments (Appendix A.4) establishing that both inference-time composition strategies preserve the target model's output distribution. These are important because speculative decoding's key appeal is its losslessness — any composition strategy that violated this property would be unacceptable regardless of its acceptance length benefits.

Foundation: Lossless verification for a fixed valid tree (Assumption A.1). A packed tree T\mathcal{T} is considered "valid" if the verifier pass on T\mathcal{T} produces, for every node, the same target-side conditional distributions q(⋅∣its path-prefix)q(\cdot \mid \text{its path-prefix}) that the target model would produce under standalone autoregressive evaluation along that node's path. For every valid tree, the speculative decoding procedure is guaranteed to produce continuations distributed according to the target model:

Pr⁡(Dec(y1:t;T)∈B∣y1:t,T)=Q(B∣y1:t)\Pr(\text{Dec}(y_{1:t}; \mathcal{T}) \in B \mid y_{1:t}, \mathcal{T}) = Q(B \mid y_{1:t})

for every measurable set of continuations BB, where QQ is the target model's distribution.

Lemma A.1 (Mixtures over valid trees remain lossless). If T\mathcal{T} is any random valid tree — possibly generated by different draft model(s) using a stochastic procedure — then unconditioning preserves the target distribution:

Pr⁡(Dec(y1:t;T)∈B∣y1:t)=E[Pr⁡(Dec∈B∣y1:t,T)∣y1:t]=E[Q(B∣y1:t)∣y1:t]=Q(B∣y1:t)\Pr(\text{Dec}(y_{1:t}; \mathcal{T}) \in B \mid y_{1:t}) = \mathbb{E}[\Pr(\text{Dec} \in B \mid y_{1:t}, \mathcal{T}) \mid y_{1:t}] = \mathbb{E}[Q(B \mid y_{1:t}) \mid y_{1:t}] = Q(B \mid y_{1:t})

This follows from the tower property of expectation. The key insight is that as long as each possible tree the procedure might select is valid (in the sense of Assumption A.1), the fact that which tree is selected depends on the input does not break losslessness — the expectation over tree randomness preserves the target distribution.

Proposition A.1 (Correctness of routing, reproduced). Let Tmath\mathcal{T}_{\text{math}} and Tchat\mathcal{T}_{\text{chat}} be valid draft trees from the same prefix. Let gg be any routing rule that depends only on draft-side quantities (confidences, entropies, tree statistics) computed before verification, and define T∗=Tg(y1:t,Tmath,Tchat)\mathcal{T}^* = \mathcal{T}_{g(y_{1:t}, \mathcal{T}_{\text{math}}, \mathcal{T}_{\text{chat}})}. Then:

Pr⁡(Dec(y1:t;T∗)∈B∣y1:t)=Q(B∣y1:t)\Pr(\text{Dec}(y_{1:t}; \mathcal{T}^*) \in B \mid y_{1:t}) = Q(B \mid y_{1:t})

Proof sketch: T∗\mathcal{T}^* is a random valid tree — it is one of the two valid input trees, selected by a draft-side function. By Lemma A.1, any random valid tree preserves the target distribution. The critical condition is that gg depends only on draft-side quantities — if gg used verifier logits (which would require running the verifier first), the tree would not be "pre-verification valid" and the lemma would not apply. Confidence and entropy are draft-side quantities, satisfying this condition.

Proposition A.2 (Correctness of merged-tree verification, reproduced). Let Tmath\mathcal{T}_{\text{math}} and Tchat\mathcal{T}_{\text{chat}} be valid trees. Construct the merged tree T∪\mathcal{T}_{\cup} by sharing the root, concatenating non-root nodes, and using attention masks that preserve each subtree's internal ancestry while preventing cross-subtree attention. Then every node in the merged verifier pass receives the same target-side conditional as in standalone verification of its source subtree, and:

Pr⁡(Dec(y1:t;T∪)∈B∣y1:t)=Q(B∣y1:t)\Pr(\text{Dec}(y_{1:t}; \mathcal{T}_{\cup}) \in B \mid y_{1:t}) = Q(B \mid y_{1:t})

Proof sketch: The proof rests on Lemma A.2 (verifier invariance under masked concatenation). For each subtree, the merged packing agrees with standalone packing on tokens, depth-based positions, and within-subtree attention. Cross-subtree attention is masked, so nodes in each subtree do not attend outside their subtree. By Lemma A.2, the verifier logits on all nodes match standalone verification. Therefore T∪\mathcal{T}_{\cup} is a valid tree, and the standard speculative acceptance rule preserves the target distribution.

Practical significance. These correctness guarantees mean that the paper's acceptance length comparisons are between methods that all produce identical output quality (the target model's distribution). The differences in acceptance length are pure measures of draft-to-target alignment under different training and composition strategies, not confounded by quality degradation. A practitioner can confidently adopt any of these composition strategies and know that the generated text will be exactly what the verifier would have produced autoregressively — the only question is how much speedup they achieve.

Summary of Design Choices and Their Justifications

The paper's technical approach is characterized by deliberate constraints that isolate the effect of draft training distribution:

  • Fixed verifier (Llama-3-8B-Instruct) throughout: Ensures acceptance length differences reflect draft quality, not verifier variation. The choice of an 8B instruction-tuned model is representative of production LLM deployments.

  • Identical draft architecture for EAGLE-2 and HASS: Eliminates confounds from model capacity differences. Both use one transformer layer, hidden size 4096, ~0.8B parameters, shared tokenizer/vocabulary with verifier.

  • Consistent training hyperparameters across all data variants: 20 epochs, learning rate 3×10−53 \times 10^{-5}, batch size 8, gradient accumulation 1. Prevents optimization differences from explaining domain effects.

  • Two single-domain, two mixed-data, three composition variants per backbone: Creates a factorial design that cleanly separates the effects of domain matching (RQ1), data mixing (RQ2), and composition strategy (RQ3).

  • Confidence routing uses mean node confidence rather than more complex signals: Keeps the routing policy interpretable and demonstrates that a simple draft-side statistic suffices for effective routing — no learned router, no verifier feedback, no cost modeling.

  • Merged-tree verification uses ancestor-preserving masking rather than letting subtrees cross-attend: This is the key to maintaining correctness — cross-attention would violate the valid-tree condition because it would change the contextual information each node sees, altering the target-side conditionals.

  • Formal correctness proofs for both inference-time composition strategies: Establishes that the lossless guarantee of speculative decoding is preserved, which is necessary for practical deployment where output quality parity with the target model is non-negotiable.

  • Single-GPU-node experimental setup: Demonstrates reproducibility on standard academic hardware and makes the findings accessible without requiring datacenter-scale infrastructure.

4. Key Insights and Innovations

Innovation 1: Draft Training Distribution as a First-Class Design Variable, Not an Implementation Detail

Prior speculative decoding research has treated the draft model's training distribution as a fixed constant — nearly all work uses ShareGPT or an equivalent generic conversational corpus, and the implicit assumption is that any reasonably broad dataset will produce an adequate proposal distribution. The focus has been entirely on how the draft model processes information (architecture: EAGLE, HASS; tree construction: dynamic vs. static; verification: cascaded, hierarchical, retrieval-assisted) rather than what information it processes. This paper's fundamental reframing is to treat the draft training distribution as an independent axis of speculative decoding quality, on par with draft architecture in determining deployment performance.

What distinguishes this from "just picking a good dataset": The paper provides the first controlled evidence that domain matching produces large, systematic, and architecturally consistent effects on acceptance length. Under HASS at temperature 0, the MathInstruct drafter achieves 5.35 acceptance length on MATH-500 but only 2.90 on MT-Bench — a 46% drop from deploying the wrong specialist (Table 1). The ShareGPT drafter shows the inverse pattern: 3.98 on MT-Bench vs. 3.98 on MATH-500. These are not small, noisy effects at the margins; they represent the difference between a speculative decoding deployment that delivers on its throughput promises and one that falls substantially short. The fact that the pattern replicates across both EAGLE-2 and HASS — architectures with fundamentally different training objectives and context alignment strategies — demonstrates that this is a property of speculative decoding itself, not an artifact of a particular drafting approach.

The significance extends beyond the empirical finding. By establishing that draft training data matters independently of draft architecture, the paper opens a new dimension for speculative decoding research. Future work on improving speculative decoding can now explore the joint optimization of architecture and training data, rather than treating the latter as a constant. This is analogous to how the pretraining literature evolved from focusing purely on model architecture to jointly optimizing architecture, data quantity, and data quality (Hoffmann et al., 2022). The paper effectively argues that speculative decoding needs a similar broadening of its optimization space.

Innovation 2: Weight-Space Averaging as a Diagnostic for Representation Incompatibility Between Domain Specialists

Weight averaging — merging the parameters of two fine-tuned models through linear interpolation — has been shown effective for multi-task learning and model editing in other contexts (Ilharco et al., 2022). The natural expectation would be that averaging a MathInstruct drafter and a ShareGPT drafter produces a model that is moderately competent at both domains, trading peak performance for robustness. The paper shows this expectation is dramatically wrong: across both backbones and both temperatures, the averaged checkpoint is the weakest variant in Table 1, with average acceptance lengths between 2.34 and 2.62 — worse than either individual specialist and substantially worse than mixed-data training or inference-time composition.

Why this is a genuine conceptual contribution: The failure of weight averaging is not just a negative result — it is a diagnostic that reveals something about the structure of the learned draft representations. If the two specialists had learned complementary features that could be linearly combined (e.g., different attention heads specializing in different domains), averaging would preserve some degree of dual-domain competence. The near-total collapse in performance suggests instead that the two specialists have learned fundamentally incompatible internal representations — the features that make MathInstruct good at reasoning and ShareGPT good at chat are not additive, and their average produces a model that is mediocre at both rather than competent at either. Figure 6 reinforces this: interpolating between the two checkpoints produces unstable behavior across the entire weight spectrum, with no interpolation coefficient recovering performance close to either specialist or to inference-time composition.

This finding has implications beyond speculative decoding. It suggests that naively averaging models fine-tuned on qualitatively different data distributions is a risky strategy in general — the approach that works for merging models fine-tuned on similar tasks (as in Ilharco et al., 2022) may fail catastrophically when the domains are more sharply distinct, as they are between conversational chat and mathematical reasoning. The paper effectively provides a counterexample that delineates the boundary conditions for weight-space merging: it works when the fine-tuning tasks share enough underlying structure that their parameter updates lie in a compatible subspace, and it fails when the updates push the model into fundamentally different regions of weight space that do not combine linearly.

The methodological move here is notable: the paper uses weight averaging not as a proposed solution but as a diagnostic baseline whose failure motivates and justifies the inference-time composition strategies that follow. This is a cleaner experimental design than simply proposing inference-time composition without a strong baseline — the failure of the simplest alternative strengthens the case that inference-time composition is genuinely necessary, not merely convenient.

Innovation 3: Confidence as a Domain-Match Signal That Enables Effective Zero-Shot Specialist Routing

The paper identifies that draft model confidence — the predicted probability assigned to generated tokens — serves as an effective zero-shot signal for determining which specialist drafter should handle an incoming prompt. Under confidence routing, the MathInstruct drafter is selected for 90.8% of GSM8K, 97.0% of MATH-500, and 93.0% of SVAMP prompts, while the ShareGPT drafter is selected for 81.2% of MT-Bench prompts (Table 2). This alignment between confidence and domain is not engineered — the routing policy uses no domain labels, no prompt classifier, and no learned routing model. It simply selects whichever specialist produces higher mean confidence in its draft tree.

What makes this non-obvious: The paper compares confidence routing against entropy-based routing and finds that entropy is far less discriminative (Table 2: near-balanced 52.5%/47.5% splits on MT-Bench, 54.6%/45.4% on GSM8K). This distinction is instructive. Entropy measures uncertainty — how spread-out the probability distribution is across all tokens — while confidence measures the model's probability estimate for its chosen token. A model can be uncertain (high entropy) about a prompt while still being the better specialist for it, if the better specialist is also uncertain because the prompt is genuinely ambiguous. Conversely, a model can be confident about the wrong answer if it's unfamiliar with the domain and overestimates the plausibility of plausible-sounding but incorrect completions. The fact that confidence — not entropy — cleanly separates domains suggests that the draft models have learned calibrated uncertainty estimates that are domain-dependent: a specialist is confident when it recognizes patterns from its training distribution, and this confidence signal is a more reliable proxy for domain match than overall distribution shape.

The significance is practical and conceptual. From a practical standpoint, confidence routing provides a simple, interpretable, and provably lossless (Proposition A.1) mechanism for deploying heterogeneous drafters in production. The system needs no domain classifier, no prompt metadata, and no verifier feedback to make the routing decision — it works purely from draft-side signals computed before verification, which is critical for the correctness guarantee. From a conceptual standpoint, the paper demonstrates that draft model confidence — typically used only for tree construction within a single drafter — encodes information about domain match that can be exploited for system-level routing decisions. This connects the internal representation of uncertainty in draft models to the external task of workload distribution, opening a bridge between model calibration research and inference systems design.

Innovation 4: Merged-Tree Verification as a Mechanism for Lossless Ensemble Speculative Decoding

The paper introduces merged-tree verification: a technique that packs draft trees from multiple specialists under a shared root, with ancestor-preserving attention masks that prevent cross-subtree information leakage, and verifies them jointly in a single verifier forward pass. This achieves the highest acceptance lengths in Table 1 (5.11 for HASS, 5.03 for EAGLE-2 at temperature 0), outperforming confidence routing and all single-drafter baselines. The key conceptual move is to treat multiple drafters not as competitors to be selected between, but as complementary proposal sources whose diversity improves the verifier's options at each speculative step.

Why this is distinct from prior tree-verification work: Previous work on tree verification (Miao et al., 2024) and dynamic tree construction (EAGLE-2, Li et al., 2024b) expanded proposal diversity by making a single drafter explore more branches — increasing tree width, depth, or branching factor. Merged-tree verification expands diversity by introducing proposals from qualitatively different distributions — the MathInstruct drafter contributes tokens it considers likely based on mathematical patterns, while the ShareGPT drafter contributes tokens based on conversational patterns. These proposals are not just more of the same; they represent alternative hypotheses about what the verifier will accept, drawn from different regions of the draft distribution space. The verifier can accept tokens from either specialist at any depth, allowing mixtures like "first two tokens from MathInstruct, third token from ShareGPT" that neither single drafter would have proposed as a complete path.

The correctness proof (Proposition A.2, Lemma A.2) is also conceptually elegant: it shows that as long as cross-subtree attention is masked and each subtree's internal structure is preserved, the verifier processes each subtree exactly as it would in standalone verification. This means merged-tree verification preserves the target distribution without any approximation — it is as lossless as standard speculative decoding, despite introducing proposals from multiple sources. The proof technique (verifier invariance under masked concatenation) is general and could apply to other multi-drafter or ensemble verification schemes beyond the two-specialist case studied here.

The trade-off — larger tree size increasing per-call verifier cost — is rigorously quantified (0.59×–0.78× speedup reduction relative to the strongest single checkpoint) and presented as a transparent design choice rather than a hidden cost. This honesty about the systems-level trade-off is itself a contribution: it frames merged-tree verification as a point on the acceptance-length-versus-per-call-cost Pareto frontier that practitioners can evaluate against their specific latency and throughput requirements. A deployment that can absorb larger per-call verifier computation (e.g., through batching, pipelining, or hardware acceleration) may prefer merged-tree verification for its higher acceptance length; a latency-sensitive deployment may prefer confidence routing despite slightly lower acceptance length. The paper provides the data to make this decision without claiming one strategy is universally superior.

Innovation 5: Depth-Aware Task Specialization — Coverage Dominates Early, Domain Match Dominates Late

The depth-wise acceptance analysis (Figure 8, Appendix tables 5–8) reveals a pattern that has implications for how draft trees should be constructed when domain specialists are available: at shallow speculative depths, mixed-data drafts often achieve the highest acceptance rates, suggesting that broad proposal coverage is most valuable for early tokens where the verifier has many acceptable continuations. At deeper depths, the task-matched specialist becomes increasingly dominant, especially on reasoning-heavy tasks — the deeper the speculative sequence goes, the more acceptance depends on sustained alignment between drafter and verifier that only a domain-matched model can provide.

Conceptual significance: This depth-dependence separates two distinct functions that a draft model serves in speculative decoding. At shallow depths, the draft model is an exploration mechanism — its job is to propose plausible continuations that the verifier can quickly screen. At deeper depths, the draft model becomes an exploitation mechanism — its job is to maintain precise distributional alignment with the verifier over long sequences where accumulated drift would otherwise cause rejection. The paper's results suggest that different drafters are optimal for these two roles: mixed-data drafters (with their broader coverage) perform exploration well, while domain-matched specialists (with their precise domain calibration) perform exploitation well.

This insight is not fully operationalized in the paper — neither confidence routing nor merged-tree verification uses depth-dependent specialist selection — but it provides a clear direction for future work. A system could, for example, construct a merged tree where shallow nodes come from a mixed-data drafter (for coverage) and deeper nodes come from domain-matched specialists (for precision), or could use confidence routing to select different specialists at different depths. The depth analysis effectively diagnoses why merged-tree verification works (it preserves both broad coverage from the "wrong" specialist and deep precision from the "right" specialist within a single verifier call) and suggests that even better composition strategies might be possible by making the depth-dependence explicit rather than implicit.

5. Experimental Analysis

Evaluation Methodology

  • Dataset. The paper evaluates on four benchmarks chosen to span conversational and mathematical reasoning domains: MT-Bench (80 multi-turn conversational questions), GSM8K (1,319 grade-school math word problems), MATH-500 (a 500-question subset of the MATH competition-level math benchmark), and SVAMP (300 math word problems with varying surface structures). All benchmarks are used as test sets — no training is performed on them. The benchmarks are evaluated at two temperatures: temperature 0 (deterministic/greedy drafting) and temperature 1 (stochastic sampling from the full distribution).

  • Base model(s). The verifier (target model) across all experiments is Meta-Llama-3-8B-Instruct, an 8-billion-parameter instruction-tuned language model (Llama Team, 2024). The draft model architecture is a lightweight LLaMA-style decoder with one transformer layer, hidden size 4096, and approximately 0.8 billion parameters, sharing the verifier's tokenizer and vocabulary. Two speculative decoding backbones are evaluated independently: EAGLE-2 (feature-level drafting with dynamic draft trees; Li et al., 2024b) and HASS (harmonized objective with Top-K distillation and context alignment; Zhang et al., 2025). The 8B verifier scale is representative of production LLM deployments, and the ~0.8B draft scale (roughly 10% of verifier parameters) represents the practical regime where the drafter is substantially cheaper than the verifier, making speculative decoding net-beneficial.

  • Metrics. The primary metric is acceptance length: the average number of consecutively accepted draft tokens per verifier call under the lossless speculative decoding constraint. Formally, for a prompt xx, verifier MTM_T, and drafter MDM_D, acceptance length is Ex∼D[A(x;MD,MT)]\mathbb{E}_{x \sim \mathcal{D}}[A(x; M_D, M_T)] where A(⋅)A(\cdot) counts the number of draft tokens accepted by the sequential application of Equation 1 before rejection occurs (or the draft tree is exhausted). Acceptance length is a pure measure of draft-to-target distributional alignment — it is invariant to hardware, batching, and systems implementation details that affect wall-clock speedup. The paper also provides speedup reduction factors relative to the strongest single checkpoint (Section 7, "Discussion and Limitations") but does not report absolute wall-clock speedup, acknowledging that end-to-end latency depends on systems-level factors beyond the scope of the study.

  • Baselines. The paper's baselines are internal to its own experimental design rather than drawn from prior published work (since prior work does not study training distribution effects). The core baselines are the single-domain checkpoints: (1) MathInstruct-trained drafter (70k mathematical reasoning examples) and (2) ShareGPT-trained drafter (70k conversational examples). These serve as both upper bounds for domain-matched performance and lower bounds for cross-domain performance. The checkpoint averaging variant serves as a composition baseline. No external speculative decoding methods or different verifier models are compared against — the paper's contribution is understanding how training distribution affects draft quality within the EAGLE-2 and HASS frameworks, not claiming superiority over alternative decoding acceleration methods.

  • Generation budget / compute accounting. The paper does not compare methods under a fixed generation budget in the traditional sense — all methods use the same speculative decoding procedure with the same tree construction logic (for a given backbone). What varies is the composition strategy overhead. For single-domain and mixed-data checkpoints, one draft tree is generated and verified. For confidence routing, two draft trees are generated (one from each specialist) but only one is verified, doubling draft-model computation. For merged-tree verification, two draft trees are generated and both are verified in one verifier pass, but the merged tree is larger (increasing attention computation quadratically). The paper reports speedup reduction factors relative to the strongest single checkpoint: confidence routing reduces speedup by 0.32×–0.47×, and merged-tree verification by 0.59×–0.78×, depending on backbone and temperature. These are coarse estimates — the paper emphasizes that end-to-end wall-clock comparison requires separate systems analysis.

  • Cross-validation / statistical protocol. No cross-validation is used. The paper evaluates each draft variant directly on the four test benchmarks and reports acceptance length aggregated over all prompts in each benchmark. There is no hyperparameter tuning on held-out validation data — the training hyperparameters (20 epochs, learning rate 3×10−53 \times 10^{-5}, batch size 8) and HASS settings (Top-K distillation with K=10K=10, three forward-alignment steps) are fixed across all experiments. The paper does not report confidence intervals, standard deviations, or statistical significance tests for acceptance length differences. The primary comparisons rely on point estimates and consistency across backbones, benchmarks, and temperatures to establish reliability rather than formal statistical testing.

Main Quantitative Results

The results are organized around the five research questions. The central evidence is Table 1 (acceptance length for all variants across four benchmarks, two backbones, and two temperatures), Table 2 (routing decisions by benchmark), Figure 6 (checkpoint averaging interpolation sweep), Figure 7 (accepted vs. rejected token entropy), and Figure 8 (acceptance by speculative depth). Each research question draws from specific rows or panels of these exhibits.

RQ1: Single-Domain Specialization (Table 1, RQ1 rows; Figure 8)

The core finding is that domain-matched drafters substantially outperform domain-mismatched drafters, with the pattern consistent across both backbones and both temperatures. Under HASS at temperature 0: ShareGPT achieves 3.98 on MT-Bench while MathInstruct achieves only 2.90 (a 27% drop from mismatch); conversely, MathInstruct achieves 5.02 on GSM8K and 5.35 on MATH-500 while ShareGPT achieves 4.09 and 3.98 respectively (drops of 19% and 26%). The same pattern holds under EAGLE-2 at temperature 0: ShareGPT achieves 3.57 on MT-Bench vs. MathInstruct's 2.54 (29% drop), while MathInstruct achieves 5.04, 5.28, and 4.81 on GSM8K, MATH-500, and SVAMP vs. ShareGPT's 3.72, 3.81, and 3.71 (drops of 26%, 28%, and 23%). At temperature 1, the specialization pattern persists with some narrowing: under EAGLE-2, MathInstruct's advantage on MATH-500 is 4.61 vs. 3.43 (26% gap), still substantial.

The depth-wise analysis (Figure 8, Appendix Tables 5–8) reveals that specialization is not uniform across speculative depth. On reasoning-heavy tasks, the MathInstruct drafter maintains higher acceptance rates at all depths, with the gap sometimes widening at deeper positions. For EAGLE-2 at temperature 0 on MATH-500 (Table 5): at depth 1, MathInstruct achieves 97.9% acceptance rate vs. ShareGPT's 90.6% (7.3 percentage point gap); at depth 5, MathInstruct achieves 81.5% vs. ShareGPT's 65.5% (16 percentage point gap). This deepening gap suggests that domain-matched drafters are not just better on average — they sustain distributional alignment over longer speculative sequences, which is precisely where accumulated drift most threatens acceptance.

RQ2: Mixed-Data Robustness (Table 1, RQ2 rows)

Mixed-data training produces checkpoints that are more robust across domains than single-domain specialists, but the relationship between mixture size and performance is not monotonic — and it interacts with decoding temperature. At temperature 0 under HASS, Mixed 70k+70k achieves the highest aggregate performance (average acceptance length 5.18, surpassing Mixed 35k+35k at 4.47 and both single-domain checkpoints at 4.10–4.12). However, at temperature 1, the ordering reverses: Mixed 35k+35k achieves 4.29 average acceptance length while Mixed 70k+70k drops to 3.69. Under EAGLE-2, the same inversion occurs: Mixed 70k+70k leads at temperature 0 (4.48 vs. 4.02 for Mixed 35k+35k), but Mixed 35k+35k leads at temperature 1 (3.81 vs. 3.26).

This temperature dependence is an unexpected finding. At temperature 0 (effectively greedy decoding), the larger mixture benefits from additional training data from both domains, learning broader coverage. At temperature 1 (stochastic sampling), the larger mixture may suffer from attempting to model two qualitatively different output distributions — the stochasticity amplifies distributional conflicts that are suppressed under greedy decoding. The paper does not fully explain this reversal, but it has practical implications: a deployment that operates at low temperature may prefer the larger mixture, while a high-temperature deployment may prefer the more focused 35k+35k mixture.

Compared to single-domain checkpoints, mixed-data training never completely erases specialization — the mixed checkpoints are consistently better than single-domain checkpoints on the mismatched domain but often worse than the matched specialist on its home domain. For example, Mixed 35k+35k under HASS at temperature 0 achieves 3.92 on MT-Bench (close to ShareGPT's 3.98, better than MathInstruct's 2.90) and 5.02 on MATH-500 (close to MathInstruct's 5.35, better than ShareGPT's 3.98). This is the "robust but not optimal" pattern — mixed-data training broadens coverage at the cost of peak domain performance, which is exactly what one would expect if the two domains require partially incompatible features.

RQ3: Composition Strategies (Table 1, RQ3 rows; Figure 6)

The headline finding is stark: inference-time composition is substantially stronger than weight-space averaging, and merged-tree verification achieves the highest acceptance length overall. Under HASS at temperature 0, the hierarchy is: Merged Trees (5.11 average) > Confidence Routed (4.80) > Mixed 70k+70k (5.18, but note: the mixed checkpoint's average is inflated by strong MATH-500 performance at the expense of MT-Bench — on MT-Bench specifically, Mixed 70k+70k achieves 4.13 while Merged Trees achieves 4.05, essentially tied) > Averaged (2.59). Under EAGLE-2 at temperature 0: Merged Trees (5.03) > Confidence Routed (4.63) > Mixed 70k+70k (4.48) > Averaged (2.42).

The failure of checkpoint averaging is not a matter of choosing the wrong interpolation coefficient. Figure 6 sweeps the interpolation weight λ from 0 (pure ShareGPT) to 1 (pure MathInstruct) and shows that no point on the curve approaches either specialist's performance or the inference-time composition methods. The acceptance length surface is sharply non-convex — averaging produces a model that is worse than either endpoint for most benchmarks, not just worse than the best specialist. This is consistent with the interpretation that the two specialists have learned incompatible internal representations whose linear combination destroys domain-specific features rather than blending them.

Confidence routing's success is validated by the routing statistics in Table 2. Under EAGLE-2 confidence routing, the MathInstruct drafter is selected for 90.8% of GSM8K prompts (1,198 out of 1,319), 97.0% of MATH-500 prompts (485 out of 500), and 93.0% of SVAMP prompts (279 out of 300). The ShareGPT drafter is selected for 81.2% of MT-Bench prompts (65 out of 80). These routing decisions are made purely from draft-side confidence — no domain labels, no prompt classifier — demonstrating that confidence serves as an effective zero-shot domain-match signal.

Merged-tree verification's advantage over confidence routing (5.11 vs. 4.80 for HASS, 5.03 vs. 4.63 for EAGLE-2 at temperature 0) demonstrates that even when routing selects the correct specialist most of the time, there is additional value in also considering proposals from the "wrong" specialist. The verifier can accept tokens from either specialist at any depth, allowing it to benefit from the diversity of both proposal distributions. The performance gap quantifies the value of this diversity: roughly 6–9% improvement in acceptance length from dual-specialist verification over single-specialist verification with optimal routing.

RQ4: Confidence vs. Entropy as Signals (Table 2; Figure 7)

Confidence-based routing produces clear benchmark-level separation; entropy-based routing does not. Under EAGLE-2 entropy routing (Table 2, right columns), the splits are near-balanced: 47.5% ShareGPT on MT-Bench, 54.6% MathInstruct on GSM8K, 62.4% MathInstruct on MATH-500, 53.0% MathInstruct on SVAMP. These ratios are much closer to random assignment than to the domain-aligned splits produced by confidence routing. The paper interprets this as evidence that entropy captures distribution shape (spread) rather than domain match (peak probability on the correct token), and the two are only loosely related when drafters are well-calibrated on their domains.

Despite being a weak routing signal, entropy remains informative as a diagnostic. Figure 7 shows that across both EAGLE-2 and HASS, across all benchmarks and checkpoint families, rejected tokens consistently have higher draft entropy than accepted tokens. For EAGLE-2 at temperature 0 (Table 3, Figure 7a), the MathInstruct checkpoint on GSM8K shows accepted-token entropy of 0.5284 and rejected-token entropy of 1.0756 (a difference of +0.5473). The ShareGPT checkpoint on MT-Bench shows accepted-token entropy of 1.0404 and rejected-token entropy of 1.6292 (+0.5887). The pattern is universal — every checkpoint on every benchmark shows positive Δ (higher entropy for rejected tokens), with no exceptions in the EAGLE-2 table (Table 3) and only two exceptions in the HASS table (Table 4: Averaged checkpoint on MATH-500 shows Δ = -0.5364 and on MT-Bench shows Δ = -0.6110 — both from the catastrophically poor Averaged variant).

The verifier-side entropy patterns (Tables 3–4, "Verifier Accepted" vs. "Verifier Rejected" columns) show the same direction: rejected tokens correspond to higher verifier entropy than accepted tokens. This is expected from the speculative acceptance rule — when the verifier is uncertain (high entropy, flat distribution), the draft model is more likely to propose a token the verifier assigns low probability, leading to rejection. The rejected-token entropy elevation thus serves as a post-hoc diagnostic of why a particular token was rejected, even though it does not predict domain match well enough for routing.

RQ5: Depth Effects (Figure 8; Appendix Tables 5–8)

Acceptance rate declines with speculative depth for all variants, all backbones, all benchmarks, and both temperatures — there are zero exceptions in Tables 5–8. This is expected: longer speculative sequences are harder to predict correctly because draft model errors compound and the target distribution becomes more constrained by the accumulated prefix. The rate of decline reveals domain specialization patterns.

On reasoning-heavy tasks with matched specialists, the decline is shallower. For EAGLE-2 at temperature 0 on MATH-500 (Table 5): MathInstruct drops from 97.9% at depth 1 to 81.5% at depth 5 (total decline of 16.4 percentage points), while ShareGPT drops from 90.6% to 65.5% (decline of 25.1 points). The shallower decline for the matched specialist means it not only starts higher but also degrades more slowly — it maintains alignment deeper into speculative sequences.

Mixed-data checkpoints show an interesting pattern: at shallow depths (1–2), they often match or exceed the best single-domain specialist, but at deeper depths (4–5), the domain-matched specialist becomes dominant. For HASS at temperature 0 on MATH-500 (Table 7): Mixed 35k+35k leads at depth 2 with 87.2% (vs. MathInstruct's 88.9% — essentially tied), but at depth 5, MathInstruct achieves 75.9% while Mixed 35k+35k achieves 59.2%. This supports the "coverage at shallow depths, precision at deep depths" interpretation: mixed-data drafters propose diverse early continuations that the verifier can accept, but struggle to maintain distributional alignment over long sequences where domain-specific patterns dominate.

The Averaged checkpoint shows catastrophic depth-wise behavior. For EAGLE-2 at temperature 0 on MATH-500 (Table 5): depth 1 acceptance is 89.3% (plausible, close to individual specialists), but depth 2 drops to 63.1%, depth 3 to 30.1%, depth 4 to 28.0%, and depth 5 to 32.2%. The collapse between depths 1 and 3 (from ~89% to ~30%) is far sharper than for any other variant, suggesting that weight averaging produces a draft model whose feature predictions are superficially plausible (good enough for one-step prediction, where the context is clean) but fundamentally unstable (unable to sustain feature quality when conditioning on its own previous outputs at deeper speculative steps). This is consistent with the HASS motivation for context alignment training — the Averaged checkpoint exhibits the context mismatch problem in extreme form.

Ablation Studies and Robustness Checks

The paper is primarily an empirical study comparing draft variants under fixed conditions, so formal ablation studies in the traditional sense (removing one component of a method and measuring the impact) are limited. However, several comparisons serve an ablative function by isolating specific effects.

  • EAGLE-2 vs. HASS backbone (Table 1, all rows): The domain specialization pattern (RQ1), mixed-data robustness pattern (RQ2), and composition strategy ranking (RQ3) are consistent across both backbones. At temperature 0, the average acceptance length ordering across the seven main variants (single-domain, mixed-data 35k+35k, mixed-data 70k+70k, Averaged, Confidence Routed, Merged Trees) is identical for EAGLE-2 and HASS: Averaged is weakest, Merged Trees is strongest, and Confidence Routed occupies an intermediate position. This demonstrates that the paper's findings are about speculative decoding and training data, not about a specific draft architecture. The absolute acceptance lengths differ between backbones (HASS generally achieves slightly higher numbers — 5.11 vs. 5.03 for Merged Trees at temperature 0 — attributable to HASS's objective and context alignment improvements), but the relative patterns are conserved.

  • Temperature 0 vs. Temperature 1 (Table 1, left vs. right panels): Temperature changes the ordering of mixed-data checkpoints (Mixed 70k+70k is best at temperature 0, Mixed 35k+35k is best at temperature 1) but does not change the core findings: single-domain specialization persists (RQ1), inference-time composition dominates weight averaging (RQ3), and Merged Trees remains the strongest composition strategy. The temperature sensitivity of mixed-data mixtures is an important practical finding — it means that tuning the mixture ratio for the deployment temperature is necessary, and a mixture optimized for greedy decoding may underperform at higher temperatures.

  • Confidence routing vs. entropy routing (Table 2): This serves as an ablation of the routing signal, holding the routing architecture fixed (binary selection between two specialists). The dramatic difference in benchmark-level splits (e.g., 90.8% vs. 54.6% MathInstruct selection on GSM8K) demonstrates that the routing signal — not the existence of two specialists — drives the effectiveness of confidence routing. Entropy routing would produce near-random selection between specialists and thus near-random acceptance length (approximately the average of the two specialists), while confidence routing produces domain-appropriate selection and acceptance length close to the better specialist.

  • Checkpoint averaging interpolation sweep (Figure 6): This ablates the choice of λ = 0.5 in the main table, showing that the failure of weight-space merging is not an artifact of a suboptimal interpolation coefficient. Acceptance length varies non-monotonically and non-smoothly with λ, never approaching either the individual specialists or the inference-time composition methods. This demonstrates that the failure mode is fundamental — the two specialists occupy incompatible regions of weight space — rather than a matter of fine-tuning the merge ratio.

  • Merged-tree verification vs. confidence routing (Table 1, RQ3 rows): This serves as an ablation of the "select-one vs. verify-both" decision, holding the available drafters (MathInstruct + ShareGPT specialists) constant. Merged-tree verification consistently outperforms confidence routing (e.g., 5.11 vs. 4.80 for HASS at temperature 0, 5.03 vs. 4.63 for EAGLE-2), demonstrating that even with near-perfect routing (MathInstruct selected for 97% of MATH-500), there is additional value in making both specialists' proposals available to the verifier. The gap quantifies the benefit of proposal diversity beyond what optimal routing alone provides.

  • Accepted vs. rejected token entropy (Figure 7, Tables 3–4): This ablates the informational value of entropy as a routing signal while preserving its diagnostic value. The consistent positive Δ for rejected tokens (with only two exceptions in the catastrophically broken Averaged checkpoint) demonstrates that entropy is a reliable post-hoc indicator of rejection likelihood. This finding is robust across both backbones, all benchmarks, and all functioning checkpoint families. It suggests that a more sophisticated routing policy could potentially incorporate entropy alongside confidence — not as a standalone routing signal, but as a weighting factor that increases confidence in the domain-match signal when token-level entropy is high.

  • Single-domain vs. mixed-data vs. composition (Table 1, across RQ1, RQ2, RQ3): The full set of 7 variants for each backbone at each temperature constitutes a factorial experiment varying (a) whether data is single-domain or mixed, (b) how multiple data sources are combined (in training data vs. in weight space vs. at inference time), and (c) the inference-time composition mechanism (routing vs. merged verification). The consistent ranking — Merged Trees ≥ Confidence Routed ≥ Mixed 70k+70k ≥ Mixed 35k+35k ≥ best single-domain specialist ≫ Averaged — holds across both backbones and both temperatures with only temperature-dependent reversals between the two mixed-data variants. This robustness across multiple axes of variation strengthens the causal interpretation that inference-time composition preserves specialization while weight-space averaging destroys it, and that merged-tree verification captures diversity benefits beyond what routing achieves.

Critical Assessment

The paper's central claims, as articulated in Section 1 (the executive summary) and supported by the experiments, must be examined against what the experimental design actually tests — and what it does not.

Claim: "Task-specific training produces clear domain specialization." This claim is well-supported by the RQ1 evidence (Table 1). The specialization pattern — MathInstruct stronger on math benchmarks, ShareGPT stronger on MT-Bench — is consistent across both backbones, both temperatures, and all four benchmarks. The effect sizes are large (20–46% relative drops from domain mismatch) and not plausibly attributable to noise. However, the claim is demonstrated only for two domains (conversational chat and mathematical reasoning) and one verifier model (Llama-3-8B-Instruct). Whether domain specialization would be equally pronounced for other domain pairs (e.g., code generation vs. translation, medical reasoning vs. legal analysis) or for other verifier model families (e.g., Mistral, Qwen, Gemma) is not tested. The claim is best understood as "domain specialization exists in speculative decoding" rather than "all domain pairs will show the same degree of specialization" — the paper establishes the existence of the phenomenon without mapping its precise boundaries.

Claim: "Inference-time composition is substantially stronger than weight-space averaging." This claim is very strongly supported. The Averaged checkpoint is the weakest variant in Table 1 under every condition (both backbones, both temperatures, all benchmarks), with acceptance lengths in the 2.34–2.62 range — substantially below all other variants. The interpolation sweep (Figure 6) eliminates the possibility that a better λ would rescue weight averaging. The finding is robust and practically important. However, the paper tests only one form of weight-space merging (point-wise linear interpolation with uniform λ across all layers). More sophisticated merging techniques — task vector arithmetic (Ilharco et al., 2022), per-layer or per-parameter weighting, Fisher-weighted merging, or learned merging coefficients — are not evaluated. The claim is therefore about naive checkpoint averaging, not about whether any possible weight-space merging technique could work. A more precise statement would be: "Uniform linear interpolation in weight space fails to preserve domain specialization," which leaves open the possibility that more sophisticated merging could succeed — though the paper's evidence (fundamental incompatibility of learned features) suggests this is unlikely.

Claim: "Merged-tree verification achieves the highest acceptance length overall." Supported for the two-specialist case under the specific experimental conditions. Merged Trees achieves 5.11 (HASS) and 5.03 (EAGLE-2) at temperature 0, which are the highest numbers in Table 1. At temperature 1, merged trees are also highest (4.75 for HASS, 4.57 for EAGLE-2). However, the claim comes with an important caveat that the paper acknowledges but the headline obscures: merged-tree verification has the largest per-call computational cost (0.59×–0.78× speedup reduction relative to the strongest single checkpoint). Acceptance length is not the same as throughput. The paper explicitly does not claim an end-to-end latency improvement for merged-tree verification without a separate systems analysis. A reader focused on deployment optimization should understand that merged-tree verification offers a trade-off (higher acceptance length, larger per-call trees) whose net benefit depends on the relative cost of draft-tree construction, verifier attention (which scales quadratically with tree size), and the acceptance-length-dependent reduction in verifier calls. The paper provides acceptance length numbers but not the systems analysis needed to resolve this trade-off.

Claim: "Confidence is more useful than entropy as a routing signal." Strongly supported by Table 2. Confidence routing produces benchmark-level splits that align with domain boundaries (90.8% MathInstruct on GSM8K, 81.2% ShareGPT on MT-Bench); entropy routing produces near-balanced splits. The finding is clear and the comparison is direct. However, the paper does not explore whether combining confidence and entropy (or other signals like tree structure, acceptance history, or prompt embeddings) could produce even better routing. The claim is specifically about confidence vs. entropy as standalone routing signals — it does not establish that confidence is the optimal routing signal, only that it is substantially better than the natural alternative (entropy). This is a reasonable scope for a first study, but practitioners building routing systems may want to explore richer signal combinations.

Genuine weaknesses and missing experiments:

  • Single verifier model. All experiments use Llama-3-8B-Instruct. The paper's findings about domain specialization depend on the verifier having distinct output distributions across domains — a verifier that produces similar token distributions for math and chat would not show specialization effects. Llama-3-8B-Instruct is a reasonable choice (widely used, instruction-tuned, representative of production LLMs), but the paper cannot claim that the findings generalize to other verifier families without replication. A verifier trained primarily on code, or one with different instruction-tuning data, might show different specialization patterns or weaker domain effects.

  • No confidence intervals or statistical testing. All results are point estimates without any quantification of uncertainty. The paper reports acceptance length to two decimal places but does not provide standard deviations, confidence intervals, or significance tests. For MT-Bench (80 questions), a difference of a few tenths of an acceptance length point may not be statistically reliable. The paper relies on consistency across backbones and temperatures to establish reliability rather than formal statistics — this is a reasonable approach for an empirical study, but it means that small numerical differences between variants should not be over-interpreted.

  • Limited domain diversity. Only two domains (conversational chat, mathematical reasoning) are studied. The paper's claim that "draft training distribution is a first-class design variable" would be strengthened by showing specialization across additional domain pairs — e.g., code generation vs. chat, or medical text vs. legal text. The current two-domain design is sufficient to establish the existence of the phenomenon but cannot characterize its breadth or identify domain properties that predict the strength of specialization.

  • No exploration of domain granularity. The paper uses two broad domains (ShareGPT for all conversational data, MathInstruct for all mathematical data). It does not explore whether finer-grained specialization (e.g., separate drafters for GSM8K vs. MATH-500, or for different subcategories of chat) would further improve acceptance length, or whether the benefits of specialization saturate at some level of domain granularity. This is an open question for future work.

  • No systems-level end-to-end measurements. The paper's primary metric is acceptance length, which is a pure measure of draft-verifier alignment but does not directly measure throughput, latency, or memory consumption. The speedup reduction factors for composition strategies (0.32×–0.78×) are stated but not derived from wall-clock measurements — they appear to be estimated from the increase in draft and verifier computation. A systems-aware evaluation would measure actual tokens-per-second on specific hardware under realistic batching and KV-cache management, which would resolve the trade-off between acceptance length and per-call cost that the paper identifies but does not fully analyze.

  • No exploration of draft model scale. All drafters use the same architecture (one transformer layer, ~0.8B parameters). It is plausible that domain specialization effects depend on draft model capacity: a very small drafter might benefit more from domain-matched training (because it has limited capacity to learn multiple domains), while a larger drafter might achieve cross-domain robustness through data mixing alone. The paper does not test this hypothesis, and the fixed draft scale limits the generality of the finding that "mixed-data training does not eliminate specialization."

  • No verification with different speculative decoding frameworks. The paper uses EAGLE-2 and HASS (feature-level drafting). It does not test whether domain specialization appears with token-level drafting (the original Leviathan et al., 2023 framework) or with self-speculative decoding variants. The paper's findings are demonstrated for feature-level drafting only, though the consistency across EAGLE-2 and HASS suggests the effect is not specific to a single feature-drafting architecture.

The conditional nature of the claims, precisely:

  • "Merged-tree verification achieves the highest acceptance length" holds for the two-specialist (MathInstruct + ShareGPT) case. It may not hold for 3+ specialists (tree size grows, attention cost increases superlinearly, diminishing returns on diversity) or for specialist pairs where the domains overlap substantially (less diversity gain from merged trees).
  • "Confidence routing improves over single-domain baselines" holds when both specialists are available and the routing signal correctly identifies domain. If the specialists are poorly calibrated (e.g., a Math drafter that is overconfident on chat), confidence routing might select the wrong specialist more often and underperform.
  • "Task-specific training improves matched-domain acceptance" holds for the tested domain pair (chat vs. math). It may not hold for domains that are distributionally similar under the verifier (e.g., two conversational datasets with different stylistic properties but similar token distributions), or for verifiers that produce similar distributions across domains.
  • "Mixed-data training improves robustness" holds in the sense of reducing worst-case cross-domain performance drops, but the optimal mixture size is temperature-dependent (70k+70k better at temperature 0, 35k+35k better at temperature 1), so practitioners should tune the mixture for their deployment temperature.

6. Limitations and Trade-offs

6.1 Single Verifier Model Family — Findings May Not Transfer to Other Target LLMs

The assumption or constraint. All experiments use Meta-Llama-3-8B-Instruct as the verifier. The paper's entire empirical case — domain specialization patterns, composition strategy rankings, confidence routing accuracy — is demonstrated on a single target model. The paper acknowledges this scope implicitly in Section 3 ("Across all experiments, the verifier is Meta-Llama-3-8B-Instruct") but does not discuss how verifier-specific its findings might be.

The consequence. The acceptance length differences that the paper attributes to draft training distribution could partially reflect interactions between the draft training data and this specific verifier's behavior. A verifier with different pretraining data, instruction-tuning data, or output distributions might produce different specialization patterns. For example, a verifier trained predominantly on code might have similar output distributions for mathematical and conversational prompts (because both are "non-code" and handled similarly), which would weaken or eliminate the domain specialization effects the paper documents. Conversely, a verifier with sharper domain-specific output characteristics might show even stronger specialization. The paper provides no evidence about whether the specialization pattern is a property of speculative decoding in general (any verifier + domain-matched drafter) or an artifact of Llama-3-8B-Instruct's particular training.

Additionally, the verifier's parameter scale (8B) is fixed. The relationship between verifier size and the value of domain-matched drafting is unexplored. A very large verifier (70B+) might have output distributions that are harder for any draft model to approximate (making domain matching more important), or might produce distributions that are more uniform across domains (making domain matching less important). A very small verifier (1B) might be easier to approximate with a generic drafter (making domain matching unnecessary). The paper's results bound this relationship at exactly one point (8B verifier with ~0.8B drafter).

What evidence exists in the paper. None. The paper does not run any experiments with alternative verifiers, does not provide an analysis of why Llama-3-8B-Instruct was chosen beyond its status as a representative model, and does not discuss the likely dependence of its findings on verifier choice. Section 7 ("Discussion and Limitations") states "We evaluate one target model, two source domains, two speculative backbones, and four benchmarks" as a limitation but does not elaborate on the specific risks of single-verifier evaluation.

Mitigation status. Not addressed. The paper's claims are conditional on the verifier being Llama-3-8B-Instruct, but this condition is not stated prominently in the conclusions. The abstract and introduction frame the findings as general properties of speculative decoding ("Task-specific training produces clear domain specialization") without qualifying that the evidence comes from a single verifier. Replicating the study with at least one additional verifier family (e.g., Mistral, Qwen, Gemma) at a comparable scale would substantially strengthen the generality claim. The paper does not propose this as future work.

6.2 No End-to-End Systems Analysis — Acceptance Length Alone Does Not Resolve the Throughput vs. Latency Trade-off

The assumption or constraint. The paper's primary metric is acceptance length — the average number of draft tokens accepted per verifier call. Section 7 ("Discussion and Limitations") explicitly states: "Acceptance length is our primary metric, so this paper does not establish end-to-end deployment trade-offs for routing or merged-tree verification." Acceptance length is a pure measure of draft-verifier alignment, but it is not equivalent to throughput or latency. The translation from acceptance length to wall-clock speedup depends on: (1) the cost of generating draft trees (which varies between composition strategies — confidence routing generates two trees but verifies one; merged-tree verification generates two trees and verifies both in a larger packed tree), (2) the cost of verifier attention (which scales quadratically with tree size), (3) the overhead of tree construction and mask building, (4) KV-cache management, and (5) batching dynamics.

The consequence. The headline finding — "Merged-tree verification achieves the highest acceptance length overall" — does not guarantee that merged-tree verification achieves the highest throughput or lowest latency. The paper quantifies this tension: merged-tree verification incurs a speedup reduction of 0.59×–0.78× relative to the strongest single checkpoint, while confidence routing incurs a smaller reduction of 0.32×–0.47×. These numbers suggest that merged-tree verification's higher acceptance length comes at a substantial per-call cost increase, but the paper does not resolve which factor dominates in practice.

Consider a concrete scenario: if the verifier processes a merged tree of size 60 nodes in one call and the acceptance length is 5.11, the system accepts ~5.11 tokens per 60-node verifier pass. If confidence routing processes a tree of size 30 nodes and achieves acceptance length 4.80, it accepts ~4.80 tokens per 30-node verifier pass. The merged-tree approach has higher tokens-per-call (5.11 vs. 4.80), but lower tokens-per-verifier-node (0.085 vs. 0.160) — it is less efficient per unit of verifier computation. Whether the net throughput effect is positive or negative depends on the fixed costs of verifier invocation (memory access, kernel launch overhead, KV-cache operations) versus the variable costs of attention computation. The paper does not provide the measurements needed to resolve this.

For latency-sensitive applications (interactive chatbots, real-time systems), the larger per-call trees of merged-tree verification may increase latency beyond acceptable bounds even if they improve throughput. Confidence routing's additional draft-model forward pass (generating the unused specialist's tree) adds latency that the paper does not quantify. A sequential system that generates and verifies one draft tree has different latency characteristics than one that generates two trees (possibly in parallel) and then verifies one or both.

What evidence exists in the paper. Section 7 ("Discussion and Limitations") provides speedup reduction estimates: "confidence routing reduces average speedup by 0.32× and 0.35× under EAGLE-2 and by 0.40× and 0.47× under HASS at temperatures 0 and 1, while merged-tree verification incurs a larger drop of 0.59× and 0.62× under EAGLE-2 and 0.72× and 0.78× under HASS." These numbers are presented as coarse estimates derived from counting draft and verifier operations rather than from wall-clock measurements. The methodology for computing these reduction factors is not detailed — it is unclear whether they account for tree size differences, draft model inference costs, attention mask construction overhead, or batching effects.

Mitigation status. Partially addressed through transparency. The paper explicitly states that end-to-end systems analysis is outside scope and that the speedup reduction numbers are rough estimates. Section 7 notes that "in a deployment setting that must serve two distinct task families, this overhead may be partly or fully offset when the best single checkpoint is weak on one of the tasks." This is a qualitative argument — it acknowledges the trade-off but does not quantify the conditions under which the trade-off favors each strategy. The paper identifies this as an area for future work but provides no guidance on how a practitioner should make the deployment decision with the available data.

6.3 Only Two Domains Evaluated — Specialization May Not Generalize to Other Domain Pairs or Granularities

The assumption or constraint. The paper studies exactly two domains: conversational chat (ShareGPT, evaluated via MT-Bench) and mathematical reasoning (MathInstruct, evaluated via GSM8K, MATH-500, and SVAMP). Section 2 ("Why This Matters: Practical Deployment and the Open-Weight Ecosystem") motivates the study with a broader claim about "multiple specialized checkpoints" for "coding, mathematics, medical reasoning, legal analysis, multilingual translation, and so on," but the experiments cover only one domain pair.

The consequence. The paper's central finding — that domain-matched drafters achieve substantially longer acceptance lengths than mismatched drafters — is demonstrated only for the chat-vs-math comparison. This is a particularly clean domain contrast: conversational text and mathematical reasoning have sharply different surface statistics (vocabulary, formatting, discourse structure), and the target model likely processes them through different internal mechanisms. Whether domain specialization effects of comparable magnitude would appear for more subtle domain distinctions is unknown. For example:

  • Code generation vs. technical documentation: Both share technical vocabulary and structured formatting; the verifier's output distributions may overlap substantially, reducing the benefit of domain-matched drafting.
  • Medical reasoning vs. legal reasoning: Both involve domain-specific terminology but share the structure of professional reasoning; specialization might be weaker than for chat-vs-math.
  • Two conversational datasets with different stylistic properties: Specialization might be minimal because the token-level distributions are too similar for a draft model to differentiate.
  • Two mathematical datasets at different difficulty levels: A drafter trained on elementary math might perform poorly on competition math — the paper does not explore within-domain specialization.

The paper's findings also do not speak to the optimal level of domain granularity. The study uses coarse domain categories (all chat, all math). A practitioner might ask: should I train one math drafter for all mathematical reasoning, or separate drafters for arithmetic, algebra, geometry, and word problems? The paper provides no evidence about whether finer-grained specialization yields additional acceptance length gains, or whether the benefits saturate at some domain breadth. This is practically consequential: maintaining many specialists increases deployment complexity and memory footprint, and the return on additional specialization is unknown.

What evidence exists in the paper. Section 7 ("Discussion and Limitations") acknowledges this: "We evaluate... two source domains." The paper's abstract and conclusions use unqualified language — "Task-specific training produces clear domain specialization" — without restricting the claim to the evaluated domains. The experimental results are consistent within the chat-vs-math domain pair but provide no evidence about domain pairs with different properties.

Mitigation status. Minimally addressed. The paper flags the two-domain limitation in Section 7 but does not analyze what properties of the chat-vs-math distinction might drive the specialization effect, which would help practitioners predict whether their domain pair would show similar behavior. No experiments with additional domain pairs, within-domain granularity variation, or domain similarity metrics are conducted. The paper does not propose a framework for predicting when domain-matched drafting will provide benefits based on measurable properties of the domains.

6.4 Checkpoint Averaging Baseline Is Naive — Does Not Rule Out More Sophisticated Weight-Space Merging

The assumption or constraint. The paper evaluates exactly one weight-space merging technique: point-wise linear interpolation with uniform λ across all parameters (Equation 11, λ = 0.5 in the main table, λ swept in Figure 6). The conclusion that "weight-space averaging is a weak baseline" and that "drafters should be kept separate and combined at inference time" is based entirely on this one merging method.

The consequence. The paper's strong negative claim about weight-space merging — that it "performs poorly" and "fails" — may overstate the case against weight-space techniques in general. Several more sophisticated merging methods exist that the paper does not evaluate:

  • Task vector arithmetic (Ilharco et al., 2022, which the paper cites): Instead of averaging full parameter vectors, this approach computes task vectors (θ_specialized - θ_base) and adds or subtracts them, potentially preserving domain-specific features that linear interpolation destroys.
  • Per-layer or per-parameter weighting: Different transformer layers or parameter groups encode different types of information. Uniform averaging treats all parameters identically, but a weighted scheme that preserves domain-specific layers (e.g., using higher weight for attention parameters from the math specialist on math prompts) might perform better.
  • Fisher-weighted merging: Weighting parameters by their Fisher information (importance for each task) could preserve critical parameters for each domain while averaging less important ones.
  • Learned merging coefficients: A small amount of training to determine optimal per-layer interpolation weights could substantially improve over uniform λ.
  • TIES-Merging or DARE: Techniques that resolve sign conflicts between task vectors or prune redundant parameter changes before merging might handle the incompatibility that the paper documents.

The paper's Figure 6 shows that acceptance length varies non-smoothly with λ, which suggests that the two specialists occupy incompatible regions of weight space. However, this does not prove that no weight-space manipulation can combine them — it proves only that linear interpolation cannot. Task arithmetic, which can add or subtract parameter updates rather than interpolating between them, might find a path through weight space that preserves more domain-specific behavior.

The practical implication is significant: a practitioner who reads "weight-space averaging fails" might conclude that maintaining separate specialists and implementing inference-time composition (with its associated systems complexity — routing logic, tree merging, increased memory for multiple checkpoints) is the only viable approach. If a more sophisticated weight-space merging technique achieves most of the benefit of inference-time composition with the deployment simplicity of a single checkpoint, that would change the practical recommendation.

What evidence exists in the paper. The paper tests only uniform linear interpolation (Figure 6). There is no ablation comparing different weight-space merging techniques, no experiment with task vectors (despite citing Ilharco et al., 2022), and no analysis of which parameters or layers are most affected by averaging. The paper's claim in Section 7 that "the relevant behavior is not preserved by naive interpolation" is accurate but narrower than the general conclusion about weight-space merging.

Mitigation status. Not addressed. The paper does not discuss more sophisticated weight-space merging alternatives, does not analyze why naive averaging fails (beyond the general observation that the interpolation surface is non-convex), and does not qualify its conclusions to limit them to uniform linear interpolation. The strong recommendation to combine drafters at inference time rather than in weight space is based on a single (naive) weight-space baseline. Future work comparing inference-time composition against the best available weight-space merging technique would clarify whether the paper's core recommendation (inference-time composition) is driven by an inherent advantage of keeping specialists separate or by the weakness of the evaluated weight-space baseline.

6.5 No Quantification of Statistical Uncertainty — Small Benchmarks and Point Estimates Limit Reliability

The assumption or constraint. The paper reports all acceptance lengths as point estimates with two decimal places (e.g., "5.11," "4.80," "3.98") without any measure of uncertainty. No standard deviations, confidence intervals, standard errors, or statistical significance tests are provided for any comparison. The evaluation benchmarks vary substantially in size: MT-Bench has 80 questions, MATH-500 has 500 questions, GSM8K has 1,319 questions, SVAMP has 300 questions.

The consequence. For the smaller benchmarks — particularly MT-Bench with 80 questions and SVAMP with 300 — the reported acceptance length differences between variants may not be statistically reliable. Consider a representative comparison from Table 1: under HASS at temperature 0, Merged Trees achieves 4.05 on MT-Bench while Confidence Routed achieves 3.93. This is a difference of 0.12 acceptance length on 80 questions. Without a standard error, we cannot assess whether this difference is meaningful or within the range of sampling variability. If the standard deviation of acceptance length across questions is, say, 2.0 tokens, the standard error on 80 questions would be approximately 2.0/80≈0.222.0 / \sqrt{80} \approx 0.22, making the 0.12 difference well within one standard error and thus not statistically distinguishable.

The problem is compounded for the difficulty-bin-like analyses and benchmark-level comparisons. Table 2 (routing decisions by benchmark) reports counts — e.g., under confidence routing, ShareGPT is selected for 65 out of 80 MT-Bench prompts (81.2%). With only 80 prompts, the 95% confidence interval for this proportion (using the normal approximation) is approximately 0.812±1.96×0.812×0.188/80≈0.812±0.0850.812 \pm 1.96 \times \sqrt{0.812 \times 0.188 / 80} \approx 0.812 \pm 0.085, or roughly 72.7% to 89.7%. The paper presents the 81.2% figure as a stable routing accuracy, but the interval spans a range where the routing accuracy could be substantially lower. This matters for practitioners evaluating whether confidence routing is reliable enough for their deployment.

The paper's strategy for establishing reliability is cross-validation through consistency: the same patterns appear across two backbones, two temperatures, and four benchmarks. This is a reasonable approach — consistent patterns across independent conditions are less likely to be noise — but it does not replace formal uncertainty quantification. A pattern can be consistent across backbones while still being within the noise range for any individual backbone-benchmark comparison. The paper provides no way to assess whether a particular numerical difference (e.g., the 0.12 gap between Merged Trees and Confidence Routed on MT-Bench) is "real" or could be reversed with a different random seed, training run, or evaluation sample.

What evidence exists in the paper. None. The paper provides no uncertainty quantification of any kind. No mention of standard deviations, confidence intervals, bootstrap estimates, or significance tests appears in the main text or appendices. The evaluation methodology (Section 3) describes the benchmarks and their sizes but does not discuss statistical power or the reliability of point estimates given those sizes.

Mitigation status. Not addressed. The paper could have reported standard deviations across benchmark questions (for within-benchmark comparisons) or across multiple training runs (to quantify the stability of draft model training). Bootstrap confidence intervals for the acceptance length differences between variants would provide a simple, assumption-light way to assess the reliability of the paper's central claims. The absence of any uncertainty quantification means that the precision implied by two-decimal-place reporting may be misleading, particularly for the smaller benchmarks. A practitioner comparing two variants on MT-Bench should not treat a 0.1–0.2 acceptance length difference as actionable without knowing whether such differences are typical of sampling variation.

6.6 No On-Policy or Dynamic Adaptation — Difficulty Estimation and Routing Are Fixed Before Generation

The assumption or constraint. The paper's composition strategies — confidence routing and merged-tree verification — make a one-time decision per prompt: which specialist to use (routing) or how to combine trees (merged verification). There is no mechanism for adapting the strategy during generation based on how the speculative decoding is progressing. The routing decision depends on pre-verification draft-side signals (confidence), not on acceptance history, verifier feedback, or dynamic difficulty assessment.

The consequence. The paper's approach cannot adapt when the initial routing decision turns out to be wrong, or when a prompt contains a mixture of domain elements. Consider a prompt that asks: "Calculate the tip on a $45.80 restaurant bill and then write a polite thank-you note to the server." This prompt spans both math and conversational domains. Confidence routing will select one specialist based on overall confidence, but the optimal drafter might change mid-generation — the math drafter for the calculation portion, the chat drafter for the thank-you note. With one-time routing, the same specialist handles the entire generation, potentially underperforming on half the tokens. Merged-tree verification mitigates this by making both specialists' proposals available at every step, but at the cost of larger per-call trees; it does not dynamically adjust the relative weight or size of the two subtrees based on which specialist is performing better.

The paper's depth-wise analysis (Figure 8, RQ5) shows that at shallow depths, mixed-data or broader-coverage drafters often perform best, while at deeper depths, domain-matched specialists become dominant. This pattern suggests that an optimal strategy might change the composition method by depth — start with a mixed drafter or merged trees for exploration in early steps, then switch to the matched specialist for exploitation in later steps. Neither confidence routing (which selects one specialist for all depths) nor merged-tree verification (which uses both specialists at all depths) implements this depth-adaptive strategy.

More generally, the paper's approach is entirely feedforward: draft trees are generated, a decision is made, and verification proceeds. There is no feedback loop where low acceptance rates on recent tokens trigger a strategy change (e.g., switching from one specialist to the other, or from routing to merged trees). In a deployment serving heterogeneous traffic, the distribution of prompt types may shift over time, and a fixed routing policy cannot adapt to these shifts without retraining or manual reconfiguration.

What evidence exists in the paper. The depth-wise acceptance analysis (Figure 8, Tables 5–8) indirectly demonstrates this limitation by showing that different strategies are optimal at different depths, but the paper does not propose or test depth-adaptive strategies. Table 2 (routing decisions) shows that confidence routing makes benchmark-level separation possible, but does not analyze within-prompt routing accuracy or cases where the routing decision was wrong and how that affected acceptance length.

Mitigation status. Not addressed. The paper does not discuss dynamic adaptation, online strategy switching, or depth-dependent composition. The routing policy is "intentionally simple and confidence-based rather than learned or cost-aware" (Section 7), which the paper presents as a deliberate scope limitation. However, the depth-wise analysis (RQ5) provides evidence that adaptation could improve performance, and the paper neither proposes such methods nor acknowledges this as a specific direction for future work. The gap between the evidence (different strategies are optimal at different depths) and the method (fixed one-time routing or static merged trees) is not highlighted as a limitation in the discussion.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper makes a methodological intervention rather than a paradigm shift. It does not propose a new speculative decoding algorithm, a new draft architecture, or a new theoretical framework. Instead, it identifies a variable — draft model training distribution — that the speculative decoding literature has systematically overlooked, and demonstrates through controlled experimentation that this variable is not noise but signal: it produces large, consistent, and practically consequential effects on acceptance length (20–46% relative drops from domain mismatch in Table 1). The paper's core contribution is to recategorize draft training data from an implementation detail to a first-class design dimension, on par with draft architecture in determining deployment performance.

The landscape shift is in what questions the field considers worth asking. Before this paper, the speculative decoding community's research agenda was almost entirely architectural: how to build better feature-level drafters (EAGLE, EAGLE-2, EAGLE-3, HASS), how to construct more efficient verification trees (SpecInfer, tree attention), how to cascade multiple drafters (hierarchical drafting), and how to incorporate retrieval or self-speculation. The training data was ShareGPT by default, and no paper examined whether a different corpus would produce a different outcome. This paper establishes that the answer is "yes — and the differences are large enough to matter." A researcher designing a new speculative decoding system now has two independent optimization axes (architecture and training data) rather than one, and a practitioner deploying speculative decoding must audit their workload composition and consider whether domain-matched drafters are worth the additional training and deployment complexity.

The resolution of a latent contradiction. The paper does not resolve a bitter empirical conflict in the literature, because no prior work had studied training distribution effects directly. However, it does resolve a tacit assumption that had been carrying explanatory weight: that draft model training data doesn't matter much because feature-level drafting learns a general approximation of the target model's internal representations. The paper's consistent finding — domain mismatch causes 20–46% acceptance length drops across both EAGLE-2 and HASS architectures — falsifies this assumption. The implication is that feature-level drafting does not learn a domain-agnostic representation of the target model; it learns a representation that is specific to the patterns in its training data, and this specificity persists even after the draft features are mapped through the target model's LM head. This reframes feature-level drafting from "learning to predict the target model's hidden states" to "learning to predict the target model's hidden states on a particular data distribution."

Research directions that become more attractive. The paper makes the study of draft-data-verifier interactions a legitimate and open research area. Before this work, a proposal to "study the effect of draft training data on speculative decoding" would have seemed like an engineering detail — something a practitioner might tune but not a contribution. After this work, such studies are clearly fruitful: the effect sizes are large, the mechanisms are non-obvious (weight averaging fails catastrophically, confidence routing works remarkably well), and the practical deployment implications are direct. Specific directions that gain credibility include: characterizing which domain properties (vocabulary overlap, structural similarity, reasoning depth) predict specialization strength; developing draft-data selection and augmentation techniques specifically optimized for speculative decoding acceptance length (not just for draft model perplexity or downstream task accuracy); and studying how draft-data effects interact with verifier scale, draft model scale, and backbone choice across a broader range of configurations.

Research directions that become less attractive. The paper's findings make it less attractive to treat draft training data as a non-decision — i.e., to simply train on the largest available general corpus and assume this produces an adequate proposal distribution. The performance gap between domain-matched and domain-mismatched drafters (Table 1) is large enough that a practitioner who defaults to ShareGPT training for a math-heavy deployment is leaving substantial throughput on the table. Similarly, the paper makes it less attractive to propose new speculative decoding algorithms without specifying or controlling for the draft training distribution — the acceptance length differences from data domain are large enough to confound architectural comparisons if not held constant. Future speculative decoding papers that train drafters on different datasets for different experiments without acknowledging this as a variable now have a burden of justification that did not exist before.

A methodological contribution: the use of weight-space averaging as a diagnostic. The paper's most transferable methodological insight may be the way it uses checkpoint averaging not as a proposed solution but as a diagnostic baseline whose failure reveals something about the structure of learned draft representations. The near-total collapse in acceptance length under weight averaging (2.34–2.62 across methods and temperatures, uniformly the weakest variant in Table 1) demonstrates that the MathInstruct and ShareGPT specialists have learned representations that are not linearly composable. This technique — training specialists on distinct domains, averaging them, and measuring performance degradation — could be applied as a general diagnostic for representation compatibility in other multi-domain fine-tuning contexts. The finding that the interpolation surface is non-convex and no λ recovers strong performance (Figure 6) suggests a more fundamental incompatibility than would be revealed by simply noting that a multi-task model underperforms single-task specialists.

A limitation that bounds the landscape shift. The paper's findings are established for one verifier (Llama-3-8B-Instruct), one draft model scale (~0.8B parameters), one draft-verifier size ratio (~10%), two speculative decoding backbones (EAGLE-2 and HASS), and two domains (conversational chat and mathematical reasoning). The landscape shift is therefore best characterized as "the field should now treat draft training distribution as a variable worth studying" rather than "we now know the general relationship between any draft training distribution and any verifier's acceptance length." The specific quantitative findings (e.g., the 27% drop from math-to-chat mismatch on HASS) are conditional on the evaluated configuration. The qualitative finding — that domain matching matters, and that inference-time composition preserves specialization while naive weight averaging destroys it — is demonstrated robustly within the paper's scope, but the boundary conditions (Does this hold for code-vs-chat? For medical-vs-legal? For 70B verifiers? For 3B drafters?) remain open. The paper opens a research program more than it closes one.

Follow-Up Research This Work Enables

Verifier model family replication with an identical experimental protocol. The paper's central empirical claim — that domain-matched draft training substantially improves acceptance length — is demonstrated on exactly one target model (Llama-3-8B-Instruct). The most direct and important follow-up is to replicate the full experimental matrix (single-domain MathInstruct and ShareGPT drafters, mixed 35k+35k and 70k+70k drafters, confidence routing, merged-tree verification, checkpoint averaging) on at least two additional verifier families at comparable scale: a Mistral-family model (e.g., Mistral-7B-Instruct) and a Qwen-family model (e.g., Qwen2-7B-Instruct). The experiment would measure whether the specialization pattern magnitude (20–46% mismatch penalty), the composition strategy ranking (Merged Trees > Confidence Routed > Mixed Data > Single-Domain > Averaged), and the confidence-as-routing-signal accuracy (81–97% correct domain assignment in Table 2) replicate across verifier families with different pretraining data, tokenizers, and instruction-tuning recipes. A strong replication would establish that domain specialization is a general property of speculative decoding, not an artifact of Llama-3's particular training. A partial replication (specialization holds but at different magnitudes, or confidence routing degrades on certain verifier families) would begin to map the boundary conditions — what properties of a verifier make domain matching more or less important — which is essential for practical deployment guidance.

Domain similarity gradient characterization — when does specialization provide diminishing returns? The paper studies exactly one domain contrast (conversational chat vs. mathematical reasoning) with sharply different surface statistics and reasoning demands. A natural extension is to measure acceptance length as a function of domain similarity to map the gradient from "identical domains" (where specialization provides no benefit) to "maximally distinct domains" (where specialization is largest). A concrete experiment: train drafters on five domains along a similarity spectrum — e.g., (1) general web text (C4), (2) Wikipedia articles, (3) technical StackExchange posts, (4) Python code, (5) mathematical proofs — and measure acceptance length for each drafter on each evaluation domain on a single verifier. The resulting 5×5 matrix would reveal whether the specialization benefit is approximately monotonic with domain similarity (measured by vocabulary overlap, embedding space distance, or target model's output distribution divergence across domains), or whether it exhibits threshold effects (no benefit until domains are "sufficiently distinct"). This experiment would be enabled by the paper's demonstration that draft-data effects are large enough to measure reliably; prior to this work, it was unclear whether such a matrix would show signal above noise.

Learned or context-aware routing policies versus confidence-only routing. The paper's confidence routing uses a simple statistic (mean node confidence) and achieves strong benchmark-level separation (Table 2: 90.8–97.0% correct specialist assignment on math benchmarks). A natural follow-up is whether richer routing signals can improve on this, particularly for the 3–19% of prompts where confidence routing selects the "wrong" specialist. Three specific approaches to test. First, a linear classifier trained on draft-side features (mean confidence, entropy, tree depth, variance of confidence across nodes) to predict which specialist will achieve higher acceptance length — this classifier could be trained on a held-out set of prompts with labeled oracle routing decisions. Second, a prompt-embedding-based routing system that uses the verifier's (or draft model's) prompt representation to predict domain before any draft tree is generated — this would avoid the cost of generating the unused specialist's tree. Third, a dynamic switching policy that starts with confidence routing but monitors acceptance rate during generation and switches specialists if acceptance drops below a threshold — this would address the mixed-domain prompt problem (e.g., "Calculate the tip and write a thank-you note") that one-time routing cannot handle. All three approaches could be evaluated against the paper's confidence-routing baseline using the same two-specialist setup and benchmarks, providing a direct comparison of whether routing sophistication buys additional acceptance length beyond what a simple confidence-max rule achieves.

Depth-adaptive composition strategies. The paper's RQ5 analysis (Figure 8, Tables 5–8) reveals that shallow speculative depths benefit from broad proposal coverage (mixed-data drafters often outperform specialists at depths 1–2) while deeper depths benefit from precise domain alignment (the task-matched specialist becomes increasingly dominant at depths 3–5). Neither confidence routing (which selects one specialist for all depths) nor merged-tree verification (which verifies both specialists at all depths) exploits this depth-dependence. A concrete follow-up experiment would test depth-adaptive tree construction methods. One variant: construct a merged tree where the first 2 levels of nodes come from a mixed-data drafter (for broad coverage at shallow depths) and nodes at levels 3+ come from the confidence-selected domain-matched specialist (for precision at deep depths), with the verifier processing the full tree in one pass. Another variant: use confidence routing but with a different specialist at each depth — at depth 1, select the specialist with higher mean confidence at depth 1 (which may differ from the overall mean confidence that drives the paper's routing), and so on. A third variant: use a width-varying merged tree where the "wrong" specialist's subtree narrows at deeper depths (fewer nodes, reflecting lower confidence) while the "right" specialist's subtree stays wide. These experiments would be evaluated against the paper's existing confidence-routing and merged-tree baselines, with the metric being acceptance length at each depth and overall. The paper's depth-wise acceptance tables (5–8) provide the baseline data needed to assess whether depth-adaptive strategies improve upon the paper's static composition methods.

Draft model scale interaction with domain specialization. The paper uses a fixed draft model size (~0.8B parameters, one transformer layer) and finds large specialization effects. An important follow-up is whether these effects scale with draft model capacity. Hypotheses cut both ways. A larger drafter (e.g., 4 layers, ~3B parameters) might have sufficient capacity to learn both domains well from mixed data, making domain-matched training less important — the specialization gap might shrink as draft capacity increases. Alternatively, a larger drafter might learn more refined domain-specific features, making the specialization gap wider because the matched specialist extracts more value from domain data while the mismatched specialist's extra capacity is wasted on irrelevant patterns. A systematic experiment would train MathInstruct and ShareGPT drafters at three draft model scales (e.g., 1 layer / 0.8B, 4 layers / 3B, and 8 layers / 6B parameters) using the paper's training recipe, then measure the acceptance length gap between matched and mismatched evaluations at each scale under the same Llama-3-8B verifier. This experiment would enable the paper's core finding to be stated conditionally: "For drafters at ~0.8B scale, domain matching provides a ~25–46% acceptance length advantage; this advantage [shrinks / grows / remains stable] as draft capacity increases to 3B and 6B." This conditional statement is essential for practitioners deciding whether to invest in multiple specialized drafters or to train one larger mixed-data drafter.

On-policy adaptive composition with acceptance-history feedback. The paper's composition strategies are feedforward: they make one decision per prompt (which specialist, or merged trees) based on pre-verification signals, and that decision is locked in for the entire generation. A richer class of strategies would use acceptance history — the sequence of which tokens were accepted or rejected during generation — to dynamically adjust the composition policy. For example, if confidence routing selects the MathInstruct specialist but the first three speculative steps achieve zero acceptance (immediate rejection each time), the system might switch to the ShareGPT specialist or to merged-tree mode. Conversely, if the initial acceptance is high, the system might increase the draft tree depth for the selected specialist (since the alignment appears strong). This is enabled by the paper's demonstration that strong routing signals exist (Table 2) but that routing is not perfect (3–19% of prompts route to the "wrong" specialist). A follow-up experiment would measure: (1) the correlation between early-generation acceptance rate and whole-generation acceptance rate under confidence routing — if early rejection strongly predicts overall poor performance, then acceptance history is a useful feedback signal; (2) the acceptance-length gain of a simple policy that switches specialists after kk consecutive rejections, with kk swept from 1 to 5; and (3) whether the gain from dynamic switching exceeds the overhead of generating a second draft tree mid-generation. The paper's existing benchmarks and two-specialist setup provide the testbed; the additional cost is instrumenting the speculative decoding loop to track acceptance history and trigger specialist switches.

Practical Applications and Downstream Use Cases

Multi-domain LLM serving endpoints with heterogeneous traffic. A common production pattern is a single LLM inference endpoint serving queries from different sources — a chatbot handling both casual conversation and math homework help, or a coding assistant that also answers natural language documentation questions. The default speculative decoding deployment uses one draft model trained on generic data, which the paper shows underperforms by 20–46% on at least one domain (Table 1). With confidence routing, the system can deploy both a MathInstruct and a ShareGPT drafter (roughly 0.8B parameters each, ~1.6B total — still much smaller than the 8B verifier) and route each incoming prompt to the appropriate specialist using only draft-side confidence signals, with no domain classifier or prompt metadata. The paper's Table 2 shows this routing achieves 81–97% correct specialist assignment across benchmarks. The acceptance-length gain is concrete: on GSM8K under HASS at temperature 0, confidence routing achieves 5.01 acceptance length (Table 1) vs. 4.09 for ShareGPT alone (the generic default) — a 22% improvement in tokens accepted per verifier call. For a deployment serving, say, 1 million GSM8K-style queries per day, this translates to roughly proportional reduction in verifier calls (and thus verifier compute cost) at the modest overhead of generating a second draft tree per query. The deployment complexity is manageable: both drafters can be loaded into GPU memory simultaneously (they share architecture and tokenizer) and confidence computation is a byproduct of draft tree generation (the drafter's predicted probabilities are already computed during autoregressive proposal). The paper's correctness guarantee (Proposition A.1) ensures the output quality is identical to the verifier's autoregressive distribution — this is a fully lossless optimization.

Cost-sensitive batch inference for mathematical reasoning evaluation. Organizations that run large-scale mathematical reasoning evaluation (e.g., benchmarking suites, competition grading, curriculum assessment) often process thousands of math problems through an LLM, where throughput and cost are the dominant concerns. The paper's results show that a MathInstruct-trained drafter achieves 5.35 acceptance length on MATH-500 under HASS at temperature 0 — a 34% improvement over the ShareGPT drafter's 3.98 (Table 1). If the batch-processing pipeline uses speculative decoding with the MathInstruct specialist instead of a generic ShareGPT drafter, each verifier call accepts 5.35 tokens instead of 3.98, reducing the number of verifier forward passes by approximately 26% for the same total output length. Since the verifier (Llama-3-8B, ~8B parameters) dominates the compute budget compared to the drafter (~0.8B parameters), this reduction in verifier calls translates directly to cost savings. The training cost of the domain-matched drafter is a one-time investment amortized across all evaluations. For an organization evaluating 100,000 MATH-500-style problems monthly, training a single MathInstruct drafter costs 20 epochs on 70k examples (roughly a few GPU-hours on the paper's 4×A100 setup) and saves ongoing inference costs proportional to the 26% reduction in verifier calls. The marginal additional cost of also deploying a merged-tree verification setup (adding a ShareGPT specialist for the 3% of MATH-500 prompts where it might help — Table 2 shows 97% routing to MathInstruct) may not justify the larger per-call tree cost, making the single MathInstruct specialist the practical optimum for math-only workloads.

Draft model training as a service — domain-specific draft checkpoints for popular verifiers. The paper's finding that domain specialization substantially improves acceptance length creates a market opportunity for model providers to release domain-specific draft checkpoints for popular verifier models. Currently, a practitioner deploying speculative decoding with Llama-3-8B-Instruct typically trains their own drafter on generic data (if at all) or uses a generic drafter released alongside the verifier. The paper's results suggest that releasing separate draft checkpoints for common workload categories — a "Llama-3-8B-Math-Drafter" trained on MathInstruct, a "Llama-3-8B-Chat-Drafter" trained on ShareGPT, a "Llama-3-8B-Code-Drafter" trained on the Stack — would provide immediate acceptance-length benefits to deployments matching those domains. The drafters are small (~0.8B parameters, ~1.5GB in FP16), cheap to distribute, and can be used independently or in combination via confidence routing. The paper's demonstration that confidence routing works without metadata (Table 2) means these drafters could be combined by platform-level middleware (e.g., vLLM or TGI serving frameworks) without requiring application-level prompt tagging. The inference-time composition infrastructure — generating multiple draft trees, computing mean confidence, routing, or packing merged trees — could be abstracted behind a speculative decoding API that accepts multiple draft checkpoints and handles composition transparently. The paper's correctness guarantees (Propositions A.1 and A.2) ensure this composition layer is a valid extension of the speculative decoding contract: output distribution remains identical to the verifier's.

On-device speculative decoding with task-switching. On-device LLM deployment (smartphones, laptops, edge devices) is severely memory-constrained, making speculative decoding particularly attractive because the drafter can be very small while the verifier runs on a more capable but power-hungry accelerator or in the cloud. The paper's domain specialization finding is directly relevant here: if an on-device system serves multiple task types (e.g., a writing assistant that also does unit conversion and basic math), loading domain-matched drafters for each task may be infeasible due to memory constraints, but a single mixed-data drafter may underperform on any specific task. The paper's confidence routing approach offers a middle ground: the system loads one drafter at a time, switching between them based on the task context (which may be known from the app UI — the user is in "math mode" vs. "writing mode"). The acceptance-length difference between matched and mismatched drafters (Table 1: 5.02 vs. 4.09 on GSM8K under HASS, a 19% gap) translates to fewer verifier calls and lower energy consumption for the same output quality. For an on-device deployment where verifier calls go to a cloud API (costing latency and bandwidth), or where the verifier runs on-device with limited battery, a 19% reduction in verifier calls per token is directly valuable. The paper's finding that mixed 35k+35k training achieves robustness at intermediate acceptance length (Table 1: 4.77 on GSM8K, 4.15 on SVAMP) provides a fallback for memory-constrained deployments that can only store one checkpoint — it outperforms the mismatched specialist on every benchmark while approaching the matched specialist. The concrete guidance from the paper is: if memory allows, store multiple specialists and switch between them; if not, use a balanced mixed-data checkpoint rather than a single-domain specialist, since the cross-domain penalty is larger than the within-domain gain sacrificed.

(Conditional) When to Prefer This Method

The paper's composition strategies (confidence routing, merged-tree verification) are positioned relative to three alternatives: (1) using a single generic drafter (the default in prior work), (2) using a single mixed-data drafter, and (3) averaging two specialists in weight space. The paper provides evidence for a clear decision framework:

  • Prefer confidence routing when the deployment serves traffic from two sharply distinct domains (e.g., conversational chat and mathematical reasoning), memory is sufficient to hold both specialist checkpoints simultaneously (each ~0.8B parameters, ~1.5GB in FP16), and the cost of generating a second (unused) draft tree per query is acceptable. The evidence: confidence routing achieves 4.80–4.63 acceptance length (HASS/EAGLE-2 at temperature 0), outperforming the strongest mixed-data checkpoint on cross-domain average, with a speedup reduction of only 0.32×–0.47× relative to the strongest single checkpoint. Confidence routing is particularly attractive when the workload composition is unknown or variable — the routing is automatic and requires no prompt tagging or domain metadata (Table 2 shows 81–97% correct assignment from confidence alone).

  • Prefer merged-tree verification when maximizing acceptance length is the primary objective and the deployment can absorb increased per-call verifier computation (larger trees → quadratic attention cost increase). The evidence: merged trees achieve the highest acceptance length in Table 1 (5.11 HASS, 5.03 EAGLE-2 at temperature 0), with a 6–9% improvement over confidence routing, but incur a larger speedup reduction of 0.59×–0.78× relative to the strongest single checkpoint. This strategy is appropriate when the verifier is the dominant latency bottleneck (so reducing verifier calls matters more than increasing per-call cost) and when proposals from both specialists consistently differ (so the diversity gain is realized — the paper's results suggest this holds for chat-vs-math, but may not hold for similar-domain specialist pairs where the draft trees largely overlap).

  • Prefer a single mixed-data checkpoint when memory is constrained to a single drafter, the workload contains a known mixture of domains, and the mixture ratio can be tuned for the deployment temperature. The evidence: Mixed 35k+35k HASS at temperature 0 achieves 3.92 on MT-Bench and 5.02 on MATH-500 — close to the domain-matched specialist on each benchmark and far better than the mismatched specialist. At temperature 1, Mixed 35k+35k (average acceptance 4.29) is preferable to Mixed 70k+70k (3.69), demonstrating that mixture size must be tuned for the decoding temperature. The mixed checkpoint avoids all routing complexity and requires no inference-time composition logic — it is the simplest deployment option that still captures most of the specialization benefit.

  • Never prefer checkpoint weight averaging. The evidence is decisive: the Averaged checkpoint is the weakest variant under every condition (Table 1: 2.34–2.62 average acceptance length across all methods and temperatures), the interpolation sweep shows no λ recovers strong performance (Figure 6), and the depth-wise analysis reveals catastrophic instability at deeper speculative steps (Tables 5–8, e.g., EAGLE-2 Averaged on MATH-500 drops from 89.3% at depth 1 to 30.1% at depth 3). Weight-space averaging of domain specialists is not merely suboptimal — it destroys the specialization that makes each draft model useful, producing a model that is worse than either specialist on every benchmark. The paper's formal correctness guarantees (Propositions A.1 and A.2) do not apply to the averaged checkpoint (since it is a new model whose internal validity as a "valid tree generator" is not established), and the empirical evidence shows it fails in practice.