Cascade Speculative Drafting for Even Faster LLM Inference

ArXiv: 2312.11462

🎯 Pitch

Cascade Speculative Drafting stacks speculative decoding recursively to eliminate autoregressive generation from neural draft models altogether, replacing the innermost drafts with a cheap bigram model. It then gives the best draft models more time on the most important tokens, yielding up to an 81% extra speedup over standard speculative decoding while guaranteeing identical output.

---

1. Executive Summary

This paper introduces Cascade Speculative Drafting (CS Drafting), a speculative-execution-based algorithm that accelerates LLM inference without altering the target model's output distribution by incorporating two orthogonal cascades — a Vertical Cascade that eliminates autoregressive generation from neural models by recursively applying speculative decoding down to a negligible-cost statistical language model (e.g., a bigram model drafting for a small neural model, which in turn drafts for the target model), and a Horizontal Cascade that improves time allocation during drafting by assigning larger draft models to generate earlier, higher-acceptance-probability tokens and progressively smaller models for later tokens where the cumulative acceptance probability decays exponentially. Evaluated on GSM8K and MMLU with FLAN-T5-XXL as the target model, CS Drafting achieves up to an 81% additional speedup over standard speculative decoding — a practical gain that operates without additional deployment overhead — establishing that speculative drafting efficiency can be substantially improved through tiered model cascades and position-aware resource allocation, while maintaining exact equivalence to the target model's output distribution only when lenience is restricted to inter-draft-model reviews and withheld from the final target model review.

2. Context and Motivation

The Core Problem: Drafting Efficiency Is the Bottleneck in Speculative Decoding

Speculative decoding, as introduced by Leviathan et al. [14] and Chen et al. [4], has emerged as one of the most practical solutions for reducing LLM inference latency. The idea is elegant: instead of running the large target model autoregressively — token by token, each generation depending on the previous — a smaller, faster draft model generates a batch of $k$ candidate tokens. The target model then reviews all $k$ tokens in a single parallel forward pass, accepting those consistent with its own output distribution and generating one additional token. When the draft model's predictions align well with the target model, the target model produces multiple tokens per run, yielding speedups of 2–3× without altering the final output distribution.

This paper identifies a critical asymmetry in speculative decoding's efficiency story: the research community has focused almost exclusively on improving the acceptance rate — making the draft model better at predicting what the target model would produce — while treating the drafting process itself as a fixed cost. The paper argues this is a mistake. The drafting process remains bottlenecked by the same autoregressive generation that speculative decoding was designed to avoid, just shifted to a smaller model. Each drafting step requires the draft model to generate $k$ tokens sequentially, running $k$ forward passes where each depends on the previous. When the draft model is called repeatedly — once per target model review step, potentially hundreds of times across a long generation — this sequential cost accumulates and becomes the limiting factor.

Leviathan et al. [14] implicitly acknowledged this bottleneck through a surprising empirical finding: the best draft model is often far smaller than intuition would suggest, sometimes two orders of magnitude smaller than the target model. A larger draft model would produce higher-quality predictions (higher acceptance rate), but the increased per-token inference cost of running that larger model $k$ times per step outweighs the benefit. This reveals the tension at the heart of speculative decoding: the draft model must be fast enough that its autoregressive cost doesn't consume the speedup gained from parallel target model execution, yet accurate enough that its tokens are accepted frequently. The paper frames this as evidence that "improving drafting efficiency is crucial for further enhancing the performance of speculative decoding" (Section 1).

Why This Matters: Scale and the Economics of LLM Inference

The practical stakes are high. LLMs like GPT-4 serve hundreds of millions of daily users in interactive applications — chatbots, virtual assistants, code completion tools — where latency directly impacts user experience and retention. The paper notes that "even a slight improvement in the latency of LLMs can greatly contribute to both the service provider and the community" (Section 1). This is not hyperbole: in production systems, the cost of serving LLM inference at scale is enormous, and any technique that reduces latency without degrading output quality translates directly to reduced hardware requirements, lower energy consumption, and improved user experience.

Speculative decoding already demonstrated a 2–3× improvement, which is substantial. But if the drafting process itself contains inefficiencies — specifically, the continued reliance on autoregressive generation from neural models and the uniform allocation of expensive model capacity across all draft token positions — then there is room for additional gains on top of what speculative decoding already achieves. The paper positions CS Drafting as capturing exactly these residual inefficiencies: not replacing speculative decoding, but improving its internal drafting mechanism to extract speedups from gains that the original formulation leaves on the table.

Prior Work and Its Limitations

Speculative Decoding and Its Variants

The foundation is speculative decoding [14, 4, 25], which the paper builds on directly rather than attempting to replace. The core idea — draft-then-verify with a rejection sampling procedure that preserves the target model's distribution exactly — is retained. But the paper identifies two structural inefficiencies in how prior work implements the drafting phase:

Limitation 1: All drafting requires autoregressive neural generation. In standard speculative decoding, the draft model generates $k$ tokens sequentially. Each of these $k$ forward passes is a neural network inference — faster than the target model, but still expensive relative to what could be achieved with non-neural alternatives. The paper observes that there is no fundamental requirement that all drafting be done by neural models. A statistical language model (e.g., a bigram model) has negligible inference cost and zero autoregressive dependency in its sampling, yet prior work hadn't integrated such models into the speculative decoding framework because their prediction quality alone is too low to serve as a direct draft model for the target model.

Limitation 2: All draft token positions receive equal computational resources. Figure 2 in the paper shows empirically what probability theory predicts: the acceptance probability of a draft token declines sharply with its position in the draft sequence. The first draft token is accepted roughly 70–80% of the time (depending on the model and task), while the 30th token is accepted less than 10% of the time. This is because acceptance at position $i$ requires acceptance at all positions $1$ through $i-1$ — a multiplicative probability chain. Yet standard speculative decoding uses the same draft model (of fixed size and cost) for every token position, meaning that expensive model capacity is spent uniformly on tokens that contribute very differently to the expected number of accepted tokens per step.

Knowledge Distillation and Model Compression

Section 6.1 briefly surveys alternative efficiency approaches: pruning [26, 22, 9, 5], knowledge distillation [11, 8], and quantization [1, 18]. These are orthogonal to CS Drafting — they modify the model itself (reducing parameters, transferring knowledge to smaller architectures, reducing numerical precision), whereas CS Drafting modifies the decoding procedure while keeping the model weights unchanged. The paper doesn't position itself against these methods but rather as complementary: a deployment could both quantize its models and apply CS Drafting for cumulative speedups.

Improvements on Speculative Decoding

The paper situates itself relative to three categories of speculative decoding enhancements:

Knowledge-distillation-based approaches (Zhou et al. [29]): These aim to improve the acceptance rate by training the draft model to better match the target model's output distribution, typically through distillation objectives. The paper doesn't dispute the value of higher acceptance rates but argues that acceptance rate improvements alone cannot overcome the fundamental inefficiency of autoregressive neural drafting — even a perfectly aligned draft model still requires $k$ sequential forward passes.

Self-drafting approaches (Zhang et al. [27], Hooper et al. [12]): These avoid deploying a separate draft model by reusing components of the target model itself — for example, using a subset of the target model's layers or adding lightweight prediction heads. This reduces the deployment overhead of maintaining a separate draft model but doesn't address the autoregressive bottleneck: the drafting still requires sequential forward passes through (part of) the target model.

Tree attention approaches (Miao et al. [15], Cai et al. [3]): These improve the acceptance rate by generating multiple candidate continuations at each draft step, effectively branching the draft tree and increasing the probability that at least one path is accepted. The paper explicitly demonstrates in Section 5.2 that CS Drafting can be combined with tree attention (via Medusa [3]) for additional gains (Table 3), showing that the two techniques address orthogonal bottlenecks — tree attention improves acceptance rate, CS Drafting improves drafting efficiency.

The Closest Prior: Staged Speculative Decoding

Spector and Re [19] proposed a method that is the most direct precursor to CS Drafting's vertical cascade. Their "staged speculative decoding" uses two layers of speculative decoding — a small model drafts for a medium model, which drafts for the target model — showing conceptual similarity to the vertical cascade. However, the paper identifies two critical limitations of this prior work:

1. It doesn't recognize the recursive nature of the approach. Staged speculative decoding uses exactly two layers. CS Drafting generalizes this to an arbitrary-depth recursion that terminates at a negligible-cost statistical language model (the Max-Gram or MaG model). The paper argues that the full benefit of the vertical cascade only emerges when the recursion reaches the bottom — a statistical model whose inference cost is so low that it can be treated as effectively zero. Two layers still require the bottom model to perform autoregressive neural generation.

2. It doesn't incorporate lenience between draft models. Lenience (Section 3.1) is a hyperparameter that loosens the acceptance criterion, allowing a reviewer to accept draft tokens with a higher tolerance for probability mismatch. In the original speculative decoding paper [14], lenience was noted as a way to trade output quality for speed. CS Drafting makes the crucial observation that lenience only affects the final output distribution if applied when the target model reviews. Lenience applied when an intermediate draft model reviews another draft model changes what tokens are proposed to the next level up, but the target model's final review — which enforces exact distribution matching — can still be performed without lenience. This means the vertical cascade can use lenience aggressively to speed up internal draft-to-draft reviews while guaranteeing that the final output distribution remains identical to the target model's autoregressive distribution. The prior staged approach missed this asymmetric application of lenience entirely.

How This Paper Positions Itself

The paper frames CS Drafting as addressing the under-studied drafting efficiency dimension of speculative decoding. Where prior work asked "how can we make the draft model's predictions more accurate?" (acceptance rate improvement) or "how can we avoid maintaining a separate draft model?" (self-drafting), this paper asks "how can we make the drafting process itself faster, independent of acceptance rate?" This framing is novel because it treats the internal structure of the drafting phase as a design space to be optimized, rather than treating drafting as a monolithic, fixed-cost step.

The paper's theoretical contribution (Section 4) formalizes this distinction through the Expected Walltime Improvement Factor (EWIF) analysis, which decomposes the factors affecting speedup into terms that depend on acceptance rate (which prior work optimizes) and terms that depend on drafting cost (which CS Drafting optimizes). The generating function analysis (Theorem 4.1, 4.3) provides a mathematical tool for reasoning about nested speculative decoding systems, and Corollary 4.4 proves that adding a negligible-cost draft model almost always improves EWIF — a formal justification for the vertical cascade.

The paper's practical contribution is demonstrating that the cascade approach works with off-the-shelf models without fine-tuning (Section 5.1: "To align our experiment with current common usage, we do not perform fine-tuning for CS Drafting"), making it immediately deployable. The Max-Gram algorithm (Section 3.3, Appendix A) provides a concrete, GPU-friendly implementation of a statistical drafting model that leverages input-output token matching (common in tasks like summarization and reasoning where the output reuses vocabulary from the input) and falls back to a bigram model for general generation. This avoids the deployment complexity of training or maintaining additional neural models beyond what speculative decoding already requires.

The paper positions both cascades as orthogonal and composable — the horizontal cascade allocates model sizes across token positions within a single draft step, the vertical cascade optimizes the generation mechanism within each position, and both can be combined (Algorithm 1). Furthermore, CS Drafting is positioned as composable with other speculative decoding enhancements (tree attention, as demonstrated with Medusa in Table 3), suggesting it functions as a drop-in improvement to the drafting phase that preserves the benefits of other complementary techniques.

3. Technical Approach

3.1 Reader Orientation

Cascade Speculative Drafting (CS Drafting) is a multi-level speculative decoding system where draft models do not generate tokens autoregressively from scratch; instead, each neural draft model is itself accelerated by an even smaller draft model, forming a recursive pyramid that bottoms out at a negligible-cost statistical language model. The system solves the problem that standard speculative decoding wastes time on two fronts — the draft model still performs expensive autoregressive neural generation for every draft token, and it allocates equal computational resources to all token positions despite later tokens having exponentially lower acceptance probability — by restructuring the drafting process as a cascade where fast statistical models generate the bulk of tokens and larger models only review, not autoregressively generate, with the largest models reserved for the highest-value token positions.

3.2 Big-Picture Architecture (Diagram in Words)

The CS Drafting system has five major components arranged in a recursive hierarchy:

1. Target Model ($M_t$) — the large LLM whose output distribution the system must preserve exactly. It serves as the final reviewer: it never autoregressively generates, only verifies draft tokens and produces one additional token per review step.

2. Neural Draft Models ($M_{d1}, M_{d2}, ...$) — a set of progressively smaller neural language models (e.g., FLAN-T5-BASE, FLAN-T5-SMALL) that review tokens proposed by models below them and propose refined tokens upward. Each neural draft model acts as both a reviewer (for the model below it) and a proposer (to the model above it).

3. Max-Gram (MaG) Statistical Model — a non-neural, negligible-cost language model at the bottom of the cascade that combines greedy input-output token matching with a bigram fallback. It generates draft tokens for the smallest neural model above it with near-zero latency and no autoregressive neural computation.

4. Vertical Cascade Mechanism — the recursive speculative decoding procedure where each model (except the bottom MaG) reviews drafts from a smaller model rather than generating autoregressively. The recursion terminates at MaG, eliminating all autoregressive neural generation.

5. Horizontal Cascade Mechanism — a within-step allocation policy where the largest draft model generates the earliest tokens (highest acceptance probability) and progressively smaller models handle later tokens (lower acceptance probability), using a hyperparameter matrix $K_{nn}$ that specifies how many tokens each model generates at each recursion level.

Information flows as follows: an input prefix enters the system → the MaG model generates initial candidate tokens using max-gram pattern matching → the smallest neural draft model reviews these tokens using speculative acceptance/rejection, then proposes refined tokens → this review-propose cycle continues upward through progressively larger neural draft models → the target model performs the final review without lenience, preserving exact distributional equivalence → accepted tokens are appended to the output and the process repeats.

3.3 Roadmap for the Deep Dive

• First, the Vertical Cascade: how recursive speculative decoding eliminates autoregressive neural generation, including the critical role of asymmetric lenience (applied between draft models but never at the target model review).
• Second, the Horizontal Cascade: how position-dependent model allocation is motivated by the exponential decay of acceptance probability, formalized through the probability chain, and implemented via the $K_{nn}$ hyperparameter matrix.
• Third, the Max-Gram (MaG) algorithm: how the bottom-level statistical model works, why input-output token matching is effective for tasks like summarization and reasoning, and the GPU-friendly implementation details.
• Fourth, the unified Algorithm 1: how the vertical and horizontal cascades are combined into a single recursive procedure, including the flow of hyperparameters, generation budgets, and the isFirstCall flag that controls lenience application.
• Fifth, the theoretical framework (EWIF and generating functions): how the paper formalizes the expected speedup, the probability generating function for speculative decoding, the extension to vertical cascades via function composition, and the analysis of horizontal cascade via partial derivatives of EWIF with respect to per-position acceptance rates.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and algorithms paper whose core idea is that speculative decoding's drafting phase can be accelerated by replacing autoregressive neural generation with recursive speculative review and by allocating model capacity proportional to token acceptance probability.

---

The Vertical Cascade: Recursive Speculative Decoding

The core insight. In standard speculative decoding, the draft model $M_d$ must run $k$ autoregressive forward passes to produce $k$ candidate tokens for each single forward pass of the target model $M_t$. Even though $M_d$ is smaller than $M_t$, this $k\times$ multiplier on draft inference cost can dominate the total runtime, which is why Leviathan et al. [14] found that very small draft models (roughly two orders of magnitude smaller than the target model) are often optimal — larger draft models produce better predictions but their per-step cost multiplied by $k$ erases the benefit.

The vertical cascade eliminates this $k\times$ multiplier entirely for neural models. The key realization is that speculative decoding's draft-then-review pattern is itself a general acceleration mechanism, not something reserved for the target-draft relationship. If we have three models — $M_{d2}$ (smallest), $M_{d1}$ (medium), $M_t$ (target) — we can arrange them such that:

1. $M_{d2}$ generates draft tokens (this is the only autoregressive generation).
2. $M_{d1}$ reviews $M_{d2}$'s drafts using speculative acceptance/rejection, producing refined tokens.
3. $M_t$ reviews $M_{d1}$'s refined tokens, producing the final output.

In this arrangement, $M_{d1}$ never generates tokens autoregressively — it only performs parallel review of $M_{d2}$'s drafts, which costs a single forward pass per review step. If $M_{d2}$ is made extremely cheap (a statistical model with negligible cost), then the only neural autoregressive generation has been pushed to the bottom of the cascade where its cost approaches zero, while $M_{d1}$ maintains high-quality predictions through its review process, and $M_t$ guarantees distributional equivalence through its final review.

Recursive generalization. The vertical cascade extends this logic to an arbitrary depth: a chain of $n$ draft models $M_{d1}, M_{d2}, ..., M_{dn}$ where each model $M_{di}$ acts as the reviewer for $M_{d(i+1)}$ and the proposer for $M_{d(i-1)}$, with $M_{dn}$ (the bottom model) being a negligible-cost statistical language model that performs the only autoregressive generation. The target model $M_t$ sits at the top, reviewing $M_{d1}$'s proposals. The paper implements this recursion directly in Algorithm 1: the function CascadeSpeculativeDraftingStep calls itself with a progressively smaller draftList, terminating when draftList is empty (the base case, where MaG generates).

Asymmetric lenience. A critical design choice enables the vertical cascade to achieve additional speedup without sacrificing output quality. Lenience, introduced in the original speculative decoding paper [14], is a hyperparameter $l \in [1, \infty)$ that loosens the acceptance criterion. Under standard speculative decoding with sampling, a draft token $x_i$ with draft probability $M_d(x)[i]$ and target probability $M_t(x)[i]$ is accepted if $M_d(x)[i] \leq M_t(x)[i]$; otherwise, it is rejected with probability $1 - M_t(x)[i] / M_d(x)[i]$. With lenience $l$, the acceptance condition becomes $M_d(x)[i] \leq l \times M_t(x)[i]$, and the rejection probability becomes $1 - (l \times M_t(x)[i]) / M_d(x)[i]$.

Lenience increases the acceptance rate (fewer tokens rejected, more tokens per target model run) at the cost of potentially producing output that differs from the target model's true distribution — because the acceptance criterion is looser, some tokens that would have been rejected under exact matching are now accepted.

The paper's key observation is that this quality-speed tradeoff only matters at the final review step. When an intermediate draft model $M_{d1}$ reviews tokens from $M_{d2}$, the output of that review is not the final output — it is a draft proposal to the next level up. If lenience is applied at this intermediate review, $M_{d1}$ accepts more tokens from $M_{d2}$, producing a longer draft for $M_t$ (or the next draft model) to review. But $M_t$'s final review (performed without lenience, i.e., $l=1$) will still enforce exact distributional equivalence. Any tokens that were "incorrectly" accepted due to intermediate lenience will be caught and rejected by the strict target model review. The net effect: intermediate lenience reduces the number of intermediate review steps needed (fewer rejections mean $M_{d1}$ can propose longer drafts per review cycle) without changing the final output distribution.

In Algorithm 1, this is implemented through the isFirstCall flag. When isFirstCall is True (meaning the target model $M_t$ is the reviewer), lenience $l$ is set to 1, enforcing exact matching. When isFirstCall is False (meaning an intermediate draft model is the reviewer), lenience can be greater than 1, enabling faster internal drafting. The paper notes (Section 3.1): "We can limit the application of lenience in the vertical cascade only when draft models review and do not apply lenience when the target model reviews. This can ensure the final output is not altered while further reducing latency."

Why this works: Corollary 4.4 intuition. The paper provides a formal proof (Corollary 4.4) that if the bottom model's cost coefficient $c_{d2}$ is negligible (i.e., $c_{d2} \ll 1$, meaning a single run of the bottom model takes near-zero time compared to the target model), then the Expected Walltime Improvement Factor (EWIF) of the vertical cascade with $M_{d1}$ and $M_{d2}$ exceeds the EWIF of using $M_{d1}$ alone with standard speculative decoding. The intuition: the vertical cascade replaces $k$ autoregressive forward passes of $M_{d1}$ with $k$ forward passes of the negligible-cost $M_{d2}$ plus one parallel review forward pass of $M_{d1}$. The cost reduces from $k \times c_{d1}$ to approximately $c_{d1}$ (plus negligible terms), while the acceptance rate at each level is preserved through the speculative review process.

---

The Horizontal Cascade: Position-Dependent Model Allocation

The empirical motivation (Figure 2). The paper measures the acceptance rate of draft tokens as a function of their position within a draft step, using FLAN-T5-SMALL, BASE, and LARGE as draft models on GSM8K and MMLU. The pattern is consistent: the first draft token has an acceptance rate of roughly 0.7–0.8 (70–80% chance of being accepted), while the 30th token has an acceptance rate of roughly 0.1 or below. The decline is approximately exponential, which follows from the probability chain: for token $i$ to be accepted, tokens $1$ through $i-1$ must all have been accepted first.

The probability chain formalization. Let $\alpha_j$ be the acceptance probability of the $j$-th draft token conditioned on all previous tokens being accepted (the marginal probability that token $j$ passes review given it is reached). The unconditional probability that token $i$ is accepted — meaning it is both reached (all previous tokens accepted) and passes review — is the product:

$P(\text{token } i \text{ accepted}) = \prod_{j=1}^{i} \alpha_j$

where $\alpha_j$ is the conditional acceptance probability for position $j$. This product decays with $i$ because each factor $\alpha_j$ is less than 1 (no draft model perfectly matches the target model).

The resource allocation problem. Standard speculative decoding uses the same draft model for all $k$ token positions, with a uniform cost per token $c_d = c(M_t, M_d)$. But the expected contribution of token $i$ to the total accepted token count — and therefore to the speedup — is $\prod_{j=1}^{i} \alpha_j$, which declines with $i$. This means later tokens contribute less to the expected speedup but cost the same to generate. The horizontal cascade addresses this misalignment by assigning cheaper models to lower-value positions.

The allocation policy. Given a set of draft models $\{M_1, M_2, ..., M_k\}$ ordered by decreasing size (and cost), the horizontal cascade assigns model $M_i$ to generate the $i$-th draft token. The largest model $M_1$ generates the first token (highest acceptance probability, highest value), $M_2$ generates the second token (lower acceptance probability), and so on, down to the smallest model for the $k$-th token (lowest acceptance probability). This stops when the smallest model — typically the MaG statistical model — is reached for the final token positions.

Cost structure. Let $c_i = c(M_t, M_i)$ be the cost coefficient for model $M_i$ (the ratio of a single forward pass of $M_i$ to a single forward pass of $M_t$). Since $M_1$ is the largest draft model, $c_1$ is the largest draft cost; since $M_k$ is the smallest (MaG), $c_k \approx 0$. The total drafting cost per step becomes $\sum_{i=1}^{k} c_i$ rather than $k \times c_1$ (if the largest model generated all tokens) or $k \times c_d$ (if a single medium model generated all tokens).

Theoretical justification (Theorem 4.5). The EWIF of the horizontal cascade is:

$T(k, \alpha_1, ..., \alpha_k, c_1, ..., c_k) = \frac{\sum_{i=0}^{k} \prod_{j=1}^{i} \alpha_j}{1 + \sum_{i=1}^{k} c_i}$

where $k$ is the number of draft tokens, $\alpha_i$ is the acceptance probability for token $i$, and $c_i$ is the cost coefficient of the model generating token $i$. The numerator $\sum_{i=0}^{k} \prod_{j=1}^{i} \alpha_j$ is the expected number of accepted tokens per step (with the convention that the empty product for $i=0$ equals 1, representing the guaranteed acceptance of 0 tokens). The denominator $1 + \sum_{i=1}^{k} c_i$ is the total cost per step: 1 for the target model's review forward pass plus the sum of all draft model forward passes.

What this equation computes: For a single speculative decoding step with position-dependent models, the expected speedup as the ratio of tokens produced (expected accepted tokens) to time spent (target model cost plus draft model costs). Standard speculative decoding is the special case where $\alpha_i = \alpha$ (constant acceptance rate) and $c_i = c$ (constant cost), reducing to the familiar $\frac{1-\alpha^{k+1}}{(1-\alpha)(1+kc)}$.

Why this form matters: The numerator separates into position-dependent factors, making explicit that later positions contribute multiplicatively smaller terms. This decomposition enables the analysis in Corollary 4.6, which computes the derivative of EWIF with respect to each position's acceptance rate:

$\frac{dT}{d\alpha_1} = \frac{\sum_{i=1}^{k} \prod_{j=2}^{i} \alpha_j}{1 + \sum_{i=1}^{k} c_i} \quad \text{vs.} \quad \frac{dT}{d\alpha_k} = \frac{\prod_{j=1}^{k-1} \alpha_j}{1 + \sum_{i=1}^{k} c_i}$

The derivative with respect to $\alpha_1$ (first token acceptance) is larger than the derivative with respect to $\alpha_k$ (last token acceptance) by a factor of approximately $\sum_{i=1}^{k} \prod_{j=2}^{i} \alpha_j / \prod_{j=1}^{k-1} \alpha_j$, which is substantially greater than 1. This proves that earlier positions have higher marginal impact on speedup, justifying the allocation of larger (more accurate, more expensive) models to earlier positions.

Simulated validation (Table 1). Using acceptance probabilities and cost coefficients from Leviathan et al. [14] on CNN/DailyMail and WMT EnDe, the paper simulates EWIF under the Bernoulli acceptance assumption. For CNN/DailyMail, the horizontal cascade with BASE generating token 1 and SMALL generating tokens 2–3 achieves EWIF of 3.03, compared to 2.96 for BASE alone (all tokens) and 2.65 for SMALL alone. For WMT EnDe, the cascade achieves 3.93 versus 3.75 for BASE alone. These simulations predate the full experiments but demonstrate that position-dependent allocation can improve upon uniform allocation even before accounting for the vertical cascade.

Implementation in Algorithm 1. The horizontal cascade is realized through the for loop over draft models (line: "for i ← 1 to n do"). For each draft model $M_{di}$ in sequence (from largest to smallest), the algorithm generates $k_i$ tokens (where $k_i$ comes from the $K_{nn}$ matrix) by repeatedly calling the vertical cascade (recursive call) with the remaining smaller draft models as the curDraftList. The loop while curGen.length - curPrefix.length is less than ki ensures that exactly $k_i$ tokens are produced by draft model $M_{di}$ and its subordinates before moving to the next (smaller) draft model for the next block of tokens.

The $K_{nn}$ hyperparameter matrix. The horizontal cascade is parameterized by an upper-triangular matrix $K_{nn}$ where $n$ is the total number of draft models. Each row $i$ specifies the generation budget for a layer of the recursive calls where model $M_{di}$ is the reviewer. Specifically:

• $K_{nn}[i][i]$ (the diagonal entry) specifies how many tokens model $M_{di}$ generates directly (via its subordinates in the vertical cascade) when it is the reviewer in a recursion layer.
• $K_{nn}[i][j]$ for $j > i$ specifies the generation budget when $M_{di}$ is the reviewer and $M_{dj}$ and smaller models are available as subordinates.

The matrix is upper-triangular because model $M_{di}$ only reviews models smaller than itself (index $j > i$). For a setup with three models (e.g., BASE, SMALL, MaG), $K_{33}$ might have entries: $K[1][1] = 8$ (BASE reviews 8 tokens from SMALL+MaG), $K[1][2] = 13$ (a different budget for a different recursion configuration), $K[2][2] = 1$ (SMALL reviews 1 token from MaG), with other entries zero.

The paper reports specific hyperparameters in Table 5. For CS Drafting with BASE, SMALL, MaG on GSM8K (model size metric): $k_{11} = 8$, $k_{12} = 13$, $k_{22} = 1$, and lenience $l = 3$. Here, $k_{11}$ and $k_{12}$ are the step limitations when the target model reviews (the first row of $K_{nn}$), and $k_{22}$ is the step limitation when the first draft model reviews the second.

---

The Max-Gram (MaG) Algorithm: Statistical Drafting at the Bottom

The design motivation. Both cascades terminate at a statistical language model that performs the only autoregressive generation in the system. This model must be: (1) extremely fast — its cost must be negligible relative to neural model inference, (2) capable of generating plausible tokens that have non-trivial acceptance probability by the reviewer above it, and (3) general-purpose — it should work across different tasks and domains without task-specific training. A standard bigram model satisfies (1) and (3) but may have low acceptance probability, limiting speedup. The MaG algorithm augments a bigram model with pattern matching to improve (2).

The core observation. In many language model generation tasks, particularly summarization, machine translation, and reasoning tasks with chain-of-thought prompts, the model's output frequently reuses words, phrases, and multi-token sequences from the input prompt. For example, a math word problem solution might repeat numbers from the problem statement; a translation might repeat proper nouns; a summarization might reuse key phrases. This means that greedy matching between the input and the end of the current generation can produce high-quality draft tokens without any learned parameters.

The MaG algorithm (Appendix A, Listing 1). The algorithm operates as follows:

1. Find maximal match: Given the input sequence (encoder_ids) and the current generation prefix (input_ids), find the longest suffix of the generation that appears as a contiguous subsequence in the input. This is done greedily: start by checking if the last token of the generation matches anywhere in the input; if so, extend the match backward token by token (checking if the last 2 tokens appear as a contiguous sequence, then last 3, etc.) until the match fails.

2. Extract continuation: Once the maximal match is found (the longest suffix of the generation that appears in the input), take the token(s) immediately following that match in the input as the predicted next token(s). The number of tokens predicted is controlled by parameter $n$ (the paper uses $n=10$ by default — "we use MaG to generate 10 tokens at once, as it is rare for more than 10 tokens to be accepted").

3. Fallback to bigram: If no match exists (the last token of the generation doesn't appear in the input at all), fall back to a bigram model based on the probability distribution of Wikipedia. This ensures the model can always produce output even for tasks where the output vocabulary diverges from the input.

GPU-friendly implementation. The paper provides a PyTorch implementation in Listing 1 that performs the matching operation efficiently on GPU using tensor operations: encoder_ids[0] == input_ids[0, -1] performs a vectorized equality check to find all positions where the last generated token appears in the input, and subsequent steps use shifted comparisons and cumulative AND operations to find maximal matches without sequential loops over token positions.

Why MaG works for CS Drafting. The MaG model has negligible cost — its parameters are essentially the tokenizer vocabulary (for the bigram component) plus the matching logic — making the memory cost "negligible" (Section 5.2). This satisfies the $c_{d} \ll 1$ condition required by Corollary 4.4. Furthermore, because the vertical cascade applies lenience at intermediate reviews, the MaG's relatively low standalone acceptance probability is compensated for: lenience loosens the acceptance criterion, allowing more MaG tokens to pass through to higher-level reviewers where they may be refined or rejected based on stricter criteria.

---

The Unified Algorithm 1: Combining Both Cascades

Algorithm structure. Algorithm 1, CascadeSpeculativeDraftingStep, is a recursive function that implements both cascades simultaneously. It takes as input:

• draftList: a list of draft models $[M_{d1}, ..., M_{dn}]$ ordered from largest to smallest (the first element is the model that will act as the reviewer for this recursion layer).
• target: the model that will review the output of this recursion layer (either $M_t$ for the top-level call or a larger draft model for intermediate calls).
• prefix: the current token sequence to extend.
• isFirstCall: a boolean flag that is True only for the top-level call where $M_t$ is the reviewer.
• Knn: the hyperparameter matrix controlling generation budgets.
• l: the lenience parameter.

Base case: MaG generation. If draftList is empty (line: "if draftList is empty then"), the recursion has reached the bottom. The algorithm takes the first (and only) model from the list (which, due to the recursion structure, is the MaG model), runs it on the current prefix to generate tokens, and returns those tokens along with their log probabilities. This is the only point in the system where autoregressive generation occurs — and it uses the statistical MaG model, not a neural network.

Recursive case: horizontal + vertical cascade integration. For each draft model $M_{di}$ in draftList (iterating from largest to smallest, the horizontal cascade loop):

1. Prepare the next recursion call: Set curTarget to $M_{di}$ (this model will review the output of the next recursion). Set curDraftList to the sublist of draftList starting from index $i+1$ (all models smaller than $M_{di}$ will serve as its draft models in the vertical cascade). Extract the appropriate submatrix from $K_{nn}$ to control generation budgets for this sub-recursion.

2. Generate $k_i$ tokens: In a while loop, repeatedly call CascadeSpeculativeDraftingStep recursively with curDraftList as the draft models and curTarget as the reviewer. Each recursive call produces a variable number of draft tokens (via speculative review of the smaller models' output). Append these tokens to curGen. Continue until curGen has grown by at least $k_i$ tokens beyond curPrefix (where $k_i$ comes from the first row of curK).

3. Advance to next model: After $M_{di}$ has produced its allocation of $k_i$ tokens, the loop continues to the next (smaller) draft model for the next block of tokens. This implements the horizontal cascade: $M_{d1}$ (largest) generates the first $k_1$ tokens, $M_{d2}$ generates the next $k_2$ tokens, and so on.

Final review step. After the horizontal cascade loop completes (all draft models have generated their allocated tokens), the accumulated curGen (all draft tokens from all models) is reviewed by the original target model using the review function. This function implements the standard speculative decoding review with acceptance/rejection sampling. The isFirstCall flag controls lenience: if True (top-level call where target is $M_t$), lenience $l$ is forced to 1 before the review, ensuring exact output distribution matching.

The review function (not shown in detail in Algorithm 1 but referenced) implements the standard speculative decoding verification procedure. Given a draft token sequence curGen and the draft model's probabilities curProbs at each position, it:

• Runs the target model on the prefix plus draft tokens in a single parallel forward pass, obtaining target probabilities at each draft position.
• For each draft token position $i$:
  • If $M_d(x)[i] \leq l \times M_t(x)[i]$, accept the token.
  • Otherwise, reject the token with probability $1 - (l \times M_t(x)[i]) / M_d(x)[i]$. If rejected, resample from the normalized residual distribution $\text{norm}(\max(0, M_t(x)[i] - M_d(x)[i]))$, and reject all subsequent tokens.
• After processing all draft tokens, generate one additional token from the target model (the bonus token that speculative decoding always produces after the verified draft).

The isFirstCall mechanism. This flag is critical for correctness. When isFirstCall is True, the algorithm sets $l \leftarrow 1$ before the review, overriding any lenience that was used internally. This ensures that the final target model review enforces exact distributional equivalence regardless of how much lenience was applied in intermediate draft-to-draft reviews. When isFirstCall is False (intermediate recursion layers), lenience $l$ can be greater than 1, allowing faster internal drafting at the cost of potentially including tokens that don't strictly match the intermediate reviewer's distribution — but since these tokens will be re-reviewed by the target model (with $l=1$), any distributional distortion is corrected before output.

Example execution (Figure 1). The paper illustrates CS Drafting with $M_t$ (target), $M_{d1}$, $M_{d2}$, and $M_{d3}$ (MaG). The execution proceeds as:

1. $M_{d3}$ (MaG) generates tokens autoregressively using max-gram pattern matching.
2. $M_{d2}$ reviews $M_{d3}$'s tokens (vertical cascade, lenience allowed), producing refined tokens.
3. $M_{d1}$ reviews $M_{d2}$'s refined tokens (vertical cascade, lenience allowed), producing further refined tokens — but only for the first block of positions (horizontal cascade: larger model on earlier tokens).
4. For the next block of positions, $M_{d2}$ directly reviews $M_{d3}$'s tokens (smaller model on later tokens).
5. $M_t$ reviews all accumulated tokens from all draft models (final review, $l=1$).

---

Theoretical Framework: EWIF and Generating Functions

The paper develops a theoretical framework for analyzing the expected speedup of cascade speculative decoding systems. The analysis proceeds in three stages: (1) defining the Expected Walltime Improvement Factor (EWIF), (2) using probability generating functions to analyze the vertical cascade, and (3) direct expectation analysis for the horizontal cascade.

EWIF definition. The Expected Walltime Improvement Factor is defined as the expected number of tokens produced per unit time, normalized by the target model's autoregressive generation rate. Under the i.i.d. assumption (each token's acceptance is an independent Bernoulli trial with probability $\alpha$), EWIF simplifies to:

$\text{EWIF} = \frac{\text{expected tokens per step}}{\text{expected time per step}}$

where expected tokens per step depends on acceptance probabilities and the draft length $k$, and expected time per step depends on the cost coefficients of all models involved. The paper notes that despite the simplifying i.i.d. assumption, EWIF "aligns with the experimental results in most instances" [14].

Probability generating function for speculative decoding (Theorem 4.1). For standard speculative decoding between $M_t$ and $M_d$ with acceptance probability $\alpha$ and draft length $k$, the probability generating function $\phi_{(\alpha,k)}(x)$ — a polynomial where the coefficient of $x^i$ is the probability of accepting exactly $i$ tokens in a single step — satisfies:

$\phi_{(\alpha,k)}(x) = 1 + (x - 1)\frac{1 - \alpha^{k+1}x^{k+1}}{1 - \alpha x}$

where $x$ is the formal variable of the generating function, $\alpha$ is the per-token acceptance probability, and $k$ is the number of draft tokens.

What this represents: The generating function encodes the entire acceptance distribution in a single expression. Expanding the right-hand side as a polynomial in $x$ yields coefficients: the coefficient of $x^i$ is $\alpha^{i-1} - \alpha^i$ for $i = 1, ..., k$, and the coefficient of $x^{k+1}$ is $\alpha^k$. This captures the fact that accepting exactly $i$ tokens (for $i < k+1$) requires tokens 1 through $i$ to be accepted (probability $\alpha^i$) and token $i+1$ to be rejected (probability $1-\alpha$), while accepting all $k+1$ tokens (the $k$ draft tokens plus the bonus target-generated token) requires all $k$ draft tokens to be accepted (probability $\alpha^k$).

Why this expression: The closed form $1 + (x-1)\frac{1-\alpha^{k+1}x^{k+1}}{1-\alpha x}$ is derived by summing the geometric series of probabilities and factoring. It's useful because differentiation yields the expected number of accepted tokens immediately: $\phi'_{(\alpha,k)}(1) = \frac{1-\alpha^{k+1}}{1-\alpha}$, which appears in the EWIF numerator. Without the generating function, computing expectations over the variable-length acceptance distribution would require case analysis.

EWIF for standard speculative decoding (Corollary 4.2). The expected number of accepted tokens per step is $\phi'_{(\alpha,k)}(1) = \frac{1-\alpha^{k+1}}{1-\alpha}$. The time per step is the target model's single forward pass (cost 1) plus $k$ draft model forward passes (cost $ck$). Therefore:

$\text{EWIF}_{\text{SD}} = \frac{\phi'_{(\alpha,k)}(1)}{ck + 1} = \frac{1 - \alpha^{k+1}}{(1 - \alpha)(ck + 1)}$

where $c = c(M_t, M_d)$ is the cost coefficient of the draft model relative to the target model.

Extension to vertical cascade (Theorem 4.3). For a vertical cascade with two draft models $M_{d1}$ and $M_{d2}$, where $M_{d2}$ drafts for $M_{d1}$ with acceptance probability $\alpha' = \alpha(M_{d1}, M_{d2})$ and draft length $k$, and $M_{d1}$ drafts for $M_t$ with acceptance probability $\alpha = \alpha(M_t, M_{d1})$, and the target model reviews $n$ times:

$\text{EWIF}_{\text{VC}} = \frac{1 - \alpha \phi^n(\alpha)}{(1 - \alpha)(1 + n c_{d1} + nk c_{d2})}$

where $\phi(x) = \phi_{(\alpha', k)}(x)$ is the generating function for the $M_{d2} \to M_{d1}$ speculative decoding step, $\phi^n(\alpha)$ means $\phi(\phi(...\phi(\alpha)...))$ composed $n$ times and evaluated at $\alpha$, and $c_{d1}, c_{d2}$ are the cost coefficients.

What this equation computes: The numerator $1 - \alpha\phi^n(\alpha)$ divided by $(1-\alpha)$ gives the expected number of tokens accepted by $M_t$ after $n$ rounds of $M_{d1}$ reviewing $M_{d2}$'s drafts. The denominator gives the total cost: 1 for the target model's forward pass, $n c_{d1}$ for $n$ forward passes of $M_{d1}$ (reviewing), and $nk c_{d2}$ for $nk$ forward passes of $M_{d2}$ (the autoregressive generation at the bottom).

The operator method for deriving the numerator. The paper introduces an operator $T_\alpha$ that maps a probability generating function to the expected number of accepted tokens under acceptance probability $\alpha$. For a single term $x^j$ (representing $j$ tokens generated by the draft system), $T_\alpha(x^j) = \frac{1-\alpha^{j+1}}{1-\alpha}$ — this is the expected number of tokens accepted when the target model reviews a draft of length $j$. By linearity, $T_\alpha$ extends to any polynomial (any generating function). Applying $T_\alpha$ to $\phi^n(x)$ (the generating function for the total tokens produced by $n$ rounds of $M_{d2} \to M_{d1}$ drafting) yields the expected tokens accepted by $M_t$.

Why this form: The composition $\phi^n(\alpha)$ captures the recursive nature — the output of one level of speculative decoding becomes the input to the next level. The generating function formalism makes this composition natural: instead of convolving probability distributions (which would require $n$-fold convolutions), we compose generating functions and evaluate. This is the key mathematical insight that makes the vertical cascade analysis tractable.

Corollary 4.4 (Vertical cascade almost always improves EWIF). The paper proves that $\phi_{\alpha',k}(\alpha) < \alpha$ for any $0 < \alpha, \alpha' < 1$ and $k > 0$. This means that the generating function evaluated at $\alpha$ produces a value strictly less than $\alpha$. Consequently, if $c_{d2} \ll 1$ (the bottom model has negligible cost, so $nk c_{d2} \approx 0$), then:

$\frac{1 - \alpha\phi^n(\alpha)}{(1-\alpha)(1 + n c_{d1} + nk c_{d2})} > \frac{1 - \alpha^{n+1}}{(1-\alpha)(1 + n c_{d1})}$

The right-hand side is the EWIF for standard speculative decoding with $M_{d1}$ as the draft model and step size $n$. The inequality proves that adding a negligible-cost model below $M_{d1}$ strictly improves EWIF — the vertical cascade always helps (under the i.i.d. assumption and negligible bottom cost).

Horizontal cascade analysis (Theorem 4.5 and Corollary 4.6). The EWIF for position-dependent models is derived directly (without generating functions) as:

$T(k, \alpha_1, ..., \alpha_k, c_1, ..., c_k) = \frac{\sum_{i=0}^{k} \prod_{j=1}^{i} \alpha_j}{1 + \sum_{i=1}^{k} c_i}$

Corollary 4.6 computes partial derivatives $\frac{dT}{d\alpha_l}$ for each position $l$:

$\frac{dT}{d\alpha_l} = \frac{\sum_{i=l}^{k} \prod_{j=1, j \neq l}^{i} \alpha_j}{1 + \sum_{i=1}^{k} c_i}$

The derivative for $\alpha_1$ (first token) is:

$\frac{dT}{d\alpha_1} = \frac{\sum_{i=1}^{k} \prod_{j=2}^{i} \alpha_j}{1 + \sum_{i=1}^{k} c_i}$

The derivative for $\alpha_k$ (last token) is:

$\frac{dT}{d\alpha_k} = \frac{\prod_{j=1}^{k-1} \alpha_j}{1 + \sum_{i=1}^{k} c_i}$

What these derivatives mean: $dT/d\alpha_l$ is the marginal improvement in EWIF from improving the acceptance rate at position $l$. Since $\sum_{i=1}^{k} \prod_{j=2}^{i} \alpha_j > \prod_{j=1}^{k-1} \alpha_j$ (the sum of products for positions 1 through $k$ exceeds the single product for position $k$), the derivative for $\alpha_1$ is strictly larger than for $\alpha_k$. This formalizes that investing model capacity (which increases $\alpha_l$) in earlier positions yields higher returns than investing in later positions — the mathematical justification for the horizontal cascade's allocation policy.

---

Design Choices and Their Justifications

Choice 1: Recursive vertical cascade instead of two-level staged decoding. The paper explicitly extends Spector and Re [19]'s two-level approach to full recursion. The justification (Section 3.1, Section 6.2) is that only full recursion to a negligible-cost statistical model eliminates all autoregressive neural generation. A two-level system still has the bottom model performing autoregressive neural generation, leaving the fundamental bottleneck partially intact.

Choice 2: Asymmetric lenience (applied internally, not at target review). This is the enabling design choice for the vertical cascade. Without lenience, intermediate speculative decoding steps would have the same acceptance rates as the target-level step, limiting the benefit of the cascade. With symmetric lenience (applied everywhere), output quality would degrade. The asymmetric approach extracts speedup from intermediate steps while preserving exact output equivalence.

Choice 3: MaG with bigram fallback instead of a pure bigram or pure N-gram model. Pure bigram models have low acceptance probability, limiting speedup. Higher-order N-gram models have exponentially larger parameter counts and still may not capture long-range dependencies. The MaG approach exploits the observation that tokens from the input are reused in the output (common in many tasks) while falling back to bigram probabilities for novel tokens. The GPU-friendly implementation (tensor-based matching) avoids the sequential scanning that would make CPU-based pattern matching a bottleneck.

Choice 4: Upper-triangular $K_{nn}$ matrix for hyperparameter specification. This structure reflects the natural constraint that a model only reviews models smaller than itself. The upper-triangular form reduces the hyperparameter count from $n^2$ to $n(n+1)/2$ while still allowing position-dependent and recursion-layer-dependent budgets. The paper reports specific values in Table 5 and notes in Appendix B that for typical usage (without hyperparameter tuning), a simple fixed matrix like [[2, 10], [0, 10]] works well.

Choice 5: Generating functions for vertical cascade analysis instead of direct expectation computation. The generating function approach (Section 4.1) avoids $n$-fold convolutions of the acceptance distribution. The operator $T_\alpha$ maps generating functions to expected acceptance counts, and composition of generating functions naturally models the output of one speculative decoding layer feeding into the next. This mathematical machinery is not strictly necessary for the algorithm but provides formal justification for why the vertical cascade improves EWIF.

Choice 6: EWIF under i.i.d. assumption as the theoretical framework. The paper acknowledges that real token acceptances are not i.i.d. — the acceptance probability depends on context, position, and the specific tokens. However, the i.i.d. assumption makes the analysis tractable while still producing predictions that "align with experimental results in most instances" [14]. This is a common tradeoff in systems analysis: simplified models that capture the first-order effects (exponential decay of acceptance probability, cost ratios between models) while abstracting away second-order complexities.

Choice 7: Greedy decoding (temperature 0) for experiments. The paper states (Section 5.1): "Since we do not observe any significant difference between sampling with temperature 1 and greedy decoding in previous speculative decoding experiments [14], and to ensure our experiments are fully reproducible, we perform sampling at temperature 0, i.e., using greedy decoding by default." Greedy decoding simplifies the acceptance/rejection procedure (it becomes deterministic: accept if $\text{argmax }M_d(x)[i] = \text{argmax }M_t(x)[i]$) and eliminates the stochasticity that would make walltime measurements noisy.

Choice 8: No fine-tuning. The paper explicitly does not fine-tune any models (Section 5.1: "To align our experiment with current common usage, we do not perform fine-tuning for CS Drafting"). This distinguishes CS Drafting from approaches like DistillSpec [29] that require distillation training, and makes it immediately deployable with off-the-shelf models. The tradeoff is potentially lower acceptance rates than could be achieved with fine-tuned draft models, but the cascade structure compensates by reducing drafting cost rather than improving acceptance rate.

4. Key Insights and Innovations

Innovation 1: Drafting Efficiency as a First-Class Design Dimension, Decoupled from Acceptance Rate

The dominant framing in speculative decoding research — from the original formulations [14, 4] through knowledge distillation variants [29] and tree-attention approaches [15, 3] — treats the drafting phase as a monolithic step whose sole optimization lever is the acceptance rate: make the draft model better at matching the target model, and speedup improves. This framing implicitly assumes that the cost structure of drafting is fixed — given a draft model, the time to produce $k$ draft tokens is simply $k$ autoregressive forward passes, and the only question is how many of those $k$ tokens survive review.

CS Drafting makes a conceptual move that separates it from all prior work: it decouples drafting efficiency from acceptance rate by treating the internal structure of the drafting process as an independent design space. The paper identifies two distinct inefficiencies — the autoregressive generation bottleneck within neural draft models, and the uniform allocation of model capacity across token positions — and addresses them through mechanisms (vertical cascade, horizontal cascade) that improve speedup without changing acceptance probabilities at any level. This is not an incremental refinement of existing speculative decoding; it is a fundamental reframing of what optimization dimensions are available.

The significance of this reframing extends beyond the specific mechanisms introduced. It implies that future speculative decoding systems should be evaluated on at least three axes — acceptance rate, draft model cost, and internal drafting architecture — rather than collapsing everything into a single "quality of the draft model" metric. The paper's empirical demonstration that the fastest speculative decoding setup (SMALL alone) is not the best setup when the vertical cascade is allowed (BASE + MaG consistently outperforms SMALL + MaG in Table 2, reversing the single-draft-model ordering) proves that this reframing has practical consequences: the optimal draft model architecture changes when the drafting process itself can be restructured.

The EWIF framework (Section 4) provides the formal language for this decoupling. By decomposing speedup into terms that depend on acceptance probability and terms that depend on cost structure, the generating function analysis shows mathematically that the two can be optimized independently — Corollary 4.4 proves that improving cost structure (adding a negligible-cost bottom model) strictly improves EWIF regardless of acceptance rates. This is the paper's most theoretically general contribution.

---

Innovation 2: The Vertical Cascade as a Recursive Generalization of Speculative Decoding — Not Just Additional Layers

Prior work by Spector and Re [19] demonstrated that two layers of speculative decoding — a small model drafting for a medium model, which drafts for the target model — could improve efficiency. This was presented as a specific architectural choice: stack two speculative decoding stages. CS Drafting transforms this from a fixed-depth architecture into a recursive algorithm with three intellectual moves that the prior work missed entirely.

First, the recursion principle. The paper recognizes that speculative decoding's draft-then-review pattern is a general computation primitive that can be composed arbitrarily, not something reserved for the target-draft boundary. Where Spector and Re stopped at two levels, CS Drafting pushes the recursion to its logical endpoint: a chain of $n$ drafting levels terminating at a statistical language model whose cost is so low that adding more levels would yield diminishing returns. This is a conceptual generalization, not merely adding more layers — it transforms a specific system design into a recursive pattern with a natural base case.

Second, asymmetric lenience as an enabling insight. The original speculative decoding paper [14] introduced lenience as a global speed-quality tradeoff: increase lenience for more speed, decrease for better quality. Spector and Re [19] applied lenience uniformly. CS Drafting's key observation — that lenience only affects final output quality when applied at the target model's review — is a genuine diagnostic insight. It separates the verification role of speculative decoding (ensuring distributional equivalence, which must be strict at the final level) from the acceleration role (generating draft tokens efficiently, which can use lenience aggressively at intermediate levels). This asymmetric application means the vertical cascade can extract speedup from intermediate lenience that prior approaches could only obtain by sacrificing output quality.

Third, the statistical model as the natural recursion base case. Prior work treated the bottom draft model as necessarily neural — a smaller version of the target model, or a distilled version. CS Drafting identifies that the recursion's termination condition should be a model whose cost is negligible relative to all others, and that a statistical language model (MaG) satisfies this while still providing non-trivial drafting utility when combined with lenience at the level above it. This is a design principle with implications beyond the specific MaG implementation: any drafting acceleration system structured as a recursion should terminate at the point where adding another level costs more than it saves.

The empirical significance is that the vertical cascade alone — BASE + MaG without the horizontal cascade — achieves up to a 70% speedup over standard speculative decoding on MMLU (Table 2), without any additional neural model deployment. Since MaG's memory cost is "negligible" (vocabulary-sized bigram parameters), this speedup comes with essentially zero additional overhead.

---

Innovation 3: Position-Dependent Resource Allocation as a New Optimization Axis in Speculative Decoding

Standard speculative decoding treats all $k$ draft token positions identically: the same model, the same cost, the same contribution to expected tokens per step. Figure 2 provides empirical evidence that this uniformity is wasteful — acceptance probability decays exponentially with position because it's a multiplicative chain — but the paper's conceptual contribution goes beyond documenting this decay. It identifies position as an allocation dimension and provides both a theoretical framework (Corollary 4.6, computing marginal returns of improving acceptance at each position) and a practical mechanism (horizontal cascade via the $K_{nn}$ matrix) for unequal resource distribution.

This is a genuinely new design axis in speculative decoding. Prior work asked "what model should draft?" — a global choice applied uniformly across positions. CS Drafting asks "what model should draft at each position?" — decomposing the drafting problem into $k$ sub-problems with different cost-value profiles. The horizontal cascade is the first mechanism to exploit this decomposition, assigning larger (more accurate, more expensive) models to high-value early positions and smaller (cheaper) models to low-value late positions.

The theoretical analysis in Corollary 4.6 proves that this allocation is optimal under the cost-value framework: the derivative of EWIF with respect to $\alpha_1$ (first-token acceptance) is larger than the derivative with respect to $\alpha_k$ (last-token acceptance), meaning that marginal investment in model capacity yields higher returns at earlier positions. This is not an empirical heuristic — it's a structural property of the probability chain that any position-aware allocation strategy should exploit.

Table 1 provides simulated validation independent of the full system: even under the simple Bernoulli acceptance assumption, a 2-model horizontal cascade (BASE on first token, SMALL on later tokens) exceeds the EWIF of either uniform strategy. This demonstrates that position-dependent allocation is a self-contained improvement, orthogonal to the vertical cascade. The fact that both cascades can be combined (Algorithm 1, Table 2) confirms that they address independent efficiency dimensions — one optimizes the generation mechanism within positions (vertical), the other optimizes resource distribution across positions (horizontal).

The practical significance is that the horizontal cascade requires no additional models beyond what speculative decoding already deploys — it merely changes how existing draft models are used. For practitioners already running speculative decoding with multiple draft model options (e.g., experimenting with different sizes), the horizontal cascade provides a principled way to compose them for better performance than any single model alone.

---

Innovation 4: The Max-Gram Algorithm as a Demonstration That Statistical Drafting Viability Depends on Task Structure, Not Just Model Size

The paper's introduction of the Max-Gram (MaG) algorithm is straightforward as a mechanism — greedy input-output token matching with a bigram fallback. The intellectual contribution is not the algorithm's complexity but rather the diagnostic insight it operationalizes: that in certain task families (summarization, machine translation, chain-of-thought reasoning), a statistical model exploiting input-output token overlap can serve as a viable bottom-level draft model in a speculative cascade, because the value of its predictions is amplified through lenient intermediate review rather than requiring high standalone acceptance.

This insight matters because it contradicts a natural assumption: that draft model quality must monotonically increase with model capacity and training. A bigram model's standalone acceptance rate is far lower than even the smallest neural draft model, and prior work that tested statistical language models as direct draft models for the target model would have found them inadequate. CS Drafting shows that a statistical model's predictions become viable not by being accurate enough to survive strict target model review directly, but by being accurate enough to survive lenient intermediate review, with the intermediate neural model then refining and extending them for the stricter review above.

The MaG algorithm's task-dependent effectiveness — it exploits the observation that "in language model generation, some words and phrases from the input query frequently reappear in the generated content" (Section 3.3) — reveals a broader principle: the optimal bottom-level drafting strategy depends on the structural relationship between input and output in the target task. For tasks with high input-output token overlap (summarization, translation, reasoning with given numbers), pattern-matching is effective. For tasks with low overlap (open-ended creative generation), different statistical strategies might be needed. The paper doesn't explore this task-dependence systematically, but the MaG design opens the question: what is the optimal bottom-level statistical model for different task families?

The GPU-friendly implementation (Appendix A, Listing 1) is also notable as an engineering contribution: it shows that CPU-based pattern matching is unnecessary, and that tensor operations can perform the matching without introducing a sequential bottleneck that would negate the speedup from avoiding neural generation.

---

Innovation 5: EWIF as a Compositional Analysis Framework for Multi-Level Speculative Systems

The Expected Walltime Improvement Factor (EWIF) and the generating function machinery (Theorems 4.1, 4.3, 4.5) are introduced as analysis tools, not as algorithm components. Their conceptual contribution is providing the first formal framework for reasoning about nested speculative decoding systems — systems where one speculative decoding stage feeds into another, recursively.

Prior to this work, the analysis of speculative decoding (e.g., in Leviathan et al. [14]) treated a single draft-target pair in isolation. The generating function approach — particularly the operator $T_\alpha$ that maps a probability generating function to expected acceptances — makes multi-level analysis tractable by enabling composition: the output distribution of one speculative decoding stage becomes the input generating function for the next, and function composition replaces the $n$-fold convolutions that would otherwise be needed.

This is a theoretical infrastructure contribution rather than an algorithmic one. The generating function approach doesn't change what CS Drafting computes at runtime, but it provides the formal justification for why the vertical cascade works (Corollary 4.4) and enables future researchers to analyze more complex cascade architectures — for instance, systems where different levels have different acceptance probabilities, draft lengths, or cost structures — without re-deriving expectations from scratch.

The operator method is particularly elegant. Instead of computing $E[\text{accepted tokens}] = \sum_i i \cdot P(\text{accept exactly } i)$ by enumerating cases, the operator maps a generating function $f(x) = \sum p_i x^i$ to an expected acceptance count under acceptance probability $\alpha$: $T_\alpha(x^j) = \frac{1-\alpha^{j+1}}{1-\alpha}$. Linearity extends this to any polynomial, enabling the analysis of empirical token-length distributions (not just theoretically modeled ones) by applying $T_\alpha$ to the observed generating function. This abstraction — separating the distribution of draft lengths from the acceptance probability — is a conceptual tool that generalizes beyond the specific cascade configurations in the paper.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper uses two standard benchmarks: GSM8K[7], a dataset of 8,500 grade-school math word problems requiring multi-step arithmetic reasoning (2–8 steps), and MMLU[10] (Massive Multitask Language Understanding), a benchmark spanning 57 diverse subjects from elementary science to advanced law. Both are used in a zero-shot chain-of-thought setup [13, 23], meaning the model receives a reasoning prompt but no task-specific examples. The paper does not specify using a particular validation/test split — experiments appear to run on the standard test sets.

• Base model(s). For encoder-decoder experiments, the target model is FLAN-T5-XXL (11 billion parameters) from the FLAN-T5 family [6], chosen because the family provides a wide size range (77M to 11B parameters) for draft model selection. Draft models are FLAN-T5-BASE (~250M parameters) and FLAN-T5-SMALL (~80M parameters). For decoder-only experiments, the target model is Vicuna-7B[28], a fine-tuned LLaMA [20] model, with a 68M-parameter model sharing the same tokenizer as the reviewing draft model. The Max-Gram (MaG) statistical model serves as the bottom-level generating draft model in all CS Drafting configurations. No models are fine-tuned — all are used off-the-shelf.

• Metrics. Two complementary speedup metrics are reported:

  • Standardized Walltime Improvement (SWI): Assumes each forward pass of a model takes constant time, measured either by model size (number of parameters, "Speedup (MS)" in tables) or by previously reported time costs from Leviathan et al. [14] ("Speedup (PW)"). SWI eliminates hardware variability and is fully reproducible.
  • Walltime: Actual elapsed time measured as tokens generated per second on the experimental GPU. Less reproducible but representative of real-world performance. All walltime experiments run on a single NVIDIA A40 GPU.

  Both metrics are expressed as speedup factors relative to autoregressive generation with the target model alone (speedup = 1 for autoregressive baseline).

• Baselines. The paper compares against:

  • Autoregressive generation: The target model generating tokens one-by-one without speculative decoding (speedup normalized to 1).
  • Speculative Decoding (S Decoding) [14]: Standard speculative decoding with a single neural draft model (either FLAN-T5-BASE or FLAN-T5-SMALL for encoder-decoder experiments; the 68M model for decoder-only experiments). This is the primary baseline — CS Drafting's speedups are reported as improvements over this method.
  • Medusa[3]: A tree-attention-based speculative decoding method that generates multiple candidate continuations to increase acceptance probability. Used as a baseline for decoder-only experiments.
  • CS Drafting variants with different cascade depths: CS Drafting with one neural model + MaG (vertical cascade only), and CS Drafting with two neural models + MaG (full vertical + horizontal cascade). These internal comparisons isolate the contributions of each cascade.

• Generation budget / compute accounting. The paper does not use a unified "generation budget" in the sense of FLOPs or token counts. Instead, speedup is measured as the ratio of walltime (or standardized walltime) for autoregressive generation to walltime for the speculative method, which inherently accounts for all model forward passes at all cascade levels. The hyperparameter matrix $K_{nn}$ controls how many tokens each model generates at each recursion level, and these hyperparameters are tuned per configuration (reported in Table 5). For fair comparison, the same target model and hardware are used across all methods. The paper notes that MaG's memory cost is "negligible" (vocabulary-sized bigram parameters) and does not add deployment overhead beyond what speculative decoding already requires.

• Cross-validation / statistical protocol. The paper does not employ cross-validation or statistical significance testing. Hyperparameter tuning is performed per dataset and configuration, with specific values reported in Table 5. For walltime measurements, the inherent variability is acknowledged ("less reproducible and sensitive to noise," Section 5.1) but no confidence intervals or error bars are reported. The paper relies on the dual-metric approach (SWI for reproducibility, walltime for realism) rather than statistical protocols to establish robustness. For the decoder-only experiments, hyperparameters are intentionally fixed across datasets to "better align with most users who do not perform hyperparameter tuning" (Appendix B), serving as a form of robustness check.

---

Main Quantitative Results

Encoder-Decoder Results: FLAN-T5 Family on GSM8K and MMLU

Headline numbers (Table 2). On GSM8K under the model-size standardized metric (Speedup MS), standard speculative decoding achieves 3.38× speedup with FLAN-T5-BASE as draft model, and 3.06× with FLAN-T5-SMALL. CS Drafting configurations consistently outperform these baselines:

• CS Drafting with BASE + MaG (vertical cascade only): 3.70× speedup — a 9.5% improvement over the best speculative decoding baseline (3.38×).
• CS Drafting with SMALL + MaG (vertical cascade only): 3.19× speedup — a 4.2% improvement over speculative decoding with SMALL (3.06×).
• CS Drafting with BASE + SMALL + MaG (both cascades): 3.88× speedup — the maximum speedup, representing an additional 44% speedup over the fastest speculative decoding baseline (computed as (3.88 - 3.38) / 3.38 × baseline, but the paper claims a 44% improvement, likely referencing a different baseline or calculation).

On MMLU under the same metric, the gains are larger:

• Standard speculative decoding: 3.97× (BASE) and 4.12× (SMALL). Notably, SMALL outperforms BASE as a draft model in standard speculative decoding on MMLU, the reverse of GSM8K.
• CS Drafting with BASE + MaG: 4.56× — a 14.9% improvement over speculative decoding with BASE (3.97×), and also outperforming speculative decoding with SMALL (4.12×) despite BASE being the neural draft model.
• CS Drafting with SMALL + MaG: 4.39× — a 6.6% improvement over speculative decoding with SMALL.
• CS Drafting with BASE + SMALL + MaG: 4.88× speedup — the maximum, representing an additional 81% speedup over the fastest speculative decoding baseline (SMALL alone at 4.12×; computed as (4.88 - 4.12) / 4.12 × baseline, approximating the paper's claim).

The Speedup (PW) column, using time costs from prior work [14], shows consistent patterns but generally lower absolute speedup values. On GSM8K, CS Drafting with BASE + SMALL + MaG achieves 3.43× vs. 2.99× for speculative decoding with BASE. On MMLU, it achieves 4.32× vs. 3.51× for speculative decoding with SMALL. The relative improvements are comparable, confirming that the gains are not an artifact of the model-size cost heuristic.

Key pattern: draft model size ordering reverses with cascade. In standard speculative decoding on MMLU, SMALL (4.12×) outperforms BASE (3.97×) — the smaller draft model is faster because its lower per-step cost outweighs its lower acceptance rate. However, in CS Drafting with the vertical cascade (neural model + MaG), BASE + MaG (4.56×) outperforms SMALL + MaG (4.39×). The paper interprets this as evidence that "with the limitation of a single draft model, the ideal size of the draft model might increase with the assistance of the MaG model" (Section 5.2). The vertical cascade reduces the cost of drafting with a larger model (since BASE no longer autoregressively generates — it only reviews MaG's drafts), allowing the higher acceptance rate of BASE to translate into net speedup gains.

Effectiveness of MaG alone (vertical cascade contribution). The vertical cascade with one neural model + MaG provides substantial gains even without the horizontal cascade. On MMLU, BASE + MaG achieves 4.56× vs. 3.97× for BASE alone (speculative decoding) — a 14.9% improvement — and on GSM8K, BASE + MaG achieves 3.70× vs. 3.38× — a 9.5% improvement. The paper emphasizes that this gain "involves no additional deployment overhead while reducing both latency and computational cost, making it a superior choice over speculative decoding" (Section 5.2), since MaG's parameters are vocabulary-sized (negligible memory) and it requires no training or fine-tuning.

Horizontal cascade contribution. The full CS Drafting (BASE + SMALL + MaG) adds the horizontal cascade on top of the vertical cascade. On GSM8K, this yields 3.88× vs. 3.70× for BASE + MaG alone — a 4.9% additional improvement from the horizontal cascade. On MMLU, it yields 4.88× vs. 4.56× — a 7.0% additional improvement. These gains, while smaller than the vertical cascade's contribution, are obtained by more efficiently allocating existing models (using SMALL for later lower-value tokens) rather than requiring additional models.

---

Decoder-Only Results: Vicuna-7B

Headline numbers (Table 3). On Vicuna-7B with walltime measured in tokens per second:

• Autoregressive: 33.34 tokens/s (GSM8K), 33.11 tokens/s (MMLU).
• Speculative Decoding: 44.31 tokens/s (GSM8K, 1.33× speedup), 43.65 tokens/s (MMLU, 1.32× speedup).
• CS Drafting: 56.72 tokens/s (GSM8K, 1.70× speedup over autoregressive, 28% improvement over speculative decoding), 55.60 tokens/s (MMLU, 1.68× speedup over autoregressive, 27% improvement over speculative decoding).
• CS Drafting + Tree Attention (Medusa combination): 63.81 tokens/s (GSM8K, 1.91× speedup over autoregressive), 63.37 tokens/s (MMLU, 1.91× speedup over autoregressive).

Comparison with Medusa. Standard Medusa [3] (tree attention without CS Drafting) achieves 61.87 tokens/s on GSM8K and 56.19 tokens/s on MMLU. CS Drafting alone (56.72 on GSM8K) underperforms Medusa on GSM8K but is comparable on MMLU. Crucially, CS Drafting + Tree Attention achieves 63.81 on GSM8K, exceeding both Medusa (61.87) and CS Drafting alone (56.72). This demonstrates composability: the two techniques address orthogonal bottlenecks — CS Drafting improves drafting efficiency (how fast draft tokens are produced), while tree attention improves acceptance rate (how many draft tokens survive review) — and their combination yields cumulative gains. The paper notes this as evidence that CS Drafting "can be integrated with other efficient designs for speculative decoding to further accelerate inference" (Section 5.2).

Absolute walltime speedups are lower than SWI speedups. The decoder-only speedups (1.3–1.9×) are substantially lower than the encoder-decoder SWI speedups (3–5×). This reflects both the difference between SWI (an idealized metric assuming constant per-model forward pass times) and walltime (which includes memory overhead, implementation inefficiencies, and hardware-specific effects), and the difference in model architectures (encoder-decoder vs. decoder-only). The paper doesn't analyze this gap in detail but the dual reporting allows readers to assess both the idealized potential and the realized performance.

---

Hyperparameter Robustness

Ablation on $K_{00}$ (generation length limit). Table 4 reports walltime for CS Drafting on GSM8K with Vicuna-7B under different values of $K_{00}$, the hyperparameter controlling the generation length limit at the top-level recursion. With $K_{00} = 1$: 56.16 tokens/s. With $K_{00} = 2$: 55.51 tokens/s (1.2% decrease). With $K_{00} = 3$: 53.73 tokens/s (4.3% decrease). The performance is relatively flat — the difference between optimal (1) and suboptimal (3) is only 4.3% — indicating that "end users who do not require maximum performance can use a simple setup to achieve near-optimal performance" (Section 5.2).

---

Hyperparameter Details (Table 5)

The paper reports full hyperparameter configurations for all encoder-decoder experiments. For CS Drafting with BASE + SMALL + MaG on GSM8K (MS metric): $k_{11} = 8$, $k_{12} = 13$, $k_{22} = 1$, lenience $l = 3$. Here, $k_{11}$ and $k_{12}$ are the step limitations when the target model reviews (corresponding to how many tokens BASE and SMALL produce for the target model to review), $k_{22}$ is the step limitation when BASE reviews SMALL + MaG, and $l$ is the lenience applied at intermediate draft-to-draft reviews. On MMLU with the same configuration: $k_{11} = 5$, $k_{12} = 19$, $k_{22} = 1$, $l = 5$. The optimal hyperparameters differ across datasets and metrics (MS vs. PW), indicating that tuning is dataset-dependent. The paper doesn't analyze why particular values are optimal or provide tuning guidelines beyond the robustness observation in Table 4.

For speculative decoding baselines, $k$ (the single draft length) is reported: e.g., BASE on GSM8K (MS) uses $k = 10$, SMALL uses $k = 11$. These are tuned independently per configuration.

---

Ablation Studies and Robustness Checks

• Ablation on horizontal cascade: The paper reports (in passing, Section 5.2) that removing the horizontal cascade from CS Drafting on GSM8K with Vicuna-7B and $K_{00} = 1$ causes a performance drop from 56.16 to 53.55 tokens/s — a 4.6% decrease. This ablation is mentioned in the main text but not in a dedicated table; it isolates the horizontal cascade's contribution independent of the vertical cascade.

• Ablation on generation length limit ($K_{00}$): Table 4 shows walltime under three $K_{00}$ values (1, 2, 3) on GSM8K with Vicuna-7B. Performance degrades slowly (56.16 → 55.51 → 53.73 tokens/s), suggesting that coarse $K_{00}$ selection is sufficient and that the method is not brittle to this hyperparameter.

• Draft model size comparison with and without cascade: The natural experiment comparing speculative decoding with BASE and SMALL against CS Drafting with BASE + MaG and SMALL + MaG serves as an ablation on the vertical cascade's effect on draft model selection. The finding that BASE outperforms SMALL in CS Drafting but not in standard speculative decoding (on MMLU) is a non-obvious result: the vertical cascade changes the cost structure enough to flip the optimal draft model size ranking. This is not a controlled ablation (it compares different systems) but serves as evidence of the vertical cascade's structural impact.

• Robustness across metrics (MS vs. PW): All major findings — CS Drafting outperforming speculative decoding, vertical cascade providing the bulk of gains, full cascade providing additional improvements — replicate across both standardized metrics (model size and prior-work time costs) for both datasets. The absolute speedup values differ (PW is consistently lower than MS), but the relative ordering of configurations is preserved. This addresses the concern that the model-size heuristic might distort comparisons (e.g., by overestimating the cost of larger draft models relative to the target model).

• Robustness across datasets (GSM8K vs. MMLU): CS Drafting improves over speculative decoding on both datasets, but the magnitude differs — up to 44% improvement on GSM8K vs. 81% on MMLU (MS metric). The paper doesn't analyze why MMLU benefits more, but one plausible factor is that MMLU's diverse subject matter creates more variation in per-token acceptance probability, making the horizontal cascade's position-dependent allocation more valuable.

• Robustness across model architectures (encoder-decoder vs. decoder-only): CS Drafting improves over speculative decoding on both FLAN-T5 (encoder-decoder) and Vicuna-7B (decoder-only), though with different absolute speedup magnitudes. The decoder-only experiments use a different draft model (68M Vicuna variant) and include the Medusa comparison. The consistent improvement across architectures supports the claim that the cascade approach is architecture-agnostic.

• Composability with tree attention (Medusa): Table 3 shows CS Drafting + Tree Attention (63.81 tokens/s) outperforming both CS Drafting alone (56.72) and Medusa alone (61.87) on GSM8K. This is a robustness check in the sense that CS Drafting's gains are not redundant with tree attention's gains — they stack. If CS Drafting and tree attention addressed the same bottleneck, combining them would yield diminishing or negative returns; the fact that they combine additively confirms the paper's claim that they target orthogonal efficiency dimensions.

• Negative result: No experiments with sampling (temperature > 0). The paper uses greedy decoding (temperature 0) throughout, stating that "we do not observe any significant difference between sampling with temperature 1 and greedy decoding in previous speculative decoding experiments" (Section 5.1, citing [14]). However, this means the results don't directly demonstrate that CS Drafting's lenience mechanism (which has different behavior under sampling vs. greedy decoding, as described in Section 3.1) preserves output distribution in the sampling case. The theoretical guarantee (asymmetric lenience → exact distribution matching) should hold under sampling, but empirical validation is absent.

• Negative result: No experiments without chain-of-thought prompting. All experiments use zero-shot chain-of-thought prompting. The MaG algorithm's effectiveness depends partly on input-output token overlap, which chain-of-thought prompting may amplify (the model is encouraged to restate problem elements in its reasoning). The paper doesn't report results with standard prompting, leaving open the question of whether CS Drafting's gains are partially dependent on the chain-of-thought format.

---

Critical Assessment

Claim 1: "CS Drafting achieves up to 81% additional speedup over speculative decoding" (Abstract, Section 5.2)

This claim is supported with qualifications. The 81% figure refers specifically to CS Drafting with BASE + SMALL + MaG on MMLU under the model-size standardized metric (Table 2), where CS Drafting achieves 4.88× speedup vs. 4.12× for the best speculative decoding baseline (SMALL alone, the faster of the two standard configurations). However, this is the most favorable comparison point. The improvement over speculative decoding varies substantially:

• On GSM8K (MS): 3.88× vs. 3.38× = ~15% improvement, not 81%.
• On GSM8K (PW): 3.43× vs. 2.99× = ~15% improvement.
• On MMLU (PW): 4.32× vs. 3.51× = ~23% improvement.
• On Vicuna-7B (walltime): 56.72 vs. 44.31 tokens/s = ~28% improvement.

The "up to 81%" framing is technically accurate (that's the maximum observed) but the single-number summary obscures the 15–28% improvements that appear more typical. A reader taking "up to 81%" as representative would be misled. The paper would be strengthened by reporting the range of improvements across configurations and by analyzing why MMLU sees dramatically larger gains than GSM8K — the current draft offers no explanation for this disparity.

Additionally, the 81% figure uses the model-size standardized metric (MS), which assumes forward pass time is proportional to parameter count. On real hardware, this assumption may not hold (memory bandwidth, kernel launch overhead, and batch-size effects create non-linearities). The PW metric (using prior-work measured times) shows a 23% improvement on the same dataset, which is more likely to reflect real-world performance. The paper reports both but emphasizes the larger MS-based figure in the abstract and claims.

Claim 2: "The vertical cascade eliminates autoregressive generation from neural models" (Abstract, Section 3.1)

This claim is conceptually accurate but empirically incomplete. The vertical cascade pushes autoregressive generation to the MaG statistical model, meaning no neural model performs sequential token-by-token generation. This is demonstrated architecturally (Algorithm 1) and theoretically (Corollary 4.4 proves EWIF improvement). However, the paper does not provide an experiment that isolates and measures the cost of autoregressive neural generation vs. the cascade alternative. There is no ablation where the vertical cascade's recursive review is replaced with autoregressive generation from the same neural draft models, holding acceptance rates constant, to quantify the exact time savings from eliminating autoregressive generation. The overall speedup numbers (Table 2) demonstrate net improvement, but they conflate the elimination of autoregressive generation with the effects of lenience, the statistical model's predictions, and the horizontal cascade. A targeted micro-benchmark — e.g., measuring the walltime of one draft step with and without the vertical cascade, controlling for acceptance rate — would make the causal claim more precise.

Claim 3: "The horizontal cascade optimizes time allocation in drafting" (Abstract, Section 3.2)

This claim is supported but the evidence is modest. The ablation (Section 5.2) showing a drop from 56.16 to 53.55 tokens/s (~4.6%) when removing the horizontal cascade on Vicuna-7B provides direct evidence that the horizontal cascade contributes positively. However, this is a small effect compared to the vertical cascade's contribution. On the FLAN-T5 experiments, the horizontal cascade's contribution can be estimated by comparing the full cascade (BASE + SMALL + MaG) against the vertical cascade alone (BASE + MaG): on GSM8K, 3.88× vs. 3.70× (~4.9% improvement); on MMLU, 4.88× vs. 4.56× (~7.0% improvement). These are real but modest gains.

The theoretical analysis (Corollary 4.6) provides strong justification for position-dependent allocation, but the empirical demonstration is limited by the small number of model sizes available (effectively two neural draft models: BASE and SMALL). With only two model sizes, the horizontal cascade amounts to a single binary decision: use the larger model for the first $k_1$ tokens and the smaller model for the next $k_2$ tokens. The full promise of the horizontal cascade — fine-grained position-dependent allocation across a continuous spectrum of model sizes — is not tested. The paper's case would be stronger with experiments using 3–4 draft model sizes, or with a controlled study varying the allocation pattern systematically.

Claim 4: "CS Drafting preserves the same output distribution as the target model" (Abstract, Section 3.1)

This claim is theoretically justified but empirically unverified. The asymmetric lenience mechanism ensures that when $l=1$ at the target model review, the standard speculative decoding rejection sampling procedure [14] guarantees exact distributional equivalence regardless of what happened in intermediate reviews. The paper correctly explains this guarantee (Section 3.1) and implements it (the isFirstCall flag in Algorithm 1).

However, the paper performs no empirical verification of distribution preservation. There is no comparison of output token distributions between autoregressive generation and CS Drafting, no measurement of KL divergence, and no human evaluation of output quality. The use of greedy decoding (temperature 0) simplifies verification — with greedy decoding, the acceptance condition is deterministic, making it easier to ensure correctness — but the theoretical guarantee under sampling (where rejection sampling with recalibrated distributions is needed) is not tested. If there were bugs in the implementation (e.g., lenience not being properly reset to 1, or the isFirstCall flag being mishandled in edge cases), the experiments would not detect them because they don't check output equivalence.

For a method whose primary selling point is lossless acceleration (unlike lenience-based methods that trade quality for speed), empirical validation of distribution preservation is a notable omission. This is especially relevant given that the vertical cascade involves multiple levels of speculative review with intermediate lenience, creating multiple points where implementation errors could propagate.

Claim 5: "CS Drafting can be integrated with other efficient designs" (Section 5.2, Table 3)

This claim is supported by the CS Drafting + Tree Attention experiments on Vicuna-7B (Table 3). The combination achieves 63.81 tokens/s on GSM8K vs. 61.87 for Medusa alone and 56.72 for CS Drafting alone, demonstrating additive gains. This is a clean demonstration of composability. However, only one combination is tested (tree attention via Medusa). Other natural combinations — with knowledge distillation [29], with self-drafting [27, 12], with quantization — are not explored. The paper can fairly claim that the demonstrated composability "suggests" broader applicability, but the evidence is limited to a single complementary method.

Cross-Cutting Weaknesses

Single hardware configuration. All walltime experiments run on a single NVIDIA A40 GPU. The absolute walltime numbers are hardware-specific, and the relative speedups may not transfer to systems with different GPU architectures, memory bandwidths, or CPU-GPU balance. The SWI metric partially addresses this by providing hardware-independent comparisons, but SWI relies on assumptions (constant per-model forward pass time, linear scaling with parameter count) that may not hold across all hardware. The paper acknowledges this limitation (Section 8): "It is possible, though unlikely, that the outcome might differ on a system with different hardware configurations."

No confidence intervals or statistical testing. Walltime measurements are noisy (affected by GPU scheduling, thermal throttling, background processes), yet the paper reports point estimates without error bars. The stability of walltime measurements across multiple runs is not reported. For SWI, there is no measurement noise (it's computed deterministically from model sizes), but there is also no assessment of how sensitive SWI is to the model-size linearity assumption.

Hyperparameter tuning cost is unaccounted for. The paper reports tuned hyperparameters for each configuration (Table 5), but the cost of this tuning — which requires running the system multiple times with different settings — is not included in the speedup calculations. In a deployment setting, hyperparameter tuning would need to be performed once per model family and task distribution, but the paper doesn't discuss how expensive this tuning is or provide guidelines for avoiding it (beyond the $K_{00}$ robustness ablation in Table 4, which only addresses one hyperparameter).

Limited ablation on lenience. The asymmetric lenience mechanism is central to the vertical cascade's effectiveness, but there is no ablation study varying lenience (e.g., $l=1, 2, 3, 5$) to show its impact on speedup and to verify that $l=1$ at the target model review (enforced by the algorithm) indeed produces identical outputs. The optimal lenience values range from $l=3$ to $l=5$ across configurations (Table 5), but the sensitivity of results to this choice is unexplored. If lenience must be carefully tuned per dataset, the practical deployment cost increases.

The gap between SWI and walltime is not analyzed. On GSM8K with BASE, speculative decoding achieves 3.38× (MS) vs. 2.99× (PW) — a 13% difference between the two standardized metrics alone. On MMLU with SMALL, speculative decoding achieves 4.12× (MS) vs. 3.51× (PW) — a 17% difference. These gaps indicate that the choice of cost model affects the reported speedup magnitude, and they raise the question of which metric better predicts actual walltime performance. The paper treats both as valid but doesn't investigate why they differ or whether the differences are systematic (e.g., does MS consistently overestimate speedup relative to PW?).

Lack of comparison with staged speculative decoding [19]. Despite citing Spector and Re [19] as the closest prior work and critiquing its limitations (Section 6.2), the paper provides no direct experimental comparison with staged speculative decoding. Such a comparison — two-level speculative decoding vs. CS Drafting's vertical cascade — would directly test the claim that full recursion to a statistical model and asymmetric lenience provide benefits beyond a two-level system. Without this comparison, the improvement over "prior staged approaches" remains a theoretical argument rather than an empirical finding.

6. Limitations and Trade-offs

6.1 The Difficulty Estimation Cost Is Unaccounted for in Reported Speedups

The assumption or constraint. CS Drafting's performance depends on hyperparameters — the $K_{nn}$ matrix specifying how many tokens each model generates at each recursion level, and the lenience parameter $l$ applied at intermediate reviews. These hyperparameters are tuned per dataset and configuration. Table 5 reports tuned values that differ substantially across datasets: for CS Drafting with BASE + SMALL + MaG, $k_{11}$ varies from 8 (GSM8K, MS) to 5 (MMLU, MS), $k_{12}$ from 13 to 19, and lenience $l$ from 3 to 5. For decoder-only experiments on Vicuna-7B, the paper uses a fixed hyperparameter set across datasets to "better align with most users who do not perform hyperparameter tuning" (Appendix B), but this represents a deliberate choice to trade performance for simplicity rather than evidence that tuning is unnecessary. The paper does not quantify the computational cost of hyperparameter search — which requires running the full CS Drafting system multiple times with different settings — nor does it amortize this cost into the reported speedup figures.

The consequence. In a deployment setting, hyperparameter tuning is a one-time cost per model family and task distribution, but the magnitude of this cost is unknown. If tuning requires, say, 50–100 evaluation runs to find the optimal $K_{nn}$ and $l$, the upfront compute investment could be substantial — potentially equivalent to the cost saved by CS Drafting over many inference queries, depending on query volume. The paper's headline speedup numbers (e.g., 81% improvement on MMLU, 44% on GSM8K) are therefore best-case figures conditional on optimal hyperparameters being known in advance, not realistic end-to-end deployment gains that include the cost of obtaining those hyperparameters. The robustness check in Table 4 — showing only a 4.3% walltime degradation when $K_{00}$ changes from 1 to 3 — partially addresses concern about sensitivity to a single hyperparameter, but $K_{00}$ is not the full hyperparameter configuration (the full $K_{nn}$ matrix for multi-model cascades has multiple entries, and lenience $l$ varies from 3 to 5 across configurations in Table 5). The paper provides no evidence that other hyperparameters exhibit similarly low sensitivity, nor does it provide tuning guidelines or heuristics that would reduce the search space.

What evidence exists in the paper. Table 5 reports optimal hyperparameters for all encoder-decoder configurations, demonstrating dataset-dependence. Table 4 provides a limited ablation on $K_{00}$ (generation length limit) for the decoder-only setting, showing modest sensitivity. Appendix B describes the hyperparameter constraints used to reduce the search space (MaG always generates 10 tokens, no lenience at target model review or between MaG and its reviewer), leaving "at most four hyperparameters: $k_{11}$, $k_{12}$, $k_{22}$, and $l$." But the paper does not report the search procedure (grid search, random search, manual tuning), the number of configurations evaluated, or the compute cost. The statement that Vicuna-7B hyperparameters were fixed across datasets suggests that tuning was performed but not accounted for.

Mitigation status. Not addressed. The paper acknowledges (implicitly) that hyperparameter tuning is a practical burden by providing fixed hyperparameters for decoder-only experiments (Appendix B) and demonstrating that $K_{00}$ is not excessively brittle (Table 4). However, it does not report tuning cost, provide tuning heuristics, or suggest methods for automatic hyperparameter selection. A practitioner deploying CS Drafting would need to budget for tuning as a hidden upfront cost.

---

6.2 Distribution Preservation Is Theoretically Guaranteed but Empirically Unverified

The assumption or constraint. The paper's central claim about lossless acceleration rests on the asymmetric lenience mechanism: lenience $l > 1$ is applied when intermediate draft models review each other (accepting tokens with a looser criterion), but $l = 1$ is enforced when the target model $M_t$ performs the final review (Algorithm 1, isFirstCall flag). The theoretical guarantee follows from the standard speculative decoding analysis [14]: when the target model reviews with $l = 1$, the rejection sampling procedure ensures that the accepted token sequence is drawn from the target model's exact autoregressive distribution, regardless of what distribution the draft tokens came from. The paper states (Section 3.1): "We can limit the application of lenience in the vertical cascade only when draft models review and do not apply lenience when the target model reviews. This can ensure the final output is not altered while further reducing latency."

The consequence. The guarantee holds in theory, but the paper provides no empirical verification that CS Drafting's output distribution matches the target model's autoregressive distribution in practice. This is important because the guarantee depends on correct implementation of multiple interacting components: the isFirstCall flag must be propagated correctly through recursive calls, lenience must be properly reset before the target model review, the acceptance/rejection sampling logic in the review function must be correct at every level, and the recalibrated distribution after rejection must be computed properly. A bug in any of these components — particularly in edge cases involving the recursive vertical cascade, where intermediate reviews may produce tokens that interact with the final review in unexpected ways — would break distribution equivalence without necessarily being detected by accuracy or speedup metrics.

The paper also conducts all experiments with greedy decoding (temperature 0), where the acceptance criterion simplifies to deterministic matching: accept if $\text{argmax }M_d(x)[i] = \text{argmax }M_t(x)[i]$ or $M_d(x)[i] \leq l \times M_t(x)[i]$. Under greedy decoding, distributional equivalence is easier to maintain (the output is deterministic for a given input), but the paper claims the method works for sampling as well (Section 3.1 describes the sampling-based acceptance criterion). No experiments with temperature > 0 are reported, leaving the sampling case entirely untested empirically.

What evidence exists in the paper. None. The paper contains no measurements of output distribution similarity (KL divergence, token distribution comparison, output diversity metrics), no human evaluation of output quality, and no comparison of generated text between CS Drafting and autoregressive generation. The claim of exact distribution preservation is supported solely by reference to the speculative decoding theoretical analysis [14] and the isFirstCall mechanism in Algorithm 1.

Mitigation status. Not addressed empirically. The paper's decision to use greedy decoding (Section 5.1: "to ensure our experiments are fully reproducible") sidesteps the empirically riskier sampling case but also eliminates the natural empirical check: under sampling, one could compare output distributions between CS Drafting and autoregressive generation over many runs to verify distributional equivalence. For a method whose primary value proposition is lossless acceleration, the absence of any empirical verification — even a simple accuracy comparison on a task where outputs can be automatically evaluated — is a significant gap.

---

6.3 Single Benchmark Domain and Model Family Limit Generality Claims

The assumption or constraint. All experiments use the FLAN-T5 family [6] for encoder-decoder models and Vicuna-7B [28] (a LLaMA variant) for decoder-only models, evaluated exclusively on GSM8K [7] (grade-school math word problems) and MMLU [10] (multi-subject knowledge benchmark), both in zero-shot chain-of-thought setup. The paper states no explicit claim about generalization to other model families, tasks, or domains, but the abstract presents CS Drafting as a general method for LLM inference acceleration, and the findings are discussed without caveats about domain-specificity.

The consequence. Both GSM8K and MMLU share a structural property that the Max-Gram (MaG) algorithm exploits: the output frequently reuses tokens, phrases, and numbers from the input. In math word problems, the solution restates quantities from the problem; in multiple-choice MMLU questions, the output often echoes terminology from the question and answer choices. The paper explicitly notes this as the motivation for MaG (Section 3.3): "in language model generation, some words and phrases from the input query frequently reappear in the generated content."

For tasks without this input-output token overlap — open-ended creative writing, dialogue generation, storytelling, opinion essays — MaG's pattern-matching mechanism would fall back to the bigram model more frequently. The bigram model's predictions, while extremely cheap to generate, have lower acceptance probability than the pattern-matched tokens, which would reduce the vertical cascade's effectiveness (fewer MaG tokens accepted at each intermediate review, requiring more review steps). The paper provides no evidence about CS Drafting's performance on such tasks, nor does it analyze what fraction of MaG tokens come from pattern matching vs. bigram fallback in the tested benchmarks.

Additionally, the FLAN-T5 family is instruction-tuned [6], which may produce more structured, formulaic outputs than base pretrained models. The consistent output structure might inflate acceptance rates at higher cascade levels, making the vertical cascade appear more effective than it would be for less instruction-tuned or differently architected models. The Vicuna-7B experiments partially address the architecture concern (decoder-only vs. encoder-decoder) but not the task-family concern (both benchmarks used chain-of-thought prompting, which likely amplifies structured output).

What evidence exists in the paper. Tables 2 and 3 report results exclusively on GSM8K and MMLU. Table 3 shows that CS Drafting works on a decoder-only architecture (Vicuna-7B), addressing one dimension of generalization. Figure 2 shows acceptance rate decay curves for FLAN-T5 models on both benchmarks, and the patterns are qualitatively similar. However, there are no experiments on tasks lacking input-output token overlap (e.g., creative writing benchmarks like WritingPrompts, dialogue tasks like DailyDialog, or code generation tasks like HumanEval where the output is syntactically constrained but not token-overlapping with the input), no experiments on base (non-instruction-tuned) models, and no analysis of the MaG fallback rate.

Mitigation status. Not addressed. The paper acknowledges no limitation regarding task or domain specificity. The choice of benchmarks is motivated implicitly by their standard usage in the field (they are "commonly used datasets," Section 5.1) rather than by their suitability for demonstrating CS Drafting. A practitioner using CS Drafting for a different task family (e.g., code generation, creative writing, dialogue) would need to evaluate whether MaG's pattern-matching assumption holds and, if not, whether a different bottom-level statistical model would be needed — a question the paper does not explore.

---

6.4 Walltime Measurements Are Hardware-Specific and Statistically Uncharacterized

The assumption or constraint. All walltime experiments run on a single NVIDIA A40 GPU (Section 5.1). The paper acknowledges this limitation in Section 8: "It is possible, though unlikely, that the outcome might differ on a system with different hardware configurations." To mitigate hardware-dependence, the paper reports Standardized Walltime Improvement (SWI) under two cost models: model size (MS) and prior-work measured times (PW). The dual reporting is intended to provide hardware-independent comparisons.

The consequence. The absolute walltime numbers (33–64 tokens/s for Vicuna-7B, Table 3) are hardware-specific and may not predict performance on different GPUs (e.g., A100, H100) or deployment configurations (e.g., multi-GPU, quantized models, different batch sizes). More importantly, the relative speedups may not transfer: CS Drafting's vertical cascade reduces the number of neural forward passes but increases the number of statistical model lookups and the complexity of the orchestration logic (recursion, multiple small tensor operations for MaG). On hardware with different relative costs for GPU computation vs. CPU coordination, or with different memory bandwidth characteristics, the optimal cascade depth and the magnitude of speedup could differ.

The SWI metrics address hardware-dependence but introduce their own assumptions. The model-size metric (MS) assumes forward pass time is proportional to parameter count, which ignores memory bandwidth, kernel launch overhead, and the fact that very small models (like MaG) may be latency-bound by fixed overheads rather than computation-bound by parameter count. The prior-work metric (PW) uses time costs from Leviathan et al. [14] measured on different hardware, potentially introducing systematic biases. The paper does not validate either SWI metric against actual walltime measurements, so the relationship between SWI-predicted speedups and real-world performance is uncalibrated.

Furthermore, walltime measurements are inherently noisy — affected by GPU scheduling, thermal state, background processes, and measurement methodology. The paper reports point estimates without error bars, confidence intervals, or any indication of measurement variance. For the Vicuna-7B experiments (Table 3), walltime varies by ~0.3 tokens/s between GSM8K and MMLU for the same autoregressive baseline (33.34 vs. 33.11), suggesting measurement noise on the order of 1%. For CS Drafting (56.72 vs. 55.60), the difference is ~2%, suggesting slightly higher variance. The paper does not report how many runs were averaged, how outliers were handled, or whether differences between configurations are statistically significant.

What evidence exists in the paper. Table 3 reports single-point walltime estimates for Vicuna-7B. Table 2 reports SWI under two cost models. Section 5.1 acknowledges that walltime is "less reproducible and sensitive to noise" compared to SWI. Section 8 states the hardware-specificity caveat. The paper does not report measurement variance, run counts, or statistical tests.

Mitigation status. Partially addressed through the dual SWI reporting. The SWI metrics provide reproducibility and hardware-independence, enabling other researchers to verify the relative ordering of methods. However, the SWI-to-walltime gap remains uncalibrated — a practitioner cannot use the SWI numbers to predict actual speedup on their hardware without their own benchmarking. The paper's statement that it is "unlikely" that hardware differences would change the outcome (Section 8) is an unsubstantiated assertion rather than a finding supported by experiments on multiple hardware configurations.

---

6.5 The Empirical Case for the Horizontal Cascade Is Thin Relative to Its Theoretical Justification

The assumption or constraint. The horizontal cascade — using larger draft models for earlier (higher-acceptance-probability) tokens and smaller models for later tokens — is justified through two lines of evidence: (1) the theoretical analysis in Corollary 4.6, which proves that marginal returns to improving acceptance rate are higher at earlier positions, and (2) empirical results showing that the full cascade (BASE + SMALL + MaG, with both cascades) outperforms the vertical cascade alone (BASE + MaG). The ablation in Section 5.2 for Vicuna-7B shows a drop from 56.16 to 53.55 tokens/s (~4.6%) when the horizontal cascade is removed.

The consequence. The horizontal cascade is a conceptually clean idea — allocate expensive model capacity where it provides the highest marginal return — but the empirical evidence that it provides substantial practical benefit is weak. On GSM8K (MS), the full cascade (3.88×) improves over the vertical cascade alone (3.70×) by only 4.9%. On MMLU (MS), the improvement is 4.88× vs. 4.56× (7.0%). On Vicuna-7B, the ablation shows 4.6%. These are real but modest gains, and they must be weighed against the added complexity: the horizontal cascade requires managing multiple neural draft models simultaneously (increased memory footprint, more complex orchestration logic), tuning a multi-entry $K_{nn}$ matrix rather than a single $k$, and deploying at least two neural draft models rather than one.

With only two neural draft model sizes available (BASE and SMALL in the FLAN-T5 experiments), the horizontal cascade reduces to a single allocation decision: how many tokens should BASE produce before handing off to SMALL (or vice versa for decoder-only). This is a coarse implementation of the position-dependent allocation principle. The theoretical analysis (Theorem 4.5, Corollary 4.6) suggests that a finer-grained allocation — e.g., 4–5 model sizes, each producing 1–2 tokens at positions matched to their cost-accuracy tradeoff — could yield larger gains, but this is empirically untested. The available evidence therefore demonstrates that the horizontal cascade's principle is sound but its practical benefit over the vertical cascade alone may not justify the added complexity in some deployment scenarios.

Moreover, the horizontal cascade's benefit is likely sensitive to the acceptance rate decay curve (Figure 2). If the acceptance rate decays slowly (high-quality draft models, tasks where the model rarely makes errors), later tokens retain higher value and the benefit of using cheaper models for them diminishes. If the acceptance rate decays very rapidly, even the first few tokens have low value and the optimal strategy may be to use the cheapest model for all positions. The paper does not analyze how the horizontal cascade's benefit varies with acceptance rate characteristics, task difficulty, or model quality.

What evidence exists in the paper. The ablation on Vicuna-7B (4.6% drop without horizontal cascade, Section 5.2) provides the only direct isolation of the horizontal cascade's contribution. The FLAN-T5 comparisons between full cascade and vertical-cascade-alone configurations (Table 2) provide indirect evidence: 4.9% improvement on GSM8K (MS), 7.0% on MMLU (MS), smaller improvements on PW metrics. Table 1 provides simulated EWIF comparisons showing the horizontal cascade outperforming uniform allocation under Bernoulli acceptance assumptions. Figure 2 provides the empirical motivation by showing acceptance rate decay curves. Corollary 4.6 provides the theoretical justification.

Mitigation status. Not addressed as a limitation. The paper presents the horizontal cascade as a core contribution alongside the vertical cascade, without discussing whether its marginal benefit justifies its complexity. The composability with tree attention (Table 3) — where CS Drafting + Tree Attention achieves 63.81 vs. 61.87 for Medusa alone — suggests that in resource-rich settings where multiple optimizations are applied, even modest gains from the horizontal cascade are valuable. But for a practitioner deciding whether to implement the horizontal cascade vs. only the vertical cascade, the paper provides no guidance on when the additional complexity is worthwhile.

---

6.6 The Gap Between SWI and Walltime Metrics Is Substantial and Unexplained

The assumption or constraint. The paper reports two types of speedup metrics: Standardized Walltime Improvement (SWI) under two cost models (model size MS, prior-work times PW) and actual walltime. The SWI metrics assume that the speedup of speculative methods can be computed from model forward pass times and acceptance rates without running on real hardware, making them "fully reproducible" (Section 5.1). The walltime metric measures actual tokens-per-second on an A40 GPU.

The consequence. The gap between SWI and walltime is large and systematic. On the FLAN-T5 experiments, SWI speedups range from 3.06× to 4.88× (Table 2, MS column). On the Vicuna-7B experiments, walltime speedups range from 1.32× to 1.91× (Table 3). Even within the FLAN-T5 experiments, the two SWI metrics differ meaningfully: speculative decoding with BASE on GSM8K achieves 3.38× (MS) vs. 2.99× (PW) — a 13% difference in the predicted speedup depending on which cost model is used. For CS Drafting with BASE + SMALL + MaG on MMLU, the difference is 4.88× (MS) vs. 4.32× (PW) — a 13% difference.

This gap has two important implications. First, it means that SWI — the metric on which the paper's headline speedup claims are based — overestimates actual walltime speedup, potentially by a large factor. The 81% improvement claim (CS Drafting over speculative decoding on MMLU) is based on SWI (MS): 4.88× vs. 4.12×. If the SWI-to-walltime ratio is consistent, the actual walltime speedup would be substantially lower. The paper provides no calibration between SWI and walltime, so a practitioner reading "81% improvement" has no way to estimate what that translates to on their hardware.

Second, it raises questions about whether the SWI cost models capture the right factors. The MS model assumes linear scaling with parameter count, but real GPU inference has fixed overheads (kernel launch, memory transfers) that disproportionately affect small models. The PW model uses time costs measured on different hardware, potentially from a different era of GPU architecture. Neither model accounts for the orchestration overhead of CS Drafting — the recursive function calls, the tensor matching in MaG, the coordination between multiple models. If this overhead is non-trivial (as it might be with many small models), SWI would systematically overestimate speedup relative to walltime.

What evidence exists in the paper. The gap is visible when comparing SWI values in Table 2 against walltime values in Table 3, though these are on different model families (FLAN-T5 vs. Vicuna-7B) and different hardware, so the comparison is not direct. Within Table 2, the MS and PW columns differ by 6–17% across configurations, demonstrating that the choice of cost model affects reported speedups. Section 5.1 describes the two metrics but does not compare them or discuss their relationship.

Mitigation status. Partially addressed by reporting both SWI and walltime for different experiment families. The paper uses SWI for the encoder-decoder experiments (where a family of model sizes is available for cost modeling) and walltime for the decoder-only experiments (where real-hardware measurements are feasible). However, the paper never measures both SWI and walltime for the same configuration, which would enable direct calibration. Section 8 acknowledges hardware-dependence as a limitation but frames it as unlikely to change outcomes. The uncalibrated SWI-to-walltime gap — and its implication that headline speedup claims may not translate to real hardware — is not discussed as a limitation.

7. Implications and Future Directions

How This Work Changes the Landscape

CS Drafting shifts the speculative decoding research agenda from a single-axis optimization problem — "how do we make the draft model's predictions more accurate?" — to a multi-axis design space where drafting efficiency and position-dependent resource allocation are independently tunable dimensions alongside acceptance rate. This is not a paradigm shift in the sense of replacing speculative decoding; the foundational draft-then-verify pattern remains intact. Rather, it is a structural reframing of what constitutes an optimizable component within the speculative decoding pipeline. Prior to this work, the drafting phase was treated as monolithic: pick a draft model, it generates $k$ tokens autoregressively, and the only question is how many of those tokens survive review. CS Drafting demonstrates that the drafting phase itself has exploitable internal structure — the generation mechanism (autoregressive vs. speculative), the cost structure per token position, and the acceptance criterion at intermediate levels — and that optimizing this structure yields speedups independent of acceptance rate improvements.

The practical consequence of this reframing is that the field's investment priorities should shift. Prior work invested heavily in improving acceptance rates: knowledge distillation to better align draft and target distributions [29], tree attention to reduce rejection probability through branching [15, 3], and self-drafting to avoid the overhead of separate models [27, 12]. These are still valuable — CS Drafting composes positively with tree attention (Table 3) — but the paper's central empirical finding is that eliminating autoregressive neural generation through the vertical cascade (BASE + MaG) provides a 70% speedup on MMLU with no additional neural models (Table 2). This gain comes entirely from restructuring the drafting process, not from making the draft model more accurate. It implies that, at the current state of the field, there are larger gains available from improving drafting efficiency than from further marginal improvements in acceptance rate — a finding that should redirect research energy toward the drafting architecture itself.

The paper also resolves, or at least clarifies, a latent tension in the speculative decoding literature: the counterintuitive finding by Leviathan et al. [14] that very small draft models (two orders of magnitude smaller than the target) are often optimal, despite their lower acceptance rates. The standard interpretation was that this reflects a fundamental tradeoff between draft model quality and cost. CS Drafting's results on MMLU — where BASE outperforms SMALL in standard speculative decoding (3.97× vs. 4.12×, SMALL is faster), but BASE + MaG (4.56×) outperforms SMALL + MaG (4.39×) in the vertical cascade — reveal that this tradeoff is not fundamental but rather an artifact of the autoregressive drafting bottleneck. When the vertical cascade eliminates BASE's autoregressive cost (by having MaG draft for it), BASE's higher acceptance rate becomes a net positive that outweighs its higher per-forward-pass cost. The optimal draft model size increases when drafting efficiency improves. This finding should cause researchers to revisit assumptions about draft model sizing: the optimization is not "pick a single draft model" but "pick a cascade architecture," and the optimal configuration depends on the cost structure of the cascade, not just the standalone quality and cost of each model.

The Max-Gram (MaG) algorithm contributes a smaller but practically significant insight: that a statistical language model exploiting task-specific input-output structure can serve as a viable bottom-level drafter when coupled with lenient intermediate review. This opens the door to a broader class of "lightweight drafters" — task-specific statistical models, retrieval-based predictors, or even cached frequent generation patterns — that would be too inaccurate to serve as direct draft models for the target but become viable when buffered by a neural intermediate reviewer with lenience.

Finally, the EWIF generating function framework (Section 4) provides the first formal analysis tools for compositional speculative decoding systems — systems where one speculative decoding stage feeds into another. The operator method ($T_\alpha$) that maps probability generating functions to expected acceptance counts under a given acceptance probability is a theoretical contribution that future researchers can use to analyze arbitrary cascade architectures without re-deriving expectations from scratch. This framework makes it possible to reason about optimal cascade depth, optimal model sizing at each level, and the interaction between lenience and acceptance rates in a unified mathematical language.

Follow-Up Research This Work Enables

Adaptive cascade depth selection based on real-time acceptance rate estimation. The paper implements a fixed cascade depth (1–2 neural models + MaG) with pre-tuned hyperparameters (Table 5), but the optimal depth likely depends on the acceptance characteristics of the specific input. A system that monitors acceptance rates at each cascade level during generation and dynamically adds or removes intermediate models — for example, if MaG tokens are being accepted at high rates by the intermediate reviewer, skip the intermediate model entirely for the next cycle and let MaG draft directly for the target — could adapt to per-example difficulty without pre-tuning. The paper's EWIF framework (Theorem 4.3) provides the mathematical machinery to compute expected speedup under different cascade depths given measured acceptance rates, making real-time optimization tractable. A concrete experiment would compare fixed-depth CS Drafting against an adaptive variant on a dataset with mixed difficulty (e.g., combining easy arithmetic problems with complex multi-step reasoning) and measure whether the adaptive variant matches or exceeds the best fixed configuration without per-dataset tuning.

Stress-testing MaG on low-overlap tasks and developing alternative bottom-level models. The MaG algorithm exploits input-output token overlap — a structural feature of summarization, translation, and chain-of-thought reasoning — but its effectiveness on tasks without this overlap (creative writing, dialogue, open-ended generation) is unknown. A targeted experiment would evaluate CS Drafting on WritingPrompts (where outputs are stories that rarely repeat input tokens verbatim), HumanEval (code generation, where output syntax differs from natural language input), and Dolly (open-ended instruction following). The critical measurement is the MaG fallback rate — the fraction of tokens generated by the bigram model rather than pattern matching — and its correlation with overall speedup. If the fallback rate is high (say, >50%) and speedup degrades substantially, the finding would motivate development of task-adaptive bottom-level models: for code generation, a cached n-gram model trained on code corpora; for dialogue, a retrieval-based model that matches against conversation history rather than only the input prompt; for creative writing, a higher-order n-gram model that captures stylistic patterns. This line of work would transform CS Drafting from a method that works well on structured generation tasks to a general-purpose acceleration framework.

Combining CS Drafting with knowledge distillation to optimize the cascade holistically. The paper uses off-the-shelf models without fine-tuning, but the cascade architecture introduces a new optimization target for distillation: rather than training a single draft model to match the target model, train a cascade of draft models where each model is optimized for its specific role. The bottom model (replacing MaG) would be trained to maximize acceptance by the intermediate model under lenience $l > 1$ (a looser objective), while the intermediate model would be trained to maximize acceptance by the target model with $l = 1$ (strict distribution matching). This "cascade-aware distillation" could produce specialized models that outperform general-purpose models of the same size. A concrete experiment: take FLAN-T5-SMALL and a smaller distilled model (e.g., 30M parameters trained via DistillSpec-style distillation [29] but with the loss function modified to optimize for lenient acceptance by FLAN-T5-BASE), configure them in a 2-level cascade with MaG at the bottom, and compare against CS Drafting with off-the-shelf models at equivalent total parameter count. The EWIF framework in Theorem 4.3 predicts that improving $\alpha' = \alpha(M_{d1}, M_{d2})$ (the intermediate acceptance rate) should improve overall EWIF, and cascade-aware distillation could target this directly.

Theoretical analysis of optimal cascade depth under a parameter budget constraint. The paper empirically demonstrates that 2–3 levels (target + 1–2 neural draft models + MaG) improve over 1 level (standard speculative decoding), but the EWIF framework (Theorem 4.3) enables a more fundamental question: given a fixed total parameter budget for all draft models combined, what is the optimal number of cascade levels and the optimal size of each model? This is the cascade analog of the Chinchilla scaling laws [Hoffmann et al., 2022] but applied to inference-time architectures. The analysis would need to model how acceptance probability $\alpha(M_i, M_{i+1})$ scales with the size ratio between adjacent models, how cost coefficients $c_i$ scale with model size, and how lenience amplifies acceptance at intermediate levels. A strong theoretical contribution would provide a closed-form or numerically computable answer to: "Given a target model of size $N$ parameters and a total draft budget of $B$ parameters, what cascade depth and per-level sizes maximize EWIF?" This work would use the generating function framework in Section 4 as its mathematical foundation and would need to validate predictions against empirical measurements on a model family with many sizes (e.g., the Pythia or OPT families).

Comprehensive empirical verification of distribution preservation under sampling. The paper's most significant empirical gap is the absence of any verification that CS Drafting with lenience preserves the target model's output distribution when sampling (temperature > 0). A necessary follow-up would run CS Drafting and autoregressive generation with the same target model on a benchmark where automatic distribution comparison is possible — for instance, generating completions on a held-out text corpus and comparing token-level and sequence-level distributions via KL divergence, or measuring output diversity via distinct-n metrics, or evaluating task performance on a benchmark with automatic metrics (e.g., BLEU for translation, ROUGE for summarization). If distributions match within statistical noise, the asymmetric lenience guarantee is empirically validated and the method can be confidently recommended for sampling-based applications. If distributions diverge — which would indicate either a bug in the isFirstCall mechanism or a subtle interaction between intermediate lenience and the recalibrated resampling distribution — the finding would be critically important for practitioners and would motivate research into correction mechanisms.

Latency-constrained CS Drafting and the interaction with batching. The paper measures throughput (tokens per second) but not latency (time to first token or time per generated token). The vertical cascade introduces additional serial dependencies — MaG generates, then the intermediate model reviews, then the target model reviews — that increase latency compared to standard speculative decoding where the draft model generates in a single autoregressive pass. For interactive applications (chatbots, code completion), latency matters more than throughput. A latency-focused experiment would measure the time from user input to the first generated token and the inter-token latency distribution for CS Drafting vs. standard speculative decoding, under the same throughput conditions. Additionally, production LLM inference often uses batching to improve hardware utilization. CS Drafting's multiple models and recursive review steps may interact poorly with batching — the MaG model's generation is not easily batched with neural model reviews because they're competing for GPU resources, and the recursive orchestration adds synchronization overhead. A systems-focused follow-up would implement CS Drafting in a serving framework (e.g., vLLM, TensorRT-LLM) that supports continuous batching and measure throughput and latency under realistic serving loads with concurrent requests, comparing against standard speculative decoding and Medusa-style tree attention under equivalent conditions.

Practical Applications and Downstream Use Cases

Cost-efficient batch processing for LLM evaluation and data generation. Organizations running large-scale batch inference — evaluating thousands of test examples, generating synthetic training data, or producing model comparisons — often use speculative decoding with a fixed draft model configuration. CS Drafting offers an immediate drop-in improvement: replacing the single draft model with a BASE + MaG vertical cascade (Table 2, MMLU: 4.56× vs. 3.97×, a 15% improvement) requires no additional neural model deployment (MaG has negligible memory cost) and can be implemented by modifying the speculative decoding orchestration logic without retraining any models. For an organization processing 1 million MMLU-style queries per day with FLAN-T5-XXL as the target model, a 15% throughput improvement translates to roughly 15% fewer GPU-hours, directly reducing cloud compute costs. The improvement is larger on tasks where the draft model's acceptance rate is high enough to benefit from the vertical cascade's cost reduction — practitioners should benchmark CS Drafting on their specific task distribution to estimate the gain, but the paper's dual SWI metrics provide a starting point for prediction.

On-device or edge deployment with limited model storage. In settings where model storage is constrained — mobile devices, embedded systems, browser-based inference — deploying multiple large draft models is infeasible, but the vertical cascade with a single neural model + MaG provides speedup without additional storage. The MaG model requires only the tokenizer vocabulary (tens of thousands of entries, kilobytes to low megabytes of storage) and no additional neural weights. A system running Vicuna-7B with the 68M draft model can add MaG at near-zero storage cost and achieve the speedup in Table 3 (56.72 tokens/s with CS Drafting vs. 44.31 with standard speculative decoding, a 28% improvement). This is particularly relevant for emerging on-device LLM deployments (e.g., Chrome's built-in Gemini Nano, Apple Intelligence on-device models) where every megabyte of additional model storage is scrutinized.

Latency-sensitive interactive systems where quality must be preserved exactly. Chatbots, virtual assistants, and code completion tools serving millions of users require both low latency and output quality indistinguishable from the uncompressed target model. CS Drafting's asymmetric lenience — faster intermediate drafting with lenient criteria, but strict distribution preservation at the target model review — directly addresses this requirement. Unlike lenience-based speculative decoding where the quality-speed tradeoff affects final outputs, CS Drafting extracts speedup from internal lenience while guaranteeing (theoretically) exact output equivalence. For a production chatbot using a target model similar to Vicuna-7B, the 28% walltime improvement (Table 3) translates to lower per-query latency, enabling either cheaper serving (fewer GPU instances for the same query volume) or lower response times (same hardware, faster generation). The implementation can use the same fixed hyperparameters across datasets (Appendix B: the [[2, 10], [0, 10]] K-matrix for decoder-only models), avoiding per-task tuning.

Self-improvement and synthetic data pipelines where inference volume dominates cost. When LLMs are used to generate training data for themselves — in methods like STaR, ReST, or constitutional AI — the target model is run on large volumes of prompts to produce completions, and these completions are then curated or filtered for training. The inference cost of generating these completions can dominate the total pipeline cost. CS Drafting's speedup (up to 4.88× SWI on MMLU, Table 2) directly reduces this cost without changing the generated outputs (due to distribution preservation). For a pipeline generating 10 million completions, a 15–30% walltime improvement (as measured on Vicuna-7B, Table 3) reduces compute time by 1–3 days on a cluster — savings that compound across multiple iterations of a self-improvement loop.