Accelerating Speculative Decoding with Block Diffusion Draft Trees

ArXiv: 2604.12989

🎯 Pitch

A single forward pass from a block diffusion drafter contains rich per-position distributions, yet standard methods verify only one trajectory—DDTree reveals that constructing a draft tree directly from these existing distributions and selecting branches to maximize expected acceptance length can boost speculative decoding speedups from 5.6× to 7.5× on MATH-500, all without extra drafting passes.

1. Executive Summary

This paper introduces DDTree (Diffusion Draft Tree), a speculative decoding method that constructs a draft tree directly from the per-position marginal distributions produced by a single block diffusion drafter forward pass — rather than exploring only one drafted trajectory per round as in vanilla DFlash. Evaluated on Qwen3-4B, Qwen3-8B, and Qwen3-Coder-30B-A3B-Instruct across reasoning (MATH-500, GSM8K, AIME), code (HumanEval, MBPP, LiveCodeBench, SWE-bench Lite), and instruction-following (MT-Bench, Alpaca) benchmarks, DDTree consistently improves over vanilla DFlash — for example, boosting speedup on MATH-500 with Qwen3-8B at temperature 0.0 from 5.56× to 7.52× — by selecting a compact set of promising continuations under a fixed node budget via a best-first heap algorithm that provably maximizes the expected acceptance length under the drafter's factorized surrogate distribution. The gains hold across all three model sizes and both greedy and temperature-1.0 sampling, establishing that the information already present in a block diffusion drafter's per-position distributions can be exploited for substantial speedup without additional drafting passes, provided the tree is constructed front-heavy enough to avoid wasting the verification budget on low-probability trajectories.

2. Context and Motivation

The Core Problem: One-Pass Drafters Underutilize Their Own Information

Speculative decoding is fundamentally a two-step process: a lightweight drafter proposes future tokens cheaply, and a large target model verifies them in parallel. The end-to-end speedup depends on a delicate balance — the drafter must be cheap enough that drafting overhead stays small, yet accurate enough that the target model frequently accepts the drafted tokens. Since the target model runs once per round regardless of how many tokens are proposed, the key metric is expected acceptance length: how many consecutive tokens does the target model agree with?

The paper addresses a specific and increasingly important gap in this two-step process. Block diffusion drafters — specifically the DFlash architecture — can generate an entire draft block in a single forward pass, producing per-position marginal distributions over the next $L$ tokens. This is a major latency advantage over autoregressive drafters like EAGLE-3, which require one forward pass per drafted token position. However, vanilla DFlash throws away most of the information it has already computed: it collapses the per-position distributions into a single drafted trajectory (typically, the greedy argmax at each position) and verifies only that one path. The drafter's forward pass has already produced rich probability information about alternative continuations at every position — alternative tokens that might be more agreeable to the target model than the single greedy path — but DFlash ignores them entirely.

This is the paper's central motivating observation: a block diffusion drafter generates an entire probability landscape over future tokens in one shot, yet vanilla speculative decoding with such a drafter explores only one point in that landscape. The information is already paid for computationally — the forward pass happened, the logits were computed, the softmaxes were applied — but it sits unused. The challenge, then, is not to produce better draft distributions (DFlash already does that well), but to make fuller use of the distributions that are already available without incurring additional drafting cost.

Why This Matters: The Practical and Theoretical Stakes

The practical motivation is direct and substantial. Speculative decoding is one of the most widely deployed techniques for accelerating LLM inference — it is used in production serving systems because it provides lossless speedups, meaning the target model's output distribution is preserved exactly while wall-clock latency drops. Any improvement to speculative decoding therefore translates to either faster response times for users or lower compute costs for providers, and often both.

The specific regime that DDTree targets — block diffusion drafters — is particularly high-stakes because DFlash already represents the state of the art. Prior to DDTree, DFlash had already demonstrated that it outperforms strong autoregressive drafters like EAGLE-3. This means that further improving DFlash is improving over the best known method. If DDTree can extract additional speedup from the same drafter without additional forward passes, it raises the ceiling on what speculative decoding can achieve.

Theoretically, the problem exposes an interesting tension in how to think about "draft quality." Vanilla DFlash measures quality by the expected acceptance rate of a single trajectory — the greedy path — under the target model. DDTree reframes the question: the draft model provides a factorized approximation of the target model's autoregressive distribution, and the goal is to select a set of continuations (a tree) that maximizes expected acceptance length under that approximation. This connects speculative decoding to a well-defined tree-construction optimization problem, where the tree is constructed from a single factored distribution rather than from path-conditioned probabilities. The fact that this problem admits a clean solution — top-B prefixes under the factorized distribution, recovered by a simple heap algorithm — reveals that the structure of block diffusion output is well-suited to principled tree construction, even though the path-conditioned probabilities that autoregressive drafters provide are absent.

Prior Approaches and Their Limitations

To understand what DDTree contributes, it is necessary to trace the evolution of tree-based speculative decoding and see where existing methods fall short specifically in the block diffusion setting.

Foundational speculative decoding. The original speculative decoding papers by Leviathan et al. and Chen et al. established the core idea: a drafter proposes multiple tokens autoregressively, and the target model verifies them in one forward pass. This provides lossless speedup, but the drafter generates tokens one at a time, meaning the drafting phase itself incurs latency proportional to the number of tokens proposed. The expected speedup is bounded by the acceptance rate divided by the drafting cost ratio — if drafting a token takes a quarter of the time of a target-model forward pass but the acceptance rate is only 50%, the speedup is modest.

Tree-based verification. A subsequent line of work recognized that verifying only a single drafted path is wasteful: the target model can verify multiple candidate continuations in one forward pass using tree attention. SpecInfer by Miao et al. introduced the key mechanism: draft tokens are organized into a tree structure where each node attends only to its ancestors and a shared prefix, allowing the target model to score all branches in a single batched forward pass. Medusa extended this by attaching multiple prediction heads to the target model itself, but still relies on tree attention for parallel verification. This established the template: construct a draft tree, then verify it efficiently.

Autoregressive draft trees: EAGLE and OPT-Tree. The EAGLE family represents the state of the art in autoregressive drafting with trees. EAGLE-1 drafts in feature space using target-model hidden states; EAGLE-2 introduces dynamic draft-tree construction that adapts the tree structure per decoding step; EAGLE-3 predicts tokens from fused multi-layer features and achieves strong performance. The EAGLE line demonstrates that tree-based verification with autoregressive drafters can yield substantial speedups.

However, autoregressive draft trees share a fundamental limitation: tree construction requires multiple drafter forward passes. To build a tree of depth $d$, an autoregressive drafter must generate tokens sequentially — first the depth-1 tokens, then condition on those to generate depth-2 tokens, and so on. Each depth requires at least one drafter forward pass. This makes the drafting cost scale with tree depth, eroding the speedup benefit of exploring more branches. The tree might improve acceptance length, but if the cost of constructing it grows proportionally, the net speedup may not improve.

OPT-Tree by Wang et al. addresses this by adaptively selecting which branches to expand based on an approximate expected acceptance length objective, but it still operates in the autoregressive setting where each expansion requires an additional forward pass. The theoretical framework — maximizing expected acceptance length under a node budget — is elegant, but the practical overhead of obtaining the necessary path-conditioned probabilities limits how far the tree can be grown before the drafting cost overtakes the benefit.

Block diffusion drafters: DFlash and the one-pass advantage. Block diffusion avoids the sequential drafting bottleneck by predicting all future tokens in a single forward pass. The model takes a masked block as input and denoises it to produce logits for every future position simultaneously. DFlash implements this with a small drafter conditioned on target-model features (intermediate hidden states from the large model), achieving state-of-the-art speculative decoding performance while requiring only one lightweight forward pass per round.

This one-pass property is the crucial architectural difference. An autoregressive drafter pays a per-depth cost to explore deeper or wider; a block diffusion drafter pays a fixed cost regardless of how many positions it predicts. However, vanilla DFlash does not exploit this architectural advantage for tree construction — it uses only a single trajectory per pass, effectively treating the block diffusion drafter as if it were an autoregressive drafter that happened to generate all tokens at once.

DART: concurrent work with continuity-aware pruning. A very recent concurrent work, DART, also constructs draft trees from one-pass parallel logits. DART's approach, however, relies on an external N-gram continuity score and a large N-gram trie maintained at runtime for tree pruning. This introduces auxiliary data structures and scoring mechanisms that are not derived from the draft model's own outputs. The paper explicitly distinguishes DDTree from DART: DDTree uses only the draft model's own per-position probabilities, constructs the tree directly from those probabilities, and provides an explicit surrogate objective that the best-first construction provably maximizes. No external N-gram model or trie is required.

The gap DDTree fills. Prior to DDTree, the speculative decoding landscape had a missing combination: a principled tree-construction method that exploits the one-pass parallel nature of block diffusion drafters to build a draft tree without additional drafter forward passes, using only the information the drafter has already produced. The EAGLE/OPT-Tree line showed how to build trees from autoregressive drafters but required per-depth passes. DFlash showed the power of one-pass block diffusion drafting but threw away the per-position probability information. DART tried to fill this gap but relied on external scoring. DDTree is the first method to construct a draft tree directly from block diffusion per-position marginals, with a provable optimality guarantee under a well-defined surrogate objective, and no additional drafting cost.

How DDTree Positions Itself

DDTree's position can be understood along three axes:

Architectural: DDTree is not a new drafter. It uses the exact same DFlash drafter as vanilla DFlash — no retraining, no architecture changes, no additional parameters. The innovation is entirely in how the existing drafter's output is used. This means DDTree inherits DFlash's strong base performance (the state-of-the-art drafting quality) and layers tree-based verification on top. It is a drop-in improvement that requires only changing the post-processing of the drafter's logits and the verification procedure.

Theoretical: DDTree formalizes the tree-construction problem under the block diffusion setting as maximizing expected acceptance length under the drafter's factorized distribution $Q(y_{1:L} | c, b) = \prod_{i=1}^L q_i(y_i | c, b)$. This is a precise surrogate objective, and the paper proves that (a) the optimal tree consists of the top-$B$ prefixes under this factorized distribution, and (b) these prefixes can be recovered by a simple best-first heap algorithm without enumerating the exponential prefix space. This theoretical scaffolding is clean and self-contained — it adapts OPT-Tree's expected acceptance length framework to a setting where the required probabilities come from one forward pass rather than multiple.

Practical: DDTree is implemented on top of Hugging Face Transformers and evaluated against the same DFlash baselines across a comprehensive benchmark suite. The paper shows consistent improvements across all model sizes (4B, 8B, 30B-MoE), all domains (reasoning, code, instruction-following), and both temperature settings (0.0 and 1.0). The budget-quality tradeoff analysis in Figure 3 shows that the optimal node budget typically sits in the 256–512 range, beyond which verifier overhead dominates, making DDTree practical to deploy with a reasonable fixed budget.

The paper also implicitly addresses a potential objection: why maximize expected acceptance length under the drafter's factorized distribution rather than the target model's true autoregressive distribution? The answer is pragmatic — the target model's path-conditioned probabilities are unavailable at tree-construction time (they would require running the target model, which defeats the purpose). The factorized surrogate is the best information available from the one-pass drafter, and the empirical results demonstrate that optimizing this surrogate translates to real speedup gains. In other words, the surrogate is good enough to be useful, even though it is not perfect.

3. Technical Approach

3.1 Reader Orientation

DDTree is a tree-based speculative decoding method that takes the per-position marginal token distributions already produced by a block diffusion drafter (specifically DFlash) in a single forward pass, and uses them to construct a compact tree of promising candidate continuations — rather than discarding most of that probability information and verifying only one path as vanilla DFlash does. It solves the problem of underutilized drafter output: a block diffusion drafter computes probability distributions over all vocabulary tokens at every future position in one shot, yet prior methods collapse those distributions into a single trajectory, leaving substantial acceptance-length gains on the table. The solution's shape is a three-stage pipeline — (1) obtain per-position marginals from the drafter, (2) run a best-first heap algorithm to select the top-B most probable prefixes under those marginals as a draft tree, and (3) verify the tree in one target-model forward pass using tree attention — all without any additional drafter forward passes.

3.2 Big-Picture Architecture (Diagram in Words)

The DDTree system has four major components that operate sequentially within each speculative decoding round:

1. Block Diffusion Drafter (DFlash): Given the current context and the bonus token from the previous round, runs one forward pass to produce per-position marginal distributions $q_i(v)$ over the vocabulary for each of the next $L$ draft positions. This is the only drafter computation — it happens once per round regardless of how many continuations are eventually explored.

2. Tree Construction Engine (Algorithm 1): Takes the per-position distributions and a user-specified node budget $B$ as input. Uses a max-heap priority queue to enumerate the $B$ highest-probability prefixes under the factorized distribution $Q(y_{1:L}) = \prod_i q_i(y_i)$, guaranteeing that the resulting set forms a valid prefix-closed tree and maximizes the expected acceptance length under $Q$. Outputs a set of token ID prefixes organized as a tree rooted at the bonus token.

3. Tree Compiler and Verifier (Target Model): Flattens the draft tree into a single sequence of token IDs, assigns position IDs by tree depth, and constructs an ancestor-only tree attention mask. Runs one target-model forward pass over this flattened sequence to compute logits for all tree nodes simultaneously. This is the only target-model computation per round.

4. Verifier Walker: Starting from the bonus token, applies the target model's decoding rule (greedy argmax or temperature-based sampling) step by step, checking at each position whether the chosen token matches a child in the draft tree. Accepts the matched path, appends it to the output sequence, and designates the first unmatched target token as the next round's bonus token. Updates the KV cache to retain only the accepted path.

Information flows strictly left-to-right: drafter forward pass → per-position distributions → tree construction → flattened token tensors + attention mask → target model forward pass → verifier walk → accepted tokens + new bonus token. There is no feedback loop within a round, and no component depends on any other component more than once per round.

3.3 Roadmap for the Deep Dive

• First, I will define the speculative decoding protocol that DDTree operates within — specifically the round structure, the bonus token convention, and what information is available when — because all subsequent components depend on understanding these boundary conditions.

• Second, I will explain what a block diffusion drafter produces and why "per-position marginals" are fundamentally different from "path-conditioned probabilities," since this distinction is the core intellectual challenge that the tree-construction objective must address.

• Third, I will derive the surrogate objective for tree construction — the expected acceptance length under the factorized distribution $Q$ — and prove why it decomposes into a simple sum of prefix probabilities. This is the mathematical foundation for everything that follows.

• Fourth, I will walk through Algorithm 1 in detail: how the max-heap enumerates prefixes in descending probability order, why the sibling/child generation rules cover the entire search space without duplicates, and why the algorithm is both correct and efficient ($O(B \log B)$).

• Fifth, I will describe the verification procedure: how the draft tree is compiled into input tensors, how tree attention masks are constructed, how the verifier walk decides accept/reject at each node, and how the KV cache is compacted afterward.

3.4 Detailed, Sentence-Based Technical Breakdown

This is a method paper whose core idea is that the per-position marginal distributions produced by a single block diffusion drafter forward pass — which prior work collapses into a single trajectory — contain enough information to construct a high-quality draft tree that substantially improves expected acceptance length without additional drafting cost. The method is neither a new drafter architecture nor a new verification mechanism, but rather a principled tree-construction algorithm and its accompanying theoretical justification.

3.4.1 The Speculative Decoding Round Structure

Before diving into DDTree's specific mechanisms, it is necessary to precisely define the speculative decoding protocol that DDTree operates within, because the availability and ordering of information at each stage constrains everything else.

Speculative decoding proceeds in rounds. At the beginning of each round, the system possesses:

• A context $c$, which includes the original prompt and all tokens generated and committed in prior rounds.
• A bonus token $b$, which is a single token already selected by the target model but not yet processed through the target model's forward pass. In the first round, $b$ comes from the prefill pass (the target model's initial forward pass over the prompt, which produces the first generated token). In subsequent rounds, $b$ is the "first unmatched target token" from the previous round's verifier walk — the token the target model chose that did not match any drafted continuation.

The critical property of the bonus token is that (a) its identity is known to the drafter, so the drafter can condition on it when predicting the next tokens, but (b) the target model has not yet been run with $b$ appended to the context, meaning that target-model features (hidden states, keys, values) are only available for the context before $b$. This is why DFlash needs target-model feature distillation: the drafter sees the bonus token identity but not fresh target-model representations of it.

Within a round, the sequence of operations is:

1. Draft: The drafter predicts future tokens given $c$ and $b$.
2. Verify: The target model scores the drafted tokens in one forward pass, with the drafted tokens appended after $b$.
3. Accept/Reject: The verifier walk compares the target model's token choices against the drafted tokens and determines how many consecutive tokens are accepted.
4. Commit: The accepted tokens are appended to the output, and the first rejected (or unmatched) target token becomes the next round's bonus token.

DDTree modifies only step 1 (the drafter produces per-position distributions instead of a single trajectory, and a tree is constructed from those distributions) and the input to step 2 (the target model receives a tree of candidates instead of a single path). Steps 3 and 4 remain structurally identical, though they operate over a tree rather than a line.

3.4.2 What a Block Diffusion Drafter Produces (and What It Does Not)

Understanding DDTree requires a precise account of the information a block diffusion drafter provides, because the method's central design choice — using a factorized surrogate distribution rather than path-conditioned probabilities — follows directly from the nature of that information.

The drafter's input. Given the context $c$ and the bonus token $b$, the block diffusion drafter takes a masked block of length $L + 1$ as input:

[b, m, m, ..., m]

where $m$ represents a mask token and $L$ is the block size (set to 16 in all DDTree experiments). The drafter's job is to predict the tokens that should replace the masks.

The drafter's output. In a single forward pass, the drafter produces logits for every masked position simultaneously. After applying softmax, this yields per-position marginal distributions $q_i(v)$ for $i = 1, \ldots, L$ and $v \in \mathcal{V}$ (the vocabulary):

$q_i(v) = \text{softmax}(\ell_i)_v$

where $\ell_i \in \mathbb{R}^{|\mathcal{V}|}$ is the vector of raw logits at draft position $i$, and $q_i(v)$ is the drafter's estimated probability that token $v$ appears at position $i$ after the bonus token, averaged over all possible completions of the earlier positions in the block.

What it computes: For each future position $i$, a probability distribution over the entire vocabulary, representing the drafter's marginal belief about which token occupies that position. The distributions at different positions are computed simultaneously from the same conditioning context $(c, b)$, without any sequential dependence within the block.

Why this form: Block diffusion models (like the masked language models they descend from) are trained to predict tokens given bidirectional or prefix context with some positions masked. The training objective encourages the model to estimate $P(\text{token at position } i \mid \text{context before block})$ without conditioning on specific choices at other masked positions. This parallel prediction is what enables the one-pass speed advantage, but it means the output is fundamentally a product of marginals rather than an autoregressive factorization.

The crucial distinction: marginals vs. path-conditioned probabilities. Under the target model $p$, the distribution over a continuation $y_{1:L}$ factorizes autoregressively:

$p(y_{1:L} \mid c, b) = \prod_{i=1}^L p(y_i \mid c, b, y_{1:i-1})$

Each factor $p(y_i \mid c, b, y_{1:i-1})$ conditions on the specific tokens $y_{1:i-1}$ chosen at earlier positions within the block. This means the probability of a prefix $y_{1:d}$ depends on the joint compatibility of the tokens — whether they form a coherent continuation together.

The block diffusion drafter provides only the marginals $q_i(y_i \mid c, b)$, which condition on $(c, b)$ but not on $y_{1:i-1}$. The natural distribution associated with the drafter's output is therefore the factorized distribution:

$Q(y_{1:L} \mid c, b) = \prod_{i=1}^L q_i(y_i \mid c, b)$

This distribution treats the token at each position as independent given $(c, b)$. It does not capture dependencies between positions within the drafted block — for instance, it assigns the same probability to a grammatically coherent two-token sequence and a nonsensical one, provided both consist of individually probable tokens at each position.

Why this matters for tree construction. If the drafter provided path-conditioned probabilities $q(y_i \mid c, b, y_{1:i-1})$ — as an autoregressive drafter does — one could compute the exact probability of any prefix under the drafter's own model by multiplying those conditional factors. Tree construction could then directly optimize expected acceptance length under the drafter. But the block diffusion drafter does not provide these conditional factors — running it multiple times with different prefixes would require multiple forward passes, defeating the purpose of using a one-pass drafter. DDTree must therefore work with the factorized surrogate $Q$, and the central theoretical contribution of the paper is showing that optimizing under this surrogate — despite its independence assumption — is both computationally tractable and empirically effective.

3.4.3 The Surrogate Objective: Expected Acceptance Length Under $Q$

The ideal objective for draft-tree construction is to maximize the expected number of tokens that the target model will accept. However, this depends on the target model's autoregressive conditional probabilities $p(y_i \mid c, b, y_{1:i-1})$, which are unavailable at tree-construction time — obtaining them would require running the target model (the very computation we are trying to minimize). DDTree therefore optimizes a surrogate objective: the expected acceptance length under the drafter's factorized distribution $Q$.

Defining acceptance length. For a candidate continuation $y_{1:L}$ and a draft tree $\mathcal{T}$, the acceptance length is defined as the longest prefix of $y_{1:L}$ that appears as a complete path in the tree:

$\alpha_{\mathcal{T}}(y_{1:L}) = \max\{d : y_{1:d} \in \mathcal{T}\}$

with $\alpha_{\mathcal{T}}(y_{1:L}) = 0$ when no depth-1 node matches. In prose: start at the root, follow the tree along the path $y_1, y_2, \ldots$, and stop when the token is not found as a child in the tree. The acceptance length counts how many consecutive tokens match.

The surrogate optimization problem. The surrogate objective is:

$\max_{\mathcal{T}: |\mathcal{T}| \leq B, \mathcal{T} \text{ valid}} \mathbb{E}_{Y_{1:L} \sim Q(\cdot \mid c, b)}\left[\alpha_{\mathcal{T}}(Y_{1:L})\right]$

where:

• $\mathcal{T}$ is a draft tree — a prefix-closed set of continuation prefixes (if a node is in the tree, all its ancestors must also be in the tree),
• $|\mathcal{T}| \leq B$ is the node budget — the tree can contain at most $B$ non-root nodes (the root is the bonus token $b$ and does not count toward the budget),
• $B$ is a user-specified integer (swept over $\{16, 32, 64, 128, 256, 512, 1024\}$ in the experiments),
• $Q(\cdot \mid c, b)$ is the factorized distribution from Equation (2), constructed from the per-position marginals $q_i$,
• $Y_{1:L} \sim Q$ means we imagine sampling a continuation from the factorized distribution and measuring how far it matches the tree.

What it computes: The expected number of consecutive positions, starting from the bonus token, where a continuation drawn from the factorized distribution $Q$ matches a path in the draft tree $\mathcal{T}$. This expectation is taken over all possible continuations, weighted by their probability under $Q$. Higher values mean the tree covers more probability mass — it is more likely that a $Q$-likely continuation will find a matching path in the tree.

Why this form: The surrogate is chosen because all the information needed to compute it — the prefix probabilities $q(u \mid c, b) = \prod_{i=1}^{|u|} q_i(u_i \mid c, b)$ — is directly available from the single drafter forward pass. No additional model calls are needed, and the optimization can be performed entirely in post-processing. The key assumption is that $Q$ is a good enough approximation to $p$ that a tree which covers high-probability prefixes under $Q$ will also tend to cover prefixes the target model is likely to produce. The empirical results (Table 1, Figure 4) validate this assumption: optimizing under $Q$ yields consistent speedups over the vanilla single-trajectory approach.

The decomposition into prefix probabilities (Proposition 1). The key mathematical insight that makes the surrogate tractable is that the expected acceptance length decomposes into a sum of prefix probabilities over the tree nodes:

$\mathbb{E}_{Y_{1:L} \sim Q(\cdot \mid c, b)}\left[\alpha_{\mathcal{T}}(Y_{1:L})\right] = \sum_{u \in \mathcal{T}} q(u \mid c, b)$

where for a prefix $u = (u_1, \ldots, u_d)$:

$q(u \mid c, b) = \prod_{i=1}^{|u|} q_i(u_i \mid c, b)$

What it computes: The right-hand side is simply the sum of the factorized probabilities of every prefix (node) in the tree. Each node contributes its own marginal probability under $Q$, regardless of the other nodes. The sum is additive — adding or removing a node changes the objective by exactly that node's probability, with no interaction terms.

Why this decomposition holds (intuition for Proposition 1). The acceptance length $\alpha_{\mathcal{T}}(Y)$ counts how many prefixes of $Y$ match the tree. The probability that the acceptance length is at least $d$ equals the probability that the length-$d$ prefix of $Y$ matches one of the depth-$d$ nodes in $\mathcal{T}$. Since these depth-$d$ events are mutually disjoint (a single continuation has exactly one length-$d$ prefix), the probability is just the sum of $q(u \mid c, b)$ over depth-$d$ nodes $u \in \mathcal{T}$. Summing over $d$ from $1$ to $L$ yields the total sum over all nodes. This is formalized in Appendix A, but the intuition is straightforward: each node in the tree contributes its own probability mass to the expected acceptance length, and contributions are additive because each continuation follows exactly one path through the tree.

Why this additive form matters. The additive decomposition means that the surrogate objective is modular — the contribution of each node is independent of which other nodes are in the tree. This simplifies the optimization enormously: to maximize the sum of at most $B$ non-negative terms under a prefix-closure constraint, one simply takes the $B$ prefixes with the highest individual probabilities $q(u \mid c, b)$. The prefix-closure constraint turns out to be automatically satisfied by this greedy selection, because for any prefix $u$, its parent prefix has strictly higher probability (since $q_i(v) < 1$ for all $i, v$, multiplying by an additional factor strictly decreases the product). Therefore, when prefixes are sorted by descending probability, every ancestor appears before its descendants — the top-$B$ set is guaranteed to be prefix-closed.

The optimal tree (Proposition 2). Formally, let $u^{(1)}, u^{(2)}, \ldots$ be all nonempty prefixes of length at most $L$, sorted so that $q(u^{(1)} \mid c, b) \geq q(u^{(2)} \mid c, b) \geq \cdots$. Then:

$\mathcal{T}_B = \{u^{(1)}, \ldots, u^{(B)}\}$

is a valid draft tree (it satisfies the prefix-closure property) and maximizes the surrogate expected acceptance length among all valid trees with at most $B$ nodes.

What it computes: The optimal draft tree under the factorized surrogate $Q$ is simply the set of the $B$ individually most probable prefixes, where probability is measured by the product of per-position marginal probabilities along the prefix. No joint optimization or trade-off between depth and breadth is needed — the modular objective decouples the selection.

Why this works despite the independence assumption. The surrogate $Q$ treats token choices at different positions as independent, which means that a path's probability under $Q$ is simply the product of the marginal probabilities of its tokens. Under this independence assumption, there is no "coherence bonus" for choosing tokens that work well together — a prefix $(u_1, u_2)$ has the same probability under $Q$ regardless of whether $(u_1, u_2)$ is a natural bigram. The optimal tree under $Q$ is therefore heavily "front-loaded" with high-probability tokens at early positions extended by high-probability tokens at later positions, regardless of whether those combinations form coherent continuations. The empirical success of DDTree suggests that, in practice, the independence assumption is not too harmful: the drafter's high-probability tokens at each position tend to be individually reasonable, and extending them greedily covers enough of the target model's likely continuations to provide substantial speedup.

3.4.4 Efficient Tree Construction: Algorithm 1

Proposition 2 tells us what the optimal tree is — the top-$B$ prefixes under $Q$ — but does not tell us how to find those prefixes efficiently. The naive approach of enumerating all possible prefixes and sorting by probability is computationally infeasible: even with a vocabulary size of $|\mathcal{V}| \approx 10^5$ and depth $L = 16$, the number of possible prefixes is astronomically large. Algorithm 1 solves this by exploiting the structure of the factorized distribution to enumerate only the most promising prefixes.

Reducing the search space (Lemma 1). The first key insight is that the optimal tree uses only the top-$K$ tokens at each position, where $K = \min(B, |\mathcal{V}|)$. Intuitively, if a prefix uses a token of rank worse than $B$ at some position, replacing it with a token of rank at most $B$ at that position yields a prefix with higher probability. Since there are $B$ slots in the tree and only $B$ possible top-rank tokens per position, any prefix using a token outside the top-$B$ can never be among the top-$B$ prefixes overall. Formally, Lemma 1 states: there exists an optimal valid draft tree maximizing the surrogate objective such that every node in the tree lies in $\mathcal{S}_K$, the set of prefixes that use only the top-$K$ tokens at each depth.

This reduces the search space from $O(|\mathcal{V}|^L)$ to $O(K^L)$, which is still exponential in $L$ but with a much smaller base — in practice, $K \leq 1024$ (the maximum node budget tested). However, even $K^L$ is far too large to enumerate when $L = 16$. Algorithm 1 uses a best-first search to enumerate only the actually-needed high-probability prefixes.

Rank-tuple representation. For notational convenience, Algorithm 1 works not with vocabulary token IDs directly but with rank tuples. At each depth $i$, the tokens are sorted by descending marginal probability $q_i(v)$:

$q_i(v_i^{(1)} \mid c, b) \geq q_i(v_i^{(2)} \mid c, b) \geq \cdots$

Let $q_i^{(k)} = q_i(v_i^{(k)} \mid c, b)$ denote the probability of the $k$-th ranked token at depth $i$. A prefix is represented by a tuple of ranks:

$\rho = (\rho_1, \rho_2, \ldots, \rho_d)$

where $1 \leq d \leq L$ and $1 \leq \rho_i \leq K$ for each $i$. The tuple $\rho = (1, 3, 2)$ means: take the most probable token at position 1, the third-most probable token at position 2, and the second-most probable token at position 3.

Log-probability scoring. The probability of a rank tuple $\rho$ under $Q$ is:

$q(\rho) = \prod_{i=1}^d q_i^{(\rho_i)}$

To improve numerical stability and convert the multiplicative objective into an additive one suitable for heap ordering, Algorithm 1 uses log-probabilities:

$\sigma(\rho) = \log q(\rho) = \sum_{i=1}^d \log q_i^{(\rho_i)}$

Since the logarithm is strictly monotonic, ordering prefixes by descending $\sigma(\rho)$ is equivalent to ordering by descending $q(\rho)$. The heap stores $(\rho, \sigma(\rho))$ pairs and pops the element with maximum $\sigma(\rho)$.

The generation rules: siblings and children. The heap is initialized with the single-element tuple $(1)$, representing the most probable token at the first draft position. When a tuple $\rho = (\rho_1, \ldots, \rho_d)$ is popped from the heap (because it has the highest $\sigma$ among all unpopped tuples), the algorithm generates at most two new tuples to push:

1. Next sibling: If $\rho_d + 1 \leq K$, push $(\rho_1, \ldots, \rho_{d-1}, \rho_d + 1)$. This explores an alternative token at the current position — same prefix up to depth $d-1$, but with the next-ranked token at depth $d$. Its score is computed by an incremental update:

   $\sigma(\text{sibling}) = \sigma(\rho) - \log q_d^{(\rho_d)} + \log q_d^{(\rho_d + 1)}$

   This subtracts the contribution of the old depth-$d$ token and adds the contribution of the new one, avoiding recomputing the sum from scratch.

2. First child: If $d < L$, push $(\rho_1, \ldots, \rho_d, 1)$. This extends the current prefix with the most probable token at the next depth. Its score is:

   $\sigma(\text{child}) = \sigma(\rho) + \log q_{d+1}^{(1)}$

   This appends the log-probability of the rank-1 token at depth $d+1$.

What this generates: The sibling rule enumerates all tokens at the current depth in descending probability order — after popping rank $\rho_d$, the sibling rule generates rank $\rho_d + 1$ so it is available for future pops. The child rule extends the current prefix downward — after popping a prefix of depth $d$, the child rule makes its depth-$(d+1)$ extension with the best available token at that depth available for exploration. Together, these two rules generate every tuple in $\mathcal{S}_K$ exactly once, and in descending order of probability.

Why this ordering works (Proposition 3 intuition). The key property is that the predecessor of any tuple (obtained by either decrementing the last rank or removing it) has strictly higher probability. Since the algorithm always generates a tuple only after its predecessor is popped, and the heap always returns the highest-probability unpopped tuple, the pop sequence is guaranteed to be nonincreasing in probability. This means the first $B$ popped tuples are exactly the top-$B$ prefixes in $\mathcal{S}_K$.

The algorithm in pseudocode (operational description).

1. Initialize: Create a max-heap $H$ containing the tuple $((1), \sigma((1)))$. Initialize an empty draft tree $\mathcal{T}$.
2. Loop: While $|\mathcal{T}| < B$ and $H$ is not empty:
   • Pop the tuple $\rho = (\rho_1, \ldots, \rho_d)$ with the largest $\sigma(\rho)$.
   • Add the corresponding token prefix $(v_1^{(\rho_1)}, \ldots, v_d^{(\rho_d)})$ to $\mathcal{T}$.
   • If $\rho_d + 1 \leq K$, push the next sibling.
   • If $d < L$, push the first child.
3. Return: $\mathcal{T}$.

Complexity. The algorithm performs at most $B$ pops and at most $2B$ pushes (since each popped tuple generates at most two new tuples). The heap size is $O(B)$ throughout. Each heap operation is $O(\log B)$, so the total complexity is $O(B \log B)$. In the experiments, $B$ ranges from 16 to 1024, making this extremely fast — the tree construction overhead is negligible compared to the target model forward pass.

Why a heap rather than sorting all $K^L$ prefixes. The number of possible prefixes in $\mathcal{S}_K$ is $\sum_{d=1}^L K^d \approx K^L$, which for $K = 256$ and $L = 16$ is approximately $256^{16} \approx 2^{128}$ — completely impossible to enumerate. The heap only generates prefixes as they are needed, exploring the search space in best-first order and stopping after $B$ prefixes. Since $B \ll K^L$ (the budget is at most 1024), the algorithm examines only a tiny fraction of the space.

3.4.5 Verification: Tree Attention and the Verifier Walk

Once the draft tree $\mathcal{T}$ is constructed, DDTree must verify all $B$ drafted nodes against the target model in a single forward pass, then walk the tree to determine the accepted path.

Flattening the tree. The draft tree is a set of prefixes rooted at the bonus token $b$. To feed it to the target model, DDTree flattens the tree into a single sequence of token IDs. The flattening order is a depth-first or breadth-first traversal — any order works as long as the attention mask correctly encodes the tree structure. Each tree node (prefix) is assigned a position ID equal to its depth in the tree: the root $b$ has position ID 0, depth-1 nodes have position ID 1, depth-2 nodes have position ID 2, and so on. This ensures that the target model's positional embeddings are applied correctly — a token at depth $d$ in the tree gets the same positional encoding regardless of which path it belongs to, consistent with the fact that it represents a candidate for the $d$-th position after $b$.

Tree attention mask. Standard causal attention allows each token to attend to all preceding tokens in the sequence. For tree verification, this would be incorrect — a token at depth $d$ along one branch of the tree should not attend to tokens at depth $d$ on other branches, because those represent alternative, mutually exclusive continuations. Instead, DDTree uses tree attention (also called ancestor-only attention): each drafted node attends to:

• The past context through the KV cache — all tokens before the current speculative decoding round, including the original prompt and all previously committed tokens.
• The root of the current tree (the bonus token $b$).
• Its ancestors in the tree — the unique path from $b$ to the current node.
• Itself — standard self-attention.

In practice, this is implemented by constructing a block-diagonal attention mask where the diagonal blocks correspond to individual paths from root to leaf, and tokens on different paths cannot attend to each other. Nodes at the same depth but on different branches have no attention edges between them. This ensures that the target model's output distribution at each node conditions only on the prefix that logically precedes it, not on alternative continuations.

Single forward pass. The flattened token sequence and the tree attention mask are fed to the target model in a single forward pass. The target model computes logits for every tree node simultaneously. The output at a node at depth $d$ (representing prefix $u_1, \ldots, u_d$) is the target model's next-token distribution $p(\cdot \mid c, b, u_1, \ldots, u_d)$ — the distribution over the $(d+1)$-th token after $b$, conditioned on the specific prefix path. This is exactly the information needed for the verifier walk.

The verifier walk. With all target-model logits available from the single forward pass, verification proceeds sequentially from the root:

1. Start at the bonus token $b$ (depth 0 in the tree).
2. At the current depth, the target model has produced a distribution over the next token, conditional on the prefix that leads to this node. Apply the target model's decoding rule to select one token:
   • At temperature $T = 0.0$ (greedy): select $y^* = \arg\max_v p(v \mid \text{prefix})$.
   • At temperature $T > 0$ (sampling): sample $y^* \sim \text{softmax}(\text{logits} / T)$.
3. Check whether $y^*$ appears as a child of the current node in the draft tree. If yes, this drafted token is accepted — move to that child node and repeat step 2. If no, the walk stops at this depth.
4. The accepted path is the sequence of tokens from the root to the deepest accepted node. These tokens are appended to the output sequence.
5. The next bonus token is the first token that was not accepted — the target model's chosen token $y^*$ at the node where the walk stopped. This bonus token is carried to the next speculative decoding round and serves as the new tree root.

What this computes: The verifier walk implements the standard speculative decoding acceptance protocol but extended to tree structures. At each step, the target model's own decoding rule (greedy or sampled) determines which token would have been generated. If that token happens to be in the draft tree as a child of the current node, it is "speculatively correct" and can be committed without an additional target model forward pass. The walk continues as long as the target model's choices match the tree — it can switch between different branches of the tree at each step because the tree may have multiple children at any node and the walk follows whichever child (if any) matches the target's choice.

Why tree attention enables this efficiently. Without tree attention, verifying $B$ candidate continuations would require $B$ separate target-model forward passes (one per path) or a single forward pass over a concatenated sequence where tokens from different branches could incorrectly attend to each other. Tree attention solves both problems: it packs all $B$ nodes into one batched forward pass while ensuring that each node's contextual representation is computed correctly — as if it were the only continuation being verified. The cost is one target-model forward pass over a sequence of length $B + 1$ (the $B$ tree nodes plus the root), which is substantially cheaper than $B$ separate passes over shorter sequences because the quadratic attention cost is dominated by the shared prefix (the past context through the KV cache), which is computed only once and reused.

KV cache update. After verification, the KV cache must be updated to reflect the newly committed tokens. DDTree compacts the cache to retain only the accepted path — the key-value pairs for all nodes on the accepted path are kept, while key-value pairs for unaccepted branches are discarded. The bonus token (the first rejected target token) is then appended to the cache, and its key-value pair is computed and stored so that the next round's drafter can condition on it. This cache compaction is standard in speculative decoding and ensures that the cache size grows only with the number of actually generated tokens, not with the number of speculatively explored paths.

3.4.6 Design Choices and Their Justifications

Why a fixed node budget $B$ rather than an adaptive stopping criterion? The node budget $B$ controls the tradeoff between exploration (more nodes → higher acceptance length) and verification cost (more nodes → longer target-model forward pass). The paper sweeps $B$ as a hyperparameter and selects the value that maximizes end-to-end speedup for each dataset-model pair (Figure 3): "As our DDTree node budget grows, acceptance length increases steadily, and the end-to-end speedup improves until it peaks around budgets of 256 to 512. Pushing the budget to 1024 increases acceptance length further, but the tradeoff is no longer favorable as the additional overhead of verifying more drafted tokens outweighs the gain from the longer accepted prefix." A fixed budget is simpler to implement than an adaptive scheme (e.g., adding nodes until marginal gain drops below a threshold) and produces predictable per-round latency, which is important for deployment.

Why $K = \min(B, |\mathcal{V}|)$ rather than a smaller or larger $K$? Lemma 1 establishes that restricting to the top-$K$ tokens at each depth does not sacrifice optimality — any prefix using a token outside the top-$B$ can be improved by substitution. Using a smaller $K$ would risk excluding prefixes that could be in the optimal tree (if $K < B$). Using a larger $K$ (e.g., $K = |\mathcal{V}|$) would expand the search space unnecessarily without changing the result — the heap would still only pop $B$ prefixes, and the extra $K - B$ tokens per position would never be explored because they have lower probability than tokens already in the heap. Setting $K = \min(B, |\mathcal{V}|)$ is the minimal sufficient value.

Why log-probability scoring rather than raw probability? Probabilities $q(u)$ are products of many factors $q_i(u_i)$, each less than 1. For deep prefixes ($d$ up to 16), the product can underflow floating-point precision. Taking the logarithm converts the product into a sum, which is numerically stable. Additionally, the heap requires a total order, and log is monotonic, so ordering by $\log q(u)$ is equivalent to ordering by $q(u)$. The incremental score update for siblings ($\sigma(\text{sibling}) = \sigma(\rho) - \log q_d^{(\rho_d)} + \log q_d^{(\rho_d+1)}$) is also simpler with log-probabilities — it requires only two additions rather than a division and multiplication.

Why the sibling/child generation rules rather than a different enumeration? The two rules — next sibling and first child — together generate every prefix in $\mathcal{S}_K$ exactly once. The sibling rule ensures all tokens at a given depth are explored in rank order; the child rule ensures that once a prefix is explored, its extension with the best token at the next depth becomes available. Alternative enumerations (e.g., generating all children when a parent is popped) would produce duplicate entries in the heap and degrade the $O(B \log B)$ bound. The chosen rules produce a spanning tree over the search space, with each node generated exactly once by its unique predecessor.

Why tree attention rather than sequential verification? Sequential verification — running the target model separately on each candidate path — would multiply the target-model cost by the number of paths, completely erasing the speculative decoding speedup. Tree attention allows one target-model forward pass to score all nodes simultaneously, making the verification cost approximately proportional to the total number of tree nodes rather than the number of branches. This is what makes tree-based speculative decoding viable — without it, exploring multiple continuations would be strictly worse than verifing a single path multiple times.

Why position IDs by tree depth rather than sequential position in the flattened sequence? The target model's positional embeddings should reflect the logical position of each token in the generated sequence, not its physical position in the flattened input. A token that is the $d$-th token after the bonus token should receive the same positional encoding regardless of which branch it sits on or where it appears in the flattened tensor. Assigning position IDs by tree depth ensures this — the target model's attention mechanism will correctly interpret a depth-3 token as being three positions after $b$, even though there may be many other depth-3 tokens (on other branches) in the same forward pass.

4. Key Insights and Innovations

Innovation 1: Reframing Block Diffusion Output as a Tree-Construction Resource, Not a Single Trajectory

The central conceptual move of DDTree is not algorithmic but perceptual: it recognizes that a block diffusion drafter's single forward pass produces a probability landscape — per-position marginal distributions over all vocabulary tokens at all future positions — and that this landscape is a resource for tree construction, not merely a means to select one greedy path. This reframing is what makes all the subsequent technical contributions possible and meaningful.

To appreciate the shift, consider the default assumption in prior speculative decoding work. Autoregressive drafters (EAGLE, EAGLE-2, EAGLE-3, OPT-Tree's autoregressive variant) generate tokens one at a time, conditioning each step on previously generated tokens. In that setting, tree construction is naturally understood as a process of sequential expansion: at each depth, run the drafter, sample or select the top-k tokens, expand the tree, and repeat. The draft model's output at depth d is intrinsically tied to a specific prefix — it is the conditional distribution given that prefix. This makes tree construction feel like navigating a branching process, where each node's children are generated by running the drafter conditioned on that node.

Block diffusion breaks this mental model. A single DFlash forward pass produces distributions for all L positions simultaneously, without conditioning on any specific intra-block token choices. The distributions are marginals, not path-conditioned. The naive interpretation — and the one vanilla DFlash adopts — is to collapse each marginal to its argmax and take the resulting trajectory as the single draft. This treats the block diffusion drafter as if it were an autoregressive drafter that happened to compute all positions in parallel, discarding the probability information at non-argmax tokens entirely. It is a natural but impoverished view: the drafter computed all those probabilities, at non-trivial computational cost, and they are simply ignored.

DDTree's reframing says: the per-position marginals are exactly what you need to construct a tree, provided you are willing to optimize under a factorized surrogate rather than true path-conditioned probabilities. The tree is not built by conditioning the drafter on different prefixes (which would require multiple forward passes, defeating the purpose of one-pass drafting), but by selecting prefixes from the factorized distribution induced by the marginals. This shifts tree construction from a generation problem (run the drafter to produce candidates) to a selection problem (pick the most promising candidates from the already-computed probability landscape). The drafter runs once; the tree is an optimized post-processing of its output.

The significance of this reframing extends beyond DDTree itself. It establishes that one-pass parallel drafters are not merely faster alternatives to autoregressive drafters — they are fundamentally different in the structure of information they provide, and this difference enables qualitatively different tree-construction strategies. An autoregressive drafter provides rich conditional information (you know exactly how probable each continuation is given its prefix) but at high per-depth cost. A block diffusion drafter provides impoverished information (only marginals, no conditioning within the block) but at zero marginal cost for additional depth or breadth. DDTree shows that the latter tradeoff — impoverished but abundant and cheap information — can be exploited through a principled surrogate optimization, achieving speedups that would be infeasible with autoregressive drafters due to the per-depth drafting cost. This insight has immediate architectural implications: it suggests that future drafter designs should be evaluated not just on single-trajectory acceptance rate, but on the quality of the full per-position distributions as a tree-construction resource.

Evidence. The empirical consequence of this reframing is visible in Figure 3: as the node budget B grows from 16 to 512, acceptance length increases steadily from roughly 9 to nearly 11 tokens on MATH-500 with Qwen3-8B, and speedup improves from roughly 6.5× to 7.5×. This gain comes entirely from better utilizing the same drafter output — no additional drafter computation. Vanilla DFlash, which uses only the argmax trajectory, is equivalent to DDTree with B = 1 (a single path) and achieves 5.56× speedup and acceptance length 7.79 (Table 1). The gap between B = 1 and B = 512 is the value of the reframing: the probability information was always there in the DFlash output; DDTree is the first method to use it for tree construction without additional forward passes.

Innovation 2: The Factorized Surrogate as a Tractability Bridge Between One-Pass Drafting and Tree Optimization

The second conceptual contribution is the formulation and justification of the factorized surrogate objective — optimizing expected acceptance length under the drafter's independent-position distribution Q rather than under the target model's true autoregressive distribution p — as the right level of abstraction for block diffusion tree construction. This is a theoretical bridge that connects the information a one-pass drafter can provide (per-position marginals) to the optimization problem one wants to solve (find a tree that the target model will accept many tokens from).

The difficulty that this bridge addresses is fundamental. The target model's acceptance behavior depends on path-conditioned probabilities p(y_i | c, b, y_{1:i-1}) — the probability the target assigns to a token given the specific prefix that precedes it. These are precisely what a single block diffusion forward pass cannot provide, because the drafter does not condition on intra-block token choices. One could obtain them by running the drafter multiple times with different prefixes, but this would reintroduce the per-depth drafting cost that block diffusion was designed to eliminate. The field therefore faced a dilemma: either pay the sequential drafting cost to get path-conditioned probabilities (the autoregressive drafter approach), or accept impoverished information and use only a single trajectory (vanilla DFlash). DDTree resolves this dilemma by showing that optimizing under the factorized surrogate Q — despite its independence assumption — yields trees that are empirically effective.

The intellectual contribution here is not the mathematical fact that the surrogate decomposes additively (Proposition 1), nor that the optimal tree under Q is the top-B prefixes (Proposition 2). These are clean but straightforward consequences of the independence assumption. The contribution is the decision to optimize this surrogate at all, and the empirical demonstration that doing so produces meaningful speedup gains despite the surrogate's obvious flaws. The independence assumption means that Q assigns the same probability to a coherent bigram and a nonsensical one, provided both consist of individually probable tokens. Under Q, there is no penalty for combining tokens that never co-occur in natural language. Yet the trees constructed under Q consistently improve over single-trajectory DFlash across 60 dataset-model-temperature settings (Table 1). This is a non-obvious empirical finding: the independence assumption is wrong enough that one might reasonably expect Q-based tree construction to waste budget on incoherent paths that the target model would never follow, but right enough that the selected paths are, in aggregate, substantially better than a single greedy trajectory.

Why does the surrogate work despite its flaws? The paper does not provide a deep analysis, but the results suggest an implicit regularization: the drafter's high-probability tokens at each position are individually reasonable continuations of the prefix context, and extending them greedily (always taking the highest-probability token at the next depth) tends to produce paths that, while not guaranteed to be coherent, are good enough that the target model accepts them at higher rates than the single greedy path. The independence assumption effectively makes the tree "front-loaded" — it prefers to extend high-probability early tokens with high-probability later tokens, regardless of joint coherence. This turns out to be a useful bias in practice, perhaps because the base model's training ensures that individually probable tokens are rarely wildly incompatible in context.

Comparison to OPT-Tree. OPT-Tree (Wang et al., 2025) also maximizes expected acceptance length, but under the drafter's own autoregressive distribution, which provides path-conditioned probabilities because the drafter is run sequentially. OPT-Tree's optimization is therefore "closer" to the true target-model objective — it uses conditional probabilities that capture intra-block dependencies — but it pays for this with multiple drafter forward passes per tree depth. DDTree's surrogate is cruder but free (in drafting cost). The fact that DDTree achieves competitive or superior speedups (the paper establishes DDTree as among the leading speculative decoding approaches, building on DFlash which already outperforms EAGLE-3) suggests that the cost of obtaining path-conditioned probabilities can exceed their benefit in the tree-construction tradeoff. This is a significant finding for the speculative decoding literature: it implies that one-pass drafters with cheap-but-crude tree construction can be more efficient than sequential drafters with expensive-but-accurate tree construction, even when the former uses an independence assumption that is manifestly false.

Evidence. Table 1 provides the primary evidence: DDTree improves mean acceptance length τ over vanilla DFlash in every single entry — for example, from 7.79 to 10.73 on MATH-500 with Qwen3-8B at T = 0.0 — demonstrating that the surrogate optimization consistently selects trees that the target model accepts deeper into. The acceptance length distribution in Figure 4 shows the mechanism: DDTree shifts probability mass from short acceptances (lengths 1–4) to long acceptances (lengths 10–16), with full-block acceptance at length 16 becoming "substantially more common." This distributional shift is exactly what the surrogate objective is designed to produce — more probability mass covered by the tree — and its translation into observed target-model behavior validates the surrogate as a useful proxy.

Innovation 3: A Provably Optimal and Efficient Tree-Construction Algorithm for Factorized Distributions

The third contribution is Algorithm 1 itself — the best-first heap construction — and the theoretical scaffolding that proves it returns the optimal tree under the factorized surrogate in O(B log B) time. While this may appear to be "just" an algorithm, the intellectual contribution lies in the recognition that the factorized structure of Q makes the tree-construction problem tractable in a way that is not obvious a priori, and in the specific sibling/child generation rules that enumerate the search space without duplicates or gaps.

The combinatorial challenge is this: given L = 16 positions and a vocabulary of ~10⁵ tokens, the number of possible prefixes is astronomically large (~10⁸⁰). The naive approach of enumerating all prefixes and sorting by probability is absurd. The brute-force approach of evaluating all paths in the top-K-reduced space SK is still O(K^L), which is infeasible for K = 256 and L = 16. Algorithm 1 solves this by exploiting a structural property of the factorized distribution: the probability of a prefix is the product of per-position probabilities, which means that replacing any token with a higher-probability token at the same position strictly increases the prefix probability. This monotonicity property implies that the search can be organized as a best-first traversal over a DAG where edges correspond to "decrement the last rank" (sibling) or "append rank 1" (child), and where every node's predecessors have strictly higher probability. The heap then enumerates prefixes in descending probability order, generating only O(B) prefixes total.

The sibling/child generation rules are the key algorithmic insight. They are minimal — each popped tuple generates at most two new tuples — yet complete — they cover every prefix in SK exactly once. The sibling rule explores alternatives at the current depth in rank order; the child rule extends downward with the best token at the next depth. Together, they form a spanning tree over SK where each node is generated by a unique predecessor. This guarantees no duplicates in the heap (which would inflate the O(B log B) bound) and no gaps (every prefix in SK is reachable from the root via a sequence of sibling and child operations).

Comparison to OPT-Tree's tree construction. OPT-Tree also constructs trees by greedily selecting nodes, but its setting is different: autoregressive drafters provide path-conditioned probabilities, so OPT-Tree can compute the exact acceptance probability for each candidate node under the drafter's own model. OPT-Tree's tree construction involves evaluating candidate expansions (running the drafter) and selecting the most promising ones, which requires per-depth forward passes. DDTree's construction is purely post-hoc — it operates on already-computed probabilities — and the heap algorithm is correspondingly simpler and cheaper. The theoretical contribution is showing that, under the factorized surrogate, the tree-construction problem collapses to a simple best-first enumeration, which is both provably correct and practically efficient enough to add negligible overhead to the speculative decoding pipeline.

Why this matters beyond DDTree. The heap algorithm is not specific to DFlash or even to speculative decoding. Any setting where one has per-position marginal probabilities and wants to select the top-B prefixes under the product distribution can use this algorithm. This includes, for example, beam search over factorized sequence models, prefix selection for masked language model decoding, or any other task that requires enumerating high-probability sequences from a set of position-wise categorical distributions without enumerating the full combinatorial space. The algorithm is simple enough to implement in a few lines of code, provably correct, and efficient in both theory and practice.

Evidence. Proposition 3 formally establishes correctness. The complexity analysis (Remark 2) establishes efficiency: O(B log B) time with O(B) heap size, meaning the construction overhead is negligible for the B values used (16–1024). The empirical validation is implicit in the speedup results: if tree construction were a bottleneck, it would appear as reduced end-to-end speedup relative to the acceptance length gains, but Figure 3 shows that acceptance length and speedup track together until verification cost dominates at high B — tree construction itself is not the limiting factor.

Innovation 4: Empirical Characterization of the Budget–Quality Frontier in One-Pass Draft Tree Construction

The fourth contribution is not a method or a theorem but an empirical finding with practical implications: the existence and shape of a budget–quality frontier for draft tree construction from one-pass block diffusion output, and specifically the observation that the optimal node budget (the one maximizing end-to-end speedup) sits in an intermediate range (256–512 nodes for the tested models) beyond which additional nodes still improve acceptance length but hurt overall speedup due to verification cost.

This finding matters because it provides practical guidance for deployment: a practitioner implementing DDTree does not need to search an unbounded space of tree sizes. The experiments show that the speedup curve as a function of node budget has a clear concave shape (Figure 3) — acceptance length increases monotonically but with diminishing returns, while verification cost grows roughly linearly with B (since the target model forward pass processes all B nodes). The product of these two curves — end-to-end speedup — therefore peaks at an intermediate B. The peak location is hardware- and implementation-dependent (the paper notes that "the optimal budget can shift across hardware platforms and implementations"), but the existence of a peak and its location in the low hundreds of nodes is likely robust.

This is a different type of finding from the typical "our method achieves X speedup." It characterizes the scaling behavior of DDTree as a function of its primary hyperparameter, providing not just a point estimate of performance but a frontier that shows how performance changes as the user trades off exploration (more nodes) against verification cost (longer target-model forward pass). This is information that a practitioner needs: if B = 256 achieves 7.4× speedup and B = 512 achieves 7.5× (as in the MATH-500 case), the marginal gain is small and a deployment might prefer the lower-latency B = 256. If a different hardware platform shifts the crossover point, the shape of the curve tells the practitioner where to look.

Comparison to prior work. Most speculative decoding papers report speedup at a fixed or best hyperparameter setting, without characterizing how performance varies with the key tradeoff parameter. Medusa, for instance, reports results with a fixed tree structure. EAGLE-2 uses dynamic tree construction but does not systematically characterize the budget–quality tradeoff. DDTree's Figure 3 is a clean demonstration of the tradeoff, and the fact that the curve is concave with a clear peak provides actionable guidance for hyperparameter selection. The paper also reports that vanilla DFlash (equivalent to B = 1 path) sits below the DDTree curve across all budgets, showing that the tree approach dominates the single-trajectory approach regardless of budget choice — the practitioner can choose B based on their latency tolerance and still beat vanilla DFlash.

Evidence. Figure 3 is the primary evidence: on MATH-500 with Qwen3-8B at T = 0.0, acceptance length increases from approximately 9.2 at B = 16 to approximately 10.8 at B = 1024, while speedup peaks at approximately 7.5× around B = 256–512 and declines slightly at B = 1024 (to roughly 7.4×, though the exact number is read from the figure and the paper reports 7.52× as the best speedup for this setting in Table 1). Table 1 reports the best B per dataset-model pair, and the values consistently fall in the 128–1024 range (implicitly, since these are the budgets that produce the reported speedups). The acceptance length histogram in Figure 4 provides mechanistic evidence: the shift toward full-block acceptances at length 16 explains why acceptance length increases with B — the tree is covering more of the target model's likely paths, and when the target model's actual continuation is in the tree, the acceptance can go all the way to block completion.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The benchmark suite spans ten datasets across three domains: reasoning (MATH-500, GSM8K, AIME 2024, AIME 2025), code (HumanEval, MBPP, LiveCodeBench, SWE-bench Lite), and general instruction/dialogue (MT-Bench, Alpaca). Per-dataset sample counts are listed in Table 2: AIME 2024 and 2025 each use 30 examples; Alpaca, GSM8K, LiveCodeBench, MATH-500, MBPP, and SWE-bench Lite each use 128; HumanEval uses 164; MT-Bench uses 80. The paper follows the original DFlash benchmark setup for these counts (Appendix B).

• Base model(s). Three target models from the Qwen3 family are evaluated: Qwen3-4B, Qwen3-8B, and Qwen3-Coder-30B-A3B-Instruct, each paired with its corresponding DFlash checkpoint available at the authors' Hugging Face repository. These span a range of scales (4B, 8B, and a 30B mixture-of-experts model) and include both general-purpose and code-specialized variants, testing whether DDTree's gains are robust to model size and domain specialization.

• Metrics. Two primary metrics are reported: speedup relative to autoregressive decoding (wall-clock time of autoregressive generation divided by wall-clock time of speculative decoding, with warmup excluded) and mean acceptance length τ (the average number of tokens accepted by the target model per speculative decoding round, including the bonus token). The paper also reports the acceptance length histogram for a case-study analysis (Figure 4). The scoring for MATH, GSM8K, and other benchmarks that require answer correctness is not part of the decoding evaluation — the paper measures only whether the generated tokens match between speculative and autoregressive decoding (which is guaranteed by the lossless property of speculative decoding), and correctness of the generated content is irrelevant to the speedup metric.

• Baselines. Two baselines are used: vanilla DFlash, which uses the same block diffusion drafter but verifies only a single drafted trajectory per round (the greedy argmax at each position), and autoregressive decoding (the target model generating one token per forward pass, used as the denominator for speedup calculations). All methods use the same DFlash drafter checkpoint — DDTree is a drop-in change to how the drafter's output is post-processed and verified, not a new drafter.

• Generation budget / compute accounting. The primary controlled variable is the node budget B — the number of non-root nodes in the draft tree — swept over {16, 32, 64, 128, 256, 512, 1024}. All methods use block size L = 16 draft positions. For DDTree, B controls how many candidate continuations the target model verifies in one forward pass. For vanilla DFlash, there is no node budget (a single trajectory of up to 16 tokens is drafted and verified). The draft model runs exactly once per round for both methods, so drafting cost is identical. Verification cost differs: DDTree's target-model forward pass processes B + 1 tokens (the root plus B tree nodes) with tree attention, while vanilla DFlash processes up to 17 tokens with standard causal attention. All runs use bfloat16 precision, a maximum of 2048 new tokens per example, and a warmup phase whose time is excluded from measurements. For the autoregressive baseline and vanilla DFlash, the target model is benchmarked with both FlashAttention-2 and standard PyTorch scaled dot product attention, and the faster result is reported (which can only improve these baselines relative to DDTree, since DDTree uses standard attention for the target model due to FlashAttention-2's lack of tree attention support).

• Cross-validation / statistical protocol. None reported. The paper evaluates on fixed test sets with fixed sample counts (Table 2), reports per-dataset speedup and acceptance length for each method at each temperature, and selects the best node budget B per dataset-model pair based on the same test data used for reporting. There is no held-out validation set for hyperparameter selection, no standard error or confidence intervals, and no significance testing. The benchmark runs are sharded across 8 H200 GPUs but no replication or variance estimation is described.

Main Quantitative Results

Overall Speedup Gains (Table 1, Figure 1)

The headline result is that DDTree improves over vanilla DFlash in every single one of the 60 dataset-model-temperature settings reported in Table 1. The magnitude of improvement varies by setting, but the consistency is total — there is no entry where vanilla DFlash outperforms DDTree.

For Qwen3-8B at temperature 0.0 on MATH-500, speedup increases from 5.56× (DFlash) to 7.52× (DDTree), and mean acceptance length τ increases from 7.79 to 10.73 tokens. This is the largest absolute improvement for this model, though percentage improvements are substantial across the board: for example, on HumanEval with Qwen3-4B at T=0.0, speedup goes from 4.81× to 6.81× (a 41.6% relative improvement), and on SWE-bench Lite with Qwen3-Coder-30B-A3B-Instruct at T=0.0, from 2.77× to 4.38× (a 58.1% relative improvement).

The Qwen3-Coder-30B-A3B-Instruct model, despite having the highest absolute speedups on code tasks (peak 8.22× on HumanEval at T=0.0 with DDTree), shows large relative gains from DDTree on tasks where vanilla DFlash is weaker: MT-Bench improves from 2.04× to 3.27× (60.3% relative), and Alpaca from 1.53× to 2.46× (60.8% relative). The gains are proportionally largest on instruction-following and dialogue tasks, where vanilla DFlash's single-trajectory approach leaves the most room for improvement.

At temperature 1.0, the same pattern holds, with DDTree consistently ahead. On MATH-500 with Qwen3-8B at T=1.0, DDTree achieves 6.59× speedup versus 4.56× for DFlash — a 44.5% relative improvement. The absolute speedups are lower at T=1.0 across all settings (since stochastic sampling generally produces less predictable token sequences that are harder to draft accurately), but the relative gain from DDTree remains substantial.

Figure 1 visualizes the temperature 0.0 results across all ten datasets for all three models, with DDTree bars consistently taller than DFlash bars. The figure uses the best node budget for each dataset-model pair, so the bars represent the ceiling of DDTree's performance (not a single fixed budget across all settings).

Acceptance Length Analysis (Table 1, Figures 3 and 4)

The mechanism behind DDTree's speedup improvements is visible in the acceptance length τ column of Table 1. Across all settings, DDTree achieves substantially higher τ than vanilla DFlash. For Qwen3-8B at T=0.0, τ increases by amounts ranging from +1.97 (Alpaca: 3.12 → 5.09) to +3.28 (LiveCodeBench: 7.22 → 10.28). These increases in τ directly translate to fewer target-model forward passes per generated token, which is the primary driver of end-to-end speedup.

Figure 4 provides the per-round acceptance length distribution on MATH-500 with Qwen3-8B at T=0.0, comparing DFlash against DDTree with B=512 (the best speedup budget for this setting). The histogram shows a clear rightward shift: DFlash has its mode around acceptance lengths 8–11, with a substantial fraction of rounds accepting fewer than 4 tokens. DDTree shifts probability mass toward longer acceptances, with the mode moving to lengths 14–16, full-block acceptance (length 16) becoming "substantially more common," and short acceptances (below length 4) becoming "much rarer." This distributional shift is what the surrogate objective is designed to produce — the tree covers more of the target model's likely continuations, so when the target model's actual path aligns with a tree path, acceptance can proceed deeper into the block.

Budget-Quality Tradeoff (Figure 3)

Figure 3 examines MATH-500 with Qwen3-8B at T=0.0 and sweeps the DDTree node budget B from 16 to 1024, plotting both speedup and acceptance length on dual y-axes. The results show:

• Acceptance length increases monotonically with B, from approximately 9.2 tokens at B=16 to approximately 10.8 tokens at B=1024. The curve is concave — gains are steepest at small B (16 → 32 → 64) and diminish at large B.

• Speedup is non-monotonic: it increases from approximately 6.5× at B=16 to a peak of approximately 7.5× around B=256–512, then slightly declines at B=1024 (to roughly 7.4× from visual inspection; Table 1 reports 7.52× as the best speedup for this setting, implying the peak is at B=512 or the best of the discrete sweep points near the peak). The paper states: "Pushing the budget to 1024 increases acceptance length further, but the tradeoff is no longer favorable as the additional overhead of verifying more drafted tokens outweighs the gain from the longer accepted prefix."

• DFlash is below the DDTree curve at all budgets: the DFlash speedup line (a single point at approximately 5.56× with acceptance length 7.79, shown as horizontal reference lines) sits below even the B=16 DDTree point, meaning that even minimal tree construction with DDTree (B=16 nodes) outperforms the single-trajectory approach.

This figure is the key evidence that the node budget B is a genuine tradeoff parameter — not a "more is better" knob — and that the optimal B depends on the balance between acceptance length gains and verification overhead. The paper notes that "the optimal budget can shift across hardware platforms and implementations," making Figure 3 a characterization of the tradeoff shape rather than a universal prescription for B.

Temperature Sensitivity (Table 1)

DDTree's improvements are not confined to greedy decoding. At temperature 1.0, across all 30 dataset-model pairs, DDTree improves speedup over DFlash. The relative gains are similar in magnitude to the T=0.0 case: for example, on MATH-500 with Qwen3-8B, the relative improvement is 44.5% at T=1.0 versus 35.3% at T=0.0. On HumanEval with Qwen3-Coder-30B-A3B-Instruct, the relative improvement is 39.7% at T=1.0 (from 5.64× → 7.88×) versus 35.0% at T=0.0 (from 6.09× → 8.22×). Across all settings, there is no systematic degradation in DDTree's relative advantage at higher temperature, suggesting that the tree construction is robust to the increased entropy of the target model's output distribution under stochastic sampling.

Domain Generality (Table 1, Figure 1)

The benchmark suite spans three distinct domains, and DDTree's gains are consistent across all of them:

• Reasoning tasks (MATH-500, GSM8K, AIME 2024, AIME 2025): Speedup improvements range from +1.67× (AIME 2025, Qwen3-8B, T=0.0: 5.32× → 6.99×) to +1.96× (MATH-500, Qwen3-8B, T=0.0: 5.56× → 7.52×). Acceptance length improvements are similarly large, with MATH-500 showing the largest τ gain (+2.94 tokens: 7.79 → 10.73).

• Code tasks (HumanEval, MBPP, LiveCodeBench, SWE-bench Lite): The Qwen3-Coder-30B-A3B-Instruct model, which is code-specialized, achieves the highest absolute speedups in the benchmark (8.22× on HumanEval at T=0.0). DDTree's improvements on code tasks are consistent, with the largest relative gain on SWE-bench Lite (+58.1% on Qwen3-Coder-30B-A3B-Instruct at T=0.0: 2.77× → 4.38×). The acceptance length gains on code tasks (e.g., +2.70 tokens on HumanEval with Qwen3-Coder-30B-A3B-Instruct: 8.02 → 10.72) are comparable to those on reasoning tasks.

• Instruction-following and dialogue (MT-Bench, Alpaca): These tasks show the largest relative gains, though from lower baselines. On Alpaca with Qwen3-4B at T=0.0, DDTree achieves 3.32× versus DFlash's 2.03× — a 63.5% relative improvement. On MT-Bench with Qwen3-Coder-30B-A3B-Instruct at T=0.0, the improvement is from 2.04× to 3.27× (60.3% relative). The absolute speedups on these tasks remain lower than on reasoning and code (peak 4.18× on MT-Bench with Qwen3-4B), but DDTree substantially closes the gap. The paper does not analyze why instruction-following tasks benefit proportionally more from tree construction, but a plausible explanation is that these tasks have higher entropy in the target model's output distribution (more valid continuations), making single-trajectory drafting less reliable and tree-based exploration more valuable.

Ablation Studies and Robustness Checks

Node budget sweep (Figure 3): The budget sweep on MATH-500 with Qwen3-8B at T=0.0 tests B ∈ {16, 32, 64, 128, 256, 512, 1024}. Acceptance length increases monotonically from ~9.2 to ~10.8 tokens, while speedup peaks at B=256–512 and declines slightly at B=1024. This establishes that the method is not brittle to budget choice within a broad range — speedup at B=256 (near-peak) is substantially higher than at B=16 (the minimum tested), and there is no cliff where performance suddenly degrades. The paper treats B as a hyperparameter selected per dataset-model pair (the values producing the best speedups in Table 1 are not explicitly listed, but Figure 1 and Table 1 report using the "best tree-node budget for each dataset-model pair").

Temperature sweep (Table 1): Every dataset-model pair is evaluated at both T=0.0 and T=1.0, serving as an implicit ablation of whether DDTree's gains depend on greedy decoding. The results show that DDTree improves over DFlash at both temperatures for all settings, with no systematic difference in relative gain. This is not a formal ablation in the sense of varying temperature continuously, but it demonstrates robustness to the sampling strategy.

Model scale sweep (Table 1, Figure 1): Three model scales are tested: 4B, 8B, and 30B-MoE. DDTree improves over DFlash at all three scales. The relative gains do not show a clear trend with model size — the 30B-MoE model sees some of the largest relative gains on certain tasks (e.g., Alpaca at T=0.0: +60.8%) and some of the smallest on others (e.g., HumanEval at T=0.0: +35.0% from an already-high 6.09× baseline). There is no evidence that DDTree's benefit systematically increases or decreases with target model scale.

FlashAttention vs. standard attention for baselines (Appendix B): The autoregressive decoding baseline and vanilla DFlash are benchmarked with both FlashAttention-2 and standard PyTorch scaled dot product attention, with the faster result reported. DDTree uses only standard attention for the target model (since FlashAttention-2 does not support tree attention patterns). This choice makes the baselines strictly stronger — if FlashAttention-2 provides any speedup for the target model, the baseline benefits from it while DDTree does not. The fact that DDTree still outperforms DFlash in every setting, despite this disadvantage, strengthens the results. However, this also means that DDTree's speedup numbers would potentially be higher if FlashAttention-2 (or an equivalent optimized kernel) supported tree attention masks, which is a hardware/implementation limitation rather than a fundamental algorithmic one.

No formal ablation of the surrogate objective vs. alternatives: The paper does not compare DDTree's factorized-surrogate tree construction against alternative tree-construction strategies for block diffusion output. For example, one could construct a tree by taking the top-k tokens at each depth independently and forming the Cartesian product (a "full" tree of size k^L or a truncated version), or by sampling paths from the factorized distribution, or by using the drafter's logits with beam-search-style scoring. No such comparisons are presented. The surrogate objective is justified theoretically (Propositions 1–3) and empirically (it produces speedup gains), but the paper does not demonstrate that it is better than simpler or more complex alternatives for the same one-pass setting.

No formal ablation of K = min(B, |V|): Lemma 1 establishes that restricting to the top-K tokens at each depth does not sacrifice optimality, and Algorithm 1 uses K = min(B, |V|). The paper does not empirically test whether smaller values of K (e.g., K = B/2 or K = √B) would produce similar speedups with lower tree-construction overhead. Since B is at most 1024 and |V| ~ 10^5, K is always at most 1024, meaning the search space reduction from |V| to K is already enormous. The remaining question — whether K could be further reduced without hurting acceptance length — is not addressed.

No comparison to autoregressive draft trees (EAGLE, OPT-Tree): The paper states that DFlash "has been shown to outperform strong autoregressive drafters such as EAGLE-3" and that DDTree builds on DFlash, implying that DDTree would also outperform autoregressive drafters. However, no direct head-to-head comparison between DDTree and EAGLE-3 or OPT-Tree is provided. The DFlash paper's comparisons to EAGLE-3 are cited but not reproduced or extended with DDTree numbers in the same experimental setup. This is a reasonable omission given that the paper's contribution is improving DFlash, not benchmarking against all speculative decoding methods, but it means the claim that DDTree is "among the leading approaches to speculative decoding" is inferred from DFlash's prior results rather than directly demonstrated.

No ablation of the block size L: All experiments use L = 16. The paper does not test whether DDTree's gains are sensitive to block size — for example, whether smaller blocks (L = 8) would see proportionally larger or smaller benefits from tree construction, or whether larger blocks (L = 32) would allow even greater acceptance length gains with appropriate B. Block size is a fixed hyperparameter inherited from DFlash.

Drafter forward pass counted once per round: The paper's accounting implicitly assumes that the block diffusion drafter runs exactly once per speculative decoding round for both DFlash and DDTree. This is correct by construction — both methods use the same DFlash drafter with the same forward pass. However, there is no explicit verification that the drafter's per-round cost is identical in wall-clock time between methods, or that the tree construction step (Algorithm 1, O(B log B)) adds negligible overhead relative to the target model forward pass. The speedup numbers would capture any meaningful overhead from tree construction, and the fact that speedups are substantially positive indicates that overhead is small, but no explicit profiling or cost breakdown is reported.

Critical Assessment

Claim: "DDTree consistently improves over vanilla DFlash across models." This claim is strongly supported. Table 1 shows improvements in all 60 dataset-model-temperature settings, spanning three model scales and ten datasets. The consistency is the strongest evidence — there are no negative results, no datasets where DDTree underperforms DFlash, and no model sizes where the benefit disappears. The magnitude of improvement varies (from +0.54× on AIME 2025 with Qwen3-8B at T=1.0 to +1.61× on SWE-bench Lite with Qwen3-Coder-30B-A3B-Instruct at T=0.0), but the direction is uniform.

However, "across models" means across three models from a single family (Qwen3). The paper does not test on models from other families (e.g., Llama, Gemma, DeepSeek) or with different tokenizers, architectures, or pretraining distributions. DFlash provides checkpoints only for Qwen3 models, which limits the evaluation. It is plausible but unverified that DDTree's gains would transfer to other model families — the method is architecturally generic (any block diffusion drafter that produces per-position marginals can be used), but the quality of the factorized surrogate depends on how well the drafter's per-position distributions approximate the target model's autoregressive distribution, which could vary across model families.

Claim: "DDTree constructs a draft tree directly from per-position distributions produced by a single block diffusion drafter forward pass." This is a description of the method, not an empirical claim — it is supported by the method description in Section 4 and Algorithm 1. The experiments validate that the resulting trees produce speedup gains, but they do not provide diagnostics on the trees themselves (e.g., what fraction of tree nodes are on paths the target model ever follows, how tree structure varies by domain, or whether the independence assumption leads to visibly incoherent drafted paths that the target model rejects). Figure 4 provides indirect evidence — the shift toward longer acceptance lengths — but a more detailed analysis of tree quality (e.g., per-depth acceptance rates, comparison of accepted vs. rejected tree branches) would strengthen the connection between the surrogate optimization and the observed speedup.

Claim: "DDTree provably maximizes the expected acceptance length under the draft model's factorized distribution." This is a theoretical claim (Proposition 3) supported by proofs in Appendix A. It is not directly tested empirically, since the "expected acceptance length under Q" is a surrogate that cannot be directly measured without knowing the true Q or exhaustively evaluating all prefixes. The empirical results show that optimizing this surrogate produces trees that the target model accepts more tokens from, but they do not verify that the returned tree is literally optimal for the surrogate — that would require comparing against an exhaustive enumeration of the top-B prefixes, which is infeasible. The theoretical guarantee is sound (given the assumptions), and the practical question is whether the surrogate is a good proxy for the true objective, which the experiments address indirectly.

Claim: Speedup gains from DDTree are substantial (e.g., 5.56× → 7.52× on MATH-500 with Qwen3-8B). This claim is numerically supported by Table 1. However, several caveats about the speedup measurement deserve scrutiny:

1. Hardware and implementation dependence. The paper notes that "the optimal budget can shift across hardware platforms and implementations" (Section 5.3), and the fact that FlashAttention-2 is used for baselines but not for DDTree's target model means DDTree's target-model forward pass is potentially slower than it could be with optimized attention. This makes DDTree's speedup numbers conservative relative to what could be achieved with tree-attention-optimized kernels, but also means the absolute speedup values are not directly portable to other serving frameworks.

2. Best budget selection on test data. The paper reports the best node budget per dataset-model pair, selected from {16, 32, 64, 128, 256, 512, 1024}. The selection criterion is the budget that achieves the highest speedup on the same test data used for reporting. With 7 budget options and no held-out validation set, there is a risk of overfitting the budget choice to the test data, though the risk is limited because (a) the budget is the only hyperparameter being selected, (b) the speedup curve is smooth and concave (Figure 3), so small changes in budget produce small changes in speedup, and (c) the gains are consistent across all datasets, so even if a suboptimal budget were chosen for some dataset, the qualitative conclusion would not change.

3. No confidence intervals or variance estimates. The paper does not report standard deviations, standard errors, or confidence intervals for any speedup or acceptance length measurement. With sample sizes ranging from 30 (AIME) to 164 (HumanEval), speedup estimates have non-trivial variance. For datasets with only 30 examples (AIME 2024, AIME 2025), a few outlier examples where DDTree performs unusually well or poorly could substantially affect the mean speedup. The lack of uncertainty quantification makes it difficult to assess whether the reported differences (e.g., 7.52× vs. 7.27× between MATH-500 and AIME 2024 on Qwen3-8B) are statistically meaningful or within noise.

4. Wall-clock measurement methodology. The paper states that a warmup phase is run and its time excluded, and that the benchmark runs are sharded across 8 H200 GPUs. However, details of the timing protocol are sparse: it is not specified how many warmup iterations are run, whether the autoregressive baseline and speculative decoding methods are measured under identical GPU memory conditions, whether batch size is 1 (typical for latency benchmarking of speculative decoding), or whether GPU clock speed or power throttling is controlled. These are standard concerns for wall-clock latency benchmarking and do not necessarily invalidate the results, but they limit the reproducibility of the exact speedup values.

Genuine weaknesses in the experimental design:

• Single model family. All experiments use Qwen3 models. This is understandable — DFlash provides checkpoints for Qwen3 — but it means the generality of DDTree across model architectures, tokenizers, and pretraining distributions is untested. A block diffusion drafter's per-position distributions might be better or worse approximations of the target model's autoregressive distribution depending on the model family, which would affect DDTree's effectiveness.

• No comparison to non-DFlash baselines. The paper compares DDTree only to vanilla DFlash and autoregressive decoding. It does not compare to Medusa, EAGLE-3, or other speculative decoding methods under the same hardware and benchmark conditions. The claim that DFlash "outperforms strong autoregressive drafters such as EAGLE-3" is cited from prior work, not reproduced, and DDTree's position relative to EAGLE-3 with optimal tree construction is unknown. A head-to-head comparison on the same hardware would substantially strengthen the paper's positioning.

• No analysis of tree quality or failure modes. The paper reports acceptance lengths and speedups but provides no qualitative or quantitative analysis of what the constructed trees look like — how deep they are on average, how many branches they have at each depth, whether the target model's rejections correlate with the independence assumption producing incoherent multi-token sequences, or whether certain types of continuations (e.g., code vs. natural language vs. mathematical notation) benefit differentially from tree construction. Figure 4 is the only distribution-level analysis, and it aggregates across all decoding rounds without distinguishing by position or domain.

• Limited analysis of the budget-selection problem. Figure 3 characterizes the tradeoff for one dataset-model pair. No equivalent analysis is provided for other datasets, models, or temperatures, so it is unknown whether the optimal budget systematically varies by domain (e.g., code tasks might benefit from larger B because they have more predictable structure) or by model scale (e.g., larger target models might have more predictable output, making smaller B sufficient). The paper selects B per dataset-model pair but does not provide the selected values or analyze patterns in the selection.

• No analysis of KV cache memory. Tree attention with B nodes requires storing key-value pairs for all B nodes during the verification forward pass. For large B (1024) and large target models, this could substantially increase peak GPU memory usage compared to vanilla DFlash (which stores key-value pairs for at most 17 tokens). The paper does not report memory usage, which is a practical concern for deployment — a method that achieves higher speedup but requires more GPU memory may not be deployable under the same hardware constraints.

Experiments that would have strengthened the paper:

1. A comparison of DDTree against DART (the concurrent work that also constructs trees from one-pass logits but uses external N-gram models). This would directly test whether DDTree's surrogate-based approach is competitive with or superior to continuity-aware pruning.

2. A formal ablation comparing DDTree's factorized-surrogate tree against alternative tree-construction strategies for the same one-pass drafter output: random sampling of paths, top-k Cartesian product with truncation, or beam-search-style scoring using the drafter's logits. This would isolate the contribution of the surrogate objective from the contribution of simply exploring more paths.

3. An experiment varying block size L to see whether DDTree's benefit scales with block size — if larger L allows even greater acceptance lengths when paired with appropriate B, this would suggest a path to further speedups.

4. An analysis of how the optimal B varies with dataset and model characteristics, providing practitioners with heuristics for budget selection without requiring a full sweep.

5. Measurement of GPU memory usage and tree-construction wall-clock time, to provide a complete picture of deployment costs beyond speedup.

6. Limitations and Trade-offs

Single Model Family, Single Drafter Architecture

The assumption or constraint. All experiments are conducted exclusively with Qwen3 models (Qwen3-4B, Qwen3-8B, Qwen3-Coder-30B-A3B-Instruct) paired with the DFlash block diffusion drafter. The paper provides no results for other model families (e.g., Llama, Gemma, DeepSeek), other tokenizers, or other block diffusion drafter architectures. DFlash itself is a specific design — a small block diffusion model conditioned on target-model features — and DDTree inherits any architectural assumptions baked into DFlash.

The paper does not explicitly claim generality beyond Qwen3. It states that DFlash has been "shown to outperform strong autoregressive drafters such as EAGLE-3" and that DDTree builds on DFlash, implying DDTree would also be competitive with autoregressive approaches. However, the crucial question is whether the factorized surrogate Q — the product of per-position marginals that DDTree optimizes — is a useful approximation of the target model's autoregressive distribution across diverse model families. This depends on how well the drafter's per-position marginals correlate with the target model's path-conditioned probabilities, which could vary substantially across architectures.

The consequence. A practitioner using a non-Qwen3 target model, or a future block diffusion drafter with different modeling choices, has no empirical evidence that DDTree will provide speedup gains. The method is architecturally generic — any block diffusion drafter that outputs per-position logits can be plugged into Algorithm 1 — but the quality of the surrogate determines whether the resulting tree actually improves acceptance length over a single trajectory. If a drafter's per-position marginals are poorly calibrated (e.g., placing high probability on tokens that never co-occur in natural continuations), the top-B prefixes under Q may be dominated by incoherent paths that the target model consistently rejects. In that regime, DDTree could waste the node budget on useless branches and provide no benefit — or even reduce speedup — compared to vanilla DFlash. The paper provides no diagnostic for assessing whether a given drafter's output is "good enough" for DDTree to work.

What evidence exists in the paper. None beyond the Qwen3 results. Table 1 shows consistent improvements across three Qwen3 variants, but these share a common pretraining recipe, tokenizer, and architecture. The acceptance length distributions (Figure 4) and budget tradeoff (Figure 3) are shown only for Qwen3-8B on MATH-500. There is no cross-family evaluation, no analysis of whether the factorized surrogate's accuracy correlates with downstream speedup, and no diagnostic for predicting when DDTree will or will not help.

Mitigation status. Not addressed. The paper does not discuss model-family generality as a limitation, does not analyze properties of the factorized surrogate across models, and does not propose methods for assessing surrogate quality before deployment. A practitioner considering DDTree for a non-Qwen3 model must run their own benchmark suite to determine effectiveness.

Difficulty Estimation Cost Is Not Accounted For — Budget Selection Requires Per-Task Sweeping

The assumption or constraint. DDTree's primary hyperparameter is the node budget B, which controls the number of tree nodes the target model verifies per round. The paper's headline speedup numbers (Table 1, Figure 1) use the best B per dataset-model pair, selected from the discrete set {16, 32, 64, 128, 256, 512, 1024}. The speedup curve in Figure 3 (MATH-500, Qwen3-8B, T=0.0) shows that speedup is non-monotonic in B, peaking at an intermediate value (256–512) before declining at 1024 as verification overhead overtakes acceptance length gains. The paper explicitly acknowledges: "the optimal budget can shift across hardware platforms and implementations."

However, the paper does not account for the cost of finding this optimal budget. In the experimental protocol, B is selected by evaluating all seven candidate budgets on the test data and reporting the best result. No held-out validation set is used for budget selection, no cross-validation is performed, and no heuristics or lightweight proxies for the optimal B are proposed. The reported speedup numbers are therefore post-hoc optimal — they represent the ceiling of DDTree's performance given oracle knowledge of which B works best, not the performance a practitioner would achieve when deploying on a new dataset without prior tuning.

The consequence. In a realistic deployment scenario, a practitioner deploying DDTree on a new task faces an unknown budget-quality curve. Finding the optimal B requires running the full benchmark (or a representative sample) at multiple budgets — which multiplies the evaluation cost by the number of budget candidates. For the 7 budgets swept in the paper, this means up to 7× the benchmarking effort to determine the best setting. If the task distribution shifts over time (e.g., user queries change), the optimal B may also shift, requiring periodic re-tuning. The paper provides no guidance on how to select B without exhaustive sweeping — no correlation with dataset properties (e.g., average sequence length, domain, entropy of target model outputs), no lightweight proxy, and no default value that is robust across settings.

Moreover, the speedup gap between a "good enough" budget and the optimal one matters. Figure 3 shows that B=128 achieves roughly 7.2× speedup versus 7.5× at B=512 on MATH-500 — a difference of ~4%. But on other tasks where the curve is steeper, using a suboptimal budget could leave more performance on the table. Without knowing the shape of the curve for a new task, a practitioner cannot assess the cost of choosing a fixed budget (e.g., B=256 for all tasks) versus task-specific tuning.

What evidence exists in the paper. Figure 3 characterizes the budget-quality tradeoff for exactly one setting (MATH-500, Qwen3-8B, T=0.0). No equivalent analysis is provided for other datasets, models, or temperatures. The paper does not report the best-B values used for each entry in Table 1, nor does it report the sensitivity of speedup to suboptimal budget choices (e.g., how much speedup degrades if a fixed B=256 is used across all datasets). The fact that speedup peaks at an intermediate B is demonstrated, but the generality of the peak location and the cost of finding it are not explored.

Mitigation status. The paper acknowledges that "the optimal budget can shift across hardware platforms and implementations" (Section 5.3) but treats this as a property of the tradeoff rather than a limitation of the evaluation methodology. No automated budget selection method, heuristic, or default value is proposed. The budget is a hyperparameter that must be tuned per deployment, and the paper provides only a case study (Figure 3) illustrating the shape of the tradeoff for one setting, not a general solution to the selection problem.

No Empirical Comparison to Autoregressive Draft-Tree Methods or DART

The assumption or constraint. The paper claims that DDTree is "among the leading approaches to speculative decoding" based on DFlash's prior demonstration of outperforming EAGLE-3. However, the paper provides no direct head-to-head comparison between DDTree and autoregressive draft-tree methods (EAGLE-2, EAGLE-3, OPT-Tree) or the concurrent DART method (which also constructs trees from one-pass logits but uses an external N-gram continuity score and trie for pruning). The only baselines evaluated are vanilla DFlash and autoregressive decoding.

This is a significant gap because the central claim of the paper — that constructing trees from block diffusion per-position marginals is an effective strategy — is evaluated only against the absence of tree construction (vanilla DFlash), not against the best alternative ways to build trees in speculative decoding. The autoregressive draft-tree methods (EAGLE family + OPT-Tree) represent the primary competing approach: they use sequential drafting with path-conditioned probabilities, which provides richer information for tree construction at the cost of per-depth drafting overhead. DDTree's theoretical advantage is that it avoids this per-depth cost, but whether this translates to better end-to-end speedup depends on the tradeoff between drafting overhead and tree quality — a tradeoff that cannot be assessed without direct comparison.

Similarly, DART represents the closest concurrent work: it constructs trees from one-pass block diffusion output, but uses a different tree-construction criterion (continuity-aware pruning with an external N-gram model). A comparison between DDTree and DART would directly test whether the factorized-surrogate approach is competitive with or superior to continuity-based pruning for the same one-pass setting.

The consequence. A practitioner deciding which speculative decoding method to implement has no empirical basis for choosing DDTree over EAGLE-3 or DART. The paper's positioning — that DFlash already beats EAGLE-3, and DDTree improves on DFlash — relies on a transitive inference (DDTree > DFlash > EAGLE-3) that may not hold in practice. DFlash's comparison to EAGLE-3 was conducted under specific hardware and benchmark conditions that may differ from DDTree's evaluation setup. Moreover, EAGLE-3 with OPT-Tree-style tree construction might gain proportionally more from tree-based verification than DFlash gains from DDTree, since autoregressive drafters provide richer conditional probability information for tree construction — the per-depth drafting cost might be offset by substantially better tree quality. Without direct measurement, this tradeoff is unknown.

What evidence exists in the paper. The paper cites DFlash's prior results ("DFlash has been shown to outperform strong autoregressive drafters such as EAGLE-3") but does not reproduce or extend those comparisons. Section 2 (Related Work) mentions DART as concurrent work and distinguishes DDTree's approach, but no experimental comparison is provided. Table 1 and Figure 1 compare DDTree only to DFlash and autoregressive decoding.

Mitigation status. Not addressed. The paper's experimental design is explicitly focused on improving DFlash, not on benchmarking against the broader speculative decoding landscape. The abstract states that DDTree "place[s] DDTree among the leading approaches to speculative decoding," but this claim is supported only by the improvement over DFlash and the transitive inference from DFlash's prior comparisons. No experiments directly test DDTree against EAGLE-3, OPT-Tree, or DART under the same hardware and benchmark conditions.

Memory Overhead of Tree Attention Is Not Measured or Discussed

The assumption or constraint. DDTree's verification step runs the target model on a flattened sequence of B + 1 tokens (the root plus B tree nodes) using tree attention. For each of these B + 1 positions, the target model must compute and store key-value (KV) pairs during the forward pass. In standard speculative decoding (vanilla DFlash with a single trajectory), the target model processes at most L + 1 = 17 tokens per round. With DDTree and B = 512 (a typical optimal budget), the target model processes 513 tokens — a ~30× increase in the number of KV pairs computed and stored during the verification forward pass.

This has direct implications for peak GPU memory usage. The target model's KV cache during verification must accommodate key-value pairs for all B tree nodes simultaneously, plus the shared prefix (the past context). For large target models (30B parameters or more), the per-token KV cache size is substantial — at bfloat16 precision with typical hidden dimensions and number of layers, the KV cache for a single token can be tens to hundreds of kilobytes. For B = 512, this multiplies to tens or hundreds of megabytes of additional memory during the verification forward pass, on top of the existing cache for the past context.

The consequence. DDTree may not be deployable under the same GPU memory constraints as vanilla DFlash. If the target model already operates near GPU memory capacity (e.g., serving a 30B model on a single GPU with long contexts), the additional KV cache memory from tree verification could cause out-of-memory errors or force the use of smaller batch sizes, lower precision, or shorter context lengths. The paper's speedup numbers — measured wall-clock time per generated token — do not capture this memory constraint. A method that achieves 7.5× speedup but requires 2× the GPU memory may be infeasible for deployments where memory, not compute, is the binding constraint.

Furthermore, the memory overhead scales with B, creating a direct tension with the budget-quality tradeoff highlighted in Figure 3. Larger B improves acceptance length and (up to a point) speedup, but also increases peak memory usage. The "optimal" B for speedup may not be the optimal B for memory-constrained deployment. A practitioner with limited GPU memory might need to choose a smaller B than the speedup-optimal value, reducing the realized speedup below the paper's reported numbers.

What evidence exists in the paper. None. The paper does not report GPU memory usage for any method, model, or budget setting. The word "memory" does not appear in the paper outside of references to "GPU" in Appendix B. There is no analysis of KV cache size scaling with B, no comparison of peak memory between DFlash and DDTree, and no discussion of whether memory constraints limit the deployable budget.

Mitigation status. Not addressed. The paper's focus is exclusively on latency (speedup) as the metric of interest, with no consideration of memory as a deployment constraint. This is a common omission in speculative decoding papers, but it is particularly relevant for DDTree because the tree attention mechanism fundamentally trades memory (storing more KV pairs) for latency (verifying more paths in one forward pass). The paper provides no guidance on how to select B under a memory budget, and no analysis of whether the memory overhead is small enough to be negligible in practice or large enough to be a binding constraint.

Verification Uses Unoptimized Attention — Speedup Numbers Conservative but Non-Portable

The assumption or constraint. DDTree's target-model verification uses standard PyTorch scaled dot product attention, because FlashAttention-2 does not support the tree attention pattern required for verifying multiple tree branches in one forward pass. By contrast, the autoregressive baseline and vanilla DFlash are benchmarked with both FlashAttention-2 and standard attention, and the faster result is reported for each. This is explicitly documented in Appendix B: "For fairness, for the autoregressive baseline and vanilla DFlash, we evaluate the target model with both standard PyTorch scaled dot product attention and FlashAttention-2, and report the faster result, which can only improve these baselines relative to DDTree."

This makes DDTree's speedup numbers conservative — if tree-attention-optimized kernels (analogous to FlashAttention for causal masks) were available, DDTree's target-model forward pass would be faster, and its speedup relative to the baseline would increase. However, it also means that the absolute speedup values reported for DDTree are tied to the performance characteristics of unoptimized PyTorch attention, which can vary substantially across hardware platforms, PyTorch versions, and CUDA configurations.

The consequence. A practitioner deploying DDTree in a production serving framework with custom attention kernels (e.g., TensorRT-LLM, vLLM, or a proprietary inference engine with tree-attention support) may observe different speedup values than those reported in Table 1. The speedup numbers are not portable across serving frameworks with different attention implementations. More importantly, the relative speedup of DDTree over DFlash — which is the paper's central result — depends on how much faster FlashAttention-2 is over standard attention on the practitioner's hardware. On hardware where FlashAttention-2 provides a large speedup for causal attention, the baseline (DFlash with FlashAttention-2) benefits disproportionately, and DDTree's advantage may be smaller than reported. Conversely, on hardware where the gap between FlashAttention-2 and standard attention is small, DDTree's advantage may be larger.

Additionally, the paper's tree attention implementation — a block-diagonal ancestor-only mask in standard PyTorch SDPA — may not be the most efficient way to implement tree verification. Production systems often use custom CUDA kernels that fuse attention computation with mask application, and the performance characteristics of tree attention under such kernels are unknown and not characterized in the paper.

What evidence exists in the paper. Appendix B explicitly documents the attention implementation discrepancy and the "faster result" reporting for baselines. The paper is transparent about this choice and frames it as conservative for DDTree. However, the paper provides no quantification of how much the FlashAttention-2 advantage affects the baseline — i.e., what fraction of DFlash's speedup comes from FlashAttention-2 versus standard attention. There is no ablation of DDTree with and without FlashAttention-2 (impossible since FlashAttention-2 does not support tree attention), and no profiling of the attention computation as a fraction of total inference time.

Mitigation status. Partially addressed through transparency. The paper explicitly documents the asymmetric attention implementation and notes that it "can only improve these baselines relative to DDTree." However, the paper does not (a) quantify the magnitude of this effect, (b) provide profiling data on attention vs. non-attention computation time, (c) discuss how the results might change with optimized tree-attention kernels, or (d) recommend attention implementations for practitioners seeking to reproduce the results. The limitation is acknowledged but not analyzed, leaving practitioners to estimate the portability of the speedup numbers on their own.

No Analysis of When the Factorized Surrogate Fails — Independence Assumption Has No Diagnostic

The assumption or constraint. DDTree constructs draft trees by maximizing expected acceptance length under the factorized surrogate distribution Q(y_{1:L} | c, b) = ∏_{i=1}^L q_i(y_i | c, b), which treats token choices at different positions within the drafted block as independent given the prefix context. This independence assumption is manifestly false for natural language — the grammatically and semantically of a continuation depends on joint compatibility between tokens, not just their individual marginal probabilities. A prefix consisting of high-probability tokens at each position may be incoherent and therefore unlikely to be produced by the target model.

The paper explicitly acknowledges this as a surrogate: "The task here is not to recover the unavailable target model path-conditioned probabilities, but to make the best use of the information available from one block diffusion drafter forward pass" (Remark 1). The empirical results show that, across all tested settings, optimizing under this surrogate produces trees that improve acceptance length. However, the paper provides no analysis of when the surrogate is a good approximation and when it breaks down — no diagnostic for detecting cases where the independence assumption leads DDTree to waste budget on incoherent paths that the target model rejects.

The consequence. DDTree's performance depends on an implicit property of the drafter: that the per-position marginals are sufficiently "compatible" that the top-B product-probability prefixes are also reasonable continuations under the target model's autoregressive distribution. This property may hold for DFlash on Qwen3 but could fail for other drafter-target pairs, for certain types of content, or for certain positions within a block. Without a diagnostic, a practitioner has no way to predict whether DDTree will help for their specific deployment or to detect degradation if the query distribution shifts.

Specific failure modes that are plausible but unmeasured:

• Early positions dominate the product: since prefix probability is the product of per-position probabilities, and each factor is <1, deep prefixes have exponentially small probability under Q. The top-B prefixes under Q may be concentrated entirely at shallow depths, meaning the tree is wide but shallow. If the target model frequently produces long continuations, a wide-shallow tree may provide little benefit because acceptance stops early when the target model's path diverges from all shallow branches.
• Incoherent multi-token sequences: the independence assumption can assign high Q-probability to a path where the token at position 3 is individually probable but incompatible with the token at position 2 — e.g., a verb after a determiner. The target model would never produce such a sequence, so nodes along that path are wasted budget. The paper provides no measurement of what fraction of tree nodes are on paths the target model ever follows.
• Domain-dependent surrogate quality: code (with rigid syntax) might have more predictable joint distributions than natural dialogue, making the independence assumption less harmful. Conversely, creative writing might have looser constraints, making the surrogate more reliable. The paper reports results across domains but does not analyze whether the surrogate's accuracy varies by domain.

What evidence exists in the paper. The acceptance length histogram (Figure 4) and Table 1 provide indirect evidence that the surrogate is useful on average, but they do not characterize the surrogate's failures. There is no per-depth breakdown of acceptance rates (do shallow nodes get accepted more often than deep nodes?), no analysis of whether accepted paths tend to be the ones with the highest Q-probability, and no comparison of DDTree's tree against an oracle tree constructed from target-model probabilities. The paper provides no examples of drafted trees, no qualitative analysis of rejected paths, and no measurement of the correlation between Q-probability and target-model acceptance probability at the node level.

Mitigation status. Not addressed. The paper justifies the surrogate pragmatically (it is the best information available from one forward pass) and demonstrates empirical effectiveness, but does not analyze the surrogate's limitations, provide diagnostics for surrogate quality, or discuss deployment scenarios where the independence assumption might be particularly harmful. The surrogate is treated as a design choice that the experiments validate in aggregate, without inquiry into when or why it might fail.

7. Implications and Future Directions

How This Work Changes the Landscape

DDTree introduces a conceptual reframing rather than a paradigm shift: it shows that the per-position marginal distributions from a one-pass block diffusion drafter — which prior work treated as intermediate computation to be collapsed into a single trajectory — are themselves a rich resource for tree construction, provided one optimizes under an appropriate surrogate objective. This is not a new drafter architecture, a new verification mechanism, or a new training procedure. It is a change in how existing information is used, and it achieves consistent, substantial speedup gains (e.g., 5.56× → 7.52× on MATH-500 with Qwen3-8B, Table 1) without any additional model forward passes. The significance lies in the implication: the one-pass nature of block diffusion is not just a latency advantage over autoregressive drafters, but an architectural property that enables qualitatively different tree-construction strategies — strategies that would be infeasible with autoregressive drafters because of their per-depth cost.

This reframing shifts the research conversation around speculative decoding drafters in a specific way. Before DDTree, the dominant framing was: how do we improve the drafter so that its single best-guess trajectory matches the target model more often? This framing leads naturally to work on better drafter architectures (EAGLE-3's fused features, DFlash's target-model conditioning), better training procedures, and larger drafter models. After DDTree, an equally valid framing becomes: given a drafter that produces per-position distributions in one pass, how do we optimally select a set of continuations to verify? This shifts attention from drafter quality alone to the drafting–verification interface — the algorithmic layer that sits between the drafter's output and the target model's forward pass. This layer was previously invisible because vanilla DFlash (and most speculative decoding methods) collapsed the drafter's output directly into a single trajectory with no intermediate optimization. DDTree shows that inserting a principled tree-construction step at this interface — even one based on a manifestly imperfect independence assumption — yields nontrivial gains.

The paper also resolves a latent tension in the speculative decoding literature: the tradeoff between drafting cost and information richness. Autoregressive drafters (EAGLE series, OPT-Tree) provide path-conditioned probabilities — each node's children are scored conditional on that specific prefix — which is rich information for tree construction but requires one forward pass per tree depth. Block diffusion drafters provide only marginal (non-path-conditioned) probabilities, which is impoverished information but comes at zero marginal cost for additional depth or breadth. The field previously lacked a framework for deciding which side of this tradeoff was preferable. DDTree demonstrates that, at least for the DFlash drafter on Qwen3 models, the impoverished-but-cheap side of the tradeoff can win: optimizing under a factorized surrogate, despite discarding intra-block dependencies, produces trees that the target model accepts substantially more tokens from, while avoiding the per-depth drafting cost that limits how large autoregressive draft trees can grow before overhead erases the gain.

The paper makes certain research directions more attractive:

• Tree-construction as a first-class optimization problem in speculative decoding. DDTree formalizes tree construction under a node budget as maximizing expected acceptance length under a surrogate distribution. This framework — budgeted optimization over a combinatorial space of prefixes — can be extended to other drafter types, other surrogate objectives, and other budget constraints (e.g., memory budgets, latency budgets). It opens a design space where the tree-construction algorithm is a tunable component independent of the drafter architecture.

• One-pass drafters as tree-construction engines. DFlash was designed for single-trajectory drafting and happened to produce per-position marginals as a byproduct of its block diffusion architecture. DDTree suggests that future one-pass drafters could be designed explicitly as tree-construction engines, with training objectives that optimize the quality of the per-position distributions for downstream tree construction (e.g., by calibrating the marginals to the target model's path-conditioned probabilities, or by penalizing incoherent multi-token sequences under the factorized surrogate).

• Verifier efficiency as the binding constraint. Figure 3 shows that the optimal node budget B is the point where acceptance length gains from more nodes are overtaken by verification overhead. This makes verifier efficiency — how cheaply the target model can score many candidate tokens — the bottleneck for further scaling tree-based speculative decoding. Improvements to tree attention implementations (custom kernels, memory-efficient attention patterns, sparsity-aware verification) would directly increase the optimal B and thus the achievable speedup.

Conversely, DDTree makes certain directions less attractive:

• Autoregressive draft trees with large depth. The per-depth drafting cost of autoregressive drafters grows linearly with tree depth. DDTree shows that even crude trees built from one-pass marginals — with no path-conditioned probabilities — can achieve competitive speedups. This raises the bar for autoregressive draft trees: to justify the per-depth cost, the tree quality improvement from path-conditioned probabilities must be substantial enough to overcome the drafting overhead. For shallow trees (depth 2–3), autoregressive drafting may still be preferable; for deeper trees, the one-pass advantage likely dominates.

• Complex tree-construction criteria that require additional model calls. DART's continuity-aware pruning with an external N-gram model represents one approach to improving tree quality beyond the factorized surrogate. DDTree shows that even the simplest possible surrogate — product of marginals — provides meaningful gains. This suggests that the marginal benefit of more sophisticated tree-construction criteria (especially those requiring additional data structures or model calls) may be limited, and that the primary bottleneck is verification efficiency, not tree-construction sophistication.

Follow-Up Research This Work Enables

Direct head-to-head comparison of DDTree against autoregressive draft-tree methods (EAGLE-3 + OPT-Tree) on identical hardware. The paper's central claim — that DDTree is "among the leading approaches to speculative decoding" — rests on a transitive inference (DDTree > DFlash, and DFlash > EAGLE-3 from prior work). This inference is fragile: DFlash's comparison to EAGLE-3 was conducted under different hardware and benchmark conditions, and EAGLE-3 with OPT-Tree-style tree construction might gain proportionally more from tree-based verification than DFlash gains from DDTree, since autoregressive drafters provide richer conditional probability information. A definitive experiment would benchmark DDTree, DFlash, EAGLE-3, and EAGLE-3 + OPT-Tree on the same GPUs, same target models (at minimum, Qwen3-8B and one non-Qwen3 model for generality), same datasets, and same maximum drafting budget (matching either total FLOPs or wall-clock time for drafting + verification). The null result — DDTree does not outperform EAGLE-3 + OPT-Tree — would reveal that path-conditioned probabilities are worth their drafting cost, refining the tradeoff picture. The positive result would establish DDTree as the unambiguous state of the art and validate the impoverished-but-cheap strategy.

Characterizing when the factorized surrogate breaks down: a per-domain, per-depth diagnostic study. DDTree's effectiveness depends on an unmeasured property: that the top-B prefixes under the product-of-marginals distribution Q are also reasonable continuations under the target model's autoregressive distribution p. This property almost certainly varies by domain (code syntax is more rigid than dialogue), by depth (the independence assumption compounds errors multiplicatively), and by drafter-target model pair. A diagnostic study would, for a given target model and dataset, compare the top-B DDTree prefixes against an "oracle" tree constructed by exhaustively scoring all top-k paths under the target model itself (feasible for small k and short blocks on a subset of examples). Key measurements: (a) What fraction of DDTree-selected nodes lie on paths the target model actually follows? (b) Does this fraction degrade with depth? (c) Does it vary systematically across domains (code vs. math vs. dialogue)? (d) What is the correlation between a node's Q-probability rank and its target-model acceptance probability? A finding that the surrogate is accurate for shallow depths but degrades sharply after depth 4–5 would motivate hybrid approaches: use DDTree for the first few positions, then fall back to single-trajectory or beam-search for deeper positions.

Training drafters explicitly for tree-construction quality under the factorized surrogate. DFlash is trained to produce per-position marginals that minimize next-token prediction loss, without any objective related to tree construction. This means the marginals are optimized for single-trajectory accuracy (the argmax at each position), not for the quality of the resulting top-B prefix set. A natural extension is to add an auxiliary training objective that penalizes incoherent multi-token sequences under the factorized distribution — for instance, by sampling pairs of adjacent tokens from the marginals and scoring them with an external language model, or by using a contrastive loss that pushes the drafter's per-position distributions toward the target model's path-conditioned distributions on training data where those are available. The experiment: fine-tune DFlash with such an auxiliary loss, then evaluate DDTree speedup. If the auxiliary loss improves DDTree's acceptance length at fixed B, it validates that drafter training and tree construction should be co-designed. If it does not help, it suggests the independence assumption is not the primary bottleneck — verification overhead or something else limits gains.

Memory-aware budget selection: optimal B under a joint latency–memory constraint. Figure 3 shows the speedup-vs-B curve for one setting, but deployment involves both latency and memory constraints. Tree verification stores KV pairs for B nodes simultaneously, and for large B and large target models, peak GPU memory can be the binding constraint before verification latency becomes the bottleneck. A memory-aware study would: (a) measure peak GPU memory usage for DDTree across B values and target model sizes, (b) identify the largest B deployable under common GPU memory budgets (e.g., 24GB, 40GB, 80GB), and (c) determine whether the latency-optimal B from Figure 3 is memory-feasible. If the latency-optimal B = 512 exceeds memory capacity for a 30B model on an 80GB GPU, the realized speedup in a memory-constrained deployment is determined by the largest feasible B, not the latency-optimal one. This study would produce deployment guidelines: for a model of size X on a GPU with memory Y and context length Z, the maximum recommended B is ___ and the expected speedup is ___.

Extending DDTree to variable-length blocks with dynamic budget allocation. DDTree uses a fixed block size L = 16 and a fixed node budget B. In practice, the optimal block size and budget likely vary by decoding step — early in generation, when the model is establishing the topic, a wider tree (larger B, shallower depth) might be more valuable; later, when generating a predictable continuation, a narrower but deeper tree might suffice. An adaptive variant would: (a) run the drafter, (b) compute the entropy of the per-position marginals (as a proxy for uncertainty), (c) dynamically allocate B across depths — more nodes at high-entropy positions, fewer at low-entropy positions — subject to a total verification budget. The experiment: compare adaptive allocation against fixed B = {128, 256, 512} on MATH-500 and MT-Bench (chosen to span structured vs. open-ended generation). If adaptive allocation matches or exceeds the best fixed B without requiring per-dataset tuning, it addresses the budget-selection limitation (Section 6) and makes DDTree more practical for deployment on mixed workloads.

DDTree with alternative surrogate objectives: sampling-based vs. optimization-based tree construction. The factorized surrogate Q is one of many possible objectives for tree construction from one-pass marginals. Alternative approaches include: (a) sampling paths from Q (Monte Carlo tree construction), (b) beam-search-style scoring where the score of a path is the product of per-position probabilities multiplied by a "coherence bonus" from an N-gram model (approaching DART's approach but with a simpler scoring function), (c) entropy-weighted tree construction that allocates more budget to positions where the drafter is uncertain, and (d) directly predicting a tree structure using a learned policy network that takes the per-position logits as input and outputs node selection decisions. An ablation study comparing these alternatives on MATH-500 and HumanEval (code) would characterize how much the specific surrogate objective matters versus the general idea of constructing a tree. If all reasonable surrogates perform similarly, the insight is that any tree beats no tree, and the specific surrogate is not a critical design choice. If the factorized surrogate substantially outperforms alternatives, it validates the theoretical framework. If a learned policy outperforms the surrogate, it opens a new direction for end-to-end optimization of the drafting-verification pipeline.

Practical Applications and Downstream Use Cases

Latency-sensitive LLM serving for code completion. Code completion (e.g., in IDEs, Copilot-style tools) is a high-volume, latency-sensitive application where users expect sub-second response times. The Qwen3-Coder-30B-A3B-Instruct results in Table 1 are directly relevant: DDTree achieves 8.22× speedup on HumanEval at T=0.0 (up from 6.09× for DFlash), and 6.54× on LiveCodeBench (up from 4.72×). For a serving provider handling millions of code-completion requests per day, a 35–40% improvement in throughput per GPU — which DDTree provides over DFlash, which itself already provides ~5× over autoregressive — translates to substantial infrastructure cost savings or the ability to serve more users with the same hardware. The block diffusion drafter's one-pass property is particularly well-suited to code because code has predictable structure (keywords, brackets, indentation patterns) that block diffusion can capture in a single forward pass, and DDTree's tree explores alternative token choices (e.g., different variable names, different API calls) that the target model might prefer over the single greedy path.

Batch inference for synthetic data generation and self-improvement pipelines. Many LLM applications involve generating large volumes of text offline — synthetic training data, model evaluations, self-play trajectories for RLHF, or knowledge distillation. In these settings, throughput (tokens per second per GPU) is the primary metric, and latency per individual request matters less. DDTree's 7.52× speedup on MATH-500 with Qwen3-8B (Table 1) means generating a fixed number of math solutions requires ~26% less GPU time than vanilla DFlash and ~87% less than autoregressive decoding. For a research lab generating millions of training examples, this difference can determine whether a project is computationally feasible within a given budget. Moreover, because DDTree is lossless (it preserves the target model's output distribution exactly), the generated data has identical quality to autoregressive decoding — the only difference is speed. This makes DDTree a drop-in acceleration for any pipeline that currently uses autoregressive sampling from a target model.

On-device or edge deployment of mid-size models with speculative decoding. The consistent gains across Qwen3-4B (the smallest tested model) — e.g., 6.58× on GSM8K, 6.81× on HumanEval at T=0.0 (Table 1) — suggest DDTree is viable for smaller target models where every millisecond of latency matters. In on-device scenarios (laptops, phones, edge servers), model size is constrained by memory and compute, and speculative decoding is one of the few techniques that can accelerate inference without changing model weights or outputs. DDTree's ability to extract additional speedup from the same DFlash drafter, without increasing the drafter's size or cost, makes it particularly attractive for resource-constrained deployments. The one-pass drafter runs once per round regardless of the tree size, so the drafting cost is fixed; the verification cost scales with B, which can be tuned to match the device's compute budget. A deployment on a consumer GPU (e.g., laptop with 8GB VRAM) running Qwen3-4B with DDTree at B=128 might achieve ~6× speedup over autoregressive decoding, making real-time interactive applications feasible where they would otherwise be too slow.

When to Prefer This Method

The paper does not explicitly articulate a decision rule for choosing DDTree over named alternatives such as EAGLE-3 or DART — the experimental comparison is only against vanilla DFlash, and the positioning relative to other methods is based on transitive inference from prior DFlash results. A forced comparison matrix would therefore be speculative and is not included.

What the paper does provide is a clear internal tradeoff: DDTree should be preferred over vanilla DFlash when the node budget B can be selected to achieve higher speedup than B=1 (the single-trajectory equivalent). Figure 3 shows that this holds for all B values tested on MATH-500 with Qwen3-8B — even B=16 outperforms vanilla DFlash. The budget-quality frontier is concave, so the optimal B sits at an intermediate value (256–512 on the tested hardware) and deploying with B too small leaves speedup on the table while B too large incurs verification overhead that erodes gains. The paper provides a case study of this frontier (Figure 3) but not a general method for locating the optimal B on new hardware or new tasks, which remains a practical limitation for adoption.