Clover: Regressive Lightweight Speculative Decoding with Sequential Knowledge

ArXiv: 2405.00263

🎯 Pitch

Training draft heads to predict independent next tokens causes a sharp accuracy drop on later heads, but Clover recovers up to 26.4% of that head-level hit rate simply by feeding earlier speculated tokens back into the speculator. This sequential regressive design brings a 57% throughput lift over the best prior method on Baichuan‑Large, with no change to the base model.

---

1. Executive Summary

This paper proposes Clover, a new speculative decoding algorithm that extends the Medusa framework by integrating sequential knowledge into the parallel decoding process through three named components— Regressive Connection (feeding previously speculated token embeddings as inputs to subsequent heads), Attention Decoder (a cross-attention block that merges hidden states from the last transformer block with previous speculated tokens), and Augmenting Block (an additional transformer layer appended to the target model to enhance features for speculation). Evaluated on Baichuan-Small (7B parameters) and Baichuan-Large (over 100B parameters) across ten task categories, Clover achieves up to 91% throughput improvement over vanilla auto-regressive decoding on Baichuan-Small and 146% on Baichuan-Large, while exceeding the prior top-performing Medusa method by up to 37% on Baichuan-Small and 57% on Baichuan-Large—a gain driven primarily by improved accuracy on latter speculative heads (+21.7% to +26.4%) where sequential dependencies matter most. The paper establishes that regressive integration of pre-speculated token information into parallel decoding substantially improves hit rates and throughput, with gains becoming more pronounced as model size increases and the speculator module represents a smaller fraction of total computation, but only when deployed in large-batch, small-token-tree serving regimes where computational bottlenecks dominate over memory constraints.

2. Context and Motivation

The Core Problem: Auto-Regressive Decoding Leaves GPUs Idle

The fundamental problem this paper tackles is the mismatch between the auto-regressive decoding paradigm of large language models and the hardware architecture of modern GPUs. To understand why this mismatch exists and why it matters, we need to walk through what actually happens when an LLM generates text.

During the auto-regressive decoding phase — after the prompt has been processed — the model generates one token at a time, and each new token depends on all previously generated tokens. At each generation step, the entire model (billions to trillions of parameters) must be loaded from GPU memory into the compute units through the GPU's memory bus. This loading step — the memory transfer — is the bottleneck: modern GPUs have enormous computational capacity (they can perform trillions of FLOPS per second), but their memory bandwidth is comparatively limited. The result is that when generating a single token, the GPU spends most of its time waiting for weights to arrive from memory, not actually computing. The paper describes this succinctly:

"billions to trillions of parameters must be loaded to the GPU cache through its limited memory bandwidth for computation, but only a small batch of tokens is actually computed. Consequently, the GPU spends most of its time on memory transfer instead of computation."

This is the memory-bound regime — the GPU's compute units are underutilized because they're starving for data. In the prefilling phase (processing the input prompt), this isn't a problem because hundreds or thousands of prompt tokens are processed in parallel, giving the GPU enough work to saturate its compute units. But during decoding, with only one token per step, the compute-to-memory ratio is terrible.

This inefficiency has direct real-world consequences: it means that LLMs serving user requests are slower and more expensive than they need to be. Every decoding step incurs the full cost of loading the model weights, but only does a tiny amount of actual computation. For applications like chatbots, code assistants, or real-time translation — where low latency is critical — this inefficiency is a major bottleneck.

Why This Problem Matters: The Economic and Practical Stakes

The stakes are both economic and architectural. From an economic perspective, LLM inference is expensive precisely because of this inefficiency. If you can generate multiple tokens per memory-load cycle instead of one, you get proportionally more work done for the same hardware cost. This translates directly to lower latency for users (faster responses) and higher throughput for service providers (more requests served per GPU).

From an architectural perspective, this problem is only getting worse as models grow. Larger models have more parameters, meaning more data to transfer per decoding step, but the compute-per-token doesn't grow proportionally. The memory bottleneck intensifies with model scale, making efficient decoding strategies increasingly important.

The Speculative Decoding Solution and Its Limitation

Speculative decoding is a family of techniques that address this bottleneck by generating multiple tokens per decoding step while preserving exact output equivalence to auto-regressive decoding. The core idea is simple: use a cheap "speculator" (or "draft model") to quickly guess several future tokens, then feed all those tokens to the large target model in a single forward pass for verification. If the speculations are correct, the target model accepts them and multiple tokens are generated at the cost of roughly one memory-load cycle. If incorrect, the target model rejects the bad tokens and only the correct prefix is kept. The paper illustrates this in Figure 3, contrasting the one-token-per-step auto-regressive approach with speculative decoding that can produce multiple tokens in a single verification pass.

This is ingenious because it shifts the bottleneck: instead of being memory-bound on the large model (loading all weights for one token), speculative decoding becomes compute-bound on the large model (verifying multiple tokens at once), which is exactly what GPUs are designed to handle efficiently.

The Medusa Breakthrough and Its Architectural Limitation

Among speculative decoding approaches, Medusa (Cai et al., 2024) represents a particularly elegant design. Rather than using a separate draft model (which requires maintaining and loading a second model, adding deployment complexity), Medusa attaches lightweight "heads" directly to the target model. These heads are small multi-layer perceptron (MLP) layers that take the hidden states from the last transformer block of the target model as input and predict tokens at various future positions — the first head predicts the next token (position $i+1$), the second predicts the token after that ($i+2$), and so on. Because these heads are tiny compared to the full model, their computation is negligible, and they piggyback on the hidden states already computed by the target model during the verification step.

The architecture is shown in Figure 1(a): multiple parallel single-layer MLP heads, each independently predicting a token at a fixed offset from the current position. This simplicity is Medusa's strength — it's easy to implement, adds minimal parameters, and requires no separate model loading.

However, this simplicity also creates a fundamental limitation. Each Medusa head is completely independent — the head predicting token $i+3$ has no access to what the head predicting token $i+2$ has predicted. The heads operate on the same frozen hidden state from the last transformer block, with no mechanism to condition on each other's outputs. This violates a core principle of language: tokens are sequentially dependent. If the first head predicts "Bayesian" and the second head is trying to predict what comes after "Bayesian," knowing that "Bayesian" was predicted should dramatically inform the second head's prediction (e.g., "optimization" or "inference" are much more likely after "Bayesian" than, say, "banana").

The paper identifies this as the key weakness:

"the Medusa head consists of only a single MLP layer that takes input solely from the final hidden states. Each layer independently speculates on a word at a specified position beyond the next, disregarding the sequential dependencies from previously predicted tokens, which often results in decreased accuracy."

This independence has two damaging consequences. First, hit rate suffers: later heads are essentially guessing blind about what earlier heads predicted, so their accuracy drops off sharply with position. Second, because the heads are independent, the verification step must construct a token tree by taking the Cartesian product of all head predictions — every combination of tokens from each head forms a path in the tree. This produces an exponentially large tree that contains many obviously nonsensical combinations (e.g., "Bayesian" followed by "banana"), wasting verification computation on tokens that have essentially zero chance of being correct.

The paper further observes that this problem is acute in a specific and practically important regime:

"In real-time serving scenarios, where the inference batch size is typically large, speculative decoding often faces computational constraints, leading to performance degradation."

Figure 2 illustrates this: as the number of computed tokens increases (due to larger batch sizes or larger token trees), the speedup from speculative decoding peaks and then declines because the system shifts from being memory-bound to being compute-bound. In large-batch serving — the realistic deployment scenario — the token tree must be kept small to avoid compute saturation. But Medusa's independent heads, with their Cartesian product tree construction, produce low-information-density trees: the tree is either exponentially large (containing many bad paths) or, when pruned to a practical size, loses coverage of good candidate sequences because the pruning algorithm lacks sequential dependency information to make intelligent cuts.

Prior Approaches to Sequential Integration in Speculators

The paper acknowledges that other researchers have recognized the value of sequential dependencies in speculators. The Related Work section (Section 5) cites several concurrent approaches:

• Zhang et al. (2024): Uses an MLP layer as a regressive block to pass information between speculator heads.
• Hydra (Ankner et al., 2024): Introduces an additional block for sequentially-dependent draft heads in the Medusa framework.
• Eagle (Li et al., 2024): Uses a regressive transformer block for speculation, rethinking how feature uncertainty should be handled.
• Chimera (Zeng et al., 2024): Proposes a Trigram Encoder and Full Context Encoder as regressive speculators, fusing information across tokens.

Where Clover distinguishes itself from these approaches is in how it implements the regressive mechanism (using cross-attention rather than concatenation + MLP or self-attention) and in its explicit focus on large-batch, small-tree deployment scenarios — a regime the paper argues has been "not sufficiently addressed in previous speculative decoding work." The cross-attention design choice is intentional: it allows the model to learn which information from previous speculations is relevant to the current prediction, rather than forcing the model to disentangle concatenated representations, which the paper's ablation shows leads to sub-optimal performance (the MLP-as-regressive-block variant loses 4.8–11.5% top-5 accuracy compared to the attention decoder).

The Paper's Positioning

Clover positions itself as an extension and refinement of Medusa, not a replacement for speculative decoding as a whole. The key insight is that Medusa's architectural simplicity — while elegant — leaves substantial performance on the table by ignoring sequential dependencies. Clover introduces just enough additional structure (regressive connections, cross-attention, an augmenting block) to capture these dependencies while keeping the speculator lightweight enough that the computational overhead is outweighed by the accuracy gain.

The paper's framing around large-batch, small-tree serving is significant because it represents a deliberate departure from the evaluation methodology of prior work. Many speculative decoding papers optimize for maximum speedup in low-batch settings with large token trees, but these conditions don't match production serving environments where many requests are batched together and the system is compute-constrained. By explicitly targeting this regime, Clover aims to deliver gains that are not just theoretically impressive but practically deployable. The $1.26\times$–$1.47\times$ throughput improvement over Medusa reported in the paper's evaluation represents a meaningful practical gain in a regime where prior methods were already close to their ceiling.

In summary, the paper addresses a specific and well-motivated gap: parallel decoding speculators (Medusa-style heads) are unnecessarily inaccurate because they ignore sequential token dependencies, and this inaccuracy is especially costly in large-batch serving where token trees must be kept small. The proposed solution is to inject sequential knowledge through lightweight architectural additions, trading a small amount of added computation per speculation for substantially higher hit rates, with the net effect being improved throughput in the deployment regimes that matter most.

3. Technical Approach

3.1 Reader Orientation

Clover is a speculative decoding system — a lightweight add-on module that attaches to an already-trained large language model to make it generate text faster, without changing what text it generates. The system solves the problem that Medusa's parallel decoding heads predict future tokens independently, ignoring what earlier heads have already predicted, which leads to inaccurate guesses and wasted verification work. The solution takes the shape of three architectural additions that inject sequential knowledge into the speculation process: a feedback loop that routes previously-speculated tokens back as inputs to later heads, a cross-attention mechanism that merges this sequential information with the broader sentence context, and an extra transformer layer that enriches the hidden states before speculation begins.

3.2 Big-Picture Architecture (Diagram in Words)

The Clover system attaches to a frozen target LLM and consists of five major components arranged in a pipeline:

1. Target LLM (frozen): The large pre-trained model whose weights are never modified during Clover training. It processes the input prompt and previously-generated tokens through its full stack of transformer blocks, producing hidden states at the final layer. These hidden states encode the full context of the conversation or generation so far.

2. Augmenting Block: An additional transformer layer appended after the target model's last layer. It takes the target model's final hidden states and transforms them into richer representations specifically optimized for the speculation task (predicting multiple future tokens rather than just the next one). This block is the only part of Clover that processes the full sequence context; its output ($h_0$) serves as the initial hidden state for the speculation chain.

3. Attention Decoder (Regressive Block): A cross-attention mechanism that implements the sequential dependency between speculative heads. For each speculative position $i$ (where $i=1$ means "the first token after the actual output"), the Attention Decoder takes two inputs: the hidden state from the previous speculation step ($h_{i-1}$) and the embedding vector of the token just speculated at position $i-1$ ($e_{i-1}$). It uses cross-attention to merge these two information sources, producing an updated hidden state $h_i$ that informs the prediction at position $i$. The first Attention Decoder call uses $h_0$ from the Augmenting Block and $e_0$ (the embedding of the actual next-token output from the target model).

4. MLP Speculation Heads: After the Attention Decoder produces $h_i$, a single-layer MLP (identical in structure to Medusa's heads) projects this hidden state to predict the token at position $i$. All heads share the same LM head (the target model's output projection layer) for generating token logits, rather than each head having its own copy — a parameter-efficiency design choice.

5. Regressive Connection (Feedback Loop): The physical routing of data that makes the sequential dependency real. After the MLP head at position $i-1$ predicts a token, that token's embedding vector $e_{i-1}$ is extracted (by looking up the predicted token ID in the transposed, normalized weight matrix of the shared LM head) and fed as input to the Attention Decoder at position $i$. This is the "regressive" pathway — information flows backward from earlier speculative positions to inform later ones, creating an auto-regressive chain within the speculation phase.

Information flows through this system as follows: the target model processes the input sequence → the Augmenting Block enriches the hidden states → the first Attention Decoder merges $h_0$ with $e_0$ (the actual next token's embedding) → the first MLP head predicts token $t_1$ → $t_1$'s embedding $e_1$ is extracted via the Regressive Connection → the second Attention Decoder merges $h_1$ with $e_1$ → the second MLP head predicts token $t_2$ → this continues for all heads → all speculated tokens are assembled into a token tree → the target model verifies the entire tree in one forward pass using Tree Attention → correct tokens are accepted, incorrect ones are discarded, and the process repeats.

3.3 Roadmap for the Deep Dive

We build understanding in this order, which mirrors both the data flow and the conceptual dependencies:

• First, the Medusa baseline mechanism and its architectural limitation — because Clover is a direct extension of Medusa, and understanding what Clover fixes requires understanding what Medusa does wrong.
• Second, the Regressive Connection — the physical feedback pathway that enables sequential information transfer, which is the foundational mechanism that the other components depend on.
• Third, the Attention Decoder — the computational block that consumes the sequential information provided by the Regressive Connection, because its design (cross-attention vs. alternatives) is the key architectural decision.
• Fourth, the Augmenting Block — the feature-enhancement layer that improves the quality of the hidden states fed into the speculation chain, which is conceptually separate from the regressive mechanism but empirically important.
• Fifth, the training procedure and parameter sharing design — how all these components are trained while keeping the target model frozen, because the training constraints and initialization choices directly affect what the architecture can learn.
• Sixth, the inference-time verification process — how speculated tokens from this architecture are assembled into a token tree and verified by the target model, closing the loop from architecture to deployment.

3.4 Detailed, Sentence-Based Technical Breakdown

This is a systems paper that extends an existing speculative decoding method (Medusa) by adding three architectural components to inject sequential knowledge into what was previously a fully parallel, independent-head speculation process. The core idea is that by making later speculative heads aware of what earlier heads predicted, hit rates improve substantially, and this accuracy gain outweighs the added computational cost of the regressive mechanism—especially in large-batch serving scenarios where token trees must be kept small and every speculative token's accuracy premium matters.

---

Medusa Baseline: Independent Parallel Heads and Their Limitation

Medusa (Cai et al., 2024), illustrated in Figure 1(a), attaches multiple lightweight speculation heads to a frozen target LLM. Each head $i$ is a single-layer MLP that takes as input the hidden states from the target model's last transformer block and predicts the token that will appear $i$ positions after the current actual output token. So the first head predicts $\text{tok}_1$ (the immediate next token), the second head predicts $\text{tok}_2$ (the token after that), and so on.

The computation for each head is completely independent:

$\text{logits}_i = \text{MLP}_i(h_{\text{last}})$

where $h_{\text{last}}$ is the hidden state vector from the target model's final transformer block, and $\text{MLP}_i$ is the single-layer perceptron for head $i$.

What it computes: each head independently projects the same frozen hidden state through its own learned MLP to produce logits over the vocabulary for a specific future position. The heads are trained to specialize — head 1 learns to predict immediate continuations, head 2 learns to predict what comes two tokens later given only the current context, etc.

Why this is limited: because all heads receive identical input ($h_{\text{last}}$), head 2 has no access to what head 1 predicted. If head 1 predicts "Bayesian" and head 2 is trying to predict what comes after "Bayesian," head 2 must guess based solely on the prefix context before "Bayesian" — it cannot condition on "Bayesian" itself. This is an information bottleneck. The paper quantifies this: Medusa heads show sharply decreasing accuracy with position, with the first head being relatively accurate but later heads dropping off dramatically. The paper's Figure 7(a) shows that in the full Clover configuration, later heads benefit the most from sequential knowledge (gaining 21.7–26.4% top-5 accuracy compared to Medusa), confirming that the independence assumption harms later positions most severely.

The second consequence is structural: because heads are independent, the token tree for verification must be constructed as a Cartesian product — every token predicted by head 1 is combined with every token predicted by head 2, and so on. For $k$ heads each producing $b$ top candidates, the tree has $b^k$ paths. Most of these paths represent nonsensical token combinations (e.g., "Bayesian" followed by "banana"), wasting verification computation and forcing aggressive pruning in large-batch scenarios where the tree must be kept small. The paper argues that a regressive architecture produces trees with "greater information density" because each path in the tree respects the sequential dependencies that actually exist in language — the tree's branching reflects genuine uncertainty about what follows a known prefix, not combinatorial explosion from independent guesses.

---

Regressive Connection: The Feedback Pathway

The Regressive Connection is the physical mechanism that routes information from previously-speculated tokens back into the speculation pipeline, depicted as the blue dotted lines in Figure 5. It is the architectural realization of the core insight: later heads should know what earlier heads predicted.

How it works mechanically:

When the MLP head at position $i-1$ produces a prediction for token $\text{tok}_{i-1}$, the system needs to convert this discrete token into a continuous representation that the next head can consume. Clover does this by extracting an embedding vector from the shared LM head — the same output projection matrix that the target model uses to produce logits. Specifically, the embedding vector $e_{i-1}$ is computed as:

$e_{i-1} = \text{one\_hot}(\text{tok}_{i-1}) \cdot W_{\text{LM}}^T$

where $\text{one\_hot}(\text{tok}_{i-1})$ is a one-hot encoding of the predicted token (a vector of length vocabulary size with a 1 at the predicted token's index and 0 elsewhere), $W_{\text{LM}}$ is the weight matrix of the target model's LM head (dimensions: hidden_size × vocabulary_size), $W_{\text{LM}}^T$ is its transpose (vocabulary_size × hidden_size), and $e_{i-1}$ is the resulting embedding vector (dimension: hidden_size).

What it computes: the matrix multiplication extracts the column of $W_{\text{LM}}^T$ corresponding to the predicted token. In other words, instead of looking up a separate learned embedding from an embedding table (which is what happens at the input to the target model), Clover uses the transposed output projection weights as the embedding source. This produces a vector that lives in the same space as the hidden states — the space where the LM head's weights were trained to map hidden states to token probabilities.

Why this form: the paper argues that this embedding distribution is "much closer to the hidden states from the last transformer block" than embeddings from a separate lookup table. The reasoning: the LM head weights were trained to transform hidden states into token logits, so their transpose naturally maps back toward the hidden-state space. This reduces the distributional gap between the embedding vectors consumed by the Attention Decoder and the hidden states they're combined with, making the fine-tuning task easier. Additionally, sharing the LM head for embedding extraction avoids introducing new parameters — the weights already exist and are already loaded in GPU memory.

A critical design note: for the first head (position $i=1$), there is no previously-speculated token. Instead, $e_0$ is the embedding of $\text{tok}_0$ — the actual next token produced by the target model during the verification step. This means the speculation chain is grounded in a real token (not a speculation) at its root, which provides a stable foundation for subsequent predictions.

The Regressive Connection introduces sequential dependency into what was previously a fully parallel process. The paper explicitly acknowledges the latency tradeoff:

"Although the critical path of the computation becomes proportional to the depth of the speculation and loses a certain amount of parallelism, the increase in speculation accuracy rather improves the overall latency."

In other words, the serial nature of the regressive chain (head $i$ must wait for head $i-1$ to finish) adds latency, but the gain in hit rate — more tokens accepted per verification step — more than compensates, reducing the total number of verification steps needed and thus decreasing end-to-end latency.

---

Attention Decoder: Merging Sequential and Contextual Information

The Attention Decoder is the computational block that implements the actual regression — it takes the sequential information from the Regressive Connection and merges it with the contextual information from the target model, producing a hidden state that informs the next token prediction. It is illustrated in Figure 5 as the central component of Clover, positioned between the Regressive Connection inputs and the MLP speculation heads.

Architecture and computation:

The Attention Decoder is a cross-attention mechanism, which means it computes attention where the query comes from one source and the keys and values come from a different source. This is distinct from self-attention (where query, key, and value all come from the same sequence) and from simple concatenation + MLP (where two vectors are concatenated and fed through a feedforward network).

For the $i$-th speculative head, the computation proceeds in four steps:

Step 1: Normalize the previous hidden state to produce the query.

$Q_i = W_Q \cdot \text{normalize}(h_{i-1})$

where $h_{i-1}$ is the hidden state output from the previous Attention Decoder step (or $h_0$ from the Augmenting Block for the first head), $\text{normalize}$ is a normalization operation (likely RMSNorm or LayerNorm, standard in transformer architectures), and $W_Q$ is a learned projection matrix mapping from hidden_size to a query dimension $d_k$. The query represents "what information about the next token am I trying to extract from the previous token?"

Step 2: Project the previous token's embedding to produce keys and values.

$K_i = W_K \cdot e_{i-1}$

$V_i = W_V \cdot e_{i-1}$

where $e_{i-1}$ is the embedding vector of the token predicted at the previous position (from the Regressive Connection), $W_K$ is a learned projection to key dimension $d_k$, and $W_V$ is a learned projection to value dimension $d_v$. The key matrix $K_i$ determines "which aspects of the previous token are relevant for matching against the query," and the value matrix $V_i$ determines "what information from the previous token should be propagated forward."

Step 3: Compute cross-attention and update the hidden state.

$h_i = h_{i-1} + \text{Attention}(Q_i, K_i, V_i)$

where $\text{Attention}(Q_i, K_i, V_i)$ computes the standard scaled dot-product attention:

$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{Q K^T}{\sqrt{d_k}}\right) V$

What this computes: the attention mechanism compares the query $Q_i$ (derived from the full context via $h_{i-1}$) against the key $K_i$ (derived from the previous speculated token $e_{i-1}$). The resulting attention weights determine how much of the value $V_i$ (the previous token's information) to add to the hidden state. The residual connection ($+ h_{i-1}$) ensures that the original context information is preserved — the attention output is an additive update, not a replacement.

Step 4: Feed the updated hidden state to the MLP head.

$\text{logits}_i = W_{\text{LM}} \cdot \text{MLP}_i(h_i)$

where $\text{MLP}_i$ is the single-layer perceptron specific to head $i$, and $W_{\text{LM}}$ is the shared LM head weight matrix.

Design rationale — why cross-attention instead of concatenation + MLP:

The paper's ablation study (Figure 7a) directly compares the Attention Decoder against an alternative: removing the Attention Decoder and instead using an MLP layer as the regressive block, which would concatenate $h_{i-1}$ and $e_{i-1}$ and feed them through a feedforward network. The results are decisive:

"By removing Attention Decoder, taking the MLP layer as the regressive block instead, the top-5 accuracy of the three heads reduces by (4.8%, 9.0%, 11.5%)."

The paper's explanation:

"This is because MLP layer itself will not distinguish different embedding vectors, make it hard to learn valid sequential knowledge from the entire sentence."

To unpack this: an MLP applied to a concatenated vector $[h_{i-1}; e_{i-1}]$ treats the concatenation as a fixed-width input where each position in the vector has a fixed weight regardless of what the previous token actually is. The MLP learns a static mapping from the combined vector to the output — it cannot dynamically re-weight which parts of $e_{i-1}$ are relevant based on what's in $h_{i-1}$. In contrast, cross-attention computes a content-dependent mixing: the attention weights are a function of the interaction between $Q_i$ (derived from $h_{i-1}$) and $K_i$ (derived from $e_{i-1}$). If the previous token is "Bayesian," the query might attend strongly to aspects of "Bayesian" that suggest mathematical continuation; if the previous token is "the," the query might attend to different aspects (grammatical number, common noun preferences). This dynamic, context-sensitive integration is what makes attention more effective for combining heterogeneous information sources.

Why the residual connection matters: the formulation $h_i = h_{i-1} + \text{Attention}(\dots)$ means that $h_i$ is always a perturbation of $h_{i-1}$. Even if the attention mechanism produces poor outputs (e.g., during early training), the hidden state doesn't collapse — the original context information from the target model is preserved. This makes training more stable and allows the Attention Decoder to learn incrementally how much to trust the sequential signal from the previous token versus the contextual signal from the full input.

Computational overhead: the paper emphasizes that the Attention Decoder has "negligible overhead due to the fact that the inputs are only two vectors per request or beam." The key observation is that the cross-attention operates on single vectors, not sequences — $Q_i$ is a single vector (one query), and $K_i$ and $V_i$ are single vectors (one key-value pair). There is no sequence dimension to scale with, so the attention computation is $O(d_k)$ rather than $O(n \cdot d_k)$ for an $n$-token sequence. This is in contrast to the target model's self-attention layers, which scale quadratically with sequence length.

Recursive propagation of hidden states: a crucial detail in the computation flow is that $h_i$ is "recursively propagated throughout the entire speculation phase, piggybacking the features from the entire input sentence." This means that $h_2$ (feeding head 3) contains information from the original context plus the first speculated token; $h_3$ (feeding head 4) contains information from the original context plus the first and second speculated tokens; and so on. The hidden state accumulates information as it propagates through the chain, which is why later heads benefit disproportionately from the regressive architecture — they have access to progressively more sequential context that was denied to them in the independent-head Medusa design.

---

Augmenting Block: Enriching Hidden States for Speculation

The Augmenting Block is an additional transformer layer appended after the target model's last transformer block. Its purpose is to transform the hidden states — which were produced by a model trained only for next-token prediction — into representations that are better suited for the multi-token speculation task.

Motivation: the target LLM was pre-trained with a single objective: given a prefix, predict the next token. Its hidden states are optimized for this single task. But Clover asks these same hidden states to support predicting tokens at positions $+2, +3, +4$ — tokens that are progressively further into the future and that depend on intermediate tokens that haven't been generated yet. The Augmenting Block provides a learned transformation to extract more predictive information for these distant positions.

Architecture options explored:

The paper experiments with four configurations for the Augmenting Block (Figure 7b):

1. Full transformer block (attention + MLP): the actual Clover configuration. This includes a self-attention layer followed by an MLP layer, with residual connections and normalization — a standard transformer layer.
2. Attention block only: just the self-attention component without the MLP.
3. MLP block only: just the feedforward network without attention.
4. No Augmenting Block: the hidden states from the target model's last layer are fed directly to the Attention Decoder (this reduces to Medusa if the Regressive Connection and Attention Decoder are also removed).

Results from Figure 7b:

"The transformer block contributes (4.9%, 9.0%, 9.4%) top-5 accuracy to the heads compared with no augmenting block, in which the attention block plays the major role (increasing 1.9%, 4.4%, 5.1% top-5 accuracy when only attention block enabled for Augmenting Block). While the MLP block only provides (1.0%, 1.2%, 0.7%) accuracy improvement."

Why attention dominates: the paper explains:

"The attention mechanism focuses on extracting relationships between tokens in the sentence, making it easier to learn for feature augmentation."

The self-attention in the Augmenting Block can redistribute information across the sequence — for example, it can pull information from earlier tokens that are particularly relevant for predicting what comes several steps after the current position. This is a different computation than what the MLP can do: an MLP applied position-wise (as is standard in transformer architectures) transforms each token's representation independently based on a fixed learned function, while attention can route information between positions based on content.

Why include the MLP at all: even though the MLP block provides only ~1% accuracy gain, the paper includes it because "it has minimal time and memory overhead." The cost is proportional to approximately $1/N_{\text{layer}}$ of total inference time (since it's one additional layer among many), and the accuracy gain, while small, is strictly positive with essentially zero downside.

Parameter overhead and design: the Augmenting Block's overhead is characterized as "just a small computation overhead (e.g. approximately $1/N_{\text{layer}}$ of inference time), while the accuracy gain from the augmenting block outweighs the time it consumes." For a model with 40 transformer layers, one extra layer adds ~2.5% to the per-step computation — a meaningful but manageable cost that the paper's end-to-end throughput results show is justified by the accuracy improvement.

---

Training Procedure and Parameter Sharing

Clover is trained with the target model completely frozen — only the newly added components (Augmenting Block, Attention Decoder, MLP heads) have their weights updated. This is a critical design constraint because it means Clover can be added to any pre-trained model without modifying the original weights, preserving all existing capabilities and making deployment straightforward (the target model weights are shared and don't need to be duplicated).

Training data and objective:

The training uses the Baichuan internal supervised fine-tuning (SFT) dataset, containing approximately 0.15B tokens (95% Chinese). The objective is next-token prediction at multiple offsets: for each position in the training sequences, head $i$ is trained to predict the token that appears $i$ positions later. This is the same multi-offset training objective used in Medusa.

The loss function is standard cross-entropy loss summed across all heads:

$\mathcal{L} = \sum_{i=1}^{K} \mathcal{L}_{\text{CE}}(\text{logits}_i, \text{target}_i)$

where $K$ is the number of heads (3 in the paper's experiments), $\text{logits}_i$ are the predicted logits from head $i$, and $\text{target}_i$ is the ground-truth token at offset $i$. In practice, each head is trained to predict its corresponding future token, with no explicit coordination loss between heads — the coordination emerges from the architecture itself (later heads receive earlier heads' predictions through the Regressive Connection and Attention Decoder).

What it computes: for each training example, the system computes the cross-entropy between each head's prediction and the actual token at that offset, then sums these losses. Backpropagation flows through the MLP heads, the Attention Decoder, and the Augmenting Block, but stops at the target model (whose weights are frozen).

Why freeze the target model: this preserves the target model's language capabilities exactly, avoids catastrophic forgetting, and makes Clover a plug-and-play add-on. It also means the target model weights are shared across all deployments, reducing memory overhead.

Parameter sharing — shared LM head:

A significant parameter-efficiency design choice: all speculation heads share the target model's original LM head ($W_{\text{LM}}$) rather than each head having its own output projection. The paper notes:

"Each medusa head equipped with an individual LM head, containing a large amount of parameters (i.e. the hidden size multiplies the vocabulary size) and make it more time-consuming for training."

The vocabulary size for LLMs is typically 50,000–250,000 tokens, and the hidden size is typically 4,096–8,192 dimensions. Multiplying these gives an LM head with hundreds of millions to billions of parameters per head. With 3 heads, Medusa would require 3 copies of this massive matrix. Clover's sharing reduces this to 1 copy, dramatically cutting trainable parameters and memory usage. The consequence is that all heads must predict tokens in the same output space — they share the same mapping from hidden states to token probabilities — but this is actually desirable because all heads are predicting tokens from the same vocabulary in the same language.

Embedding extraction from shared LM head:

The same shared LM head is used for the Regressive Connection's embedding extraction (the $e_i$ vectors). Instead of maintaining a separate embedding table, the system computes:

$e_i = \text{one\_hot}(t_i) \cdot W_{\text{LM}}^T$

where $W_{\text{LM}}^T$ is the transpose of the (normalized) LM head weight matrix. The paper notes that this embedding distribution is closer to the hidden state space than a separate embedding table would be, reducing the difficulty of the Attention Decoder's integration task.

Initialization strategy:

The paper specifies precise initialization schemes for each component:

• Augmenting Block: initialized identically to the last transformer block in the target model. This is a warm-start strategy — the Augmenting Block begins as an exact copy of a layer that already knows how to process the target model's hidden states, then fine-tunes to specialize for the speculation task.
• MLP heads: initialized using the same configuration as Medusa's method (the paper cites the Medusa technical report for the exact scheme, which typically involves small random weights).
• Attention Decoder: "the weights of Q and K are initialized with identical matrix with Gaussian noise added, while the V matrix is set to all zero." This is a carefully designed initialization: (1) Q and K starting identical encourages the attention to initially treat all information from the previous token equally (since the query and key projections are the same, attention weights will be uniform initially); (2) the added Gaussian noise breaks symmetry so different dimensions can learn different patterns; and (3) setting V to zero means the attention output is initially zero, so $h_i = h_{i-1} + 0$ — the residual connection dominates at the start of training. The Attention Decoder learns gradually to add sequential information by growing V away from zero, which is a stable training strategy (the model starts by ignoring sequential information and gradually learns when it's useful).

Training hyperparameters:

The paper reports these exact values:

"We train the heads for 1 epoch, with $(\beta_1 = 0.9, \beta_2 = 0.999)$ for the AdamW optimizer. The learning rate is set to 1e-3 for Baichuan Small, and 6e-4 for Baichuan Large. Cosine decay is applied to the learning rate."

The trainable parameters are approximately 0.2B for Baichuan Small (7B total) and 2B for Baichuan Large (over 100B total), representing roughly 2–3% of the total model parameters. Training takes 2 hours on 8× A800 GPUs for the small model and 32× H800 GPUs for the large model — a modest training cost relative to pre-training.

Why one epoch: speculative decoding heads are fine-tuned for a relatively narrow task (predicting tokens at fixed offsets given rich hidden states), and with 0.15B training tokens, one epoch provides sufficient data to learn the mapping without overfitting. The frozen target model provides strong regularization — the heads can only learn patterns that are useful given the fixed representations they receive.

---

Inference-Time Verification: Token Tree Construction and Tree Attention

At inference time, Clover's speculation heads produce a set of candidate tokens that must be verified by the target model. The verification process uses Tree Attention (Miao et al., 2024) to check all speculated tokens in a single forward pass of the target model.

Token tree construction:

Unlike Medusa's Cartesian product approach (every combination of head-1 predictions × head-2 predictions × head-3 predictions), Clover's regressive architecture produces a naturally narrower tree. Because head 2's predictions are conditioned on head 1's specific predictions (through the Regressive Connection), the tree structure reflects genuine sequential uncertainty rather than combinatorial explosion from independence.

The paper describes the tree construction implicitly through the regressive mechanism: head 1 produces $b$ candidates for position 1. For each of these $b$ candidates, head 2 conditions on that specific candidate (via the Regressive Connection feeding $e_1$ to the Attention Decoder) and produces $b$ candidates for position 2. This means the tree has $b$ branches at the first level, and each branch has $b$ sub-branches at the second level, producing $b^K$ leaves for $K$ heads — the same asymptotic size as Medusa, but critically, each path in the tree is coherent because later tokens were predicted given the earlier tokens on that path. The paper claims this tree has "more comprehensive dependency information" and is "easy for pruning and less likely to meet computation bound on modern GPUs, while introducing negligible information loss."

In the paper's evaluation, the token tree size is set to 4 (meaning $b$ is chosen so that the total number of nodes in the tree is 4, excluding the root). This is an intentionally small tree, chosen for large-batch scenarios where compute constraints limit how many speculated tokens can be verified per step. The paper's Appendix A.2 (Figure 8) investigates the effect of tree size:

"As the token tree size grows larger, the acceptance length is still increases but at a slower rate. Note that the horizontal axis in Figure 8 is in exponential scale, while the vertical axis is linear."

This means that doubling the tree size produces less-than-doubling gains in accepted tokens per step — diminishing returns. In large-batch scenarios, the extra computation from a larger tree pushes the system closer to compute-bound territory, potentially reducing overall throughput even if per-step acceptance improves. The paper's choice of tree size 4 is a practical compromise for the deployment regime it targets.

Causal mask construction:

The token tree is represented as a 2-D causal attention mask, shown in Figure 4. The mask encodes the tree topology: each token can attend to itself and to its ancestors in the tree, but not to tokens in other branches. This is implemented by setting the mask entry $M_{ij} = 1$ if token $j$ is an ancestor of token $i$ in the tree (or if $i = j$), and $M_{ij} = 0$ otherwise. The target model's self-attention layers use this mask to compute attention scores only along valid tree paths, producing logits for all speculated tokens in parallel.

Verification and acceptance:

After the target model produces logits for all speculated tokens, a verification algorithm (typically greedy or speculative sampling) compares each speculated token against the target model's predicted distribution at that position. Tokens that match the target model's prediction are accepted; the first token that doesn't match (or the end of the tree) triggers rejection of that and all subsequent tokens on that branch. The accepted tokens become the official output, and a new speculation cycle begins from the last accepted position.

The key performance metric is extra tokens per step — the number of accepted speculative tokens beyond the first (actual) token. This metric appears in Figure 6 of the paper. Clover achieves 50–76% more extra tokens per step than Medusa across all tasks, which directly translates to higher throughput (tokens/second) because each verification step produces more output tokens on average while costing roughly the same computation.

Why regressive trees help in compute-bound regimes:

The paper argues (Section 1, with Figure 2) that in large-batch serving, the system is compute-bound — the GPU's compute units are saturated, so adding more speculative tokens per step eventually hurts throughput because the verification cost grows faster than the acceptance gain. In this regime, the information density of the token tree matters more than its size. Clover's sequentially-coherent tree, where each path represents a linguistically plausible continuation, achieves higher acceptance rates per tree node than Medusa's Cartesian product tree, where many paths are linguistically nonsensical. This means Clover gets more accepted tokens out of a small tree than Medusa does, which is exactly the advantage that matters when the tree must be kept small to avoid compute saturation.

A nuance not fully explored in the paper but implicit in the design: the regressive architecture means that the tree's branching factor can be smaller because later tokens are better-targeted. If head 2 knows head 1 predicted "Bayesian," it can focus its probability mass on words that commonly follow "Bayesian" rather than spreading mass across the entire vocabulary. The top-$b$ candidates from this concentrated distribution are more likely to be correct than the top-$b$ candidates from an unconditional distribution. The tree size (total nodes) may be similar, but the correct-path probability is higher — and when the tree is aggressively pruned to a small fixed size, this probability difference directly translates to throughput improvement.

4. Key Insights and Innovations

Innovation 1: Regressive Sequential Knowledge as the Missing Dimension in Parallel Decoding

The paper's central conceptual move is diagnosing that the performance ceiling of parallel decoding speculators like Medusa is not primarily about insufficient model capacity or poor training objectives — it is about information flow architecture. Specifically, the field had implicitly accepted that parallel decoding heads should operate independently on a shared frozen representation, treating the independence as a necessary cost of parallelism. Clover challenges this framing directly: the independence is not necessary, and the cost of breaking it (serializing the speculation chain) is more than compensated by the accuracy gain.

What makes this intellectually distinctive is that it reconceptualizes the speculation problem. Before Clover, the prevailing mental model was: "given the current hidden state, predict tokens at various future offsets." This is a statistical estimation framing — each head is a regression model from hidden states to future tokens, and the quality of each head depends on how well it can extract long-range predictive information from a single representation. Clover shifts the framing to: "given the current hidden state and knowledge of intermediate tokens, predict the next token in sequence." This is an auto-regressive generation framing — the speculator is a miniature language model that generates tokens one at a time, conditioning on its own previous outputs. The difference is profound: in the statistical estimation framing, later heads are inevitably worse because they must predict tokens further into the future from the same information. In the auto-regressive framing, later heads can be just as accurate as earlier ones because they have strictly more information (the earlier heads' predictions). The paper's ablation results (Figure 7a) bear this out: the regressive architecture benefits later heads disproportionately (21.7–26.4% top-5 accuracy improvement for the third head vs. 11.7% for the first), precisely because those later heads were most starved of information in the independent-head design.

This is not an incremental tweak — it is a conceptual reframing of what the speculator should do. Medusa's heads are parallel regressors; Clover's heads are sequential generators. The paper provides evidence that this reframing matters most in the deployment regime the field had underexplored: large-batch, small-tree serving. In this regime, prior work had already pushed parallel decoding close to a compute ceiling (Figure 2), and the only way to break through is to increase the information density of the token tree rather than its size. The regressive framing directly targets information density: each path in a sequentially-conditioned tree represents a linguistically coherent continuation, making the tree's fixed node budget go further in terms of expected accepted tokens.

Compared to concurrent regressive approaches cited in Section 5 (Eagle, Hydra, Chimera), Clover's distinctive contribution at the idea level is not the observation that sequential dependencies matter — several groups independently arrived at that — but rather the explicit formulation of the regressive design as a response to the large-batch compute-bound regime and the corresponding focus on tree information density rather than raw speculation length as the optimization target.

---

Innovation 2: Cross-Attention as a Mechanism for Heterogeneous Information Fusion in Speculators

At the architectural level, Clover's key innovation is the use of cross-attention — rather than concatenation-then-MLP or self-attention — as the mechanism for merging sequential information from previous speculations with contextual information from the target model's hidden states. This is a specific design choice that carries significant conceptual weight, and the paper's ablation study (Figure 7a) provides direct evidence for why it matters.

The conceptual problem is this: the speculator must combine two very different kinds of information. The hidden states ($h_{i-1}$) encode the full input context — the conversation history, the prompt, everything the model knows about what has been said. The previous token embedding ($e_{i-1}$) encodes a single discrete fact: "the model just predicted token X." These are not simply two vectors to be blended — they have fundamentally different structures, dimensionalities, and semantic roles. An MLP that concatenates them and applies a fixed transformation implicitly assumes a static relationship: the same mixing weights apply regardless of what $e_{i-1}$ actually contains. But if the previous token is "Bayesian," the relevant contextual information for predicting the next token concerns mathematical and statistical continuation patterns; if the previous token is "the," the relevant contextual information concerns grammatical agreement and noun phrase structure. A static mixing function cannot adapt to these different information demands.

Cross-attention solves this by making the integration content-dependent. The query $Q_i$, derived from the hidden state, encodes a question: "what do I need to know about the previous token to predict what comes next, given everything I know about the context?" The key $K_i$, derived from the embedding, encodes: "here is what the previous token contains." The attention weights — which are a function of the interaction between $Q_i$ and $K_i$ — determine dynamically how much of each aspect of the previous token's information to incorporate. When the previous token is "Bayesian," the query can learn to attend strongly to embedding dimensions that correlate with mathematical domains; when the previous token is "the," the query can attend to dimensions that correlate with grammatical constraints. This dynamic routing is exactly what cross-attention is designed for, and it is fundamentally impossible in a concatenation-based MLP architecture.

The ablation results quantify this: replacing the Attention Decoder with an MLP regressive block costs 4.8–11.5% top-5 accuracy, with larger losses on later heads. This gap represents the value of content-dependent integration. The paper's initialization strategy for the Attention Decoder — Q and K initialized identically with Gaussian noise, V initialized to zero — further reveals the design philosophy: the model starts in a state where it ignores sequential information entirely (V = 0 means the residual connection dominates, $h_i \approx h_{i-1}$), then learns gradually when and how to incorporate the previous token's signal. This is a principled initialization scheme that reflects a hypothesis about the learning dynamics: the sequential signal should be incorporated cautiously, only where it demonstrably helps, rather than being forced into the hidden state from the start.

Compared to the regressive blocks in concurrent work — Eagle's transformer decoder layer, Hydra's additional attention block, Chimera's Trigram Encoder — Clover's cross-attention is distinct in being a purely heterogeneous fusion operator. It does not apply self-attention over a sequence of speculated tokens (which would be more expensive and potentially mix information in less targeted ways); it applies exactly one query against exactly one key-value pair. This minimal design reflects a deeper insight: the information the speculator needs from the previous token is limited and specific, not a full sequence context. Over-parameterizing this fusion step would add overhead without proportionate gain.

---

Innovation 3: The Augmenting Block as a Task-Adaptation Layer Between Pre-Training and Speculation Objectives

The Augmenting Block is conceptually interesting for a reason that goes beyond its modest accuracy contribution (4.9% top-5 accuracy improvement, per Figure 7b). It represents an architectural acknowledgment of a task mismatch between what the target model was trained to do and what the speculator needs from its representations.

The target model's hidden states are the product of pre-training (and possibly fine-tuning) for a single objective: next-token prediction. Every layer, every attention head, every neuron has been optimized to produce representations that are maximally informative about the immediate next token. But Clover's speculator asks a different question: "given this representation, predict tokens at positions +1, +2, and +3 simultaneously." These are not the same task. Information that is critical for predicting the immediate next token (e.g., the exact grammatical constraints on the next word) may be irrelevant or even distracting for predicting tokens two or three positions ahead (where different grammatical constraints apply). Conversely, information that predicts the topic or structure of the next several tokens may be latent in the hidden states but not expressed in a form that a simple MLP head can easily extract.

The Augmenting Block is a learned transformation that bridges this gap. By appending an additional transformer layer and fine-tuning it specifically for the multi-offset prediction task, the system can reorganize the hidden states to surface information that matters for longer-range speculation. The finding that attention contributes most of the gain (1.9–5.1% top-5 accuracy) while MLP contributes only ~1% (Figure 7b) is revealing: attention can redistribute information across the sequence, bringing forward context from earlier positions that is relevant for multi-step prediction, while position-wise MLP transformation alone cannot route information between tokens.

This is not a novel architectural primitive — adding layers is standard. What makes it an innovation is the diagnostic framing: the paper identifies the task mismatch as a concrete source of inefficiency in parallel decoding, measures it empirically through ablation, and shows that a single additional layer is sufficient to recover most of the lost performance. The initialization from the target model's last layer is a warm-start that says: "start from a representation that already works for next-token prediction, then adjust it for multi-token prediction." This framing suggests a broader principle for speculative decoding systems: the interface between the target model and the speculator should not be taken as a given (just whatever hidden states the model produces) — it is a design surface that can be optimized, and even small optimizations (a single additional transformer layer, ~1/N_layer overhead) can yield meaningful gains.

The Augmenting Block also represents a separation of concerns that is architecturally clean: the target model does what it was trained to do (produce next-token-optimized hidden states), and the Augmenting Block does the adaptation work. This means Clover can be attached to any pre-trained model without modifying it, and the adaptation cost is borne entirely by the small added component. In a landscape where model weights are increasingly treated as fixed assets (due to training cost, safety evaluations, and deployment consistency), this plug-and-play property is practically significant.

---

Innovation 4: The Large-Batch, Small-Tree Regime as a Distinct Evaluation and Optimization Target

While not a technical contribution in the architectural sense, the paper's explicit choice to evaluate in the large-batch, small-tree regime — and to argue that this regime represents real deployment conditions — is an intellectually significant framing move. It reorients the speculative decoding conversation away from what has been the dominant optimization target in prior work (maximum speedup at low batch sizes with large token trees) toward conditions that the paper claims are more representative of production serving.

The key diagnostic is Figure 2, which shows speculative decoding throughput as a function of the number of computed tokens. The curve rises, peaks, and then declines — an inflection point that represents the transition from memory-bound to compute-bound operation. In low-batch scenarios, the system is memory-bound, and speculative decoding helps by amortizing the memory cost of loading model weights across more tokens. But in high-batch scenarios — typical of serving systems that process many concurrent requests — the system is already compute-bound, and adding more speculative tokens per step eventually hurts because the verification computation cost outpaces the acceptance gain.

This inflection point creates a fundamentally different optimization landscape than what prior work explored. In the memory-bound regime (left side of Figure 2), the optimal strategy is to maximize the number of speculated tokens per step — bigger trees, more aggressive speculation, longer sequences. In the compute-bound regime (right side of Figure 2), the optimal strategy is to maximize the information density of each speculated token — smaller trees, higher hit rates, more efficient use of the verification computation budget.

The paper argues that prior work, including Medusa, was implicitly optimized for the memory-bound regime. Medusa's independent heads produce Cartesian product trees that are very large (exponentially many paths) but have low information density (many paths are linguistically nonsensical). This is fine when the system is memory-bound and tree size is unconstrained, but it becomes a liability in the compute-bound regime where tree size must be capped. Clover's regressive architecture is explicitly optimized for the opposite end of the curve: by conditioning later heads on earlier predictions, the tree paths are all linguistically coherent, meaning a small tree (size 4 in the paper's experiments) captures more of the probability mass that matters.

This reframing has implications beyond Clover. It suggests that speculative decoding research should be evaluated at multiple points along the batch-size/tree-size spectrum, not just at the single point of maximum speedup. A method that looks impressive at batch size 1 may be counterproductive at batch size 32, and vice versa. The paper provides evidence for this in Table 1, where the throughput advantage of both Medusa and Clover over vanilla decoding diminishes (and in some cases reverses) as batch size increases. At batch size 48 on the CA task, vanilla decoding actually outperforms both speculative methods on Baichuan Small (2217.4 tokens/second for vanilla vs. 1352.3 for Clover and 1246.0 for Medusa) — a striking illustration that speculative decoding is not universally beneficial and that the deployment regime fundamentally determines which method is optimal.

The paper's contribution here is not the observation that speculative decoding performance varies with batch size (this is inherent in the technique) but rather the deliberate choice to optimize for the compute-bound regime and the corresponding architectural decisions that follow from that choice. This is a systems-level design philosophy: identify the deployment constraint first, then design the architecture to operate optimally under that constraint. Clover's regressive architecture, its small-tree evaluation protocol, and its emphasis on head accuracy over tree size are all consequences of this philosophy. The paper makes a persuasive case that large-batch serving is the realistic scenario for production LLM deployment, and that methods optimized for this regime will deliver more practical value than methods optimized for low-batch benchmarks, even if the latter produce more impressive speedup numbers.

5. Experimental Analysis

Evaluation Methodology

• Dataset. Training uses the Baichuan internal supervised fine-tuning (SFT) dataset, containing approximately 0.15B tokens (95% Chinese). Evaluation is conducted on a separate internal Baichuan dataset covering 10 task categories: retrieval augmentation (RA), multi-turn conversation (MC), code, information process (IP), creation (CA), logical reasoning (RS), math, tabular (Tab), question answering (QA), and medical suggestion (Med). Each task contains 100 dialogues.
• Base model(s). Baichuan-Small (7B parameters) and Baichuan-Large (over 100B parameters), both from the Baichuan model family. The paper states these models are used to evaluate Clover across substantially different scales, with the large model providing evidence that gains become more pronounced as model size increases and the speculator module represents a smaller fraction of total computation.
• Metrics. Two primary metrics are used, following prior speculative decoding work: (1) extra tokens per step — the number of accepted speculative tokens beyond the first (actual) token generated by the target model, directly measuring speculator accuracy independent of the target model's own output; and (2) tokens per second — end-to-end throughput including all computation. For ablation studies, top-k accuracy of each speculative head is reported to provide intuitive insight into architectural component contributions.
• Baselines. Three baselines are used: (1) Vanilla decoding — standard auto-regressive decoding with no speculation; (2) Medusa (Baichuan) — the Medusa method (Cai et al., 2024) implemented on the Baichuan models with 3 heads, using the same inference engine, tree construction, and tree sampling algorithm as Clover; and (3) the paper's own Clover (Baichuan) is compared against both. The number of LM heads is fixed at 3 for all methods, ensuring architectural comparability at the head-count level.
• Generation budget / compute accounting. Compute is measured in two complementary ways: (1) token tree size — the number of speculated tokens verified per step, which directly controls the computational budget of the verification phase and is set to 4 for both Clover and Medusa in all experiments (this corresponds to using a small, fixed-size tree to match large-batch deployment constraints); and (2) tokens per second — which accounts for all computational overhead including the Augmenting Block, Attention Decoder, and Regressive Connection in Clover's case. The paper does not report FLOP counts but relies on end-to-end throughput to ensure that any additional computational cost from Clover's components is accounted for in the final comparison.
• Cross-validation / statistical protocol. The paper does not report cross-validation or statistical significance testing. All results appear to be single-run evaluations on the fixed 100-dialogue-per-task test set. The training-validation split for early stopping during speculator training is not described.

Main Quantitative Results

Extra Tokens Per Step: Clover Generates 50–76% More Accepted Speculative Tokens Than Medusa

Figure 6 presents the average number of extra tokens generated per decoding step (excluding the actual token produced by the target model) across all 10 task categories on Baichuan Large. This metric isolates speculator performance: it measures how many predicted tokens survive verification beyond the ground-truth next token.

The headline finding:

"Clover generates 50% – 76% more extra tokens per step than Medusa method on all tasks"

This is a substantial and consistent gain. The exact per-task values are not tabulated numerically in the main paper (only shown as a bar chart in Figure 6), but the visual clearly shows Clover exceeding Medusa on every category, with the relative advantage appearing largest on tasks like Math, RS (logical reasoning), and Med (medical suggestion).

What this metric means operationally: for each forward pass of the target model (the dominant cost in inference), Clover's verification step accepts significantly more speculative tokens on average than Medusa's does. Since the token tree size is fixed at 4 for both methods — meaning both speculate the same number of total tokens per step — the difference in accepted tokens directly reflects the higher accuracy of Clover's speculations. Clover achieves higher throughput because the same verification compute budget buys more accepted tokens.

The paper attributes this gain specifically to the regressive architecture:

"the regressive architecture not only improves the precision of the speculations but also generates a token tree with more comprehensive dependency information"

This is worth unpacking: with the tree size constrained to 4, both methods have the same number of speculated tokens to work with. But Clover's tree paths are sequentially coherent — each path represents a plausible multi-token continuation — while Medusa's Cartesian product tree contains many paths that are linguistically nonsensical combinations of independent head predictions. When the tree is aggressively pruned to size 4, Medusa is forced to discard many paths that might be correct (because its pruning algorithm, lacking sequential dependency information, cannot distinguish coherent from incoherent paths), while Clover's smaller effective branching factor means its top-ranked paths are more likely to be correct.

End-to-End Throughput: Clover Achieves Up to 91% Improvement Over Vanilla Decoding on Baichuan-Small and 146% on Baichuan-Large

Table 1 reports tokens per second for two representative tasks (CA and Math) across six batch sizes (4, 8, 16, 24, 32, 48) on both Baichuan-Small and Baichuan-Large. The full results for all 10 tasks on Baichuan-Large appear in Appendix Table 2.

Baichuan-Small, Math task (Table 1, rows for Small/Math):

Several patterns emerge:

First, Clover consistently outperforms Medusa at every batch size on this task, with relative improvements ranging from +4% to +26%. The average improvement across batch sizes where both speculative methods outperform vanilla (4–24) is approximately +18% over Medusa.

Second, the improvement over vanilla decoding is dramatic at small batch sizes (+91% at batch size 4) but diminishes and eventually reverses as batch size increases. At batch size 32, vanilla decoding (1202.4 tokens/s) surpasses Clover (1137.7 tokens/s), and at batch size 48, the gap widens with vanilla at 1874.0 vs. Clover at 1462.8. This is the phenomenon predicted in Figure 2: as batch size grows, the system becomes compute-bound, and the overhead of speculative decoding (verifying speculated tokens, most of which will be rejected) outweighs the benefit of reduced memory-load cycles per token. The paper explicitly contextualizes this:

"the advantage of both speculation decoding methods generally diminishes with increasing batch size. This is because speculative decoding with larger batch size is getting closer to the computational bounded."

Third, the paper notes a performance fluctuation artifact:

"The performance fluctuation is due to unpredictable random factors during the inference and a not fully optimized implementation of our engine."

This is an important caveat: the throughput numbers contain noise from implementation factors rather than algorithmic properties. For example, the jump in vanilla decoding throughput from 903.2 at batch size 32 to 2217.4 at batch size 48 on the Small/CA task (Table 1, top section) — a 2.45× increase — is almost certainly a measurement artifact, not a real property of the decoding algorithm. Such fluctuations make it difficult to draw precise quantitative conclusions about the exact magnitude of Clover's advantage at specific batch sizes, though the consistent direction (Clover > Medusa at all points) remains informative.

Baichuan-Large, Math task (Table 1, rows for Large/Math):

On the large model, three important differences from the small model emerge:

First, the improvement over vanilla decoding is larger at every batch size — +146% vs. +91% at batch size 4, and notably, vanilla decoding never surpasses Clover even at batch size 48 (+36% advantage persists). This supports the paper's claim that:

"The advantages of our system over Medusa are even more pronounced for larger model sizes, as the speculator module makes up a smaller proportion of the overall model."

The reasoning: on a larger model, the target model's forward pass is even more expensive relative to the speculator's overhead. Clover's additional components (Augmenting Block, Attention Decoder) add a roughly fixed absolute cost, but this cost is a smaller fraction of total inference time when the target model is larger. The accuracy gain from these components — which translates to more tokens accepted per verification step — therefore produces a larger net throughput improvement.

Second, the Clover-to-Medusa advantage is larger on average on the large model than on the small model. Across the six batch sizes for Math on Large, Clover improves over Medusa by +24% to +37%, with an average of approximately +31%. The paper reports the range across all scenarios as:

"at most $1.26\times$ and $1.47\times$ for Baichuan Small and Baichuan Large, respectively"

The 1.47× figure corresponds to a +47% throughput improvement, which the paper's footnote clarifies is calculated as "the ratio of Clover to Medusa throughput."

Third, the "diminishing returns with batch size" pattern is less severe on the large model. Even at batch size 48, Clover achieves 1067.8 tokens/s vs. 780.9 for vanilla (+36%). This suggests that larger models shift the compute-bound threshold further to the right — they can sustain speculative decoding benefits at higher batch sizes because the target model's verification pass is so expensive that the system remains memory-bound (and thus speculative-decoding-friendly) even as batch size increases.

Baichuan-Large, all 10 tasks (Appendix Table 2): The full table shows Clover outperforming Medusa at every batch size on every task, with only one exception: QA at batch size 48, where vanilla decoding (805.4 tokens/s) slightly edges out Clover (802.3 tokens/s) — a difference of approximately 0.4% that is well within the noise level mentioned by the authors. This near-universal dominance across 10 diverse task categories (60 comparisons: 10 tasks × 6 batch sizes, with the one QA exception being negligible) strengthens the claim that Clover's advantage over Medusa is robust to task variation.

The relative improvement over Medusa varies substantially across tasks. On some tasks (RA, MC), the advantage is modest at low batch sizes (e.g., RA at batch size 4: Clover 120.7 vs. Medusa 108.0, about +12%). On others (Tab, Med), it's substantial across all batch sizes (Tab at batch size 4: Clover 178.8 vs. Medusa 123.3, about +45%). The paper does not analyze this task-dependent variation — it is not clear whether some tasks inherently benefit more from sequential knowledge (perhaps tasks with more formulaic or predictable multi-token patterns) or whether this is noise from the small 100-dialogue-per-task sample.

The Relationship Between Tokens Per Step and Tokens Per Second

An important cross-validation between Figure 6 (extra tokens per step) and Table 1 (tokens per second) is that they should be consistent: if Clover generates 50–76% more extra tokens per step than Medusa, and the per-step computational cost is similar (same tree size, small overhead from Clover's additional components), then the throughput improvement should be proportional. The paper reports Clover-to-Medusa throughput improvements ranging from approximately 1.25× to 1.47× across scenarios. The 1.50×–1.76× implied by the extra-tokens-per-step metric (50–76% more) is slightly higher than the throughput range, which is expected — the extra tokens per step metric doesn't account for the additional computational cost of Clover's Augmenting Block and Attention Decoder, while tokens per second does. The throughput improvement being lower than the pure accuracy improvement by a moderate margin confirms that Clover's architectural overhead is real but modest relative to the accuracy gain.

Token Tree Size Diminishing Returns (Appendix A.2)

Figure 8 (Appendix A.2) explores how extra tokens per step scales with token tree size for both Clover and Medusa on the MC and Math tasks using Baichuan-Small. The key finding:

"As the token tree size grows larger, the acceptance length is still increases but at a slower rate. Note that the horizontal axis in Figure 8 is in exponential scale, while the vertical axis is linear."

The exponential-to-linear relationship means that doubling the tree size (a multiplicative increase) produces only an additive increase in accepted tokens. For example, moving from tree size 2 to 4 might add 0.5 extra tokens per step, but moving from 4 to 8 might add only 0.3 more. This diminishing returns curve is the empirical justification for the paper's choice to set tree size to 4 in the main evaluation: beyond this point, the additional computation from larger trees pushes the system toward compute saturation (especially in large-batch scenarios) without proportionate throughput gains.

The paper does not show Clover-vs-Medusa comparisons at different tree sizes — Figure 8 appears to show only Clover's scaling behavior — so we cannot determine whether Clover's diminishing returns curve is flatter or steeper than Medusa's. This would have been informative: if Clover's sequentially-coherent tree achieves most of its potential acceptance length at smaller tree sizes (because its top-ranked paths are concentrated on correct continuations), then the advantage over Medusa might be even larger at tree sizes smaller than 4 (where Medusa's combinatorial explosion forces it to discard even more correct paths). Conversely, at very large tree sizes, the gap might close because both methods eventually cover most plausible paths. The paper does not explore this tradeoff space.

Ablation Studies and Robustness Checks

All ablation results are reported in Figure 7, using top-k accuracy of each of the three heads as the metric on Baichuan-Small. Top-k accuracy measures the fraction of times the correct token is among the top-k predictions of a given head — a direct measure of speculator quality that is independent of the verification algorithm and tree construction.

Ablation on architectural components (Figure 7a): Starting from the full Clover configuration, components are removed one at a time to isolate their contributions. The paper reports top-5 accuracy degradation for each head:

• Full Clover → Remove Attention Decoder (use MLP as regressive block instead): Head 1 loses 4.8%, head 2 loses 9.0%, head 3 loses 11.5% top-5 accuracy. This is the largest single-component ablation effect, confirming that the cross-attention mechanism is the most important architectural addition. The increasing loss from head 1 to head 3 is consistent with the paper's claim that sequential knowledge is most valuable for later positions, since the cross-attention's role is precisely to propagate sequential information forward.

• Remove Attention Decoder → Further remove Regressive Connection (now no sequential information flow between heads): Head 1 loses an additional 2.6%, head 2 loses 4.6%, head 3 loses 6.0%. This measures the value of the feedback pathway itself, separate from the specific mechanism (attention vs. MLP) used to process it. The loss is smaller than the Attention Decoder removal, suggesting that having a regressive connection at all is important, but having the right regressive mechanism (cross-attention) is even more important. Again, losses increase with head position.

• Remove Regressive Connection → Further remove Augmenting Block (now identical to Medusa): Head 1 loses 4.3%, head 2 loses 8.1%, head 3 loses 8.9%. The Augmenting Block contributes roughly equally to all three head positions (slightly more to later heads), which makes sense — it enriches the hidden states fed to the entire speculation chain, so all heads benefit from better input representations.

• Total Clover vs. Medusa (removing all three components): The cumulative accuracy loss across all three removals is: head 1: 4.8 + 2.6 + 4.3 = 11.7% top-5 accuracy loss; head 2: 9.0 + 4.6 + 8.1 = 21.7%; head 3: 11.5 + 6.0 + 8.9 = 26.4%. The paper cites these cumulative figures to support its claim:

"Clover approach brings sequential knowledge from pre-generated speculative tokens as well as the input sentence, improving performance of all speculative heads, especially the latter two."

The numbers are additive because the ablation is performed sequentially — removing each additional component degrades the already-degraded model further — but the paper reports the full Clover-to-Medusa difference directly, which is the most interpretable comparison.

A limitation of this ablation methodology is that it does not test interactions between components. The value of the Attention Decoder might depend on the Augmenting Block being present (if enriched hidden states make better queries, the cross-attention might be more effective), or vice versa. The sequential removal approach assumes independence of effects, which may not hold.

Ablation on Augmenting Block architecture (Figure 7b): Four configurations are compared, all within the full Clover framework (with Regressive Connection and Attention Decoder present):

• Full transformer block (attention + MLP): Baseline, reported as contributing 4.9%, 9.0%, 9.4% top-5 accuracy compared to no Augmenting Block (the numbers are the gap between "Transformer Block" and "No Augmenting Block" in Figure 7b).
• Attention block only: Contributes 1.9%, 4.4%, 5.1% top-5 accuracy compared to no Augmenting Block — this captures the majority (roughly 39–54%) of the full transformer block's benefit. The paper concludes that "the attention block plays the major role."
• MLP block only: Contributes 1.0%, 1.2%, 0.7% — a small but consistent gain.
• No Augmenting Block: The baseline from which contributions are measured.

The paper draws two conclusions. First, attention is the dominant mechanism in the Augmenting Block, because it can redistribute information across sequence positions. Second, the MLP is optional (providing only ~1% gain) but included because its overhead is negligible. This is a practical engineering decision, not a theoretical claim.

A missing ablation that would have strengthened this analysis: comparing the Augmenting Block's attention-only configuration against a configuration where the Augmenting Block is removed but an extra attention layer is added before the Attention Decoder. This would distinguish between two hypotheses: (1) the Augmenting Block helps because any additional computation enriches the hidden states, vs. (2) the Augmenting Block helps specifically because it is positioned between the target model and the speculator, allowing it to adapt the representations for the multi-offset prediction task.

Robustness check across tasks (Appendix Table 2): The full 10-task results on Baichuan-Large serve as an implicit robustness check. The consistent Clover > Medusa ordering across 60 comparisons (6 batch sizes × 10 tasks) — with the single negligible exception noted above — demonstrates that the architectural advantages are not specific to particular task types. The paper does not analyze task-specific patterns in the relative improvement, which would have been informative: tasks that involve more formulaic completions (Code, Math) might show larger gains from sequential knowledge than tasks with more open-ended responses (CA, MC). The bar chart in Figure 6, showing extra tokens per step, provides some evidence for this: Math shows relatively high extra tokens per step for Clover, while RA and MC show lower absolute numbers for both methods, though the relative Clover-to-Medusa ratio appears roughly consistent across tasks.

Negative result — MLP as regressive block: The ablation in Figure 7a includes a key negative result: using an MLP layer instead of the Attention Decoder as the regressive block causes substantial accuracy degradation (4.8–11.5% top-5 accuracy loss). The paper explains this as the MLP's inability to "distinguish different embedding vectors," but a more precise interpretation is that concatenation-based fusion forces a static mixing of information sources, whereas attention provides dynamic, content-dependent integration. This negative result justifies the cross-attention design choice and provides guidance for future work: not all regressive mechanisms are equally effective, and the specific choice of fusion operator matters substantially.

Training stability — ReST experiment (Appendix K?): The paper mentions (in Section 6, the conclusion) that a ReST-based optimization of the revision model degraded performance, but this detail is beyond the scope of the current experimental analysis section as described. If Appendix K is within scope, this would be a notable negative result about the sensitivity of the sequential training procedure.

Critical Assessment

Does Clover genuinely demonstrate that sequential knowledge improves speculative decoding throughput?

The evidence for this claim is strong in the regime tested. Clover consistently generates more extra tokens per step (Figure 6) and achieves higher throughput (Tables 1, 2) than Medusa across two model sizes and 10 task categories. The ablation study (Figure 7a) isolates the specific contribution of the regressive mechanism, showing that the Attention Decoder and Regressive Connection together account for the majority of the accuracy improvement, particularly on later heads where sequential dependencies are most informative. The causal chain is clear: regressive architecture → higher speculator accuracy → more tokens accepted per verification step → higher throughput → larger gains on larger models where speculator overhead is proportionally smaller.

However, several qualifications are necessary. First, the evidence is limited to a single model family (Baichuan). The paper does not demonstrate that the architectural improvements generalize to models with different training procedures, vocabulary sizes, or architectural choices (e.g., grouped-query attention vs. full multi-head attention). The Attention Decoder's effectiveness might depend on properties of Baichuan's hidden state representations that do not hold universally.

Second, the paper's central throughput improvement claim — "up to 91% on Baichuan-Small and 146% on Baichuan-Large" improvement over vanilla decoding — refers to the best-case batch size (4) and specific tasks, not average improvement. At more realistic batch sizes (24–48), the improvement is more modest (e.g., 19–67% on Small, 23–62% on Large for the Math task), and at the largest batch sizes, vanilla decoding sometimes outperforms both Clover and Medusa. The "up to" framing is standard in systems papers, but readers should understand that deployment gains depend critically on the operating point.

Third, the paper reports only single-run results with a self-acknowledged "not fully optimized implementation." The throughput numbers contain visible artifacts (the 2.45× jump in vanilla throughput from batch size 32 to 48 on Small/CA) that suggest measurement noise. Without multiple runs and variance estimates, it is difficult to assess whether the reported 4–26% Clover-to-Medusa throughput improvements on Baichuan-Small are statistically reliable or within the noise floor of the measurement system.

Does the paper demonstrate that the large-batch, small-tree regime is where Clover's advantages matter most?

The paper argues this conceptually (Figure 2, Section 1) and evaluates in this regime (tree size fixed at 4), but it does not provide the key comparison that would substantiate the claim: Clover vs. Medusa at varying tree sizes and batch sizes, showing that Clover's relative advantage is larger at small tree sizes and large batch sizes than at large tree sizes and small batch sizes. Without this, the claim that Clover is specifically optimized for the large-batch regime — rather than simply being generally better than Medusa — is unverified.

Figure 8 shows extra tokens per step vs. tree size for Clover only, not for Medusa. A comparative plot would reveal whether Clover achieves a larger fraction of its maximum acceptance length at small tree sizes (suggesting higher information density) or whether the curves are simply shifted upward (suggesting a uniform accuracy improvement that would benefit any regime equally). The paper's argument about information density and tree coherence is theoretically plausible but empirically untested.

The batch-size sweep in Table 1 partially addresses this: the relative Clover-to-Medusa advantage does not show a clear monotonic trend with batch size. On Baichuan-Large Math, the Clover-to-Medusa ratio is roughly 1.30 at batch size 4, 1.27 at batch size 8, 1.37 at batch size 16, 1.28 at batch size 24, 1.24 at batch size 32, and 1.37 at batch size 48 — hovering around 1.30 with no clear increasing or decreasing pattern. If Clover were specifically optimized for large batches, one would expect the ratio to increase with batch size. The data do not show this pattern.

Is the ablation study sufficient to establish the independent contribution of each component?

The sequential removal ablation (Figure 7a) measures each component's contribution when removed from the full system, but it does not test whether components interact synergistically. The Augmenting Block might be more valuable in the presence of the Attention Decoder (because richer hidden states produce better queries, making cross-attention more effective), or the Attention Decoder might partially compensate for the absence of the Augmenting Block (by extracting relevant information from the previous token that compensates for impoverished hidden states). The additive reporting of accuracy losses (11.7%, 21.7%, 26.4% cumulative) implicitly assumes independence.

A more rigorous ablation would test all 8 combinations of the three binary components (present/absent) and report the full interaction table. With 3 components and 3 head positions, this is 24 measurements — feasible to include. The paper's sequential approach provides a directional signal but does not rule out interaction effects that could change the interpretation of relative component importance.

The paper also does not ablate the number of heads (fixed at 3), the specific initialization scheme for the Attention Decoder (Q and K initialized identically with Gaussian noise, V to zero), or the choice of using the shared LM head for embedding extraction vs. a separate embedding table. These are design decisions that could interact with the regressive mechanism's effectiveness.

Does the paper adequately address the latency overhead of serialization?

The paper acknowledges the latency tradeoff explicitly:

"Although the critical path of the computation becomes proportional to the depth of the speculation and loses a certain amount of parallelism, the increase in speculation accuracy rather improves the overall latency."

But this claim is supported only by end-to-end throughput numbers, which conflate latency and throughput effects. In a batched serving system, throughput (tokens per second across all requests) can improve even if per-request latency (time to generate a complete response) stays the same or degrades, because batching allows overlapping computation. The paper does not report per-request latency, time-to-first-token, or any metric that would isolate the serialization overhead. For latency-sensitive applications (where a user is waiting for a response), the serial regressive chain — which forces head i to wait for head i−1 — could increase wall-clock time even if throughput improves. This is a genuine omission in a paper whose primary motivation is accelerating LLM inference for real-time serving.

Missing comparisons and experiments that would strengthen the paper:

• Comparison against other regressive speculators (Eagle, Hydra, Chimera): The paper positions Clover within a landscape of concurrent approaches in Section 5 but does not provide any comparative evaluation against them. While these methods were developed concurrently and direct comparison may not have been feasible, the claims about Clover's specific advantages (cross-attention over concatenation-MLP, Augmenting Block design) remain relative to Medusa only, not to other regressive approaches.

• Evaluation on standard public benchmarks: The paper uses only internal Baichuan datasets for both training and evaluation. Reproducibility and comparability with prior work would be enhanced by evaluation on standard benchmarks like MT-Bench, HumanEval, or GSM8K that are commonly used in speculative decoding papers.

• Analysis of failure modes: Figure 6 shows extra tokens per step — tokens that are accepted. It would be equally informative to analyze rejected tokens: what kinds of speculations does Clover get wrong, and are the error patterns different from Medusa? Do Clover's later heads ever compound errors from earlier heads (a potential downside of the regressive architecture)?

• Sensitivity to head count: All experiments use 3 heads. How does the Clover-to-Medusa gap change with more heads (5, 7)? If sequential knowledge benefits later heads disproportionally, as the paper claims, then the advantage should grow with more heads. Conversely, the serial chain latency grows linearly with head count, so at some point the latency cost might outweigh the accuracy gain.

• Evaluation at batch size 1: The paper's Figure 2 motivates the large-batch focus, but including batch-size-1 results would establish a baseline for how Clover performs in the memory-bound regime where prior work was optimized. If Clover underperforms Medusa at batch size 1 (due to serialization overhead that isn't amortized by batching), that would strengthen the paper's claim of being specifically designed for large-batch deployment. If Clover still outperforms Medusa at batch size 1, then the regressive architecture is simply better across the board, and the large-batch framing is less distinctive.

6. Limitations and Trade-offs

6.1 Evaluation Restricted to a Single Model Family with No Public Benchmark Results

The assumption or constraint. All experiments — both training and evaluation — are conducted exclusively on Baichuan models (Baichuan-Small at 7B and Baichuan-Large at over 100B parameters) using internal Baichuan datasets. The training data is described as the "Baichuan internal supervised fine-tuning (SFT) dataset, containing approximately 0.15B tokens, 95% of which are Chinese" (Section 4.1). The evaluation data is "another internal Baichuan dataset" spanning 10 task categories with 100 dialogues each. No results are reported on any public benchmark (e.g., MT-Bench, HumanEval, GSM8K, MATH) or any non-Baichuan model family (e.g., LLaMA, Mistral, Qwen). The paper does not claim that Clover's architectural choices are specific to Baichuan's design, but it provides no evidence of generality.

The consequence. A practitioner considering Clover for a non-Baichuan model cannot estimate expected gains from this paper. The Attention Decoder's cross-attention mechanism depends on properties of the target model's hidden states — their dimensionality, their distributional characteristics, how they encode long-range dependencies. If Baichuan's hidden states have particular properties (e.g., unusually strong positional encoding, specific normalization schemes, or training-data-induced representational geometries) that make cross-attention effective for merging sequential information, Clover might not transfer. Similarly, the Augmenting Block is initialized from the target model's last transformer layer — if a different model family's layers have different dimensionalities or architectural patterns (e.g., grouped-query attention vs. full multi-head attention, different normalization placement), the initialization strategy may need adaptation.

The 95% Chinese training data raises a further concern: the paper claims Clover generates 50–76% more extra tokens per step than Medusa (Section 4.2, Figure 6), but if these gains are partially driven by linguistic properties of Chinese (e.g., character-level vs. word-level tokenization, different sequential dependency patterns in Chinese syntax), the numbers may not transfer to English-dominant or code-dominant deployments. The paper does not analyze whether Clover's gains are language-dependent.

What evidence exists in the paper. The paper provides no cross-model or cross-language evaluation. Section 5 (Related Works) positions Clover among several concurrent regressive approaches (Eagle, Hydra, Chimera), but no comparative evaluation against any of them — or even against Medusa on a non-Baichuan model — is presented. The paper's claim that Baichuan models are used because they are "representative of the capabilities of many contemporary LLMs" (Section 1 framing, though this exact quote appears in the Executive Summary paraphrase rather than the paper) is asserted, not demonstrated. Appendix Table 2 shows consistent Clover > Medusa across all 10 internal task categories, which demonstrates robustness to task variation within the Baichuan evaluation set, but says nothing about robustness to model architecture variation.

Mitigation status. The paper does not address this limitation. No public benchmark results are provided. No code or model weights are released (the paper contains no repository link or artifact availability statement). The limitation is structural — reproducing these results requires access to Baichuan models and internal Baichuan datasets, neither of which is publicly available under the terms described in the paper. A practitioner seeking to evaluate whether Clover would benefit their deployment on a different model must implement Clover from the paper's description and run their own evaluation, with no calibration point against the paper's reported numbers.

6.2 Serial Speculation Chain Introduces Latency That Throughput Numbers Conceal

The assumption or constraint. Clover's Regressive Connection creates a serial dependency: head $i$ must wait for head $i-1$ to produce a token before it can begin computation. The paper explicitly acknowledges this tradeoff in Section 3.1:

"Although the critical path of the computation becomes proportional to the depth of the speculation and loses a certain amount of parallelism, the increase in speculation accuracy rather improves the overall latency."

The claim is that the accuracy gain (more tokens accepted per verification step, reducing the number of target-model forward passes needed) outweighs the added serialization cost within the speculation phase. However, the paper's primary evaluation metric is throughput (tokens per second, Tables 1 and 2) in batched serving scenarios. Throughput and latency are not the same quantity, and in batched settings, throughput can improve even as per-request latency degrades because batching allows overlapping computation across requests.

The consequence. For latency-sensitive applications — interactive chatbots, real-time code completion, voice assistants — per-request wall-clock time matters more than aggregate throughput. Clover's serial chain of speculation heads adds a latency penalty that the paper never quantifies. Specifically: with $K$ heads, Clover must perform $K$ sequential steps of (Attention Decoder + MLP head + embedding extraction) before the target model can begin verification. Each step is individually lightweight, but they cannot be parallelized. Medusa's independent heads, by contrast, can compute all $K$ predictions in parallel from the same hidden state. For a sufficiently latency-sensitive deployment, this serial overhead — even if small in absolute terms — could make Clover slower than Medusa on a per-request basis even if it achieves higher tokens-per-second in aggregate.

The problem compounds with more heads. If a practitioner wants to increase speculation depth (e.g., 5 or 7 heads instead of 3), Clover's serial chain grows proportionally longer, while Medusa's parallel heads add no additional serial depth (only more parallel MLP computations). The paper evaluates only 3-head configurations, leaving the latency-vs-head-count tradeoff unexplored. Figure 7a's results showing that Clover's advantage grows with head position (later heads benefit more from sequential knowledge) create a tension: more heads would likely increase Clover's accuracy advantage over Medusa, but would also increase the serial latency penalty. The paper provides no framework for determining the optimal head count in a latency-constrained setting.

What evidence exists in the paper. The paper reports no latency-specific metrics. There is no time-to-first-token measurement, no per-request latency distribution, no analysis of how the serial speculation chain affects worst-case or tail latency. Tables 1 and 2 report only throughput (tokens/second) at various batch sizes. The paper's Figure 2 (which shows throughput declining at high token counts due to compute-bound behavior) is about computational saturation, not serial latency — it would apply to any speculative decoding method regardless of whether the speculator is serial or parallel.

The paper's claim that "the increase in speculation accuracy rather improves the overall latency" (Section 3.1) is not supported by any latency measurement. It is an inference from the throughput numbers, assuming that higher tokens-per-second implies lower per-request latency. This assumption holds in a single-request, throughput-limited setting but not necessarily in batched serving where multiple requests' computation can be interleaved.

Mitigation status. Not addressed. The paper does not acknowledge latency as a distinct concern from throughput. No latency-aware evaluation protocol is proposed. The limitation is inherent to the regressive architecture: serial dependency is the mechanism by which Clover achieves its accuracy gains, and this dependency cannot be eliminated without abandoning the Regressive Connection entirely. A partial mitigation — using a smaller number of heads or accepting the serial depth — is available but not analyzed.

6.3 Difficulty Estimation and Deployment-Regime Selection Are Unaddressed

The assumption or constraint. The paper argues that Clover is specifically designed for large-batch, small-tree serving scenarios (Section 1, with motivating Figure 2), and evaluates exclusively in this regime (batch sizes 4–48, token tree size fixed at 4). But the paper provides no mechanism for a practitioner to determine, at deployment time, whether their specific workload is in the regime where Clover (or speculative decoding in general) is beneficial. Figure 2 indicates that speculative decoding speedup "reaches an inflection point and gradually diminishes due to computational limitations" as computed tokens increase, and the experimental results in Table 1 confirm this: at batch size 48 on Baichuan-Small CA, vanilla decoding (2217.4 tokens/s) substantially outperforms both Clover (1352.3 tokens/s) and Medusa (1246.0 tokens/s). The paper does not discuss how a serving system should decide whether to enable speculative decoding — or which method to use — based on current load conditions.

The consequence. A practitioner deploying Clover in a production system cannot rely on the paper for guidance on when to use it. The throughput numbers in Table 1 show that speculative decoding (both Clover and Medusa) is not universally beneficial — it hurts throughput at high batch sizes on Baichuan-Small. The crossover point varies by model size (Baichuan-Large retains speculative decoding benefits at higher batch sizes) and by task (some tasks in Appendix Table 2 show diminishing returns earlier than others). Without a principled selection criterion, the practitioner must either (a) run their own comprehensive sweep of batch sizes and tasks to find the crossover point for their specific workload, or (b) implement an adaptive system that switches between speculative and vanilla decoding based on runtime conditions — a significant engineering effort that the paper does not address.

More subtly, the paper provides no guidance on selecting the token tree size for a given deployment. Tree size 4 is used throughout the evaluation, with Appendix A.2 (Figure 8) showing diminishing returns for larger trees. But the optimal tree size likely depends on batch size, model size, and hardware characteristics (memory bandwidth vs. compute capacity). A practitioner with different hardware than the paper's A800/H800 GPUs cannot determine the right tree size from the information provided.

What evidence exists in the paper. Table 1 provides the key evidence: the crossover points where vanilla decoding surpasses speculative decoding. On Baichuan-Small Math, vanilla overtakes Clover at batch size 32 (1202.4 vs. 1137.7 tokens/s). On Baichuan-Small CA, vanilla overtakes at batch size 48 (2217.4 vs. 1352.3). On Baichuan-Large, vanilla never overtakes Clover within the tested range, but the gap narrows (e.g., CA at batch size 48: 938.2 vs. 887.8, only +5%). These crossover points are presented as observations, not as the basis for a decision framework. The paper does not discuss them in the main text beyond noting that "the advantage of both speculation decoding methods generally diminishes with increasing batch size."

Figure 2, which motivates the large-batch focus, is described as showing behavior "on a model with approximately 30B parameters, supposing speculation length is 5 with 0.4 acceptance rate." This is a different model and speculation configuration from the main experiments, and the figure is illustrative rather than experimentally validated — it shows a conceptual curve, not measured data points.

Mitigation status. Not addressed. The paper does not propose an adaptive deployment strategy, a model of the compute-bound vs. memory-bound transition, or even heuristics for when to enable speculative decoding. The limitation is acknowledged only implicitly through the presentation of diminishing-returns data. Future work on dynamic speculation policies — adaptively enabling/disabling speculation or adjusting tree size based on runtime load measurements — would directly address this gap, but the paper does not outline such an approach.

6.4 Computation Overhead of the Augmenting Block and Attention Decoder Is Not Isolated or Characterized

The assumption or constraint. Clover adds three computational components beyond the frozen target model: the Augmenting Block (an additional transformer layer), the Attention Decoder (cross-attention at each speculation step), and the Regressive Connection (embedding extraction from the shared LM head). The paper claims these have "negligible overhead" (Section 3.2) and that "the gain in head accuracy from the additional components proposed in Clover outweighs their computational overhead" (Section 4.2). However, the paper never reports the actual computational cost of these components — no FLOP counts, no latency breakdown, no memory measurements. The overhead is implicitly accounted for in the end-to-end tokens-per-second metric, but this conflates the overhead with the accuracy gain, making it impossible to determine how much of Clover's throughput advantage comes from better speculation accuracy vs. how much is lost to the added per-step computation.

The consequence. A practitioner cannot determine whether Clover's components are cost-effective for their specific hardware configuration. The Augmenting Block is "approximately 1/N_layer of inference time" (Section 3.3) — for a 40-layer model, ~2.5% overhead. The Attention Decoder's cross-attention operates on "only two vectors per request or beam" (Section 3.2), which the paper claims is negligible. But these costs are incurred on every decoding step, not just on steps where speculation succeeds. If the speculation accuracy gain is small on a particular task or model, the fixed per-step overhead might dominate, making Clover slower than Medusa despite higher per-step acceptance. The paper provides no way to estimate this breakeven point.

The memory overhead is similarly uncharacterized. The Augmenting Block adds one full transformer layer's worth of parameters (attention weights, MLP weights, normalization parameters). The Attention Decoder adds $W_Q$, $W_K$, and $W_V$ matrices (each of size hidden_size × d_k or hidden_size × d_v). The paper reports total trainable parameters as "approximately 0.2B and 2B" for Baichuan-Small and Baichuan-Large respectively (Section 4.1), but does not break this down by component. For GPU-memory-constrained deployments (e.g., running on edge devices or with large batch sizes that already strain memory), the additional parameter footprint could matter even if the FLOP overhead is small.

What evidence exists in the paper. The ablation study (Figure 7) provides indirect evidence about the cost-benefit tradeoff: it shows accuracy improvements from each component, and the end-to-end throughput numbers (Tables 1, 2) show that Clover's net throughput exceeds Medusa's despite the overhead. But there is no direct cost measurement. The paper does not report:

• Time per decoding step broken down by component (target model forward pass, Augmenting Block, Attention Decoder, MLP heads, verification).
• Memory usage of Clover components vs. the target model.
• Throughput at a fixed accuracy target (e.g., what throughput does Medusa achieve if you increase its tree size to match Clover's acceptance rate? This would isolate whether Clover's advantage comes from better accuracy or from a more efficient accuracy-vs-cost operating point).

Mitigation status. Partially addressed through the end-to-end throughput comparison, which shows that the net effect (accuracy gain minus overhead) is positive across all tested scenarios. However, the lack of cost decomposition means that a practitioner cannot predict whether the net effect would remain positive under different conditions (different hardware, different model architecture, different batch sizes, different tasks). The paper does not suggest future work on cost characterization or provide profiling data that would enable such predictions.

6.5 Fixed Head Count and Fixed Tree Size with No Sensitivity Analysis to Either Parameter

The assumption or constraint. All experiments use exactly 3 speculative heads and a token tree size of exactly 4. The paper justifies the tree size choice in Appendix A.2 (Figure 8) by showing diminishing returns from larger trees, but this analysis is shown only for Clover (not Medusa) on two tasks (MC and Math) using only Baichuan-Small. The head count of 3 receives no justification at all — it appears in Section 4.1 as "the number of lm head is 3" without explanation. The paper's central finding — that later heads benefit disproportionately from sequential knowledge (21.7–26.4% top-5 accuracy gain for head 3 vs. 11.7% for head 1, from the ablation in Figure 7a) — strongly suggests that head count is a critical hyperparameter. If sequential knowledge matters more for later positions, adding more heads should increase Clover's relative advantage over Medusa, but only up to the point where the serial latency chain becomes prohibitive.

The consequence. The paper provides no guidance on the most important deployment decision a practitioner faces: how many heads to use. With 3 heads, Clover's serial chain depth is 3 (plus the initial Augmenting Block pass). With 5 or 7 heads, the depth grows to 5 or 7, increasing the serial latency linearly. The accuracy gain from additional heads — unmeasured in the paper — would need to offset this latency cost. Without head-count sensitivity data, a practitioner cannot optimize this tradeoff.

The tree size of 4 is similarly under-analyzed. Figure 8 shows Clover's extra tokens per step increasing with tree size on a log-linear scale, but the paper does not show the corresponding throughput numbers at different tree sizes. A tree size of 4 might be optimal for the specific batch sizes and hardware tested (A800/H800 GPUs at batch sizes 4–48), but a practitioner with different hardware or batch sizes has no basis for selecting a different value. The paper's claim that Clover's regressive architecture produces trees with "greater information density" (Section 3.1) implies that Clover should achieve a larger fraction of its maximum acceptance at smaller tree sizes than Medusa — a directly testable prediction that would strengthen the paper's central argument. But no comparative tree-size sweep between Clover and Medusa is provided.

What evidence exists in the paper. Figure 8 provides Clover-only tree-size scaling on only two tasks. The ablation in Figure 7a shows the per-head accuracy pattern that motivates the "more heads benefit more" claim. The paper reports that the head count was fixed for "fairness of the comparison" (Section 4.1: "the same inference engine, tree construction and tree sampling algorithm are used for all scenarios"), but fairness in comparison does not require fixing a parameter that the proposed method is specifically designed to exploit more effectively.

Mitigation status. Not addressed. The paper does not discuss head count as a tunable parameter, does not report results with different head counts, and does not provide a heuristic or model for selecting the optimal head count given a latency or throughput target. The paper's conclusion that "the auto-regressive mechanism is an effective approach to improve the accuracy of speculation" (Section 6) is supported at the tested head count of 3, but the scaling behavior of this mechanism to deeper speculation chains remains unknown.

6.6 Single-Run Evaluation with Acknowledged Implementation Artifacts

The assumption or constraint. The paper reports throughput numbers from what appears to be a single evaluation run per configuration. No confidence intervals, standard deviations, or multiple-run averages are reported. The paper explicitly acknowledges that the implementation is not fully optimized:

"The performance fluctuation is due to unpredictable random factors during the inference and a not fully optimized implementation of our engine." (Section 4.2, Table 1 footnote)

This acknowledgment appears in the context of explaining anomalous throughput numbers in Table 1, such as the Baichuan-Small CA task where vanilla decoding jumps from 903.2 tokens/s at batch size 32 to 2217.4 tokens/s at batch size 48 — a 2.45× increase that is almost certainly a measurement artifact, not a real property of auto-regressive decoding.

The consequence. The reported Clover-to-Medusa throughput improvements — ranging from approximately 4% to 37% in Table 1 — are based on point estimates with unknown variance. If the measurement noise is on the order of 10–20% (which the anomalous jumps in Table 1 suggest is plausible), several of the smaller reported improvements (e.g., +4% at Baichuan-Small Math batch size 16, where Clover achieves 673.3 vs. Medusa's 645.2) may not be statistically distinguishable from zero. The paper's strongest quantitative claims — "up to 91% on Baichuan-Small and 146% on Baichuan-Large" improvement over vanilla decoding, and "up to 37% on Baichuan-Small and 57% on Baichuan-Large" over Medusa — are maxima across batch sizes and tasks, reported without variance estimates. A practitioner cannot determine whether these maxima represent reliable performance differences or favorable draws from a noisy measurement process.

The implementation quality issue compounds this uncertainty. An unoptimized inference engine may introduce bottlenecks (e.g., kernel launch overhead, memory allocation patterns, suboptimal CUDA graph usage) that disproportionately affect one method over another. If Clover's additional components (Augmenting Block, Attention Decoder) trigger different code paths than Medusa's simpler architecture, implementation artifacts could bias the comparison in either direction. The paper does not describe the inference engine implementation, the GPU kernel configurations, or any profiling that would allow a reader to assess whether the comparison is confounded by engineering factors rather than algorithmic ones.

What evidence exists in the paper. The anomalous throughput numbers in Table 1 serve as the primary evidence of measurement noise. Specifically:

• Baichuan-Small CA: vanilla throughput jumps from 903.2 (bs=32) to 2217.4 (bs=48) — a 2.45× increase for a 1.5× increase in batch size, which is physically implausible for a compute-bound operation.
• Baichuan-Small Math: vanilla throughput goes from 591.6 (bs=24) to 1202.4 (bs=32) to 1874.0 (bs=48), with speculative methods showing much smaller increases over the same range. The vanilla curve is not monotonic with respect to batch size in a way that suggests measurement instability.

These artifacts are noted by the authors but not investigated. No analysis is provided of their potential impact on the Clover-vs-Medusa comparison at the affected batch sizes.

Mitigation status. The paper acknowledges the issue in a single sentence but does not address it methodologically — no repeat measurements, no variance reporting, no profiling to identify the source of fluctuations. The authors' characterization of the engine as "not fully optimized" suggests that future work with a production-quality implementation might resolve these artifacts, but this does not help a practitioner evaluating whether to adopt Clover based on the current evidence. The limitation is particularly concerning for the claims about specific percentage improvements, which are the paper's headline results and the basis for its assertion of Clover's superiority over Medusa.

7. Implications and Future Directions

How This Work Changes the Landscape

Clover does not introduce a new paradigm for speculative decoding — it operates squarely within the established framework of lightweight integrated speculators attached to a frozen target model. Rather, it makes a diagnostic and architectural refinement that shifts what the field should consider the default design for parallel decoding heads. The significance lies not in inventing regressive speculators (several concurrent approaches did this independently, as Section 5 acknowledges) but in providing a clean, well-ablated demonstration that the independence assumption in Medusa-style parallel heads is the primary bottleneck, and that breaking it with a specific mechanism — cross-attention rather than concatenation — yields gains that are large (50–76% more accepted tokens per step, Figure 6), consistent across model sizes (7B to 100B+), and, critically, most pronounced at later head positions where information starvation is most severe (21.7–26.4% top-5 accuracy gain for the third head vs. 11.7% for the first, Figure 7a).

This matters for the field in three specific ways:

First, it reframes the speculator design problem from statistical estimation to sequential generation. Before Clover, the implicit mental model for parallel decoding was: "given a rich hidden state, train regressors to predict tokens at various offsets." This framing treats head accuracy as a function of representation quality — better hidden states → better predictions. Clover demonstrates that even with the same hidden states (the Augmenting Block is separable from the regressive mechanism, as the ablation shows), letting later heads condition on earlier predictions provides a larger accuracy boost than enriching the hidden states themselves. This suggests that the dominant source of error in Medusa is not insufficient representational capacity but structural information blockage — later heads are denied access to intermediate tokens that are highly predictive of what comes next. The implication is that future work on integrated speculators should treat the speculation architecture as a miniature language model rather than a collection of independent regressors, with all the design considerations (sequential conditioning, error propagation, beam search vs. sampling) that entails.

Second, it introduces the large-batch, small-tree regime as a first-class evaluation target with distinct optimization criteria. The paper's Figure 2 and Table 1 together make a concrete empirical argument that speculative decoding performance curves are non-monotonic in batch size and tree size — there exists an inflection point beyond which additional speculated tokens hurt throughput because verification computation saturates the GPU. This is not a new theoretical observation, but Clover is the first speculative decoding paper to make this the central optimization target and to design architecture specifically for it. The consequence is that not all accuracy gains are equally valuable: an architectural change that improves head accuracy by 10% at the cost of doubling the token tree size might be net-negative in the compute-bound regime, while a change that improves accuracy by 5% with no tree-size increase (or that allows equivalent accuracy from a smaller tree) might be net-positive. Clover's regressive architecture falls into the latter category — it improves accuracy while making the tree more information-dense, so a small tree (size 4) captures more of the probability mass. This shifts evaluation norms: future speculative decoding papers should report throughput across a range of batch sizes and tree sizes, not just the single point of maximum speedup.

Third, it provides the strongest empirical evidence to date that cross-attention is specifically well-suited for heterogeneous information fusion in speculators. The ablation comparing Attention Decoder against MLP-as-regressive-block (Figure 7a) is the paper's most actionable finding for practitioners building their own speculators. The 4.8–11.5% top-5 accuracy gap is large enough to be engineeringly significant, and the paper's explanation — that MLP concatenation cannot dynamically re-weight information from the previous token based on content — is mechanically grounded. This finding partially reconciles the landscape of concurrent regressive approaches: Zhang et al. (2024) and possibly early Hydra versions used MLP-based regression, while Eagle (Li et al., 2024) uses a transformer decoder layer (which includes cross-attention-like operations). Clover's ablation suggests the latter approach should systematically outperform the former, providing a testable prediction that future work can verify or falsify.

What the paper does NOT change: it does not address the fundamental limits of speculative decoding. On hard problems where the target model's next-token predictions are poorly calibrated or where multi-token continuations are genuinely unpredictable from local context, no speculator architecture — regressive or otherwise — will achieve high hit rates. The paper's results on the hardest tasks (e.g., Math) show that Clover improves over Medusa but still leaves substantial room between speculative and oracle throughput. The memory-bandwidth bottleneck that speculative decoding addresses is architectural, not algorithmic, and Clover's contribution is to push the efficiency frontier within the speculative decoding paradigm, not to transcend it.

Follow-Up Research This Work Enables

Cross-attention vs. transformer decoder for regressive speculation: a controlled comparison on public models. The paper's central architectural claim is that cross-attention (query from hidden state, key/value from previous token embedding) outperforms MLP concatenation for merging sequential information. But the paper does not compare against the most natural alternative: a full transformer decoder layer (self-attention over the growing sequence of speculated tokens), as used in Eagle (Li et al., 2024). A controlled experiment would implement Clover's Attention Decoder, an MLP concatenation baseline, and a transformer-decoder baseline on the same frozen target model (preferably a public one like LLaMA-2-7B or Mistral-7B), with identical training data and head counts, and measure both head accuracy and end-to-end throughput at multiple batch sizes and tree sizes. The key hypothesis: transformer decoder layers may achieve higher accuracy than cross-attention (by letting speculated tokens attend to each other), but at higher computational cost that may be net-negative in the large-batch, small-tree regime Clover targets. This tradeoff curve — accuracy vs. speculator overhead — would define the Pareto frontier for regressive speculator design and determine whether Clover's minimal cross-attention is Pareto-optimal or whether slightly more expensive architectures dominate it.

Does the regressive advantage scale with head count, or is there a ceiling? Figure 7a shows that Clover's accuracy gain over Medusa grows with head position (11.7% for head 1, 21.7% for head 2, 26.4% for head 3). This monotonic trend invites extrapolation: would head 4 see a 30%+ gain? Head 5? Or is there a saturation point — perhaps at 3–4 heads, beyond which even conditioned predictions become too uncertain to benefit from additional sequential context? A sweep of head counts from 1 to 8 on a fixed model and dataset, measuring both per-head top-k accuracy and end-to-end throughput at a fixed batch size and tree size, would characterize this scaling behavior. The throughput measurement is critical because the serial chain latency grows linearly with head count — there will be an optimal head count where marginal accuracy gain equals marginal latency cost. This optimum likely depends on model size (larger models have proportionally cheaper speculator overhead, shifting the optimum to more heads) and task (predictable tasks like code may sustain longer effective speculation chains than open-ended dialogue). Finding this optimum is the most immediate practical question for anyone deploying Clover.

Do sequentially-coherent token trees actually have higher information density, and can this be measured directly? The paper argues that Clover's regressive architecture produces token trees where paths are linguistically coherent, and that this coherence translates to higher acceptance rates at small tree sizes. But the paper never measures "information density" directly — it infers it from the extra-tokens-per-step metric. A direct measurement would: for a fixed tree size budget (e.g., 4, 8, 16 nodes), construct the optimal tree from Clover's sequentially-conditioned predictions and from Medusa's independent predictions, then measure the probability that the correct multi-token continuation exists as a path in each tree. This "tree recall@k" metric would quantify exactly how much of the accuracy gap is attributable to tree structure vs. raw head accuracy. The experiment would also test a specific prediction: Clover's tree recall should saturate at smaller tree sizes than Medusa's, because its paths are concentrated on plausible continuations rather than spread combinatorially. If this prediction holds, it validates the paper's information-density argument. If not — if Clover's advantage comes entirely from higher per-head accuracy with no tree-structure benefit — then the regressive architecture's tree-coherence property is incidental rather than causal.

Can the Augmenting Block be replaced with a task-adaptive lightweight alternative, such as LoRA on the target model's last layer? The Augmenting Block adds a full transformer layer (~1/N_layer overhead) and contributes 4.9–9.4% top-5 accuracy (Figure 7b). This is a meaningful accuracy gain, but the full-layer approach may be overparameterized — Figure 7b shows that attention alone captures most of the benefit (1.9–5.1%). A natural question: could the same or better accuracy be achieved by applying a low-rank adaptation (LoRA) to the target model's last transformer block, fine-tuning only those low-rank weights on the multi-offset prediction objective, and using the adapted hidden states directly? This would eliminate the Augmenting Block as a separate component, reducing both parameter count and per-step latency. The experiment would compare: (a) Clover with full Augmenting Block (the paper's configuration), (b) Clover with LoRA applied to the target model's last layer (keeping the target model otherwise frozen), and (c) Clover with no Augmenting Block. If LoRA matches or exceeds the full block's accuracy gain, it represents a strictly better design point — same accuracy, lower overhead, simpler implementation. If LoRA underperforms, it suggests the Augmenting Block's value comes from adding depth (an extra nonlinear transformation) rather than just adapting representations, which would be an important negative result for understanding what the Augmenting Block actually does.

Does the regressive chain propagate or amplify errors, and can error propagation be measured and mitigated? A known failure mode of auto-regressive generation is that early errors compound — a mistake at position 1 biases the model toward mistakes at positions 2, 3, and beyond. Clover's regressive chain creates exactly this risk: if head 1 predicts a wrong token, the Attention Decoder at head 2 conditions on that wrong token's embedding, potentially steering head 2 toward tokens that are plausible continuations of the wrong prefix but incorrect for the true prefix. The paper's extra-tokens-per-step and throughput metrics aggregate over correct and incorrect speculations and cannot distinguish between (a) Clover making fewer independent errors per head vs. (b) Clover making an early error that cascades to later heads (which would look like multiple rejected tokens in verification but might still achieve higher throughput than Medusa if the base hit rate is high enough). A diagnostic experiment would: track, for each verification step, whether the accepted token sequence came from a path where all heads predicted correctly or a path where an early correct prediction was followed by later errors. Comparing these patterns between Clover and Medusa would reveal whether the regressive architecture introduces cascade failures that partially offset its per-head accuracy gains. If cascade failures are common, mitigation strategies become relevant — for example, using top-k sampling at each regressive step rather than greedy selection, or training the Attention Decoder to be robust to incorrect conditioning tokens (e.g., by occasionally feeding ground-truth rather than predicted embeddings during training).

Practical Applications and Downstream Use Cases

Large-batch serving of LLMs in production chat systems. This is the scenario the paper explicitly targets and where its results are most directly applicable. In a production chat deployment — where dozens or hundreds of concurrent user requests are batched together for GPU efficiency — the system is likely compute-bound rather than memory-bound. Clover's architecture is designed for exactly this regime: the token tree is kept small (size 4 in the paper's experiments) to avoid saturating compute, and the regressive mechanism ensures that each node in that small tree has high probability of being accepted. The paper's numbers on Baichuan-Large (Table 2) show that Clover improves throughput over vanilla decoding across all 10 task categories at batch sizes 4–48, with gains ranging from approximately +20% to +146% depending on batch size and task. For a service operator serving millions of requests per day, a consistent 30–50% throughput improvement (the typical range at moderate batch sizes in Tables 1 and 2) translates directly to serving the same traffic with 25–33% fewer GPUs, or serving proportionally more traffic with the same infrastructure. The key deployment consideration is that Clover's advantage over Medusa is most pronounced on larger models — the paper reports up to 1.47× Clover-to-Medusa throughput on Baichuan-Large vs. 1.26× on Baichuan-Small — so the business case is strongest for large-model deployments where inference cost per token is highest.

Speculative decoding for code completion in IDEs. Code completion is a latency-sensitive, relatively low-batch application (each developer's keystrokes generate independent requests), but it shares a property with Clover's design that makes the architecture particularly well-suited: code has strong local sequential dependencies. After a token like def, the next several tokens are highly constrained (function_name, (, parameter_list, )). After import, package names follow predictable patterns. Clover's regressive mechanism, which conditions later heads on earlier predictions, should excel in these structured-prediction settings because head 1 predicting def gives heads 2 and 3 very strong priors about what follows. The paper's results on the Code task (Appendix Table 2) show Clover achieving throughput of 165.6–717.6 tokens/s across batch sizes on Baichuan-Large, consistently outperforming Medusa (130.2–562.3 tokens/s). While the paper evaluates in batched settings, the per-head accuracy gains reported in Figure 7 — particularly the large gains on later heads — suggest that Clover would also improve single-request latency for code completion relative to Medusa, since higher acceptance rates mean fewer verification round-trips. A direct latency evaluation at batch size 1 on code benchmarks (HumanEval, MBPP) would confirm this, but the existing accuracy evidence is suggestive.

Training data generation and self-improvement pipelines. When LLMs are used to generate training data — for distillation, for instruction-tuning dataset creation, or for self-play in RLHF pipelines — the generation throughput directly determines the volume and cost of data that can be produced. In these batch generation scenarios, the system is typically compute-bound (many sequences being generated in parallel), matching Clover's target regime. The paper's training setup (Section 4.1) provides a concrete reference point: training Clover's speculator on a 0.15B-token dataset takes 2 hours on 8× A800 GPUs for a 7B model. This is a one-time cost that then amortizes over all future inference. For an organization generating billions of tokens of synthetic data, a 30–50% throughput improvement (the typical Clover-over-vanilla range at moderate batch sizes) would reduce data generation time and cost proportionally. Moreover, since the target model is frozen and only the speculator is trained, Clover can be added to an already-deployed model without risking degradation of generation quality — the target model's weights are unchanged, and the verification step ensures exact output equivalence to vanilla decoding. This makes Clover a low-risk efficiency upgrade for existing generation pipelines.

When to Prefer This Method

The paper provides sufficient evidence to articulate clear deployment-condition preferences, though not as an explicit decision matrix in the text. The conditions are derived from the intersection of Clover's architectural properties and the empirical results:

• Prefer Clover over Medusa when: (1) you are deploying a large model (100B+ parameters), where the speculator overhead is proportionally smallest and the paper shows the largest relative gains (up to 1.47× over Medusa on Baichuan-Large vs. 1.26× on Baichuan-Small); (2) your serving workload operates at moderate-to-large batch sizes (8–32) where the system is shifting toward compute-bound but speculative decoding still provides net benefits, and the token tree must be kept small to avoid compute saturation — this is the regime Clover was explicitly designed for; (3) your task domain has strong local sequential dependencies (code, math, structured data generation) where knowing the previous token substantially constrains the next token distribution, making the regressive mechanism most valuable; and (4) you have the engineering capacity to implement and train the Attention Decoder and Augmenting Block, accepting that the training cost (~2 hours on 8× A800 for a 7B model, Section 4.1) is a one-time investment for persistent inference savings.

• Prefer Medusa over Clover when: (1) latency is paramount and the serial depth of the regressive chain (proportional to head count) cannot be tolerated — Medusa's fully parallel heads add no serial dependency within the speculation phase, so per-request wall-clock time may be lower even if throughput is worse; (2) you are deploying on a small model (under 7B parameters) at very low batch sizes (1–2), where the system is strongly memory-bound and the overhead of Clover's Augmenting Block and Attention Decoder is a larger fraction of total computation — the paper does not evaluate this regime, and Medusa's simpler architecture may be more efficient when compute constraints are absent; (3) implementation simplicity and ease of maintenance are the dominant concerns — Medusa requires only additional MLP heads, which are trivial to implement and train compared to Clover's cross-attention decoder and augmenting transformer layer; and (4) you lack the training data or compute budget to fine-tune Clover's additional components (the paper uses 0.15B training tokens), and must use a speculator with minimal training requirements.

• Prefer vanilla auto-regressive decoding over any speculative method when: the paper's Table 1 provides the critical evidence — at sufficiently high batch sizes, speculative decoding overhead outweighs its benefits. On Baichuan-Small, vanilla decoding surpasses Clover at batch size 32 (Math task) and batch size 48 (CA task). On Baichuan-Large, vanilla never surpasses Clover within the tested range (4–48), but the gap narrows substantially (e.g., CA at batch size 48: 938.2 vs. 887.8, only +5%). A practitioner should measure their specific workload's throughput at target batch sizes with and without speculation enabled — the crossover point depends on model size, hardware characteristics (memory bandwidth vs. compute capacity), and task properties. The paper does not provide a predictive model for this crossover, but it does provide the diagnostic: if speculative decoding throughput is not monotonically increasing with batch size, you are in or near the compute-bound regime where the benefit is marginal or negative.